Service managers require precise enough measures of service performance to make particular decisions. Generalizability theory (G-theory) has begun to be used to design service assessment studies for decision making. However, initial applications address a limited range of decisions and may underestimate the amount of data needed for decision making if the variance due to the hidden-occasions (time-of-observation) facet is substantial. This study uses test-retest mystery shopping data to investigate the variance due to the main and interaction effects of test occasions and the consequences of ignoring them for different managerial decisions. Accounting for the hidden-occasions facet reveals a need to collect more than twice as many data when benchmarking services and over 10 times as many data to segment service assessors on the basis of their evaluative responses. Thus, accounting for variation due to occasions is crucial for G-theory applications to deliver assessment data of the required quality.
Barnes, Stuart J. and Richard T. Vidgen (2002), “An Integrative Approach to the Assessment of E-Commerce Quality,” Journal ofElectronic Commerce Research, 3 (August), 114-27.
2.
Beck, Jeff and Li Miao (2003), “Mystery Shopping in Lodging Properties as a Measurement of Service Quality,” Journal of Quality Assurance in Hospitality & Tourism, 4 (1/2), 1-21.
3.
Behara, Ravi S. and Richard B. Chase (1993), “Service Quality Deployment: Quality Service by Design,” in Perspectives in Operations Management: Essays in Honor of Elwood S. Buffa, Rakesh V. Sarin, ed. Norwell, MA: Kluwer, 87-99.
4.
Brennan, Robert L. (2000), “Performance Assessments From the Perspective of Generalizability Theory,” Applied Psychological Measurement , 24 (December), 339-53.
5.
———. (2001), Generalizability Theory. New York: Springer-Verlag.
6.
Bolton, Ruth N. and James H. Drew (1994), “Linking Customer Satisfaction to Service Operations and Outcomes,” in Service Quality: New Directions in Theory and Practice, Roland T. Rust and Richard L. Oliver, eds. Thousand Oaks, CA: Sage, 173-200.
7.
Calvert, Philip (2005), “It's a Mystery: Mystery Shopping in New Zealand's Public Libraries,” Library Review, 54 (1), 24-35.
8.
Cronbach, Lee J., Goldine C. Gleser, Harinder Nanda , and Nageswari Rajaratnam (1972), The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles . New York: John Wiley.
9.
———, Robert L. Linn, Robert L. Brennan, and Edward H. Haertel (1997), “Generalizability Analysis for Performance Assessments of Student Achievement or School Effectiveness ,” Educational and Psychological Measurement, 57 (June), 373-99.
10.
Cronin, J. Joseph and Steven A. Taylor (1992), “Measuring Service Quality: A Reexamination and Extension,” Journal of Marketing, 56 (July), 55-68.
11.
Dawson, Janet and Jill Hillier (1995), “Competitor Mystery Shopping: Methodological Considerations and Implications for MRS Code of Conduct,” Journal of the Market Research Society, 37 (4), 417-27.
12.
Finn, Adam (2001), “Mystery Shopper Benchmarking of Durable Goods Chains and Stores,” Journal of Service Research , 3 (May), 310-20.
13.
——— and Ujwal Kayandé (1997), “Reliability Assessment and Optimization of Marketing Measurement,” Journal of Marketing Research, 34 (May), 262-75.
14.
——— and Ujwal Kayandé (1999), “Unmasking a Phantom: A Psychometric Assessment of Mystery Shopping,” Journal of Retailing, 75 (Summer), 195-217.
15.
Gaba, David M. (2004), “What Makes a Good Anesthesiologist?” Anesthesiology, 101 (November), 1061-63.
16.
Garver, Michael S. (2003), “Best Practices in Identifying Customer-Driven Improvement Opportunities,” Industrial Marketing Management , 32 (August), 455-66.
17.
Greguras, G.J. and C. Robie (1998), “A New Look at Within-Source Interrater Reliability of 360-Degree Feedback Ratings,” Journal of Applied Psychology, 83 (6), 960-68.
18.
Hauser, John R., Duncan I. Simester, and Birger Wernerfelt (1994), “Customer Satisfaction Incentives,” Marketing Science, 13 (Fall), 327-50.
19.
Hoyt, William T. and Michael-David Kerns (1999), “Magnitude and Moderators of Bias in Observer Ratings: A Meta-Analysis,” Psychological Methods, 4 (December), 403-24.
20.
Jagodzinski, Wolfgang, Steffen M. Kuhnel, and Peter Schmidt (1987), “Is There a `Socratic Effect' in Nonexperimental Panel Studies? Consistency of an Attitude Toward Guestworkers,” Sociological Methods and Research, 15 (February), 259-302.
21.
Kardes, Frank R., Maria L. Cronley, Manuel C. Pontes, and David C. Houghton (2001), “Down the Garden Path: The Role of Conditional Inference Processes in Self-Persuasion,” Journal of Consumer Psychology, 11 (3), 159-68.
22.
Martilla, John A. and John C. James (1977), “Importance-Performance Analysis,” Journal of Marketing, 41 (January), 77-79.
23.
McGuire, William J. (1960). “A Syllogistic Analysis of Cognitive Relationships ,” in Attitude Organization and Change, M. J. Rosenberg, C. I. Hovland , W. J. McGuire, R. P. Abelson, and J. W. Brehm, eds. New Haven, CT: Yale University Press , 65-111.
24.
Nunnally, Jum (1978), Psychometric Theory, 2nd ed. New York: McGraw-Hill.
25.
Parasuraman, A., Valarie A. Zeithaml, and Leonard L. Berry (1988), “SERVQUAL: A Multi-Item Scale for Measuring Consumer Perceptions of Service Quality,” Journal of Retailing, 64 (Spring), 12-40.
26.
Searle, Shayle R., George Casella, and Charles E. McCulloch (1992), Variance Components. New York: Wiley-Interscience.
27.
Shavelson, Richard J. and Noreen M. Webb (1991), Generalizability Theory: A Primer. Newbury Park, CA: Sage.
28.
Simester, Duncan I., John R. Hauser, Birger Wernerfelt, and Roland T. Rust( 2000), “Implementing Quality Improvement Programs Designed to Enhance Customer Satisfaction: Quasi-Experiments in the United States and Spain,” Journal of Marketing Research, 37 (February), 102-12.
29.
Sitzia, John (1999), “How Valid and Reliable are Patient Satisfaction Data? An Analysis of 195 Studies,” International Journal for Quality in Health Care, 11 (4), 319-28.
30.
Thompson, Bruce and Colleen Cook (2002), “Stability of the Reliability of LibQUAL+™ Scores: A Reliability Generalization Meta-Analysis Study,” Educational and Psychological Measurement, 62 (August), 735-43.
31.
Thompson, Robert S. (2001), “Reliability, Validity and Bias in Peer Evaluations of Self-Directed Interdependent Work Teams,” ASEE Annual Conference Proceedings, 8495-532.
32.
Trost, A. and W. Bungard (2004), “The Interrater Reliability of Employee Survey Results,” Zeitschrift fur Arbeits- und Organisationspsychologie , 48 (3), 122-31.
33.
Webb, Noreen M., Jonah Schlackman, and Brenda Sugrue ( 2000), “The Dependability and Interchangeability of Assessment Methods in Science,” Applied Measurement in Education , 13 (July), 277-301.
34.
Van der Weile, Ton, Martin Hesselink, and Jos van Iwaarden (2005), “Mystery Shopping: A Tool to Develop Insight Into Customer Service Provision,” Total Quality Management, 16 (June), 529-41.
35.
Wilson, Alan M. (1998a), “The Role of Mystery Shopping in the Measurement of Service Performance,” Managing Service Quality, 8 (6), 414-20.
36.
———. (1998b), “The Use of Mystery Shopping in the Measurement of Service Delivery,” Services Industries Journal, 18 (July), 148-63.
37.
Wolfinbarger, Mary and Mary C. Gilly (2003), “e-TailQ: Dimensionalizing, Measuring and Predicting e-Tail Quality,” Journal of Retailing, 79 (Fall), 183-98.
38.
Yang, Z. and M. Jun (2002), “Consumer Perception of e-Service Quality: From Internet Purchaser and Non-Purchaser Perspectives,” Journal of Business Strategies, 19 (Spring), 19-41.
39.
Yoo, Boonghee and Naveen Donthu ( 2001), “Developing a Scale to Measure the Perceived Quality of an Internet Shopping Site (SITE-QUAL),” Quarterly Journal ofElectronic Commerce, 2 (1), 31-45.
40.
Zeithaml, Valarie A. (2000), “Service Quality, Profitability and the Economic Worth of Customers: What We Know and What We Need to Learn,” Journal of the Academy of Marketing Science, 28 (Winter), 67-85.
