How authentic are inquiry calls made by simulated clients, or ‘mystery shoppers’, to service organizations, when compared to real callers? We analysed 48 simulated and 63 real inquiry calls to different veterinary practices in the United Kingdom and Ireland. The data were transcribed for conversation analysis, as well as coded for a variety of call categories including reason for the call, call outcome and turn design features. Analysis revealed systematic differences between real and simulated calls in terms of (1) reasons for the call, call outcome and call duration and (2) how callers refer to their pets in service requests and follow-up questions about their animal. Our qualitative analyses were supported with statistical summaries and tests. The findings reveal the limitations of mystery shopper methodology for the assessment of service provision. We also discuss the implications of the findings for the use of simulated encounters and the development of conversational agents.
AlbertSAHousleyWStokoeE (2019) In case of emergency, order pizza: An urgent case of action formation and recognition. In: Proceedings of the 1st international conference on conversational user interfaces (CUI 2019), Dublin, 22–23 August. New York: ACM. DOI: 10.1145/3342775.334280.
2.
AtkinsS (2018) Assessing health professionals’ communication through role-play: An interactional analysis of simulated versus actual general practice consultations. Discourse Studies21(2): 109–134.
3.
AtkinsSRobertsCHawthorneK, et al. (2016) Simulated consultations: A sociolinguistic perspective. BMC Medical Education16: 16.
BokkenLRethansJJvan HeurnL, et al. (2009) Students’ views on the use of real patients and simulated patients in undergraduate medical education. Academic Medicine84(7): 958–963.
6.
CalvertP (2005) It’s a mystery: Mystery shopping in New Zealand’s public libraries. Library Review54(1): 24–35.
7.
ClarkHHFox TreeJ (2002) Using uh and um in spontaneous speaking. Cognition84(1): 73–111.
8.
Couper-KuhlenE (2001) Constructing Reason-for-the-Call Turns in Everyday Telephone Conversation. InLiSt No. 25. Konstanz: Interaction and Linguistic Structures.
9.
De la CroixASkeltonJ (2009) The reality of role-play: Interruptions and amount of talk in simulated consultations. Medical Education43: 695–703.
10.
De RuiterJPAlbertS (2017) An appeal for a methodological fusion of conversation analysis and experimental psychology. Research on Language and Social Interaction50(1): 90–107.
FisherNIKordupleskiRE (2018) Good and bad market research: A critical review of Net Promoter Score. Applied Stochastic Models in Business and Industry35(1): 138–151.
13.
GarfinkelH (1967) Studies in Ethnomethodology. Englewood Cliffs, NJ: Prentice Hall.
14.
GlasierAMannersRLoudonaJC, et al. (2010) Community pharmacists providing emergency contraception give little advice about future contraceptive use: A mystery shopper study. Contraception82(6): 538–542.
15.
GoffmanEA (1974) Frame Analysis: An Essay on the Organization of Experience. Boston, MA: Northeastern University Press.
16.
HepburnA (2004) Crying: Notes on description, transcription, and interaction. Research on Language and Social Interaction37: 251–290.
17.
HepburnABoldenGB (2017) Transcribing for Social Research. London: Sage.
18.
HeritageJ (1984) A change-of-state token and aspects of its sequential placement. In: AtkinsonJMHeritageJ (eds) Structures of Social Action: Studies in Conversation Analysis. Cambridge: Cambridge University Press, pp. 299–345.
19.
HeritageJSorjonenM (1994) Constituting and maintaining activities across sequences: And-prefacing as a feature of question design. Language in Society23: 1–29.
20.
JacobSSchiffinoNBiardB (2018) The mystery shopper: A tool to measure public service delivery?International Review of Administrative Sciences84(1): 164–184.
LevinsonSC (2012) Action formation and ascription. In: SidnellJStiversT (eds) The Handbook of Conversation Analysis. Oxford: John Wiley & Sons, pp. 101–130.
23.
MounseyALBovbjergVWhiteL, et al. (2006) Do students develop better motivational interviewing skills through role-play with standardised patients or with student colleagues?Medical Education40(8): 775–780.
24.
NorrisPT (2002) Purchasing restricted medicines in New Zealand pharmacies: Results from a ‘mystery shopper’ study. Pharmacy World and Science24(4): 149–153.
25.
OttB (2016) The limits of control in service work: Interactive routines and interactional competence. In: VallasS (ed.) Research in the Sociology of Work, vol. 29. Bingley: Emerald Group, pp. 155–183.
26.
OwenR (2019) Net Promoter Score and its successful application. In: KompellaK (ed.) Marketing Wisdom: Management for Professionals. Singapore: Springer, pp. 17–29.
PilnickATrussonDBeekeS, et al. (2018) Using conversation analysis to inform role play and simulated interaction in communications skills training for healthcare professionals: Identifying avenues for further development through a scoping review. BMC Medical Education18(1): 627.
29.
ReevesS (2017) Some conversational challenges of talking with machines. In: Talking with conversational agents in collaborative action, workshop at the 20th ACM conference on computer-supported cooperative work and social computing (CSCW ’17), Portland, OR, 25 February–1 March.
30.
SacksH (1992) Lectures on Conversation (ed. JeffersonG), vol. I. Oxford: Blackwell.
31.
SchegloffEA (1986) The routine as achievement. Human Studies9: 111–151.
32.
SchegloffEA (1996a) Confirming allusions: Toward an empirical account of action. American Journal of Sociology102(1): 161–216.
33.
SchegloffEA (1996b) Some practices for referring to persons in talk-in-interaction: A partial sketch of a systematics. In: FoxB (ed.) Studies in Anaphora. Amsterdam: John Benjamins, pp. 437–485.
34.
SchegloffEA (2010) Some other ‘Uh(m)’s’. Discourse Processes47(2): 130–174.
35.
SchegloffEASacksH (1973) Opening up closings. Semiotica8(4): 289–327.
36.
StiversT (2012) Sequence organisation. In: StiversTSidnellJ (eds) The Handbook of Conversation Analysis. Oxford: John Wiley & Sons, pp. 191–209.
37.
StiversT (2015) Coding social interaction: A heretical approach in conversation analysis?Research on Language and Social Interaction48(1): 1–19.
38.
StokoeE (2013) The (in)authenticity of simulated talk: Comparing role-played and actual interaction and the implications for communication training. Research on Language and Social Interaction46(2): 165–185.
39.
StokoeESikvelandRO (2017) The conversation analytic role-play method: Simulation, endogenous impact and interactional nudges. In: PinkSForsVO’DellT (eds) Theoretical Scholarship and Applied Practice. Oxford: Berghahn Books, pp. 73–96.
40.
Van HasseltVBRomanoSJVecchiGM (2008) Role playing: Applications in hostage and crisis negotiation skills training. Behavior Modification32(2): 248–263.
41.
WilsonAM (1998) The role of mystery shopping in the measurement of service performance. Managing Service Quality: An International Journal8(6): 414–420.
