The use of Amazon’s Mechanical Turk (MTurk) in management research has increased over 2,117% in recent years, from 6 papers in 2012 to 133 in 2019. Among scholars, though, there is a mixture of excitement about the practical and logistical benefits of using MTurk and skepticism about the validity of the data. Given that the practice is rapidly increasing but scholarly opinions diverge, the Journal of Management commissioned this review and consideration of best practices. We hope the recommendations provided here will serve as a catalyst for more robust, reproducible, and trustworthy MTurk-based research in management and related fields.
AguinisH.BanksG. C.RogelbergS. G.CascioW. F. in press. Actionable recommendations for narrowing the science-practice gap in open science. Organizational Behavior and Human Decision Processes, 158: 27-35.
2.
AguinisH.RamaniR. S.AlabduljaderN.2018. What you see is what you get? Enhancing methodological transparency in management research. Academy of Management Annals, 12: 83-110.
3.
AguinisH.RamaniR. S.AlabduljaderN.2020. Best-practice recommendations for producers, evaluators, and users of methodological literature reviews. Organizational Research Methods. Advance online publication. doi:10.1177/1094428120943281
4.
AguinisH.SolarinoA. M.2019. Transparency and replicability in qualitative research: The case of interviews with elite informants. Strategic Management Journal, 40: 1291-1315.
5.
AlonsoO.MizzaroS.2012. Using crowdsourcing for TREC relevance assessment. Information Processing & Management, 48: 1053-1066.
6.
AntinJ.ShawA.2012. Social desirability bias and self-reports of motivation: A study of Amazon Mechanical Turk in the US and India. In ACM Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: 2925-2934. New York: ACM.
BaderF.BaumeisterB.BergerR.KeuschniggM. in press. On the transportability of laboratory results. Sociological Methods & Research. Advance online publication. doi:10.1177/0049124119826151
9.
BarchardK. A.WilliamsJ.2008. Practical advice for conducting ethical online experiments and questionnaires for United States psychologists. Behavior Research Methods, 40: 1111-1128.
10.
BarendsA. J.de VriesR. E.2019. Noncompliant responding: Comparing exclusion criteria in MTurk personality research to improve data quality. Personality and Individual Differences, 143: 84-89.
11.
BedersonB. B.QuinnA. J.2011. Web workers unite! Addressing challenges of online laborers. ACM CHI Extended Abstracts on Human Factors in Computing Systems, 2011: 97-106.
12.
BehrendT. S.SharekD. J.MeadeA. W.WiebeE. N.2011. The viability of crowdsourcing for survey research. Behavior Research Methods, 43: 800-813.
13.
Bergvall-KårebornB.HowcroftD.2014. Amazon Mechanical Turk and the commodification of labor. New Technology, Work, and Employment, 29: 213-223.
14.
BerinskyA. J.HuberG. A.LenzG. S.2012. Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20: 351-368.
15.
BerinskyA. J.MargolisM. F.SancesM. W.2014. Separating the shirkers from the workers? Making sure respondents pay attention on self-administered surveys. American Journal of Political Science, 58: 739-753.
16.
BernerthJ.AguinisH.2016. A critical review and best-practice recommendations for control variable usage. Personnel Psychology, 69: 229-283.
17.
BrawleyA. M.PuryC. L.2016. Work experiences on MTurk: Job satisfaction, turnover, and information sharing. Computers in Human Behavior, 54: 531-546.
18.
BrinkW. D.EatonT. V.GrenierJ. H.ReffettA.2019. Deterring unethical behavior in online labor markets. Journal of Business Ethics, 156: 71-88.
19.
BuchananE. M.ScofieldJ. E.2018. Methods to detect low quality data and its implication for psychological research. Behavior Research Methods, 50: 1-11.
20.
BuhrmesterM. D.TalaifarS.GoslingS. D.2018. An evaluation of Amazon’s Mechanical Turk, its rapid rise, and its effective use. Perspectives on Psychological Science, 13: 149-154.
21.
BungeE.CookH. M.BondM.WilliamsonR. E.CanoM.BarreraA. Z. . . . MuñozR. F.2018. Comparing Amazon Mechanical Turk with unpaid internet resources in online clinical trials. Internet Interventions, 12: 68-73.
22.
Callison-BurchC.DredzeM.2010. Creating speech and language data with Amazon’s Mechanical Turk. In Proceedings of the NAACL HLT workshop on creating speech and language data with Amazon’s Mechanical Turk: 1-12. Stroudburg, PA: Association for Computational Linguistics.
23.
CaseyL. S.ChandlerJ.LevineA. S.ProctorA.StrolovitchD. Z.2017. Intertemporal differences among MTurk workers: Time-based sample variations and implications for online data collection. SAGE Open, 7: 2158244017712774.
24.
CaslerK.BickelL.HackettE.2013. Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29: 2156-2160.
25.
ChandlerJ.MuellerP.PaolacciG.2014. Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46: 112-130.
26.
ChandlerJ. J.PaolacciG.2017. Lie for a dime: When most prescreening responses are honest but most study participants are impostors. Social Psychological and Personality Science, 8: 500-508.
27.
ChandlerJ.PaolacciG.PeerE.MuellerP.RatliffK. A.2015. Using non-naive participants can reduce effect sizes. Psychological Science, 26: 1131-1139.
28.
ChandlerJ.RosenzweigC.MossA. J.RobinsonJ.LitmanL.2019. Online panels in social science research: Expanding sampling methods beyond Mechanical Turk. Behavior Research Methods, 51: 2022-2038.
29.
CheungJ. H.BurnsD. K.SinclairR. R.SliterM.2017. Amazon Mechanical Turk in organizational psychology: An evaluation and practical recommendations. Journal of Business and Psychology, 32: 347-361.
30.
CliffordS.JeritJ.2014. Is there a cost to convenience? An experimental comparison of data quality in laboratory and online studies. Journal of Experimental Political Science, 1: 120-131.
31.
ColmanD. E.VineyardJ.LetzringT. D.2018. Exploring beyond simple demographic variables: Differences between traditional laboratory samples and crowdsourced online samples on the Big Five personality traits. Personality and Individual Differences, 133: 41-46.
32.
CrumpM. J.McDonnellJ. V.GureckisT. M.2013. Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLOS ONE, 8: e57410.
33.
De QuidtJ.HaushoferJ.RothC.2018. Measuring and bounding experimenter demand. American Economic Review, 108: 3266-3302.
34.
DengX.JoshiK. D.GalliersR. D.2016. The duality of empowerment and marginalization in microtask crowdsourcing: Giving voice to the less powerful through value sensitive design. MIS Quarterly, 40: 279-302.
35.
DennisS. A.GoodsonB. M.PearsonC.2019. Virtual private servers and the limitations of IP-based screening procedures: Lessons from the MTurk quality crisis of 2018 (Publication No. 3233954). SSRN.
36.
FeitosaJ.JosephD. L.NewmanD. A.2015. Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English. Personality and Individual Differences, 75: 47-52.
37.
FieselerC.BucherE.HoffmannC. P.2017. Unfairness by design? The perceived fairness of digital labor on crowdworking platforms. Journal of Business Ethics, 156: 987-1005.
38.
GleibsI. H.2017. Are all “research fields” equal? Rethinking practice for the use of data from crowdsourcing market places. Behavior Research Methods, 49: 1333-1342.
39.
GoodmanJ. K.CryderC. E.CheemaA.2013. Data collection in a flat world: The advantages and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26: 213-224.
40.
HambyT.TaylorW.2016. Survey satisficing inflates reliability and validity measures: An experimental comparison of college and Amazon Mechanical Turk samples. Educational and Psychological Measurement, 76: 912-932.
41.
HeerJ.BostockM.2010. Crowdsourcing graphical perception: Using Mechanical Turk to assess visualization design. In ACM Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: 203-212. New York: ACM.
42.
HongM.SteedleJ. T.ChengY.2020. Methods of detecting insufficient effort responding: Comparisons and practical recommendations. Educational and Psychological Measurement, 80: 312-345.
43.
HortonJ. J.ChiltonL. B.2010. The labor economics of paid crowdsourcing. In Proceedings of the ACM Conference on Electronic Commerce: 209-218. New York: ACM.
44.
HuangJ. L.BowlingN. A.LiuM.LiY.2015. Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30: 299-311.
45.
HydockC.2018. Assessing and overcoming participant dishonesty in online data collection. Behavior Research Methods, 50: 1563-1567.
46.
JohnsonJ. A.2005. Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39: 103-129.
47.
KanI. P.DrummeyA. B.2018. Do imposters threaten data quality? An examination of worker misrepresentation and downstream consequences in Amazon's Mechanical Turk workforce. Computers in Human Behavior, 83: 243-253.
48.
KeesJ.BerryC.BurtonS.SheehanK.2017. An analysis of data quality: Professional panels, student subject pools, and Amazon’s Mechanical Turk. Journal of Advertising, 46: 141-155.
49.
KeithM. G.TayL.HarmsP. D.2017. Systems perspective of Amazon Mechanical Turk for organizational research: Review and recommendations. Frontiers in Psychology, 8: 1-19.
50.
KennedyR.CliffordS.BurleighT.JewellR.WaggonerP.2018. The shape of and solutions to the MTurk quality crisis (Publication No. 3272468). SSRN.
51.
LandersR. N.BehrendT. S.2015. An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology, 8: 142-164.
52.
LevayK. E.FreeseJ.DruckmanJ. N.2016. The demographic and political composition of Mechanical Turk samples. Sage Open, 6: 2158244016636433.
53.
LitmanL.RobinsonJ.RosenzweigC.2015. The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk. Behavior Research Methods, 47: 519-528.
54.
LiuB.SundarS. S.2018. Microworkers as research participants: Does underpaying Turkers lead to cognitive dissonance?Computers in Human Behavior, 24: 89-101.
55.
LovettM.BajabaS.LovettM.SimmeringM. J.2018. Data quality from crowdsourced surveys: A mixed method inquiry into perceptions of Amazon’s Mechanical Turk Masters. Applied Psychology, 67: 339-366.
56.
ManiaciM. R.RoggeR. D.2014. Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48: 61-83.
57.
MasonW.SuriS.2012. Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44: 1-23.
58.
MummoloJ.PetersonE.2019. Demand effects in survey experiments: An empirical assessment. American Political Science Review, 113: 517-529.
59.
NeckaE. A.CacioppoS.NormanG. J.CacioppoJ. T.2016. Measuring the prevalence of problematic respondent behaviors among MTurk, campus, and community participants. PLOS ONE, 11: e0157732.
60.
PaolacciG.ChandlerJ.IpeirotisP. G.2010. Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5: 411-419.
61.
ParéG.TrudelM. C.JaanaM.KitsiouS.2015. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management, 52: 183-199.
62.
PearlJ.BareinboimE.2014. External validity: From do-calculus to transportability across populations. Statistical Science, 29: 579-595.
63.
PeerE.VosgerauJ.AcquistiA.2014. Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46: 1023-1031.
64.
PorterC. O.OutlawR.GaleJ. P.ChoT. S.2019. The use of online panel data in management research: A review and recommendations. Journal of Management, 45: 319-344.
65.
RamseyS. R.ThompsonK. L.McKenzieM.RosenbaumA.2016. Psychological research in the internet age: The quality of web-based data. Computers in Human Behavior, 58: 354-360.
66.
RouseS. V.2015. A reliability analysis of Mechanical Turk data. Computers in Human Behavior, 43: 304-307.
67.
SchulzeT.SeedorfS.GeigerD.KaufmannN.SchaderM.2011. Exploring task properties in crowdsourcing: An empirical study on Mechanical Turk. In ECIS Proceedings: 122. http://aisel.aisnet.org/ecis2011/122.
68.
SiegelJ. T.NavarroM. A.ThomsonA. L.2015. The impact of overtly listing eligibility requirements on MTurk: An investigation involving organ donation, recruitment scripts, and feelings of elevation. Social Science & Medicine, 142: 256-260.
69.
SprouseJ.2011. A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior Research Methods, 43: 155-167.
70.
StewartN.UngemachC.HarrisA. J.BartelsD. M.NewellB. R.PaolacciG.ChandlerJ.2015. The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making, 10: 479-491.
71.
StritchJ. M.PedersenM. J.TaggartG.2017. The opportunities and limitations of using Mechanical Turk (MTurk) in public administration and management scholarship. International Public Management Journal, 20: 489-511.
72.
SummervilleA.ChartierC. R.2013. Pseudo-dyadic “interaction” on Amazon’s Mechanical Turk. Behavior Research Methods, 45: 116-124.
73.
ThomasK. A.CliffordS.2017. Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments. Computers in Human Behavior, 77: 184-197.
74.
Tosti-KharasJ.ConleyC.2016. Coding psychological constructs in text using Mechanical Turk: A reliable, accurate, and efficient alternative. Frontiers in Psychology, 7: 741.
75.
von AhnL.BlumM.HopperN. J.LangfordJ.2003. CAPTCHA: Using hard AI problems for security. Lecture Notes in Computer Science, 2656: 294-311.
76.
WalterS. L.SeibertS. E.GoeringD.O’BoyleE. H.2019. A tale of two sample sources: Do results from online panel data and conventional data converge?Journal of Business and Psychology, 34: 425-452.
77.
WeinbergJ.FreeseJ.McElhattanD.2014. Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample. Sociological Science, 1: 292-310.
78.
WesslingK. S.HuberJ.NetzerO.2017. MTurk character misrepresentation: Assessment and solutions. Journal of Consumer Research, 44: 211-230.
79.
WoodD.HarmsP. D.LowmanG. H.DeSimoneJ. A.2017. Response speed and response consistency as mutually validating indicators of data quality in online samples. Social Psychological and Personality Science, 8: 454-464.
80.
ZackE. S.KennedyJ.LongJ. S.2019. Can nonprobability samples be used for social science research? A cautionary tale. Survey Research Methods, 13: 215-227.
81.
ZhouH.FishbachA.2016. The threat of experimenting on the web: How unattended selective attrition leads to surprising yet false research conclusions. Journal of Personality and Social Psychology, 111: 493-504.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
