Over the past 2 decades, many social scientists have expanded their data-collection capabilities by using various online research tools. In the 2011 article “Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data?” in Perspectives on Psychological Science, Buhrmester, Kwang, and Gosling introduced researchers to what was then considered to be a promising but nascent research platform. Since then, thousands of social scientists from seemingly every field have conducted research using the platform. Here, we reflect on the impact of Mechanical Turk on the social sciences and our article’s role in its rise, provide the newest data-driven recommendations to help researchers effectively use the platform, and highlight other online research platforms worth consideration.
AntounC.ZhangC.ConradF. G.SchoberM. F. (2016). Comparisons of online recruitment strategies for convenience samples: Craigslist, Google AdWords, Facebook, and Amazon Mechanical Turk. Field Methods, 28, 231–246.
2.
ArditteK. A.ÇekD.ShawA. M.TimpanoK. R. (2016). The importance of assessing clinical phenomena in Mechanical Turk research. Psychological Assessment, 28, 684–691.
3.
BerinskyA. J.HuberG. A.LenzG. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20, 351–368.
4.
BerinskyA. J.MargolisM. F.SancesM. W. (2016). Can we turn shirkers into workers?Journal of Experimental Social Psychology, 60, 20–28.
5.
BlackJ.ReynoldsW. M. (2013). Examining the relationship of perfectionism, depression, and optimism: Testing for mediation and moderation. Personality and Individual Differences, 54, 426–431.
6.
BuchananT.WilliamsJ. E. (2010). Ethical issues in psychological research on the Internet. In GoslingS.JohnsonJ. (Eds.), Advanced methods for conducting online behavioral research (pp. 255–271). Washington, DC: American Psychological Association.
7.
BuhrmesterM. D.KwangT.GoslingS. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data?Perspectives on Psychological Science, 6, 3–5.
8.
CallegaroM.BakerR. P.BethlehemJ.GöritzA. S.KrosnickJ. A.LavrakasP. J. (Eds.). (2014). Online panel research: A data quality perspective. Chichester, England: John Wiley.
9.
CaslerK.BickelL.HackettE. (2013). Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29, 2156–2160.
10.
ChandlerJ.MuellerP.PaolacciG. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112–130.
11.
ChandlerJ.PaolacciG. (2017). Lie for a dime: When most prescreening responses are honest but most study participants are imposters. Social Psychological & Personality Science, 8, 500–508. doi:10.1177/1948550617698203
12.
ChandlerJ.ShapiroD. (2016). Conducting clinical research using crowdsourced convenience samples. Annual Review of Clinical Psychology, 12, 53–81. doi:10.1146/annurev-clinpsy-021815-093623.
13.
CliffordS.JeritJ. (2016). Cheating on political knowledge questions in online surveys: An assessment of the problem and solutions. Public Opinion Quarterly, 80, 858–887.
14.
FraleyR. C. (2004). How to conduct behavioral research over the internet: A beginner’s guide to HTML and CGI/Perl. New York, NY: Guilford.
15.
GaikwadS. N. S.WhitingM. E.GamageD.MullingsC. A.MajetiD.GoyalS.GilbeeA. . . . BernsteinM. S. (2017, February). The Daemo Crowdsourcing Marketplace. In Companion of the 2017 Association for Computing Machinery Conference on Computer Supported Cooperative Work and Social Computing (pp. 1-4). doi:10.1145/3022198.3023270
16.
GillanC. M.DawN. D. (2016). Taking psychiatry research online. Neuron, 91, 19–23.
17.
GleibsI. H. (2017). Are all “research fields” equal? Rethinking practice for the use of data from crowdsourcing market places. Behavior Research Methods, 49, 1333–1342. doi:10.3758/s13428-016-0789-y
18.
GoodmanJ. K.CryderC. E.CheemaA. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26, 213–224.
19.
GoslingS. D.JohnsonJ. A. (2010). Advanced methods for conducting online behavioral research. Washington, DC: American Psychological Association.
20.
GoslingS. D.MasonW. (2015). Internet research in psychology. Annual Review of Psychology, 66, 877–902.
21.
GoslingS. D.VazireS.SrivastavaS.JohnO. P. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about Internet questionnaires. American Psychologist, 59, 93–104.
22.
HauserD. J.SchwarzN. (2015). It’s a trap!: Instructional manipulation checks prompt systematic thinking on “tricky” tasks. SAGE Open, 5(2), 1–6. doi:10.1177/2158244015584617
23.
HauserD. J.SchwarzN. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than subject pool participants. Behavior Research Methods, 48, 400–407. doi:10.3758/s13428-015-0578-z.
24.
HarariG. M.GoslingS. D.WangR.ChenF.ChenZ.CampbellA. T. (2017). Patterns of behavior change in students over an academic term: A preliminary study of activity and sociability behaviors using smartphone sensing methods. Computers in Human Behavior, 67, 129–138.
25.
HortonJ. J.RandD. G.ZeckhauserR. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14, 399–425.
26.
KosinskiM.MatzS. C.GoslingS. D.PopovV.StillwellD. (2015). Facebook as a research tool for the social sciences: Opportunities, challenges, ethical considerations, and practical guidelines. American Psychologist, 70, 543–556.
27.
LitmanL.RobinsonJ.RosenzweigC. (2015). The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk. Behavior Research Methods, 47, 519–528.
28.
LowryP. B.D’ArcyJ.HammerB.MoodyG. D. (2016). “Cargo Cult” science in traditional organization and information systems survey research: A case for using nontraditional methods of data collection, including Mechanical Turk and online panels. The Journal of Strategic Information Systems, 25, 232–240.
29.
MasonW.SuriS. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44, 1–23.
30.
MasonW.WattsD. J. (2009). Financial incentives and the “performance of crowds.”Association for Computing Machinery Explorations Newsletter, 11, 100–108.
31.
MasonW.WattsD. J. (2011). Collective problem solving in networks. Proceedings of the National Academy of Sciences, 109, 764–769. doi:10.1073/pnas.1110069108
32.
OppenheimerD. M.MeyvisT.DavidenkoN. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872.
33.
PaolacciG.ChandlerJ.IpeirotisP. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.
34.
PeerE.VosgerauJ.AcquistiA. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023–1031.
35.
PeerE.BrandimarteL.SamatS.AcquistiA. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153–163.
36.
RamseyS. R.ThompsonK. L.McKenzieM.RosenbaumA. (2016). Psychological research in the internet age: The quality of web-based data. Computers in Human Behavior, 58, 354–360.
37.
RandD.G. (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172–179.
38.
RizviS.BobocelD. R. (2016). Promoting forgiveness through psychological distance. Social Psychological & Personality Science, 7, 875–883.
39.
RossJ.IraniL.SilbermanM.ZaldivarA.TomlinsonB. (2010). Who are the crowdworkers? Shifting demographics in mechanical turk. In CHI ’10 Extended Abstracts on Human Factors in Computing Systems (pp. 2863–2872). doi:10.1145/1753846.1753873
40.
SheehanK. B.PittmanM. (2016). Amazon’s Mechanical Turk for academics: The HIT handbook for social science research. Irvine, CA: Melvin & Leigh Publishers.
41.
StewartN.UngemachC.HarrisA. J.BartelsD. M.NewellB. R.PaolacciG.ChandlerJ. (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making, 10, 479–491.
42.
Van DijckJ. (2013). The culture of connectivity: A critical history of social media. Oxford, England: Oxford University Press.
43.
ZhouH.FishbachA.ShaddyF.SteinmetzJ.BregantJ.SchroederJ.Choshen-HillelS. (2016). The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions. Journal of Personality and Social Psychology, 222, 493–504. doi:10.1037/pspa0000056
