The advent of online platforms such as Amazon’s Mechanical Turk (MTurk) has expanded considerably researchers’ options for collecting research data. Many researchers, however, express understandable skepticism of the viability of using platforms such as MTurk. In this article, we provide a background on the use of MTurk as a mechanism for collecting research data. We then review what is currently known about the advantages and issues associated with using MTurk and highlight important areas for future research. We conclude by discussing implications of the use of crowdsourcing platforms such as MTurk for education research.
BartneckC.DuenserA.MoltchanovaE.ZawieskaK. (2015). Comparing the similarity of responses received from studies in Amazon’s Mechanical Turk to studies conducted online and with direct recruitment. PLOS ONE, 10, 1–23. https://doi.org/10.1371/journal.pone.0121595
2.
BehrendT. S.SharekD. J.MeadeA. W.WiebeE. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43(3), 800–813. doi:10.3758/s13428-011-0081-0
3.
BerinskyA. J.HuberG. A.LenzG. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20, 351–368. https://doi.org/10.1093/pan/mpr057
4.
BuhrmesterM.KwangT.GoslingS. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data?Perspectives on Psychological Science, 6(1), 3–5. doi:10.1177/1745691610393980
5.
CaslerK.BickelL.HackettE. (2013). Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29, 2156–2160. https://doi.org/10.1016/j.chb.2013.05.009
6.
ChandlerJ.MuellerP.PaolacciG. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavioral Research Methods, 46(1), 112–130. doi:10.3758/s13428-013-0365-7
7.
CroneD. L.WilliamsL. A. (2016). Crowdsourcing participants for psychological research in Australia: A test of MicroWorkers. Australian Journal of Psychology, 69(1), 39–47. doi:10.1111/ajpy.12110
8.
GadirajuU.DemartiniG.DifallahD. E.CatastaM. (2016). It’s getting crowded! How to use crowdsourcing effectively for web science research. In Proceedings of the 8th ACM Conference on Web Science (pp. 1–11). New York, NY: ACM. doi:10.1145/2908131.2908140
9.
GoodmanJ. K.CryderC. E.CheemaA. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26, 213–224. doi:10.1002/bdm.1753
10.
HauserD. J.SchwartzN. (2015). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavioral Research Methods, 48(1), 400–407. doi:10.3758/s13428-015-0578-z
11.
HoldenC. J.DennieT.HicksA. D. (2013). Assessing the reliability of the M5-120 on Amazon’s Mechanical Turk. Computers in Human Behavior, 29, 1749–1754. https://doi.org/10.1016/j.chb.2013.02.020
12.
JohnsonD. R.BordenL. A. (2012). Participants at your fingertips: Using Amazon’s Mechanical Turk to increase student–faculty collaborative research. Teaching of Psychology, 39(4), 245–251. doi:10.1177/0098628312456615
13.
KosaraR.ZiemkiewiczC. (2010). Do Mechanical Turks dream of square pie charts? In Proceedings of the 3rd BELIV’10 Workshop: Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (pp. 63–70). New York, NY: ACM.
14.
LandersR. N.BehrendT. S. (2015). An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology, 8(2), 142–164. https://doi.org/10.1017/iop.2015.13
15.
MasonW.SuriS. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavioral Research Methods, 44(1), 1–23. doi:10.3758/s13428-011-0124-6
16.
PalaciosM.Martinez-CorralA.NisarA.GrijalvoM. (2016). Crowdsourcing and organizational forms: Emerging trends and research implications. Journal of Business Research, 69(5), 1834–1839. https://doi.org/10.1016/j.jbusres.2015.10.065
17.
PaolacciG.ChandlerJ. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23(3), 184–188. doi:10.1177/0963721414531598
18.
PaolacciG.ChandlerJ.IpeirotisP. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419. https://ssrn.com/abstract=1626226
19.
PeerE.SamatS.BrandimarteL.AcquistiA. (2015). Data quality in online research: Challenges and solutions. Advances in Consumer Research, 43, 18–22.
20.
PeerE.VosgerauJ.AcquistiA. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavioral Research Methods, 46, 1023–1031. doi:10.3758/s13428-013-0434-y
21.
PetersonR. A.MerunkaD. R. (2014). Convenience samples of college students and research reproducibility. Journal of Business Research, 67(5), 1035–1041. https://doi.org/10.1016/j.jbusres.2013.08.010
SchmidtG. B. (2015). Fifty days an MTurk worker: The social and motivational context for Amazon Mechanical Turk workers. Industrial and Organizational Psychology, 8(2), 165–237. https://doi.org/10.1017/iop.2015.20
24.
ShankD. B. (2016). Using crowdsourcing websites for sociological research: The case of Amazon Mechanical Turk. American Sociologist, 47(1), 47–55. doi:10.1007/s12108-015-9266-9
25.
ShapiroD. N.ChandlerJ.MuellerP. A. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1(2), 213–220. doi:10.1177/2167702612469015
26.
SmithN. A.SabatI. E.MartinezL. R.WeaverK.XuS. (2015). A convenient solution: Using MTurk to sample from hard-to-reach populations. Industrial and Organizational Psychology, 8(2), 220–228. https://doi.org/10.1017/iop.2015.29
27.
VakhariaD.LeaseM. (2015). Beyond Mechanical Turk: An analysis of paid crowd work platforms. In Proceedings of iConference (pp. 1–17). Newport Beach, CA: iSchools.
28.
WilliamsC. (2013). Crowdsourcing research: A methodology for investigating state crime. State Crime Journal, 2(1), 30–51. doi:10.13169/statecrime.2.1.0030
29.
ZhaoY.ZhuQ. (2014). Evaluation on crowdsourcing research: Current status and future direction. Information Systems Frontiers, 16(3), 417–434. doi:10.1007/s10796-012-9350-4
