Too Anecdotal to Be True? Mechanical Turk Is Not All Bots and Bad Data: Response to Webb and Tangney (2022)

Abstract

In response to Webb and Tangney (2022) we call into question the conclusion that data collected on Amazon’s Mechanical Turk (MTurk) was “at best—only 2.6% valid” (p. 1). We suggest that Webb and Tangney made certain choices during the study-design and data-collection process that adversely affected the quality of the data collected. As a result, the anecdotal experience of these authors provides weak evidence that MTurk provides low-quality data as implied. In our commentary we highlight best practice recommendations and make suggestions for more effectively collecting and screening online panel data.

Keywords

crowdsourcing research data data screening and quality Mechanical Turk

Get full access to this article

View all access options for this article.

References

Anderson

C. A.

Allen

J. J.

Plante

Quigley-McBride

Lovett

Rokkum

J. N.

(2019). The MTurkification of social and personality psychology. Personality and Social Psychology Bulletin, 45, 842–850. https://doi.org/10.1177/0146167218798821

Behrend

T. S.

Sharek

D. J.

Meade

A. W.

Wiebe

E. N.

(2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43, 800–813. https://doi.org/10.3758/s13428-011-0081-0

Berinsky

A. J.

Huber

G. A.

Lenz

G. S.

(2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20, 351–368. https://doi.org/10.1093/pan/mpr057

Berry

D. T. R.

Wetter

M. W.

Baer

R. A.

Larsen

Clark

Monroe

(1992). MMPI-2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4, 340–345. https://doi.org/10.1037/1040-3590.4.3.340

Bowling

N. A.

Gibson

A. M.

DeSimone

J. A.

(2022). Stop with the questions already! Does data quality suffer for scales positioned near the end of a lengthy questionnaire? Journal of Business and Psychology, 37, 1099–1116. https://doi.org/10.1007/s10869-021-09787-8

Bowling

N. A.

Gibson

A. M.

Houpt

J. W.

Brower

C. K.

(2021). Will the questions ever end? Person-level increases in careless responding during questionnaire completion. Organizational Research Methods, 24, 718–738. https://doi.org/10.1177/1094428120947794

Bowling

N. A.

Huang

J. L.

Bragg

C. B.

Khazon

Liu

Blackmore

C. E.

(2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111, 218–229. https://doi.org/10.1037/pspp0000085

Brawley

A. M.

Pury

C. L. S.

(2016). Work experiences on MTurk: Job satisfaction, turnover, and information sharing. Computers in Human Behavior, 54, 531–546. https://doi.org/10.1016/j.chb.2015.08.031

Chandler

Mueller

Paolacci

(2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112–130. https://doi.org/10.3758/s13428-013-0365-7

10.

Cho

Allen

T. D.

(2012). Relationship between work interference with family and parent–child interactive behavior: Can guilt help? Journal of Vocational Behavior, 80, 276–287. https://doi.org/10.1016/j.jvb.2011.12.002

11.

Curran

P. G.

(2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19. https://doi.org/10.1016/j.jesp.2015.07.006

12.

Dennis

S. A.

Goodson

B. M.

Pearson

C. A.

(2020). Online worker fraud and evolving threats to the integrity of MTurk data: A discussion of virtual private servers and the limitations of IP-based screening procedures. Behavioral Research in Accounting, 32(1), 119–134. https://doi.org/10.2308/bria-18-044

13.

DeSimone

J. A.

Harms

P. D.

DeSimone

A. J.

(2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36, 171–181. https://doi.org/10.1002/job.1962

14.

Feitosa

Joseph

D. L.

Newman

D. A.

(2015). Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English. Personality and Individual Differences, 75, 47–52. https://doi.org/10.1016/j.paid.2014.11.017

15.

Foe

Larson

E. L.

(2016). Reading level and comprehension of research consent forms: An integrative review. Journal of Empirical Research on Human Research Ethics, 11, 31–46. https://doi.org/10.1177/1556264616637483

16.

Garnett

Curtin

Stone

(2022). Suicide mortality in the United States, 2000–2020 (NCHS Data Brief No. 433). National Center for Health Statistics. https://doi.org/10.15620/cdc:114217

17.

Goodman

J. K.

Cryder

C. E.

Cheema

(2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26, 213–224. https://doi.org/10.1002/bdm.1753

18.

Huang

J. L.

Curran

P. G.

Keeney

Poposki

E. M.

DeShon

R. P.

(2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99–114. https://doi.org/10.1007/s10869-011-9231-8

19.

Keith

M. G.

Stevenor

B. A.

McAbee

S. T.

(2022). Scale mean and variance differences in MTurk and non-MTurk samples: A meta-analysis. Journal of Personnel Psychology, 22(1), 1–12. https://doi.org/10.1027/1866-5888/a000309

20.

Keith

M. G.

Tay

Harms

P. D.

(2017). Systems perspective of Amazon Mechanical Turk for organizational research: Review and recommendations. Frontiers in Psychology, 8, Article 1359. https://doi.org/10.3389/fpsyg.2017.01359

21.

Litman

Robinson

Abberbock

(2017). TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior Research Methods, 49, 433–442. https://doi.org/10.3758/s13428-016-0727-z

22.

Maniaci

M. R.

Rogge

R. D.

(2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61–83. https://doi.org/10.1016/j.jrp.2013.09.008

23.

McKay

A. S.

Garcia

D. M.

Clapper

J. P.

Shultz

K. S.

(2018). The attentive and the careless: Examining antecedents of careless responding. Computers in Human Behavior, 84, 295–303. https://doi.org/10.1016/j.chb.2018.03.007

24.

Meade

A. W.

Craig

S. B.

(2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455. https://doi.org/10.1037/a0028085

25.

Moss

A. J.

Rosenzweig

Jaffe

S. N.

Gautam

Robinson

Litman

(2021). Bots or inattentive humans? Identifying sources of low-quality data in online platforms. PsyArXiv. https://doi.org/10.31234/osf.io/wr8ds

26.

Paolacci

Chandler

Stern

L. N.

(2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.

27.

Peer

Brandimarte

Samat

Acquisti

(2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153–163. https://doi.org/10.1016/j.jesp.2017.01.006

28.

Peer

Rothschild

Gordon

Evernden

Damer

(2022). Data quality of platforms and panels for online behavioral research. Behavior Research Methods, 54, 1643–1662. https://doi.org/10.3758/s13428-021-01694-3

29.

Piscopo

K. D.

(2017). Suicidality and death by suicide among middle-aged adults in the United States. Center for Behavioral Health Statistics and Quality. https://www.samhsa.gov/data/sites/default/files/report_3370/ShortReport-3370.html

30.

Robinson

Rosenzweig

Moss

A. J.

Litman

(2019). Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool. PLOS ONE, 14(12), Article e0226394. https://doi.org/10.1371/journal.pone.0226394

31.

Steelman

Z. R.

Hammer

B. I.

Limayem

(2014). Data collection in the digital age: Innovative alternatives to student samples. MIS Quarterly, 38, 355–378. https://doi.org/10.25300/MISQ/2014/38.2.02

32.

Storozuk

Ashley

Delage

Maloney

E. A.

(2020). Got bots? Practical recommendations to protect online survey data from bot attacks. Quantitative Methods for Psychology, 16, 472–481. https://doi.org/10.20982/tqmp.16.5.p472

33.

Varnhagen

C. K.

Gushta

Daniels

Peters

T. C.

Parmar

Law

Hirsch

Sadler Takach

Johnson

(2005). How informed is online informed consent? Ethics & Behavior, 15, 37–48. https://doi.org/10.1207/s15327019eb1501_3

34.

Walter

S. L.

Seibert

S. E.

Goering

O’Boyle

E. H.

(2019). A tale of two sample sources: Do results from online panel data and conventional data converge? Journal of Business and Psychology, 34, 425–452. https://doi.org/10.1007/s10869-018-9552-y

35.

Ward

M. K.

Meade

A. W.

(2018). Applying social psychology to prevent careless responding during online surveys. Applied Psychology, 67, 231–263. https://doi.org/10.1111/apps.12118

36.

Ward

M. K.

Pond

S. B.

(2015). Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys. Computers in Human Behavior, 48, 554–568. https://doi.org/10.1016/j.chb.2015.01.070

37.

Webb

M. A.

Tangney

J. P.

(2022). Too good to be true: Bots and bad data from Mechanical Turk. Perspectives on Psychological Science. Advance online publication. https://doi.org/10.1177/17456916221120027

38.

Zickar

M. J.

Keith

M. G.

(2023). Innovations in sampling: Improving the appropriateness and quality of samples in organizational research. Annual Review of Organizational Psychology and Organizational Behavior, 10, 315–337. https://doi.org/10.1146/annurev-orgpsych-120920-052946