Sage Journals: Discover world-class research

Abstract

The potential role of brief online studies in changing the types of research and theories likely to evolve is examined in the context of earlier changes in theory and methods in social and personality psychology, changes that favored low-difficulty, high-volume studies. An evolutionary metaphor suggests that the current publication environment of social and personality psychology is a highly competitive one, and that academic survival and reproduction processes (getting a job, tenure/promotion, grants, awards, good graduate students) can result in the extinction of important research domains. Tracking the prevalence of brief online studies, exemplified by studies using Amazon Mechanical Turk, in three top journals (Journal of Personality and Social Psychology, Personality and Social Psychology Bulletin, Journal of Experimental Social Psychology) reveals a dramatic increase in their frequency and proportion. Implications, suggestions, and questions concerning this trend for the field and questions for its practitioners are discussed.

Keywords

MTurk research methods/assessment philosophy of science theory

Get full access to this article

View all access options for this article.

References

Brawley

A. M.

Pury

C. L. S.

(2016). Work experiences on MTurk: Job satisfaction, turnover, and information sharing. Computers in Human Behavior, 54, 531-546. doi:10.1016/j.chb.2015.08.031

Cheung

J. H.

Burns

D. K.

Sinclair

R. R.

Sliter

(2017). Amazon Mechanical Turk in organizational psychology: An evaluation and practical recommendations. Journal of Business and Psychology, 32, 347-361. doi:10.1007/s10869-016-9458-5

Crandall

C. S.

Sherman

J. W.

(2016). On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology, 66, 93-99. doi:10.1016/j.jesp.2015.10.002

Finkel

E. J.

Eastwick

P. W.

Reis

H. T.

(2015). Best research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science. Journal of Personality and Social Psychology, 108, 275-297. doi:10.1037/pspi0000007

Finkel

E. J.

Eastwick

P. W.

Reis

H. T.

(2017). Replicability and other features of a high-quality science: Toward a balanced and empirical approach. Journal of Personality and Social Psychology, 113, 244-253. doi:10.1037/pspi0000075

Fleischer

Mead

A. D.

Huang

(2015). Inattentive responding in MTurk and other online samples. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8, 196-202. doi:10.1017/iop.2015.25

Goodman

J. K.

Cryder

C. E.

Cheema

(2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26, 213-224. doi:10.1002/bdm.1753

Harms

P. D.

DeSimone

J. A.

(2015). Caution! MTurk workers ahead—Fines doubled. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8, 183-190. doi:10.1017/iop.2015.23

Hauser

D. J.

Schwarz

(2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48, 400-407. doi:10.3758/s13428-015-0578-z

10.

Landers

R. N.

Behrend

T. S.

(2015). An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8, 142-164.

11.

Meehl

P. E.

(1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1, 108-141.

12.

Reis

H. T.

Stiller

(1992). Publication trends in JPSP: A three-decade review. Personality and Social Psychology Bulletin, 18, 465-472. doi:10.1177/0146167292184011

13.

Saleem

Anderson

C. A.

(2013). Arabs as terrorists: Effects of stereotypes within violent contexts on attitudes, perceptions and affect. Psychology of Violence, 3, 84-99. doi:10.1037/a0030038

14.

Saleem

Anderson

C. A.

Barlett

C. P.

(2015). Assessing helping and hurting behaviors through the tangram help/hurt task. Personality and Social Psychology Bulletin, 41, 1345-1362.

15.

Saleem

Prot

Anderson

C. A.

Lemieux

A. F.

(2017). Exposure to Muslims in media and support for public policies harming Muslims. Communication Research, 44, 841-869.

16.

Schmidt

G. B.

(2015). Fifty days an MTurk worker: The social and motivational context for Amazon Mechanical Turk workers. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8, 165-171. doi:10.1017/iop.2015.20

17.

Siegel

J. T.

Navarro

M. A.

Thomson

A. L.

(2015). The impact of overtly listing eligibility requirements on MTurk: An investigation involving organ donation, recruitment scripts, and feelings of elevation. Social Science & Medicine, 142, 256-260. doi:10.1016/j.socscimed.2015.08.020

18.

Simons

D. J.

Shoda

Lindsay

D. S.

(2017). Constraints on generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science, 12, 1123-1128. doi:10.1177/1745691617708630

19.

Smith

N. A.

Sabat

I. E.

Martinez

L. R.

Weaver

(2015). A convenient solution: Using MTurk to sample from hard-to-reach populations. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8, 220-228. doi:10.1017/iop.2015.29

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

The MTurkification of Social and Personality Psychology

Abstract

Keywords

Get full access to this article

References

Supplementary Material