Although participants with psychiatric symptoms, specific risk factors, or rare demographic characteristics can be difficult to identify and recruit for participation in research, participants with these characteristics are crucial for research in the social, behavioral, and clinical sciences. Online research in general and crowdsourcing software in particular may offer a solution. However, no research to date has examined the utility of crowdsourcing software for conducting research on psychopathology. In the current study, we examined the prevalence of several psychiatric disorders and related problems, as well as the reliability and validity of participant reports on these domains, among users of Amazon’s Mechanical Turk. Findings suggest that crowdsourcing software offers several advantages for clinical research while providing insight into potential problems, such as misrepresentation, that researchers should address when collecting data online.
AharoniE.Sinnott-ArmstrongW.KiehlK. A. (2012). Can psychopathic offenders discern moral wrongs? A new look at the moral/conventional distinction. Journal of Abnormal Psychology, 121, 484–497. doi:10.1037/a0024796.
2.
AltmanE. G.HedekerD.PetersonJ. L.DavisJ. M. (1997). The Altman Self-Rating Mania Scale. Biological Psychiatry, 42, 948–955. doi:10.1016/S0006-3223(96)00548-3
3.
AnderssonG.CarlbringP.HolmströmA.SparthanE.FurmarkT.Nilsson-IhrfeltE.. . . EkseliusL. (2006). Internet-based self-help with therapist feedback and in vivo group exposure for social phobia: A randomized controlled trial. Journal of Consulting and Clinical Psychology, 74, 677–686. doi:10.1037/0022-006X.74.4.677
4.
ArbisiP. A.Ben-PorathY. S. (1995). An MMPI-2 infrequent response scale for use with psychopathological populations: The Infrequency-Psychopathology Scale, F(p). Psychological Assessment, 7, 424–431. doi:10.1037/1040-3590.7.4.424
5.
BarghJ. A.McKennaK. Y. A. (2004). The Internet and social life. Annual Review of Psychology, 55, 573–590. doi:10.1146/annurev.psych.55.090902.141922
6.
BeckA. T.EpsteinN.BrownG.SteerR. A. (1988). An inventory for measuring clinical anxiety: Psychometric properties. Journal of Consulting and Clinical Psychology, 56, 893–897. doi:10.1037/0022-006X.56.6.893
7.
BeckA. T.WardC. H.MendelsonM.MockJ.ErbaughJ. (1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 561–571. doi:10.1001/archpsyc.1961.01710120031004
8.
BerinskyA. J.HuberG. A.LenzG. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20, 351–368. doi:10.1093/pan/mpr057
9.
BernatJ. A.RonfeldtH. M.CalhounK. S.AriasI. (1998). Prevalence of traumatic events and peritraumatic predictors of posttraumatic stress symptoms in a nonclinical sample of college students. Journal of Traumatic Stress, 11, 645–664. doi:10.1023/A:1024485130934
10.
BinikY. M.MahK.KieslerS. (1999). Ethical issues in conducting sex research on the Internet. Journal of Sex Research, 36, 82–90. doi:10.1080/00224499909551971
11.
BlakeD. D.WeathersF. W.NagyL. M.KaloupekD. G.GusmanF. D.CharneyD. S.KeaneT. M. (1995). The development of a Clinician-Administered PTSD Scale. Journal of Traumatic Stress, 8, 75–90. doi:10.1007/BF02105408
12.
BrownR. L.RoundsL. A. (1995). Conjoint screening questionnaires for alcohol and other drug abuse: Criterion validity in a primary care practice. Wisconsin Medical Journal, 94, 135–140.
13.
BuhrmesterM.KwangT.GoslingS. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data?Perspectives on Psychological Science, 6, 3–5. doi:10.1177/1745691610393980
14.
ButcherJ. N.DahlstromW. G.GrahamJ. R.TellegenA.KaemmerB. (1989). MMPI–2: Manual for administration and scoring. Minneapolis, MN: University of Minnesota Press.
15.
ByrnesJ. P.MillerD. C.SchaferW. D. (1999). Gender differences in risk taking: A meta-analysis. Psychological Bulletin, 125, 367–383. doi:10.1037/0033-2909.125.3.367
16.
CarlbringP.GunnarsdóttirM.HedensjöL.AnderssonG.EkseliusL.FurmarkT. (2007). Treatment of social phobia: Randomised trial of Internet-delivered cognitive-behavioural therapy with telephone support. British Journal of Psychiatry, 190, 123–128. doi:10.1192/bjp.bp.105.020107
17.
ChandlerJ.MuellerP.PaolacciG. (2013). Methodological concerns and advanced uses of Amazon Mechanical Turk in psychological research. Manuscript submitted for publication.
18.
CorriganP. (2004). How stigma interferes with mental health care. American Psychologist, 59, 614–625. doi:10.1037/0003-066X.59.7.614
19.
CouperM. (2008). Designing effective Web surveys. New York, NY: Cambridge University Press.
20.
DienerE.EmmonsR. A.LarsenR. J.GriffinS. (1985). The Satisfaction With Life Scale. Journal of Personality Assessment, 49, 71–75. doi:10.1207/s15327752jpa4901_13
21.
DooleyD.CatalanoR.WilsonG. (1994). Depression and unemployment: Panel findings from the Epidemiologic Catchment Area Study. American Journal of Community Psychology, 22, 745–765. doi:10.1007/BF02521557
22.
DownsJ. S.HolbrookM.PeelE. (2012, October). Screening participants on Mechanical Turk: Techniques and justifications. Paper presented at the annual conference of the Association for Consumer Research, Vancouver, BC, Canada.
23.
Estellés-ArolasE.Ladrün-de-GuevaraF. (2012). Towards an integrated crowdsourcing definition. Journal of Information Science, 38, 189–200.
HortonJ. J.ChiltonL. B. (2010). The labor economics of paid crowdsourcing. In Proceedings from EC ’10: The 11th ACM Conference on Electronic Commerce (pp. 209–218). New York, NY: ACM. doi:10.1145/1807342.1807376
26.
JoinsonA. N. (2001). Self-disclosure in computer mediated communication: The role of self-awareness and visual anonymity. European Journal of Social Psychology, 31, 177–192. doi:10.1002/ejsp.36
27.
JonesD. N.PaulhusD. L. (2011). The role of impulsivity in the dark triad of personality. Personality and Individual Differences, 51, 679–682. doi:10.1016/j.paid.2011.04.011
28.
KesslerR. C.BerglundP. A.DemlerO.JinR.WaltersE. E. (2005). Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication (NCS-R). Archives of General Psychiatry, 62, 593–602. doi:10.1001/archpsyc.62.6.593
29.
KesslerR. C.ChiuW. T.DemlerO.WaltersE. E. (2005). Prevalence, severity, and comorbidity of twelve-month DSM-IV disorders in the National Comorbidity Survey Replication (NCS-R). Archives of General Psychiatry, 62, 617–627. doi:10.1001/archpsyc.62.6.617
30.
KitturA.ChiE. H.SuhB. (2008). Crowdsourcing user studies with Mechanical Turk. In Proceedings of the ACM Conference on Human Factors in Computing Systems (pp. 453–456). New York, NY: ACM.
31.
LambertM. V.SeniorC.PhillipsM. L.DavidA. S. (2000). Depersonalization in cyberspace. Journal of Nervous and Mental Disease, 188, 764–771. doi:10.1097/00005053-200011000-00007
32.
LexB. W. (1991). Some gender differences in alcohol and polysubstance users. Health Psychology, 10, 121–132. doi:10.1037/0278-6133.10.2.121
33.
LiebowitzM. R. (1987). Social phobia. Modern Problems of Pharmacopsychiatry, 22, 141–173.
34.
MasonW.SuriS. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44, 1–23. doi:10.3758/s13428-011-0124-6
35.
MorrisR.PicardR. (2012). Crowdsourcing collective emotional intelligence. In MaloneT. W.von AhnL. (Eds.), Proceedings of Collective Intelligence 2012. arXiv:1204.2991
NunnallyJ. C. (1978). Psychometric theory. New York, NY: McGraw-Hill.
38.
O’NeilK. M.PenrodsD. (2001). Methodological variables in web-based research that may affect results: Sample type, monetary incentives, and personal information. Behavioral Research Methods, Instruments, & Computers, 33, 226–233. doi:10.3758/BF03195369
39.
PaolacciG.ChandlerJ.IpeirotisP. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.
40.
PatelM. X.DokuV.TennakoonL. (2003). Challenges in recruitment of research participants. Advances in Psychiatric Treatment, 9, 229–238. doi:10.1192/apt.9.3.229
41.
RandD. G. (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172–179. doi:10.1016/j.jtbi.2011.03.004
42.
RinckM.BeckerE. S. (2005). A comparison of attentional biases and memory biases in women with social phobia and major depression. Journal of Abnormal Psychology, 114, 62–74. doi:10.1037/0021-843X.114.1.62
43.
RodriguezE.AllenJ. A.FrongilloE. A.ChandraP. (1999). Unemployment, depression, and health: A look at the African-American community. Journal of Epidemiology and Community Health, 53, 335–342. doi:10.1136/jech.53.6.335
44.
RosenthalR.RosnowR. L. (2009). Artifacts in behavioral research. New York, NY: Oxford University Press.
45.
SartoriusN. (1998). Stigma: What can psychiatrists do about it?Lancet, 352, 1058–1059. doi:10.1016/S0140-6736(98)08008-8
46.
ShapiroD. N.StewartA. J. (2011). Parenting stress, perceived child regard, and depression in stepmothers and biological mothers. Family Relations, 60, 533–544. doi:10.1111/j.1741-3729.2011.00665.x
47.
ShepherdR. M.EdelmannR. J. (2005). Reasons for Internet use and social anxiety. Personality and Individual Differences, 39, 949–958. doi:10.1016/j.paid.2005.04.001
48.
SimmonsJ.NelsonL.SimonsohnU. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632
49.
SnowdenL. R.CheungF. K. (1990). Use of inpatient mental health services by members of ethnic minority groups. American Psychologist, 45, 347–355. doi:10.1037/0003-066X .45.3.347
50.
StevensS. B.MorrisT. L. (2007). College dating and social anxiety: Using the Internet as a means of connecting to others. CyberPsychology & Behavior, 10, 680–688. doi:10.1089/cpb.2007.9970
51.
SuriS.GoldsteinD. G.MasonW. A. (2011, August). Honesty in an online labor market. In von AhnL.IpeirotisP. G. (Eds.), Papers from the 2011 AAAI Workshop. Menlo Park, CA: AAAI Press.
VranaS.LauterbachD. (1994). Prevalence of traumatic events and post-traumatic psychological symptoms in a nonclinical sample of college students. Journal of Traumatic Stress, 7, 289–302. doi:10.1007/BF02102949
56.
WeidmanA. C.FernandezK. C.LevinsonC. A.AugustineA. A.LarsenR. J.RodebaughT. L. (2012). Compensatory Internet use among individuals higher in social anxiety and its implications for well-being. Personality and Individual Differences, 53, 191–195. doi:10.1016/j.paid.2012.03.003
57.
WeinstockL. S. (1999). Gender differences in the presentation and management of social anxiety disorder. Journal of Clinical Psychiatry, 60, 9–13. doi:10.1007/BF02102949
58.
YanceyA. K.OrtegaA. N.KumanyikaS. K. (2006). Effective recruitment and retention of minority research participants. Annual Review of Public Health, 27, 1–28. doi:10.1146/annurev.publhealth.27.021405.102113
