Although the benefits of crowdsourcing research models have been outlined elsewhere, very little attention has been paid to the application of these models to cross-cultural behavioral research. In this manuscript, we delineate two types of crowdsourcing initiatives—researcher crowdsourced and participant crowdsourced. Researcher crowdsourced refers to initiatives where researchers are gathered to work toward a shared goal. Participant crowdsourced refers to those which allow a researcher to gather a large number of participants within a short time frame. We explore the utility of each type of initiative while providing readers with a framework that can be used when deciding whether researcher or participant crowdsourcing initiatives would be most fruitful for their work. Perceived strengths of a researcher crowdsourced initiative with a cross-cultural focus is based on contributor data from Psi Chi’s Network for International Collaborative Exchange (NICE) and is integrated into this framework. Claims are made for the utility of both researcher and participant crowdsourcing as a way to increase generalizability and reliability, decrease time burdens, democratize research, educate individuals on open science, and provide mentorship. These claims are supported with data from NICE contributors.
AlabriA.HunterJ. (2010). Enhancing the quality and trust of citizen science data. In 2010 IEEE sixth international conference on e-science (pp. 81–88). IEEE.
2.
AlzahraniJ. A. (2011). Overcoming barriers to improve research productivity in Saudi Arabia. International Journal of Business and Social Science, 2(19), 50–57.
3.
AndersonS. F.MaxwellS. E. (2017). Addressing the “replication crisis”: Using original studies to design replication studies with appropriate statistical power. Multivariate Behavioral Research, 52(3), 305–324. https://doi.org/10.1080/00273171.2017.1289361
4.
ArnettJ. J. (2008). The neglected 95%: why American psychology needs to become less American. American Psychologist, 63(7), 602–614. https://doi.org/10.1037/14805-008
5.
ArzaV.FressoliM. (2017). Systematizing benefits of open science practices. Information Services & Use, 37(4), 463–474. https://doi.org/10.3233/ISU-170861
6.
AsendorpfJ. B.ConnerM.De FruytF.De HouwerJ.DenissenJ. J.FiedlerK.FiedlerS.FunderD. C.KlieglR.NosekB. A.PeruginiM. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27(2), 108–119. https://doi.org/10.1002/per.1919
BuhrmesterM.KwangT.GoslingS. D. (2011). Amazon’s mechanical Turk a new source of inexpensive, yet high-quality, data?Perspectives on Psychological Science, 6(1), 3–5. https://doi.org/10.1037/14805-009
ColeE. R.StewartA. J. (2001). Invidious comparisons: Imagining a psychology of race and gender beyond differences. Political Psychology, 22(2), 293–308. https://doi.org/10.1111/0162-895X.00240
14.
CouperM. P.TraugottM. W.LamiasM. J. (2001). Web survey design and administration. Public Opinion Quarterly, 65(2), 230–253. https://doi.org/10.1086/322199
Del SavioL.PrainsackB.BuyxA. (2016). Crowdsourcing the Human Gut. Is crowdsourcing also ‘citizen science’?Journal of Science Communication, 15(3), 1–16.
17.
DickinsonJ. L.ZuckerbergB.BonterD. N. (2010). Citizen science as an ecological research tool: Challenges and benefits. Annual Review of Ecology, Evolution, and Systematics, 41(1), 149–172.
18.
EbersoleC. R.AthertonO. E.BelangerA. L.SkulborstadH. M.AllenJ. M.BanksJ. B.BaranskiE.BernsteinM. J.BonfiglioD. B.BoucherL.BrownE. R. (2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68–82. https://doi.org/10.1016/j.jesp.2015.10.012
EitzelM. V.CappadonnaJ. L.Santos-LangC.DuerrR. E.VirapongseA.WestS. E.KybaC.BowserA.CooperC. B.SforziA.MetcalfeA. N. (2017). Citizen science terminology matters: Exploring key terms. Citizen Science: Theory and Practice, 2(1), 2. http://doi.org/10.5334/cstp.96
21.
FisherC. B.HoagwoodK.BoyceC.DusterT.FrankD. A.GrissoT.LevineR. J.MacklinR.SpencerM. B.TakanishiR.TrimbleJ. E. (2002). Research ethics for mental health science involving ethnic minority children and youths. American Psychologist, 57(12), 1024. https://doi.org/10.1037/0003-066X.57.12.1024
22.
FiskeS. T. (2010). Venus and Mars or down to Earth: Stereotypes and realities of gender differences. Perspectives on Psychological Science, 5(6), 688–692. https://doi.org/10.1177/1745691610388768
23.
GelfandM. J.RaverJ. L.NishiiL.LeslieL. M.LunJ.LimB. C.DuanL.AlmaliachA.AngS.ArnadottirJ.AycanZ. (2011). Differences between tight and loose cultures: A 33-nation study. Science, 332(6033), 1100–1104. https://doi.org/10.1126/science.1197754
GraheJ. E.CuccoloK.LeightonD. C.Cramblet AlvarezL. D. (2020). Open science promotes diverse, just, and sustainable research and educational outcomes. Psychology Learning & Teaching, 19(1), 5–20.
26.
HandlerR. A.ConillR. F. (2016). Open data, crowdsourcing and game mechanics. A case study on civic participation in the digital age. Computer Supported Cooperative Work (CSCW), 25(2–3), 153–166. https://doi.org/10.1007/s10606-016-9250-0
27.
HarmsP. D.DeSimoneJ. A. (2015). Caution! MTurk workers ahead—Fines doubled. Industrial and Organizational Psychology, 8(2), 183–190. https://doi.org/10.1017/iop.2015.23
28.
HauserD. J.PaolacciG.ChandlerJ. J. (2018). Common Concerns with MTurk as a Participant Pool: Evidence and solutions. In KardesF.HerrP.SchwarzN (Eds), Handbook in Research Methods in Consumer Psychology. (1st Ed., 319-337). Routledge. https://doi.org/10.31234/osf.io/uq45c
29.
HauserD. J.SchwarzN. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48(1), 400–407. https://doi.org/10.3758/s13428-015-0578-z
JamesonD. A. (1994). Strategies for overcoming barriers inherent in cross-cultural research. Bulletin of the Association for Business Communication, 57(3), 39–41.
33.
JohnL. K.LoewensteinG.PrelecD. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953
34.
KamC. D.WilkingJ. R.ZechmeisterE. J. (2007). Beyond the “narrow data base”: Another convenience sample for experimental research. Political Behavior, 29(4), 415–440. https://doi.org/10.1007/s11109-007-9037-6
35.
KennedyR.CliffordS.BurleighT.JewellR.WaggonerP. (2018). The shape of and solutions to the MTurk quality crisis. Available at SSRN 3272468.
KidwellM. C.LazarevićL. B.BaranskiE.HardwickeT. E.PiechowskiS.FalkenbergL. S.KennettC.SlowikA.SonnleitnerC.Hess-HoldenC.ErringtonT. M. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biology, 14(5), e1002456. http://doi.org/10.1371/journal.pbio.1002456
38.
KleinR.RatliffK.VianelloM.AdamsR.Jr.BahníkS.BernsteinM.BocianK.BrandtM.BrooksB.BrumbaughC.CemalcilarZ. (2014). Data from investigating variation in replicability: A “many labs” replication project. Journal of Open Psychology Data, 2(1), e4. http://doi.org/10.5334/jopd.ad
39.
KleinR. A.VianelloM.HasselmanF.AdamsB. G.AdamsR. B.Jr.AlperS.AveyardM.AxtJ. R.BabalolaM. T.BahníkŠ.BatraR. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. https://doi.org/10.1177/2515245918810225
40.
LarivièreV.GingrasY.SugimotoC. R.TsouA. (2015). Team size matters: Collaboration and scientific impact since 1900. Journal of the Association for Information Science and Technology, 66(7), 1323–1332. https://doi.org/10.1002/asi.23266
41.
LaursenS.HunterA. B.SeymourE.ThiryH.MeltonG. (2010). Undergraduate research in the sciences: Engaging Students in Real Science. John Wiley & Sons.
42.
LeeY. J.AridaJ. A.DonovanH. S. (2017). The application of crowdsourcing approaches to cancer research: A systematic review. Cancer Medicine, 6(11), 2595–2605. https://doi.org/10.1002/cam4.1165
MillerJ. D.CroweM.WeissB.Maples-KellerJ. L.LynamD. R. (2017). Using online, crowdsourcing platforms for data collection in personality disorder research: The example of Amazon’s Mechanical Turk. Personality Disorders: Theory, Research, and Treatment, 8(1), 26–34. https://doi.org/10.1037/per0000191
48.
MoshontzH.CampbellL.EbersoleC. R.IJzermanH.UrryH. L.ForscherP. S.GraheJ. E.McCarthyR. J.MusserE. D.AntfolkJ.CastilleC. M.EvansT. R.FiedlerS.FlakeJ. K.ForeroD. A.JanssenS. M. J.KeeneJ. R.ProtzkoJ.AczelB.. . . ChartierC. R (2018). The psychological science accelerator: Advancing psychology through a distributed collaborative network. Advances in Methods and Practices in Psychological Science, 1(4), 501–515. https://doi.org/10.1177/2515245918797607
49.
MunafòM. R.NosekB. A.BishopD. V.ButtonK. S.ChambersC. D.Du SertN. P.. . . IoannidisJ. P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 1–9. https://doi.org/10.1038/s41562-016-0021
50.
NosekB. A.EbersoleC. R.DeHavenA. C.MellorD. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114
51.
NosekB. A.LakensD. (2014). A method to increase the credibility of published results. Social Psychology, 45(3), 137–141. http://doi.org/10.1027/1864-9335/a000192.
PeerE.VosgerauJ.AcquistiA. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46(4), 1023–1031. https://doi.org/10.3758/s13428-013-0434-y
54.
PiwowarH. A.DayR. S.FridsmaD. B. (2007). Sharing detailed research data is associated with increased citation rate. PLoS One, 2(3), e308. https://doi.org/10.1371/journal.pone.0000308
55.
SantoS. A.EngstromM. E.ReetzL.SchweinleW. E.ReedK. (2009). Faculty productivity barriers and supports at a school of education. Innovative Higher Education, 34(2), 117–129. https://doi.org/10.1007/s10755-009-9098-z
56.
SchmidtS. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2), 90–100. https://doi.org/10.1037/a0015108
57.
SchweinsbergM.MadanN.VianelloM.SommerS. A.JordanJ.TierneyW.AwtreyE.ZhuL. L.DiermeierD.HeinzeJ. E.SrinivasanM. (2016). The pipeline project: Pre-publication independent replications of a single laboratory’s research pipeline. Journal of Experimental Social Psychology, 66, 55–67. https://doi.org/10.1016/j.jesp.2015.10.001
58.
ShankD. B. (2016). Using crowdsourcing websites for sociological research: The case of Amazon Mechanical Turk. The American Sociologist, 47(1), 47–55. https://doi.org/10.1007/s12108-015-9266-9
59.
Sierra-MercadoD.Lázaro-MuñozG. (2018). Enhance diversity among researchers to promote participant trust in precision medicine research. The American Journal of Bioethics, 18(4), 44–46. https://doi.org/10.1080/15265161.2018.1431323
60.
SilberzahnR.UhlmannE. L.MartinD. P.AnselmiP.AustF.AwtreyE.BahníkŠ.BaiF.BannardC.BonnierE.CarlssonR. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. https://doi.org/10.1177/2515245917747646
SwamiV.TranU. S.BarronD.AfhamiR.AiméA.AlmenaraC. A.. . . ArgyridesM. (2020). The Breast Size Satisfaction Survey (BSSS): Breast size dissatisfaction and its antecedents and outcomes in women from 40 nations. Body Image, 32, 199–217. https://doi.org/10.1016/j.bodyim.2020.01.006
63.
SyedM.AzmitiaM.CooperC. R. (2011). Identity and academic success among underrepresented ethnic minorities: An interdisciplinary review and integration. Journal of Social Issues, 67(3), 442–468. https://doi.org/10.1111/j.1540-4560.2011.01709.x
64.
TahamtanI.AfsharA. S.AhamdzadehK. (2016). Factors affecting number of citations: a comprehensive review of the literature. Scientometrics, 107(3), 1195–1225. https://doi.org/10.1007/s11192-016-1889-2.
van’t VeerA. E.Giner-SorollaR. (2016). Pre-registration in social psychology—A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12. https://doi.org/10.1016/j.jesp.2016.03.004
WaggeJ.BaciuC.BanasK.NadlerJ. T.SchwarzS.WeisbergY.IJzermanH.LegateN.GraheJ. (2018). A Demonstration of the collaborative replication and education project: Replication attempts of the red-romance effect. Collabra: Psychology, 5(1), 5. https://doi.org/10.1525/collabra.177
69.
WaggonerP.KennedyR.CliffordS. (2019). Detecting fraud in online surveys by tracing, scoring, and visualizing IP addresses. Journal of Open Source Software, 4(37), 1285. https://doi.org/10.21105/joss.01285
70.
WhitlaP. (2009). Crowdsourcing and its application in marketing activities. Contemporary Management Research, 5(1), 15–28. https://doi.org/10.7903/cmr.1145
71.
WigginsA.CrowstonK. (2011). From conservation to crowdsourcing: A typology of citizen science. In 2011 44th Hawaii international conference on system sciences (pp. 1–10). IEEE.
72.
WuchtyS.JonesB. F.UzziB. (2007). The increasing dominance of teams in production of knowledge. Science, 316(5827), 1036–1039. https://doi.org/10.1126/science.1136099
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
