Management scholars have long depended on convenience samples to conduct research involving human participants. However, the past decade has seen an emergence of a new convenience sample: online panels and online panel participants. The data these participants provide—online panel data (OPD)—has been embraced by many management scholars owing to the numerous benefits it provides over “traditional” convenience samples. Despite those advantages, OPD has not been warmly received by all. Currently, there is a divide in the field over the appropriateness of OPD in management scholarship. Our review takes aim at the divide with the goal of providing a common understanding of OPD and its utility and providing recommendations regarding when and how to use OPD and how and where to publish it. To accomplish these goals, we inventoried and reviewed OPD use across 13 management journals spanning 2006 to 2017. Our search resulted in 804 OPD-based studies across 439 articles. Notably, our search also identified 26 online panel platforms (“brokers”) used to connect researchers with online panel participants. Importantly, we offer specific guidance to authors, reviewers, and editors, having implications for both micro and macro management scholars.
AguinisH.LawalS. O.2013. eLancing: A review and research agenda for bridging the science–practice gap. Human Resource Management Review, 23: 6-17.
2.
AppelbaumM.CooperH.KlineR. B.Mayo-WilsonE.NezuA. M.RaoS. M.2018. Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board Task Force report. American Psychologist, 73: 3-25.
3.
AustF.DiedenhofenB.UllrichS.MuschJ.2013. Seriousness checks are useful to improve data validity in online research. Behavior Research Methods, 45: 527-535.
4.
BaerM. D.BundyJ.GarudN.KimJ. K.2018. The benefits and burdens of organizational reputation for employee well-being: A conservation of resource approach. Personnel Psychology, 71: 571-595.
5.
BehrendT. S.SharekD. J.MeadeA. W.WiebeE. N.2011. The viability of crowdsourcing for survey research. Behavior Research Methods, 43: 800-813.
6.
BonoJ. E.GlombT. M.ShenW.KimE.KochA. J.2013. Building positive resources: Effects of positive events and positive reflection on work stress and health. Academy of Management Journal, 56: 1601-1627.
7.
BoswellW. R.Olson-BuchananJ. B.HarrisT. B.2014. I cannot afford to have a life: Employee adaptation to feelings of job insecurity. Personnel Psychology, 67: 887-915.
8.
BrawleyA. M.PuryC. L.2016. Work experiences on MTurk: Job satisfaction, turnover, and information sharing. Computers in Human Behavior, 54: 531-546.
9.
BuhrmesterM.KwangT.GoslingS. D.2011. Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data?Perspectives on Psychological Science, 6: 3-5.
10.
CallegaroM.BakerR. P.BethlehemJ.GöritzA. S.KrosnickJ. A.LavrakasP. J.2014. Online panel research: A data quality perspective. New York: Wiley.
11.
ChandlerJ.MuellerP.PaolacciG.2014. Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46: 112-130.
12.
ChandlerJ.PaolacciG.PeerE.MuellerP.RatliffK. A.2015. Using nonnaive participants can reduce effect sizes. Psychological Science, 26: 1131-1139.
13.
ChandlerJ.ShapiroD.2016. Conducting clinical research using crowdsourced convenience samples. Annual Review of Clinical Psychology, 12: 53-81.
14.
CheungJ. H.BurnsD. K.SinclairR. R.SliterM.2017. Amazon Mechanical Turk in organizational psychology: An evaluation and practical recommendations. Journal of Business Psychology, 32: 347-361.
15.
ChuaR. Y. J.MorrisM. W.IngramP.2009. Guanxi vs. networking: Distinctive configurations of affect- and cognition-based trust in the networks of Chinese vs. American managers. Journal of International Business Studies, 40: 490-509.
16.
ClarkT.2011. Gaining and maintaining access: Exploring the mechanisms that support and challenge the relationship between gatekeepers and researchers. Qualitative Social Work, 10: 485-502.
17.
CohenJ.1992. A power primer. Psychological Bulletin, 112: 155-159.
18.
CrillyD.NiN.JiangY.2016. Do-no-harm versus do-good social responsibility: Attributional thinking and the liability of foreignness. Strategic Management Journal, 37: 1316-1329.
19.
CroneD. L.WilliamsL. A.2017. Crowdsourcing participants for psychological research in Australia: A test of microworkers. Australian Journal of Psychology, 69: 39-47.
20.
CrumpM. J.McDonnellJ. V.GureckisT. M.2013. Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLoS One, 8: e57410.
21.
CunliffeA. L.AlcadipaniR.2016. The politics of access in fieldwork: Immersion, backstage dramas, and deception. Organizational Research Methods, 19: 535-561.
22.
DeVoeS. E.HouseJ.2016. Replications with MTurkers who are naïve versus experienced with academic studies: A comment on Connors, Khamitov, Moroz, Campbell, and Henderson (2015). Journal of Experimental Social Psychology, 67: 65-67.
23.
DownsJ. S.HolbrookM. B.PeelE.2012. Screening participants on Mechanical Turk: Techniques and justifications. Advances in Consumer Research, 40: 113-114.
24.
ElenkovD. S.JudgeW.WrightP.2005. Strategic leadership and executive innovation influence: An international multi-cluster comparative study. Strategic Management Journal, 26: 665-682.
25.
FeitosaJ.JosephD. L.NewmanD. A.2015. Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English. Personality and Individual Differences, 75: 47-52.
26.
FowlerF. J.2014. Survey research methods. Thousand Oaks, CA: Sage.
27.
GamacheD. L.McNamaraG.MannorM. J.JohnsonR. E.2015. Motivated to acquire? The impact of CEO regulatory focus on firm acquisitions. Academy of Management Journal, 58: 1261-1282.
28.
GleibsI. H.2017. Are all “research fields” equal? Rethinking practice for the use of data from crowdsourcing market places. Behavior Research Methods, 49: 1333-1342.
29.
GoodmanJ. K.CryderC. E.CheemaA.2013. Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26: 213-224.
GöritzA. S.2007. Using online panels in psychological research. In JoinsonA. N.McKennaK. Y. A.PostmesT.ReipsU.-D. (Eds.), The Oxford handbook of Internet psychology: 473-485. Oxford, UK: Oxford University Press.
32.
HarmonD. J.KimP. H.MayerK. J.2015. Breaking the letter vs. spirit of the law: How the interpretation of contract violations affects trust and the management of relationships. Strategic Management Journal, 36: 497-517.
33.
HauserD. J.SchwarzN.2016. Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48: 400-407.
34.
HuangJ. L.BowlingN. A.LiuM.LiY.2015. Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30: 299-311.
35.
IlgenD. R.1986. Laboratory research: A question of when, not if. In LockeE. A. (Ed.), Generalizing from laboratory to field settings: 257-267. Lexington, MA: Heath.
36.
InskoC. A.WildschutT.CohenT. R.2013. Interindividual–intergroup discontinuity in the prisoner’s dilemma game: How common fate, proximity, and similarity affect intergroup competition. Organizational Behavior and Human Decision Processes, 120: 168-180.
37.
JudgeT. A.CableD. M.ColbertA. E.RynesS. L.2007. What causes a management article to be cited—article, author, or journal?Academy of Management Journal, 50: 491-506.
38.
JudgeT. A.IliesR.ScottB. A.2006. Work–family conflict and emotions: Effects at work and at home. Personnel Psychology, 59: 779-814.
39.
KeithM. G.HarmsP. D.2016. Is Mechanical Turk the answer to our sampling woes?Industrial and Organizational Psychology, 9: 162-167.
40.
KleinR. A.RatliffK. A.VianelloM.AdamsR. B.Jr.BahnikS.BernsteinM. J.. . . NosekB. A.2014. Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45: 142-152.
41.
LandersR. N.BehrendT. S.2015. An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology, 8: 142-164.
42.
LiH.KuoC.RussellM. G.1999. The impact of perceived channel utilities, shopping orientations, and demographics on the consumer’s online buying behavior. Journal of Computer-Mediated Communication, 5: 1-23.
43.
LitmanL.RobinsonJ.RosenzweigC.2015. The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk. Behavior Research Methods, 47: 519-528.
44.
LittleL. M.MajorV. S.HinojosaA. S.NelsonD. L.2015. Professional image maintenance: How women navigate pregnancy in the workplace. Academy of Management Journal, 58: 8-37.
45.
LovettM.BajabaS.LovettM.SimmeringM. J.2018. Data quality from crowdsourced surveys: A mixed method inquiry into perceptions of Amazon’s Mechanical Turk masters. Applied Psychology: An International Review, 67: 339-366.
46.
MasonW.SuriS.2012. Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44: 1-23.
MillerJ. D.CroweM.WeissB.Maples-KellerJ. L.LynamD. R.2017. Using online, crowdsourcing platforms for data collection in personality disorder research: The example of Amazon’s Mechanical Turk. Personality Disorders: Theory, Research, and Treatment, 8: 26-34.
49.
MochonD.FrederickS.2013. Anchoring in sequential judgments. Organizational Behavior and Human Decision Processes, 122: 69-73.
50.
NgT. W.FeldmanD. C.2012. The effects of organizational and community embeddedness on work-to-family and family-to-work conflict. Journal of Applied Psychology, 97: 1233-1251.
51.
NgT. W.FeldmanD. C.2015. Idiosyncratic deals and voice behavior. Journal of Management, 41: 893-928.
52.
PaolacciG.ChandlerJ.2014. Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23: 184-188.
53.
PaolacciG.ChandlerJ.IpeirotisP. G.2010. Running experiments on Amazon Mechanical Turk. Judgement and Decision Making, 5: 411-419.
54.
PeerE.BrandimarteL.SamatS.AcquistiA.2017. Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70: 153-163.
55.
PeerE.VosgerauJ.AcquistiA.2014. Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46: 1023-1031.
56.
PetersonR. A.MerunkaD. R.2014. Convenience samples of college students and research reproducibility. Journal of Business Research, 67: 1035-1041.
57.
PhillipsJ. M.GullyS. M.McCarthyJ. E.CastellanoW. G.KimM. S.2014. Recruiting global travelers: The role of global travel recruitment messages and individual differences in perceived fit, attraction, and job pursuit intentions. Personnel Psychology, 67, 153-201.
58.
PiccoloR. F.ColquittJ. A.2006. Transformational leadership and job behaviors: The mediating role of core job characteristics. Academy of Management Journal, 49: 327-340.
59.
PodsakoffP. M.MacKenzieS. B.LeeJ.-Y.PodsakoffN. P.2003. Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88: 879-903.
60.
RamseyS. R.ThompsonK. L.McKenzieM.RosenbaumA.2016. Psychological research in the internet age: The quality of web-based data. Computers in Human Behavior, 58: 354-360.
61.
RouseS. V.2015. A reliability analysis of Mechanical Turk data. Computers in Human Behavior, 43: 304-307.
62.
ShadishW. R.CookT. D.CampbellD. T.2002. Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton-Mifflin.
63.
ShapiraZ.ShaverJ. M.2014. Confounding changes in averages with marginal effects: How anchoring can destroy economic value in strategic investment assessments. Strategic Management Journal, 35: 1414-1426.
64.
SimmonsJ. P.NelsonL. D.SimonsohnU.2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22: 1359-1366.
65.
SmithN. A.SabatI. E.MartinezL. R.WeaverK.XuS.2015. A convenient solution: Using MTurk to sample from hard-to-reach populations. Industrial and Organizational Psychology, 8: 220-228.
66.
StritchJ. M.PedersenM. J.TaggartG.2017. The opportunities and limitations of using Mechanical Turk (Mturk) in public administration and management scholarship. International Public Management Journal, 20: 489-511.
67.
SwaabR. I.PhillipsK. W.SchaererM.2016. Secret conversation opportunities facilitate minority influence in virtual groups: The influence on majority power information processing, and decision quality. Organizational Behavior and Human Decision Processes, 133, 17-22.
68.
Texas A&M University & University of Georgia. 2018. Texas A&M/University of Georgia rankings of management department research productivity. Retrieved from http://www.tamugarankings.com/rankings/2017-2/
69.
Tosti-KharasJ.ConleyC.2016. Coding psychological constructs in text using Mechanical Turk: A reliable, accurate, and efficient alternative. Frontiers in Psychology, 7: 741.
70.
TracyS. J.2013. Qualitative research methods: Collecting evidence, crafting analysis, communicating impact. New York: Wiley-Blackwell.
WalterS.SeibertS.GoeringD. D.O’BoyleE. H.2018. A tale of two sample sources: Do results from online panel data and conventional data converge?Journal of Business and Psychology. Advance online publication. doi:10.1007/s10869-018-9552-y
73.
WooS. E.KeithM.ThorntonM. A.2015. Amazon Mechanical Turk for industrial and organizational psychology: Advantages, challenges, and practical recommendations. Industrial and Organizational Psychology, 8: 171-179.
74.
WowakA. J.MannorM. J.ArrfeltM.McNamaraG.2016. Earthquake or glacier? How CEO charisma manifests in firm strategy over time. Strategic Management Journal, 37: 586-603.
75.
YamagishiT.MifuneN.LiY.ShinadaM.HashimotoH.HoritaY.. . . SimunovicD.2013. Is behavioral pro-sociality game-specific? Pro-social preference and expectations of pro-sociality. Organizational Behavior and Human Decision Processes, 120: 260-271.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
