Is there a point within a self-report questionnaire where participants will start responding carelessly? If so, then after how many items do participants reach that point? And what can researchers do to encourage participants to remain careful throughout the entirety of a questionnaire? We conducted two studies (Study 1 N = 358; Study 2 N = 129) to address these questions. Our results found (a) consistent evidence that participants responded more carelessly as they progressed further into a questionnaire, (b) mixed evidence that participants who were warned that carelessness would be punished displayed smaller increases in carelessness, and (c) mixed evidence that increases in carelessness were greater within an unproctored online study (Study 1) than within a proctored laboratory study (Study 2). These findings help address when and why careless responding is likely to occur, and they suggest effective preventive strategies.
Ben-PorathY. S.TellegenA. (2008). The Minnesota Multiphasic Personality Inventory-2 Restructured Form: Manual for administration, scoring, and interpretation. University of Minnesota Press.
2.
BerryD. T.WetterM. W.BaerR. A.LarsenL.ClarkC.MonroeK. (1992). MMPI–2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4, 340–345.
3.
BowlingN. A.HuangJ. L.BraggC. B.KhazonS.LiuM.BlackmoreC. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111, 218–229.
4.
CarpenterB.GelmanA.HoffmanM. D.LeeD.GoodrichB.BetancourtM.BrubakerM.GuoJ.LiP.RiddellA. (2017). Stan: A probabilistic programming language. Journal of Statistical Software, 76.
5.
CostaP. T.McCraeR. R. (1992). Revised NEO Personality Inventory (NEO-PI-R) and NEO Five-Factor Inventory (NEO-FFI) professional manual. Psychological Assessment Resources, Inc.
6.
CredéM. (2010). Random responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement, 70, 596–612.
7.
CurranP. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19.
8.
DeSimoneJ. A.HarmsP. D.DeSimoneA. J. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36, 171–181.
9.
ElliottM. R.GalloJ. J.Ten HaveT. R.BognerH. R.KatzI. R. (2005). Using a Bayesian latent growth curve model to identify trajectories of positive affect and negative events following myocardial infarction. Biostatistics, 6, 119–143.
10.
FervahaG.RemingtonG. (2013). Invalid responding in questionnaire-based research: Implications for the study of schizotypy. Psychological Assessment, 25, 1355–1360.
11.
GalesicM.BosnjakM. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73, 349–360.
12.
GibsonA. M.BowlingN. A. (2020). The effects of questionnaire length and behavioral consequences on careless responding. European Journal of Psychological Assessment, 36, 410–420.
13.
GoldbergL. R. (1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. In MervieldeIDearyI.De FruytF.OstendorfF. (Eds.), Personality psychology in Europe (Vol. 7, pp. 7–28). Tilburg University Press.
HuangJ. L.BowlingN. A.LiuM.LiY. (2015). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30, 299–311.
18.
HuangJ. L.CurranP. G.KeeneyJ.PoposkiE. M.DeShonR. P. (2012). Detecting and deterring insufficient effort respond to surveys. Journal of Business and Psychology, 27, 99–114.
19.
HuangJ. L.LiuM.BowlingN. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100, 828–845.
20.
JohnsonJ. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39, 103–129.
21.
KrosnickJ. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.
22.
KrosnickJ. A.HolbrookA. L.BerentM. K.CarsonR. T.HanemannW. M.KoppR. J.MitchellR. C.PresserS.RuudP. A.SmithV. K.MoodyW. R.GreenM. C.ConawayM. (2002). The impact of “no opinion” response options on data quality: Non-attitude reduction or an invitation to satisfice?Public Opinion Quarterly, 66, 371–403.
23.
KussM.JäkelF.WichmannF. A. (2005). Bayesian inference for psychometric functions. Journal of Vision, 5, 478–492.
24.
ManiaciM. R.RoggeR. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61–83.
25.
McArdleJ. J. (1988). Dynamic but structural equation modeling of repeated measures data. In NesselroadeJ. R.CattellR. B. (Eds.), Handbook of multivariate experimental psychology (pp. 561–614). Springer.
26.
McGrathR. E.MitchellM.KimB. H.HoughL. (2010). Evidence for response bias as a source of error variance in applied assessment. Psychological Bulletin, 136, 450–470.
27.
MeadeA. W.CraigS. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.
28.
MoonY. (1999). The effects of physical distance and response latency on persuasion in computer-mediated communication and human—computer communication. Journal of Experimental Psychology: Applied, 5, 379–392.
29.
RoghanizadM. M.BohnsV. K. (2017). Ask in person: You’re less persuasive than you think over email. Journal of Experimental Social Psychology, 69, 223–226.
30.
Stan Development Team. (2018). Stan modeling language users guide and reference manual, version 2.18.0. http://mc-stan.org
31.
TourangeauR. (1984). Cognitive science and survey methods. In JabineT. B.StrafM. L.TanurJ. M.TourangeauR. (Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73–100). National Academy Press.
32.
WardM. K.MeadeA. W. (2018). Applying social psychology to prevent careless responding during online surveys. Applied Psychology: An International Review, 67, 231–263.
33.
WardM. K.PondS. B. (2015). Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys. Computers in Human Behavior, 48, 554–568.
34.
WrightD. B.LondonK. (2009). Modern regression techniques using R. Sage.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
