Will the Questions Ever End? Person-Level Increases in Careless Responding During Questionnaire Completion

Abstract

Is there a point within a self-report questionnaire where participants will start responding carelessly? If so, then after how many items do participants reach that point? And what can researchers do to encourage participants to remain careful throughout the entirety of a questionnaire? We conducted two studies (Study 1 N = 358; Study 2 N = 129) to address these questions. Our results found (a) consistent evidence that participants responded more carelessly as they progressed further into a questionnaire, (b) mixed evidence that participants who were warned that carelessness would be punished displayed smaller increases in carelessness, and (c) mixed evidence that increases in carelessness were greater within an unproctored online study (Study 1) than within a proctored laboratory study (Study 2). These findings help address when and why careless responding is likely to occur, and they suggest effective preventive strategies.

Keywords

careless responding insufficient effort responding random responding response effort questionnaire length

Get full access to this article

View all access options for this article.

References

Ben-Porath

Y. S.

Tellegen

(2008). The Minnesota Multiphasic Personality Inventory-2 Restructured Form: Manual for administration, scoring, and interpretation. University of Minnesota Press.

Berry

D. T.

Wetter

M. W.

Baer

R. A.

Larsen

Clark

Monroe

(1992). MMPI–2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4, 340–345.

Bowling

N. A.

Huang

J. L.

Bragg

C. B.

Khazon

Liu

Blackmore

C. E.

(2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111, 218–229.

Carpenter

Gelman

Hoffman

M. D.

Lee

Goodrich

Betancourt

Brubaker

Guo

Riddell

(2017). Stan: A probabilistic programming language. Journal of Statistical Software, 76.

Costa

P. T.

McCrae

R. R.

(1992). Revised NEO Personality Inventory (NEO-PI-R) and NEO Five-Factor Inventory (NEO-FFI) professional manual. Psychological Assessment Resources, Inc.

Credé

(2010). Random responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement, 70, 596–612.

Curran

P. G.

(2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19.

DeSimone

J. A.

Harms

P. D.

DeSimone

A. J.

(2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36, 171–181.

Elliott

M. R.

Gallo

J. J.

Ten Have

T. R.

Bogner

H. R.

Katz

I. R.

(2005). Using a Bayesian latent growth curve model to identify trajectories of positive affect and negative events following myocardial infarction. Biostatistics, 6, 119–143.

10.

Fervaha

Remington

(2013). Invalid responding in questionnaire-based research: Implications for the study of schizotypy. Psychological Assessment, 25, 1355–1360.

11.

Galesic

Bosnjak

(2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73, 349–360.

12.

Gibson

A. M.

Bowling

N. A.

(2020). The effects of questionnaire length and behavioral consequences on careless responding. European Journal of Psychological Assessment, 36, 410–420.

13.

Goldberg

L. R.

(1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. In Mervielde

Deary

De Fruyt

Ostendorf

(Eds.), Personality psychology in Europe (Vol. 7, pp. 7–28). Tilburg University Press.

14.

Gough

H. G.

Bradley

(1996). California Psychological Inventory: Administrator’s guide (3rd ed.). Consulting Psychologists Press.

15.

Herzog

A. R.

Bachman

J. G.

(1981). Effects of questionnaire length on response quality. Public Opinion Quarterly, 45, 549–559.

16.

Houpt

(2017). Careless responding [Computer software]. http://www.github.com/jhoupt / CarelessResponding

17.

Huang

J. L.

Bowling

N. A.

Liu

(2015). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30, 299–311.

18.

Huang

J. L.

Curran

P. G.

Keeney

Poposki

E. M.

DeShon

R. P.

(2012). Detecting and deterring insufficient effort respond to surveys. Journal of Business and Psychology, 27, 99–114.

19.

Huang

J. L.

Liu

Bowling

N. A.

(2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100, 828–845.

20.

Johnson

J. A.

(2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39, 103–129.

21.

Krosnick

J. A.

(1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.

22.

Krosnick

J. A.

Holbrook

A. L.

Berent

M. K.

Carson

R. T.

Hanemann

W. M.

Kopp

R. J.

Mitchell

R. C.

Presser

Ruud

P. A.

Smith

V. K.

Moody

W. R.

Green

M. C.

Conaway

(2002). The impact of “no opinion” response options on data quality: Non-attitude reduction or an invitation to satisfice? Public Opinion Quarterly, 66, 371–403.

23.

Kuss

Jäkel

Wichmann

F. A.

(2005). Bayesian inference for psychometric functions. Journal of Vision, 5, 478–492.

24.

Maniaci

M. R.

Rogge

R. D.

(2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61–83.

25.

McArdle

J. J.

(1988). Dynamic but structural equation modeling of repeated measures data. In Nesselroade

J. R.

Cattell

R. B.

(Eds.), Handbook of multivariate experimental psychology (pp. 561–614). Springer.

26.

McGrath

R. E.

Mitchell

Kim

B. H.

Hough

(2010). Evidence for response bias as a source of error variance in applied assessment. Psychological Bulletin, 136, 450–470.

27.

Meade

A. W.

Craig

S. B.

(2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.

28.

Moon

(1999). The effects of physical distance and response latency on persuasion in computer-mediated communication and human—computer communication. Journal of Experimental Psychology: Applied, 5, 379–392.

29.

Roghanizad

M. M.

Bohns

V. K.

(2017). Ask in person: You’re less persuasive than you think over email. Journal of Experimental Social Psychology, 69, 223–226.

30.

Stan Development Team. (2018). Stan modeling language users guide and reference manual, version 2.18.0. http://mc-stan.org

31.

Tourangeau

(1984). Cognitive science and survey methods. In Jabine

T. B.

Straf

M. L.

Tanur

J. M.

Tourangeau

(Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73–100). National Academy Press.

32.

Ward

M. K.

Meade

A. W.

(2018). Applying social psychology to prevent careless responding during online surveys. Applied Psychology: An International Review, 67, 231–263.

33.

Ward

M. K.

Pond

S. B.

(2015). Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys. Computers in Human Behavior, 48, 554–568.

34.

Wright

D. B.

London

(2009). Modern regression techniques using R. Sage.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.14 MB

0.06 MB

0.22 MB