The Quick and the Careless: The Construct Validity of Page Time as a Measure of Insufficient Effort Responding to Surveys

Abstract

Several recent studies have examined the prevention, causes, and consequences of insufficient effort responding (IER) to surveys. Scientific progress in this area, however, rests on the availability of construct-valid IER measures. In the current paper we describe the potential merits of the page time index, which is computed by counting the number of questionnaire pages to which a participant has responded more quickly than two seconds per item (see Huang et al., 2012). We conducted three studies (total N = 1,056) to examine the page time index's construct validity. Across these studies, we found that page time converged highly with other IER indices, that it was sensitive to an experimental manipulation warning participants to respond carefully, and that it predicted the extent to which participants were unable to recognize item content. We also found that page time's validity was superior to that of total completion time and that the two-seconds-per-item rule yielded a construct-valid page time score for items of various word lengths. Given its apparent validity, we provide practical recommendations for the use of the page time index.

Keywords

insufficient effort responding careless responding participant inattention random responding response effort

Get full access to this article

View all access options for this article.

References

Aguinis

Bradley

K. J.

(2014). Best practice recommendations for designing and implementing experimental vignette methodology studies. Organizational Research Methods, 17(4), 351-371. https://doi.org/10.1177/1094428114547952

Bowling

N. A.

Gibson

A. M.

Houpt

J. W.

Brower

C. K.

(2021). Will the questions ever end? Within-person increases in careless responding during questionnaire completion. Organizational Research Methods, 24(4), 718-738. https://doi.org/10.1177/1094428120947794

Bowling

N. A.

Huang

J. L.

Bragg

C. B.

Khazon

Liu

Blackmore

C. E.

(2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111(2), 218-229. https://doi.org/10.1037/pspp0000085

Campbell

D. T.

Fiske

D. W.

(1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81-105. https://doi.org/10.1037/h0046016

Cor

M. K.

Haertel

Krosnick

J. A.

Malhotra

(2012). Improving ability measurement in surveys by following the principles of IRT: The Wordsum vocabulary test in the General Social Survey. Social Science Research, 41(5), 1003-1016. https://doi.org/10.1016/j.ssresearch.2012.05.007

Credé

(2010). Random responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement, 70(4), 596-612. https://doi.org/10.1177/0013164410366686

Cronbach

L. J.

Meehl

P. E.

(1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281-302. https://doi.org/10.1037/h0040957

Curran

P. G.

(2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4-19. https://doi.org/10.1016/j.jesp.2015.07.006

DeSimone

J. A.

Harms

P. D.

DeSimone

A. J.

(2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36(2), 171-181. https://doi.org/10.1002/job.1962

10.

Donnellan

M. B.

Oswald

F. L.

Baird

B. M.

Lucas

R. E.

(2006). The mini-IPIP scales: Tiny-yet-effective measures of the big five factors of personality. Psychological Assessment, 18(2), 192-203. https://doi.org/10.1037/1040-3590.18.2.192

11.

Francavilla

N. M.

Meade

A. W.

Young

A. L.

(2019). Social interaction and internet-based surveys: Examining the effects of virtual and in-person proctors on careless response. Applied Psychology, 68(2), 223-249. https://doi.org/10.1111/apps.12159

12.

Galesic

Bosnjak

(2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73(2), 349-360. https://doi.org/10.1093/poq/nfp031

13.

Gibson

A. M.

Bowling

N. A.

(2020). The effects of questionnaire length and behavioral consequences on careless responding. European Journal of Psychological Assessment, 36(2), 410-420. https://doi.org/10.1027/1015-5759/a000526

14.

Goldberg

L. R.

(1992). The development of markers for the big-five factor structure. Psychological Assessment, 4(1), 26-42. https://doi.org/10.1037/1040-3590.4.1.26

15.

Hayton

J. C.

Allen

D. G.

Scarpello

(2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7(2), 191-205. https://doi.org/10.1177/1094428104263675

16.

Hinkin

T. R.

(1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1), 104-121. https://doi.org/10.1177/109442819800100106

17.

Huang

J. L.

(2014). Does cleanliness influence moral judgments? Response effort moderates the effect of cleanliness priming on moral judgment. Frontiers in Psychology, 5(1276), 1-8. https://doi.org/10.3389/fpsyg.2014.01276

18.

Huang

J. L.

Bowling

N. A.

Liu

(2015). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30, 299-311. https://doi.org/10.1007/s10869-014-9357-6

19.

Huang

J. L.

Curran

P. G.

Keeney

Poposki

E. M.

DeShon

R. P.

(2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(5), 99-114. https://doi.org/10.1007/s10869-011-9231-8

20.

Huang

J. L.

DeSimone

J. A.

(2021). Insufficient effort responding as a potential confound between survey measures and objective tests. Journal of Business and Psychology, 36(5), 807-828. https://doi.org/10.1007/s10869-020-09707-2

21.

Huang

J. L.

Liu

Bowling

N. A.

(2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100(3), 828-845. https://doi.org/10.1037/a0038510

22.

James

L. R.

(1998). Measurement of personality via conditional reasoning. Organizational Research Methods, 1(2), 131-163. https://doi.org/10.1177/109442819812001

23.

Johnson

J. A.

(2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39(1), 103-129. https://doi.org/10.1016/j.jrp.2004.09.009

24.

Johnson

J. W.

(2000). A heuristic method for estimating the relative weight of predictor variables in multiple regression. Multivariate Behavioral Research, 35(1), 1-19. https://doi.org/10.1207/S15327906MBR3501_1

25.

Kam

C. C. S.

Chan

G. H. H.

(2018). Examination of the validity of instructed response items in identifying careless respondents. Personality and Individual Differences, 129, 83-87. https://doi.org/10.1016/j.paid.2018.03.022

26.

Krosnick

J. A.

(1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213-236. https://doi.org/10.1002/acp.2350050305

27.

LeBreton

J. M.

Hargis

M. B.

Griepentrog

Oswald

F. L.

Ployhart

R. E.

(2007). A multidimensional approach for evaluating variables in organizational research and practice. Personnel Psychology, 60(2), 475-498. https://doi.org/10.1111/j.1744-6570.2007.00080.x

28.

Lee

I. A.

Preacher

K. J.

(2013, September). Calculation for the test of the difference between two dependent correlations with one variable in common [Computer software]. Available from http://quantpsy.org.

29.

Maniaci

M. R.

Rogge

R. D.

(2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61-83. https://doi.org/10.1016/j.jrp.2013.09.008

30.

McKay

A. S.

Garcia

D. M.

Clapper

J. P.

Shultz

K. S.

(2018). The attentive and the careless: Examining the relationship between benevolent and malevolent personality traits with careless responding in online surveys. Computers in Human Behavior, 84, 295-303. https://doi.org/10.1016/j.chb.2018.03.007

31.

Meade

A. W.

Craig

S. B.

(2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437-455. https://doi.org/10.1037/a0028085

32.

Murphy

K. R.

Davidshofer

C. O.

(2005). Psychological testing: Principles and applications (6th ed.). Pearson Prentice Hall.

33.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science (new York, N Y ), 349(6251), 1-8. https://doi.org/10.1126/science.aac4716

34.

Pashler

Wagenmakers

E. J.

(2012). Editors’ introduction to the special section on replicability in psychological science a crisis of confidence? Perspectives on Psychological Science, 7(6), 528-530. https://doi.org/10.1177/1745691612465253

35.

Schmitt

Stults

D. M.

(1985). Factors defined by negatively keyed items: The result of careless respondents? Applied Psychological Measurement, 9(4), 367-373. https://doi.org/10.1177/014662168500900405

36.

Spector

P. E.

(1991). Summated rating scale construction: An introduction. Sage.

37.

Steiger

J. H.

(1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2), 245-251. https://doi.org/10.1037/0033-2909.87.2.245

38.

Tabachnick

B. G.

Fidell

L. S.

Ullman

J. B.

(2007). Using multivariate statistics. Pearson.

39.

Tonidandel

LeBreton

J. M.

(2015). RWA Web: A free, comprehensive, web-based, and user-friendly tool for relative weight analyses. Journal of Business and Psychology, 30, 207-216. https://doi.org/10.1007/s10869-014-9351-z

40.

Tourangeau

(1984). Cognitive science and survey methods. In Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines (Vol. 15, pp. 73-100).

41.

Velicer

W. F.

(1976). Determining the number of components from the matrix of partial correlations. Psychometrika, 41(3), 321-327. https://doi.org/10.1007/BF02293557

42.

Ward

M. K.

Meade

A. W.

(2018). Applying social psychology to prevent careless responding during online surveys. Applied Psychology: An International Review, 67(2), 231-263. https://doi.org/10.1111/apps.12118

43.

Ward

M. K.

Pond

S. B.

(2015). Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys. Computers in Human Behavior, 48, 554-568. https://doi.org/10.1016/j.chb.2015.01.070

44.

Wise

S. L.

(2019). An information-based approach to identifying rapid-guessing thresholds. Applied Measurement in Education, 32(4), 325-336. https://doi.org/10.1080/08957347.2019.1660350

45.

Yentes

R. D.

Wilhelm

(2018). careless: Procedures for computing indices of careless responding. R package version, 1, 2018.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.05 MB