Several recent studies have examined the prevention, causes, and consequences of insufficient effort responding (IER) to surveys. Scientific progress in this area, however, rests on the availability of construct-valid IER measures. In the current paper we describe the potential merits of the page time index, which is computed by counting the number of questionnaire pages to which a participant has responded more quickly than two seconds per item (see
Huang et al., 2012). We conducted three studies (total N = 1,056) to examine the page time index's construct validity. Across these studies, we found that page time converged highly with other IER indices, that it was sensitive to an experimental manipulation warning participants to respond carefully, and that it predicted the extent to which participants were unable to recognize item content. We also found that page time's validity was superior to that of total completion time and that the two-seconds-per-item rule yielded a construct-valid page time score for items of various word lengths. Given its apparent validity, we provide practical recommendations for the use of the page time index.
AguinisH.BradleyK. J. (2014). Best practice recommendations for designing and implementing experimental vignette methodology studies. Organizational Research Methods, 17(4), 351-371. https://doi.org/10.1177/1094428114547952
2.
BowlingN. A.GibsonA. M.HouptJ. W.BrowerC. K. (2021). Will the questions ever end? Within-person increases in careless responding during questionnaire completion. Organizational Research Methods, 24(4), 718-738. https://doi.org/10.1177/1094428120947794
3.
BowlingN. A.HuangJ. L.BraggC. B.KhazonS.LiuM.BlackmoreC. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111(2), 218-229. https://doi.org/10.1037/pspp0000085
4.
CampbellD. T.FiskeD. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81-105. https://doi.org/10.1037/h0046016
5.
CorM. K.HaertelE.KrosnickJ. A.MalhotraN. (2012). Improving ability measurement in surveys by following the principles of IRT: The Wordsum vocabulary test in the General Social Survey. Social Science Research, 41(5), 1003-1016. https://doi.org/10.1016/j.ssresearch.2012.05.007
6.
CredéM. (2010). Random responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement, 70(4), 596-612. https://doi.org/10.1177/0013164410366686
7.
CronbachL. J.MeehlP. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281-302. https://doi.org/10.1037/h0040957
8.
CurranP. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4-19. https://doi.org/10.1016/j.jesp.2015.07.006
9.
DeSimoneJ. A.HarmsP. D.DeSimoneA. J. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36(2), 171-181. https://doi.org/10.1002/job.1962
10.
DonnellanM. B.OswaldF. L.BairdB. M.LucasR. E. (2006). The mini-IPIP scales: Tiny-yet-effective measures of the big five factors of personality. Psychological Assessment, 18(2), 192-203. https://doi.org/10.1037/1040-3590.18.2.192
11.
FrancavillaN. M.MeadeA. W.YoungA. L. (2019). Social interaction and internet-based surveys: Examining the effects of virtual and in-person proctors on careless response. Applied Psychology, 68(2), 223-249. https://doi.org/10.1111/apps.12159
12.
GalesicM.BosnjakM. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73(2), 349-360. https://doi.org/10.1093/poq/nfp031
13.
GibsonA. M.BowlingN. A. (2020). The effects of questionnaire length and behavioral consequences on careless responding. European Journal of Psychological Assessment, 36(2), 410-420. https://doi.org/10.1027/1015-5759/a000526
14.
GoldbergL. R. (1992). The development of markers for the big-five factor structure. Psychological Assessment, 4(1), 26-42. https://doi.org/10.1037/1040-3590.4.1.26
15.
HaytonJ. C.AllenD. G.ScarpelloV. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7(2), 191-205. https://doi.org/10.1177/1094428104263675
16.
HinkinT. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1), 104-121. https://doi.org/10.1177/109442819800100106
17.
HuangJ. L. (2014). Does cleanliness influence moral judgments? Response effort moderates the effect of cleanliness priming on moral judgment. Frontiers in Psychology, 5(1276), 1-8. https://doi.org/10.3389/fpsyg.2014.01276
18.
HuangJ. L.BowlingN. A.LiuM.LiY. (2015). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30, 299-311. https://doi.org/10.1007/s10869-014-9357-6
19.
HuangJ. L.CurranP. G.KeeneyJ.PoposkiE. M.DeShonR. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(5), 99-114. https://doi.org/10.1007/s10869-011-9231-8
20.
HuangJ. L.DeSimoneJ. A. (2021). Insufficient effort responding as a potential confound between survey measures and objective tests. Journal of Business and Psychology, 36(5), 807-828. https://doi.org/10.1007/s10869-020-09707-2
21.
HuangJ. L.LiuM.BowlingN. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100(3), 828-845. https://doi.org/10.1037/a0038510
22.
JamesL. R. (1998). Measurement of personality via conditional reasoning. Organizational Research Methods, 1(2), 131-163. https://doi.org/10.1177/109442819812001
23.
JohnsonJ. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39(1), 103-129. https://doi.org/10.1016/j.jrp.2004.09.009
24.
JohnsonJ. W. (2000). A heuristic method for estimating the relative weight of predictor variables in multiple regression. Multivariate Behavioral Research, 35(1), 1-19. https://doi.org/10.1207/S15327906MBR3501_1
25.
KamC. C. S.ChanG. H. H. (2018). Examination of the validity of instructed response items in identifying careless respondents. Personality and Individual Differences, 129, 83-87. https://doi.org/10.1016/j.paid.2018.03.022
26.
KrosnickJ. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213-236. https://doi.org/10.1002/acp.2350050305
27.
LeBretonJ. M.HargisM. B.GriepentrogB.OswaldF. L.PloyhartR. E. (2007). A multidimensional approach for evaluating variables in organizational research and practice. Personnel Psychology, 60(2), 475-498. https://doi.org/10.1111/j.1744-6570.2007.00080.x
28.
LeeI. A.PreacherK. J. (2013, September). Calculation for the test of the difference between two dependent correlations with one variable in common [Computer software]. Available fromhttp://quantpsy.org.
29.
ManiaciM. R.RoggeR. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61-83. https://doi.org/10.1016/j.jrp.2013.09.008
30.
McKayA. S.GarciaD. M.ClapperJ. P.ShultzK. S. (2018). The attentive and the careless: Examining the relationship between benevolent and malevolent personality traits with careless responding in online surveys. Computers in Human Behavior, 84, 295-303. https://doi.org/10.1016/j.chb.2018.03.007
31.
MeadeA. W.CraigS. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437-455. https://doi.org/10.1037/a0028085
32.
MurphyK. R.DavidshoferC. O. (2005). Psychological testing: Principles and applications (6th ed.). Pearson Prentice Hall.
33.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science (new York, N Y ), 349(6251), 1-8. https://doi.org/10.1126/science.aac4716
34.
PashlerH.WagenmakersE. J. (2012). Editors’ introduction to the special section on replicability in psychological science a crisis of confidence?Perspectives on Psychological Science, 7(6), 528-530. https://doi.org/10.1177/1745691612465253
35.
SchmittN.StultsD. M. (1985). Factors defined by negatively keyed items: The result of careless respondents?Applied Psychological Measurement, 9(4), 367-373. https://doi.org/10.1177/014662168500900405
36.
SpectorP. E. (1991). Summated rating scale construction: An introduction. Sage.
TabachnickB. G.FidellL. S.UllmanJ. B. (2007). Using multivariate statistics. Pearson.
39.
TonidandelS.LeBretonJ. M. (2015). RWA Web: A free, comprehensive, web-based, and user-friendly tool for relative weight analyses. Journal of Business and Psychology, 30, 207-216. https://doi.org/10.1007/s10869-014-9351-z
40.
TourangeauR. (1984). Cognitive science and survey methods. In Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines (Vol. 15, pp. 73-100).
41.
VelicerW. F. (1976). Determining the number of components from the matrix of partial correlations. Psychometrika, 41(3), 321-327. https://doi.org/10.1007/BF02293557
42.
WardM. K.MeadeA. W. (2018). Applying social psychology to prevent careless responding during online surveys. Applied Psychology: An International Review, 67(2), 231-263. https://doi.org/10.1111/apps.12118
43.
WardM. K.PondS. B. (2015). Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys. Computers in Human Behavior, 48, 554-568. https://doi.org/10.1016/j.chb.2015.01.070
44.
WiseS. L. (2019). An information-based approach to identifying rapid-guessing thresholds. Applied Measurement in Education, 32(4), 325-336. https://doi.org/10.1080/08957347.2019.1660350
45.
YentesR. D.WilhelmF. (2018). careless: Procedures for computing indices of careless responding. R package version, 1, 2018.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
