Researchers have become increasingly interested in response times to survey items as a measure of cognitive effort. We used machine learning to develop a prediction model of response times based on 41 attributes of survey items (e.g., question length, response format, linguistic features) collected in a large, general population sample. The developed algorithm can be used to derive reference values for expected response times for most commonly used survey items.
Get full access to this article
View all access options for this article.
References
1.
AlattarL.MesselM.RogofskyD.. 2018. An introduction to the understanding America study Internet panel. Social Security Bulletin78:13–28.
2.
BaisF.BaisF.SchoutenB.LugtigP.ToepoelV.Arends-TòthJ.DouhouS.KierujN.MorrenM.VisC.. 2019. Can survey item characteristics relevant to measurement error be coded reliably? A case study on 11 Dutch general population surveys. Sociological Methods & Research48:263–95.
3.
BaldwinP.YanevaV.MeeJ.ClauserB. E.HaL. A.. 2021. Using natural language processing to predict item response times and improve test construction. Journal of Educational Measurement58:4–30.
4.
BassiliJ. N.ScottB. S.. 1996. Response latency as a signal to question problems in survey research. Public Opinion Quarterly60:390–99.
5.
BowlingN. A.HuangJ. L.BrowerC. K.BraggC. B.. 2021. The quick and the careless: The construct validity of page time as a measure of insufficient effort responding to surveys. Organizational Research Methods26:323–52.
CouperM. P.KreuterF.. 2013. Using paradata to explore item level response times in surveys. Journal of the Royal Statistical Society: Series A (Statistics in Society)176:271–86.
8.
FazioR. H.1990. Multiple processes by which attitudes guide behavior: The MODE model as an integrative framework. In Advances in experimental social psychology, ed. ZannaM. P., pp. 75–109) . San Diego: Academic Press.
9.
FriedmanJ. H.2002. Stochastic gradient boosting. Computational Statistics & Data Analysis38:367–78.
10.
GoldhammerF.NaumannJ.StelterA.TóthK.RölkeH.KliemeE.. 2014. The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology106:608–26.
11.
HöhneJ. K.SchlosserS.KrebsD.. 2017. Investigating cognitive effort and response quality of question formats in web surveys using paradata. Field Methods29:365–82.
12.
JamesG. M.WittenD.HastieT.TibshiraniR.. 2013. An introduction to statistical learning. Berlin: Springer.
13.
JohnsonM.2004. Timepieces: Components of survey question response latencies. Political Psychology25:679–702.
14.
KrosnickJ. A.1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology5:213–36.
15.
LenznerT.2012. Effects of survey question comprehensibility on response quality. Field Methods24:409–28.
16.
LenznerT.KaczmirekL.LenznerA.. 2010. Cognitive burden of survey questions and response times: A psycholinguistic experiment. Applied Cognitive Psychology24:1003–20.
17.
LischetzkeT.CuccodoroG.GaugerA.TodeschiniL.EidM.. 2005. Measuring affective clarity indirectly: Individual differences in response latencies of state. Emotion5:431–45.
MolenaarD.RózsaS.KõN.. 2021. Modeling asymmetry in the time–distance relation of ordinal personality items. Applied Psychological Measurement45:178–94.
20.
OECD. 2019. Beyond proficiency: Using log files to understand respondent behaviour in the survey of adult skills. Paris: OECD Skills Studies OECD Publishing.
21.
PennebakerJ. W.BoydR. L.JordanK.BlackburnK.. 2015. The development and psychometric properties of LIWC2015. Austin: University of Texas.
22.
QiY.2012. Random forest for bioinformatics. In Ensemble machine learning: Methods and applications, ed. ZhangC., pp. 307–23). Berlin: Springer.
23.
RatcliffR. 1993. Methods for dealing with reaction time outliers. Psychological Bulletin114:510–32.
24.
RobinsonM. D.CloreG. L.. 2002. Belief and feeling: Evidence for an accessibility model of emotional self-report. Psychological Bulletin128:934–60.
25.
SchneiderS.MayM.StoneA. A.. 2018. Careless responding in Internet-based quality of life assessments. Quality of Life Research27:1077–88.
26.
ThissenD.1983. Timed testing: An approach using item response theory. In New horizons in testing, ed. WeissD. J., pp. 179–203) . New York: Academic Press.
27.
TourangeauR.RipsL. J.RasinskiK.. 2000. The psychology of survey response. Cambridge: Cambridge University Press.
28.
UlitzschE.PohlS.KhorramdelL.KroehneU.von DavierM.. 2021. A response-time-based latent response mixture model for identifying and modeling careless and insufficient effort responding in survey data. Psychometrika. 87:593–619.
29.
van der LindenW. J.van Krimpen-StoopE. M. L. A.. 2003. Using response times to detect aberrant responses in computerized adaptive testing. Psychometrika68:251–65.
30.
van der MaasH. L.MolenaarD.MarisG.KievitR. A.BorsboomD.. 2011. Cognitive psychology meets psychometric theory: On the relation between process models for decision making and latent variable models for individual differences. Psychological Review118:339–56.
31.
WalentynowiczM.SchneiderS.StoneA. A.. 2018. The effects of time frames on self-report. PloS One13:e0201655.
32.
YanT.OlsonK.. 2013. Analyzing paradata to investigate measurement error. In Improving surveys with paradata: Analytic uses of process information, ed. KreuerF., pp. 73–96). Hoboken, NJ: John Wiley.
33.
YanT.TourangeauR.. 2008. Fast times and easy questions: The effects of age, experience and question complexity on web survey response times. Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition22:51–68.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
