Tracking respondents’ eyes while they complete a survey reveals that (a) they do not read instructions, survey questions, and answer options carefully enough, investing only as little as 32% of the required time; (b) their attention diminishes over the course of the survey; and (c) their self-reports of the survey experience do not reflect actual survey completion behavior. As much as 15% of survey data may be negatively affected by systematic respondent inattention. From these findings, we derive practical recommendations on how to improve pre-testing of surveys and how to reduce the likelihood of survey respondents ignoring instructions and not reading survey questions and answer options.
BaerR. A.BallengerJ.BerryD. T. R.WetterM. W. (1997). Detection of random responding on the MMPI-A. Journal of Personality Assessment, 68, 139–151.
2.
Ben-NunP. (2008). Respondent fatigue. Encyclopedia of Survey Research Methods, 2, 742–743.
3.
BradleyM.DalyA. (1994). Use of the logit scaling approach to test for rank-order and fatigue effects in stated preference data. Transportation, 21, 167–184.
4.
CarverR. P. (1990). Reading rate: A review of research and theory. San Diego, CA: Academic Press.
5.
DolnicarS. (2013). Asking good survey questions. Journal of Travel Research, 52, 551–574.
DolnicarS.GrünB. (2014). Including don’t know answer options in brand image surveys improves data quality. International Journal of Market Research, 56, 33–50.
8.
DolnicarS.RossiterJ. R. (2008). The low stability of brand-attribute associations is partly due to market research methodology. International Journal of Research in Marketing, 25, 104–108.
9.
GalesicM.BosnjakM. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73, 349–360.
10.
GalesicM.TourangeauR.CouperM. P.ConradF. G. (2008). Eye-Tracking data new insights on response order effects and other cognitive shortcuts in survey responding. Public Opinion Quarterly, 72, 892–913.
11.
GlöcknerA.HerboldA. K. (2011). An eye-tracking study on information processing in risky decisions: Evidence for compensatory strategies based on automatic processes. Journal of Behavioral Decision Making, 24, 71–98.
12.
HartS. G.StavelandL. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology, 52, 139–183.
13.
JohnsonJ. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39, 103–129.
14.
JohnsonM. D.LehmannD. R.HorneD. R. (1990). The effects of fatigue on judgments of interproduct similarity. International Journal of Research in Marketing, 7, 35–43.
15.
KaminskaO.FoulshamT. (2016). Eye-tracking social desirability bias. Bulletin de Méthodologie Sociologique, 130, 73–89.
16.
KamoenN.HollemanB.MakP.SandersT.Van Den BerghH. (2011). Agree or disagree? Cognitive processes in answering contrastive survey questions. Discourse Processes, 48, 355–385.
17.
KrosnickJ. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.
18.
LenznerT.KaczmirekL.GalesicM. (2011). Seeing through the eyes of the respondent: An eye-tracking study on survey question comprehension. International Journal of Public Opinion Research, 23, 361–373.
19.
LenznerT.KaczmirekL.GalesicM. (2014). Left feels right: A usability study on the position of answer boxes in web surveys. Social Science Computer Review, 32, 743–764.
20.
MeadeA. W.CraigS. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.
21.
MeißnerM.DeckerR. (2010). Eye-tracking information processing in choice-based conjoint analysis. International Journal of Market Research, 52, 593–612.
22.
MenoldN. (2017). Rating-Scale Labeling in Online Surveys: An Experimental Comparison of Verbal and Numeric Rating Scales with Respect to Measurement Quality and Respondents’ Cognitive Processes. Sociological Methods & Research, 0049124117729694.
23.
NeuertC.LenznerT. (2016). Incorporating eye tracking into cognitive interviewing to pretest survey questions. International Journal of Social Research Methodology, 19, 501–519.
24.
NoyesJ. M.BruneauD. P. (2007). A self-analysis of the NASA-TLX workload measure. Ergonomics, 50, 514–519.
25.
OsgoodC. E. (1952). The nature and measurement of meaning. Psychological Bulletin, 49, 197–237.
26.
PaasL.DolnicarS.KarlssonL. (2018). Instructional manipulation checks: A longitudinal analysis with implications for MTurk. International Journal of Research in Marketing, 35, 258–269.
RaynerK. (2009). Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology, 62, 1457–1506.
29.
RedlineC. D.DillmanD. A. (2002). The influence of alternative visual designs on respondents’ performance with branching instructions in self-administered questionnaires. In GrovesR. M.DilmanD. A.EltingeJ. L.LittleR. J. A. (Eds.), Survey nonresponse (pp. 179–193). New York, NY: Wiley.
RossiterJ. R.DolnicarS.GrünB. (2015). Why the level-free forced-choice binary measure of brand benefit beliefs works so well. International Journal of Market Research, 57, 239–256.
32.
SavageS. J.WaldmanD. M. (2008). Learning and fatigue during choice experiments: A comparison of online and mail survey modes. Journal of Applied Econometrics, 23, 351–371.
33.
SharpL. M.FrankelJ. (1983). Respondent burden: A test of some common assumptions. Public Opinion Quarterly, 47, 36–53.
TourangeauR.RipsL. J.RasinskiK. (2000). The psychology of survey response. Cambridge, UK: Cambridge University Press.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
