“I know what you’re going to ask me” Why respondents don’t read survey questions

Abstract

Tracking respondents’ eyes while they complete a survey reveals that (a) they do not read instructions, survey questions, and answer options carefully enough, investing only as little as 32% of the required time; (b) their attention diminishes over the course of the survey; and (c) their self-reports of the survey experience do not reflect actual survey completion behavior. As much as 15% of survey data may be negatively affected by systematic respondent inattention. From these findings, we derive practical recommendations on how to improve pre-testing of surveys and how to reduce the likelihood of survey respondents ignoring instructions and not reading survey questions and answer options.

Keywords

answer options eye-tracking questionnaire design respondent inattention survey instructions

Get full access to this article

View all access options for this article.

References

Baer

R. A.

Ballenger

Berry

D. T. R.

Wetter

M. W.

(1997). Detection of random responding on the MMPI-A. Journal of Personality Assessment, 68, 139–151.

Ben-Nun

(2008). Respondent fatigue. Encyclopedia of Survey Research Methods, 2, 742–743.

Bradley

Daly

(1994). Use of the logit scaling approach to test for rank-order and fatigue effects in stated preference data. Transportation, 21, 167–184.

Carver

R. P.

(1990). Reading rate: A review of research and theory. San Diego, CA: Academic Press.

Dolnicar

(2013). Asking good survey questions. Journal of Travel Research, 52, 551–574.

Dolnicar

(2018). Peer-to-peer accommodation networks: Pushing the boundaries. Oxford, UK: Goodfellow Publishers.

Dolnicar

Grün

(2014). Including don’t know answer options in brand image surveys improves data quality. International Journal of Market Research, 56, 33–50.

Dolnicar

Rossiter

J. R.

(2008). The low stability of brand-attribute associations is partly due to market research methodology. International Journal of Research in Marketing, 25, 104–108.

Galesic

Bosnjak

(2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73, 349–360.

10.

Galesic

Tourangeau

Couper

M. P.

Conrad

F. G.

(2008). Eye-Tracking data new insights on response order effects and other cognitive shortcuts in survey responding. Public Opinion Quarterly, 72, 892–913.

11.

Glöckner

Herbold

A. K.

(2011). An eye-tracking study on information processing in risky decisions: Evidence for compensatory strategies based on automatic processes. Journal of Behavioral Decision Making, 24, 71–98.

12.

Hart

S. G.

Staveland

L. E.

(1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology, 52, 139–183.

13.

Johnson

J. A.

(2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39, 103–129.

14.

Johnson

M. D.

Lehmann

D. R.

Horne

D. R.

(1990). The effects of fatigue on judgments of interproduct similarity. International Journal of Research in Marketing, 7, 35–43.

15.

Kaminska

Foulsham

(2016). Eye-tracking social desirability bias. Bulletin de Méthodologie Sociologique, 130, 73–89.

16.

Kamoen

Holleman

Mak

Sanders

Van Den Bergh

(2011). Agree or disagree? Cognitive processes in answering contrastive survey questions. Discourse Processes, 48, 355–385.

17.

Krosnick

J. A.

(1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.

18.

Lenzner

Kaczmirek

Galesic

(2011). Seeing through the eyes of the respondent: An eye-tracking study on survey question comprehension. International Journal of Public Opinion Research, 23, 361–373.

19.

Lenzner

Kaczmirek

Galesic

(2014). Left feels right: A usability study on the position of answer boxes in web surveys. Social Science Computer Review, 32, 743–764.

20.

Meade

A. W.

Craig

S. B.

(2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.

21.

Meißner

Decker

(2010). Eye-tracking information processing in choice-based conjoint analysis. International Journal of Market Research, 52, 593–612.

22.

Menold

. (2017). Rating-Scale Labeling in Online Surveys: An Experimental Comparison of Verbal and Numeric Rating Scales with Respect to Measurement Quality and Respondents’ Cognitive Processes. Sociological Methods & Research, 0049124117729694.

23.

Neuert

Lenzner

(2016). Incorporating eye tracking into cognitive interviewing to pretest survey questions. International Journal of Social Research Methodology, 19, 501–519.

24.

Noyes

J. M.

Bruneau

D. P.

(2007). A self-analysis of the NASA-TLX workload measure. Ergonomics, 50, 514–519.

25.

Osgood

C. E.

(1952). The nature and measurement of meaning. Psychological Bulletin, 49, 197–237.

26.

Paas

Dolnicar

Karlsson

(2018). Instructional manipulation checks: A longitudinal analysis with implications for MTurk. International Journal of Research in Marketing, 35, 258–269.

27.

Potaka

(2007). What the eye doesn’t see: A feasibility study to evaluate eye-tracking technology as a tool for paper-based questionnaire development. In Proceedings from quest Statistics Canada, Ottowa, Ontario Canada. Retrieved from https://wwwn.cdc.gov/QBank/QUEST/2007/QUEST%202007%20Proceedings-all%20papers.pdf#page=172 (pp. 162–171).

28.

Rayner

(2009). Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology, 62, 1457–1506.

29.

Redline

C. D.

Dillman

D. A.

(2002). The influence of alternative visual designs on respondents’ performance with branching instructions in self-administered questionnaires. In Groves

R. M.

Dilman

D. A.

Eltinge

J. L.

Little

R. J. A.

(Eds.), Survey nonresponse (pp. 179–193). New York, NY: Wiley.

30.

Redline

C. D.

Lankford

C. P.

(2001, May). Eye-movement analysis: A new tool for evaluating the design of visually administered instruments. Paper presented at the American Association of Public Opinion Research. Retrieved from https://pdfs.semanticscholar.org\\5c73\\fdba13bcd10a0fc5afe75fb3201b47dbe10f.pdf

31.

Rossiter

J. R.

Dolnicar

Grün

(2015). Why the level-free forced-choice binary measure of brand benefit beliefs works so well. International Journal of Market Research, 57, 239–256.

32.

Savage

S. J.

Waldman

D. M.

(2008). Learning and fatigue during choice experiments: A comparison of online and mail survey modes. Journal of Applied Econometrics, 23, 351–371.

33.

Sharp

L. M.

Frankel

(1983). Respondent burden: A test of some common assumptions. Public Opinion Quarterly, 47, 36–53.

34.

Tobii Technology User Manual. (2003). Retrieved from https://www.tobiipro.com/siteassets/tobii-pro/user-manuals/tobii-pro-studio-user-manual.pdf

35.

Tourangeau

Rips

L. J.

Rasinski

(2000). The psychology of survey response. Cambridge, UK: Cambridge University Press.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.61 MB