To collect high-quality data, survey designers aim to develop questions that each respondent can understand as intended. A critical step to this end is designing questions that minimize the respondents’ burden by reducing the cognitive effort required to comprehend and answer them. One promising technique for identifying problematic survey questions is eye tracking. This article investigates the potential of eye movements and pupil dilations as indicators for evaluating survey questions. Respondents were randomly assigned to either a problematic or an improved version of six experimental questions. By analyzing fixation times, fixation counts, and pupil diameters, it was examined whether these parameters could be used to distinguish between the two versions. Identifying the improved version worked best by comparing fixation times, whereas in most cases, it was not possible to differentiate between versions on the basis of pupil data. Limitations and practical implications of the findings are discussed.
BeattyJ. (1982). Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin, 91, 276–292.
2.
BeattyJ.KahnemanD. (1966). Pupillary changes in two memory tasks. Psychonomic Science, 5, 371–372.
3.
BeattyJ.Lucero-WagonerB. (2000). The pupillary system. Handbook of Psychophysiology, 2, 142–162.
4.
BiemerP. P.LybergL. E. (2003). Introduction to survey quality. Hoboken, NJ: John Wiley.
5.
ConradF. G.SchoberM. F. (2000). Clarifying question meaning in a household telephone survey. Public Opinion Quarterly, 64, 1–28.
6.
FowlerF. J. (2001). Why it is easy to write bad questions. ZUMA Nachrichten, 25, 49–66.
7.
FowlerF. J. (2013). Survey research methods (5th ed.). Thousand Oaks, CA: Sage.
8.
GalesicM.YanT. (2011). Use of eye tracking for studying survey response processes. In DasM.EsterP.KaczmirekL. (Eds.), Social and behavioral research and the internet: Advances in applied methods and research strategies (pp. 349–370). New York, NY: Routledge.
9.
GraesserA. C.CaiZ.LouwerseM. M.DanielF. (2006). Question understanding aid (QUAID). A web facility that tests question comprehensibility. Public Opinion Quarterly, 70, 3–22.
10.
HessE. H.PoltJ. M. (1964). Pupil size in relation to mental activity during simple problem-solving. Science, 143, 1190–1192.
11.
HöhneJ. K.LenznerT. (2018). New insights on the cognitive processing of agree/disagree and item-specific questions. Journal of Survey Statistics and Methodology, 6, 401–417.
12.
HoosainR. (1973). The Processing of Negation. Journal of Verbal Learning and Verbal Behaviour, 12, 618–626.
13.
HorwitzR.KreuterF.ConradF. (2017). Using mouse movements to predict Web survey response difficulty. Social Science Computer Review, 35, 388–405.
14.
IqbalS. T.ZhengX. S.BaileyB. P. (2004). Task-evoked pupillary response to mental workload in human–computer interaction. Proceedings of the Conference on Human Factors in Computing Systems (CHI’04), Vienna, pp. 1477–1480.
15.
ISSP Research Group. (2012). International social survey programme: Environment III—ISSP 2010. GESIS Data Archive, Cologne. ZA5500 Data File Version 2.0.0. doi:10.4232/1.11418
16.
ISSP Research Group. (2016). International social survey programme: Citizenship II—ISSP 2014. GESIS Data Archive, Cologne. ZA6670 Data File Version 2.0.0. doi:10.4232/1.12590
17.
JustM. A.CarpenterP. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87, 329–354.
18.
JustM. A.CarpenterP. A. (1993). The intensity dimension of thought: Pupillometric indices of sentence processing. Canadian Journal of Experimental Psychology, 47, 310–339.
19.
KamoenN.HollemanB.MakP.SandersT.Van Den BerghH. (2017). Why are negative questions difficult to answer? On the processing of linguistic contrasts in surveys. Public Opinion Quarterly, 81, 613–635.
20.
KrosnickJ. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology, 5, 213–236.
21.
KrosnickJ. A. (1999). Survey research. Annual Review of Psychology, 50, 537–567.
22.
KrugerJ. L.HeferE.MatthewG. (2013). Measuring the impact of subtitles on cognitive load: Eye tracking and dynamic audiovisual texts. In Proceedings of the 2013 Conference on Eye Tracking South Africa (pp. 62–66). ACM.
23.
LaengB.SiroisS.GredebäckG. (2012). Pupillometry: A window to the reconscious?Perspectives on Psychological Science, 7, 18–27.
24.
LenznerT.KaczmirekL.GalesicM. (2011). Seeing through the eyes of the respondent: An eye-tracking study on survey question comprehension. International Journal of Public Opinion Research, 23, 361–373.
25.
LenznerT.KaczmirekL.GalesicM. (2014). Left feels right: A usability study on the position of answer boxes in web surveys. Social Science Computer Review, 32, 743–764.
26.
LenznerT.NeuertC.OttoW. (2014). German Internet Panel (GIP) Reforms Monitor (2014). Kognitiver Pretest (GESIS Projektbericht). doi:10.17173/pretest20
27.
LenznerT.OttoW.AdamsF.DischK.NeuertC.MenoldN. (2015). Conceptions of democracy and preferences over democratic procedures. Kognitiver Online-Pretest (GESIS Projektbericht). doi:10.17173/pretest2
28.
LenznerT.OttoW.NeuertC.BeitzC.SchmidtR.StieglerA. (2016). Comparative Study of Electoral Systems (CSES) Module 5. Cognitive Pretest (GESIS Project Report). doi:10.17173/pretest27
29.
LiversedgeS. P.FindlayJ. M. (2000). Saccadic eye movements and cognition. Trends in Cognitive Sciences, 4, 6–14.
30.
MillerK. (2014). Introduction. In MillerK.CheppV.WillsonS.PadillaJ. L. (Eds.), Cognitive interviewing methodology (pp. 1–6). New York, NY: John Wiley.
31.
NeuertC. E.LenznerT. (2016). Incorporating eye tracking into cognitive interviewing to pretest survey questions. International Journal of Social Research Methodology, 19, 501–519.
32.
PorstR. (2011). Fragebogen. Ein Arbeitsbuch [Questionnaire. A workbook]. Wiesbaden, Germany: VS Verlag für Sozialwissenschaften.
33.
PorterG.TrosciankoT.GilchristI. D. (2007). Effort during visual search and counting: Insights from pupillometry. Quarterly Journal of Experimental Psychology, 60, 211–229.
34.
PresserS.CouperM. P.LesslerJ. T.MartinE.MartinJ.RothgebJ. M.SingerE. (2004). Methods for testing and evaluating survey questions. Public Opinion Quarterly, 68, 109–130.
35.
RaynerK. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372.
36.
RaynerK. (2009). Eye movements and attention in reading, scene perception, and visual search. Quarterly Journal of Experimental Psychology, 62, 1457–1506.
37.
SiegleG. J.SteinhauerS. R.StengerV. A.KoneckyR.CarterC. S. (2003). Use of concurrent pupil dilation assessment to inform interpretation and analysis of fMRI data. Neuroimage, 20, 114–124.
TourangeauR.RipsL. J.RasinskiK. (2000). The psychology of survey response. Cambridge, England: Cambridge University Press.
40.
YanT.TourangeauR. (2008). Fast times and easy questions: The effects of age, experience and question complexity on web survey response times. Applied Cognitive Psychology, 22, 51–68.
41.
YanT.WilliamsD.MaitlandA.TourangeauR. (2016, May). Use of eye-tracking to measure response Burden. Paper presented at 2016 AAPOR Annual Conference, Austin, Texas.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
