Sage Journals: Discover world-class research

Abstract

Many studies have shown that vague or ambiguous questions are often interpreted idiosyncratically by respondents and thus can increase measurement error. This article provides some evidence that the cognitive effort required to comprehend survey questions affects data quality in a similar way. A web survey experiment revealed that respondents receiving less comprehensible questions provided lower-quality responses (as indicated by breakoff rates, number of nonsubstantive responses, number of neutral responses, and over-time consistency) than respondents receiving control questions that were easier to comprehend. Moreover, interaction effects of question comprehensibility with respondents’ verbal skills and their motivation to answer surveys were found. These findings indicate that survey designers should minimize the cognitive effort required to comprehend their questions and the article suggests specific ways how to do so.

Keywords

questionnaire design question wording response effects response quality web survey

Get full access to this article

View all access options for this article.

References

Aiken

L. S.,

West

S. G.

. 1991. Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage.

Bless

Wänke

Bohner

Fellhauer

R. R.

Schwarz

. 1994. Need for cognition: Eine skala zur erfassung von engagement und freude bei denkaufgaben [Need for cognition: A scale measuring engagement and happiness in cognitive tasks]. Zeitschrift für Sozialpsychologie 25:147–54.

Bradburn

N. M.,

Miles

. 1979. Vague quantifiers. Public Opinion Quarterly 43:92–101.

Cacioppo

J. T.,

Petty

R. E.

. 1982. The need for cognition. Journal of Personality and Social Psychology 42:116–31.

Collani

2009. Eine Deutsche Skala zum Bedürfnis nach Bewertung (Need to Evaluate) [A German scale for the need to evaluate]. In Zusammenstellung sozialwissenschaftlicher Items und Skalen. ZIS Version 13.00, ed. Glöckner-Rist

, np. Bonn, Germany: GESIS.

Duffy

S. A.

Morris

R. K.

Rayner

. 1988. Lexical ambiguity and fixation times in reading. Journal of Memory and Language 27:429–46.

Federico

C. M.,

Schneider

M. C.

. 2007. Political expertise and the use of ideology: Moderating effects of evaluative motivation. Public Opinion Quarterly 71:221–52.

Fowler

F. J.

1992. How unclear terms affect survey data. Public Opinion Quarterly 56:218–31.

Galesic

2006. Dropouts on the web: Effects of interest and burden experienced during an online survey. Journal of Official Statistics 22:313–28.

10.

Graesser

A. C.

Cai

Louwerse

M. M.

Daniel

. 2006. Question understanding aid (QUAID). A web facility that tests question comprehensibility. Public Opinion Quarterly 70:3–22.

11.

Haviland

S. E.,

Clark

H. H.

. 1974. What’s new? Acquiring new information as a process in comprehension. Journal of Verbal Learning and Verbal Behavior 13:512–21.

12.

Horning

A. S.

1979. On defining redundancy in language: Case notes. Journal of Reading 22:312–22.

13.

Inhoff

A. W.

Rayner

1986. Parafoveal word processing during eye fixations in reading: Effects of word frequency. Perception & Psychophysics 40:431–39.

14.

Jarvis

W. B. G.

Petty

R. E.

. 1996. The need to evaluate. Journal of Personality and Social Psychology 70:172–94.

15.

Kimball

1973. Seven principles of surface structure parsing in natural language. Cognition 2:15–47.

16.

Kintsch

W.,

Keenan

J. M.

. 1973. Reading rate and retention as a function of the number of propositions in the base structure of sentences. Cognitive Psychology 5:257–79.

17.

Knäuper

Belli

R. F.

Hill

D. H.

Herzog

A. R.

. 1997. Question difficulty and respondents’ cognitive ability: The effect on data quality. Journal of Official Statistics 13:181–99.

18.

Krosnick

J. A.

1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5:213–36.

19.

Krosnick

J. A.

Holbrook

A. L.

Berent

M. K.

Carson

R. T.

Hanemann

W. M.

Kopp

R. J.

Mitchell

R. C.

Presser

Ruud

P. A.

Smith

V. K.

Moody

W. R.

Green

M. C.

Conaway

. 2002. The impact of “no opinion” response options on data quality. Public Opinion Quarterly 66:371–403.

20.

Lenzner

Kaczmirek

Galesic

. 2011. Seeing through the eyes of the respondent: An eye-tracking study on survey question comprehension. International Journal of Public Opinion Research 23:361–73.

21.

Lenzner

Kaczmirek

Lenzner

. 2010. Cognitive burden of survey questions and response times: A psycholinguistic experiment. Applied Cognitive Psychology 24:1003–20.

22.

Lessler

J. T.,

Forsyth

B. H.

. 1996. A coding system for appraising questionnaires. In Answering questions, eds. Schwarz

Sudman

, 259–91. San Francisco: Jossey-Bass.

23.

Long

J. S.

1997. Regression models for categorical and limited dependent variables. Thousand Oaks, CA: Sage.

24.

Mosier

C. I.

1941. A psychometric study of meaning. Journal of Social Psychology 13:123–40.

25.

Myers

J. L.

Cook

A. E.

Kambe

Mason

R. A.

O’Brien

E. J.

. 2000. Semantic and episodic effects on bridging inferences. Discourse Processes 29:179–99.

26.

Paulhus

D. L.

1991. Measurement and control of response bias. In Measures of personality and social psychological attitudes, eds. Robinson

J. P.

Shaver

P. R.

Wrightsman

L. S.

, 17–60. San Diego, CA: Academic Press.

27.

Peytchev

2009. Survey breakoff. Public Opinion Quarterly 73:74–97.

28.

Poe

G. S.

Seeman

McLaughlin

Mehl

Dietz

. 1988. Don’t know boxes in factual questions in a mail questionnaire. Public Opinion Quarterly 52:212–22.

29.

Saris

W. E.,

Gallhofer

I. N.

. 2007. Design, evaluation, and analysis of questionnaires for survey research. Hoboken, NJ: Wiley.

30.

Schaeffer

N. C.

1991. Hardly ever or constantly? Group comparisons using vague quantifiers. Public Opinion Quarterly 55:395–23.

31.

Schmidt

K.-H.,

Metzler

. 1992. Wortschatztest (WST) [Vocabulary test (WST)]. Weinheim, Germany: Beltz Test GmbH.

32.

Smith

T. W.

1987. That which we call welfare by any other name would smell sweeter. Public Opinion Quarterly 51:75–83.

33.

Stöber

1999. Die Soziale-Erwünschtheits-Skala-17 (SES-17): Entwicklung und erste Befunde zu Reliabilität und Validität [The social desirability scale-17 (SDS 17): Development and first findings on reliability and validity]. Diagnostica 45:173–77.

34.

Sturgis

P.,

Smith

. 2010. Assessing the validity of generalized trust questions: What kind of trust are we measuring? International Journal of Public Opinion Research 22:74–92.

35.

Toepoel

Vis

Das

van Soest

. 2009. Design of web questionnaires. An information-processing perspective for the effect of response categories. Sociological Methods & Research 37:371–92.

36.

Tourangeau

Rips

L. J.

Rasinski

. 2000. The psychology of survey response. Cambridge, UK: University Press.

37.

Velez

P.,

Ashworth

S. D.

. 2007. The impact of item readability on the endorsement of the midpoint response in surveys. Survey Research Methods 1:69–74.

38.

Whisman

M. A.

McClelland

G. H.

. 2005. Designing, testing, and interpreting interactions and moderator effects in family research. Journal of Family Psychology 19:111–20.

39.

Ziegler

Kemper

C. J.

Rammstedt

. In press. The Vocabulary and Overclaiming Test (VOC-T). Journal of Individual Differences.

Effects of Survey Question Comprehensibility on Response Quality

Abstract

Keywords

Get full access to this article

References