Sage Journals: Discover world-class research

Abstract

Considerable research has examined the effect of response option order in ordinal bipolar questions such as satisfaction questions. However, no research we know of has examined the effect of the order of presentation of concepts in the question stem or whether stem order moderates response option order. In this article, we experimentally test the main and interaction effects of both stem and response option order for items in self-administered surveys on response distributions and answer changes in eight satisfied/dissatisfied questions. We find consistent evidence that response option order impacts answers. We also find that the order of “satisfied” or “dissatisfied” in the question stem impacts response distributions for four of our eight items but does not moderate the effect of response option order. We discuss the implications of our findings for questionnaire design and secondary data analyses.

Get full access to this article

View all access options for this article.

References

American Association for Public Opinion Research (AAPOR). 2016. Standard definitions: Final dispositions of case codes and outcome rates for surveys. http://www.aapor.org/AAPOR_Main/media/publications/Standard-Definitions20169theditionfinal.pdf. (accessed April 18, 2017).

Belson

W. A.

1966. The effects of reversing the presentation order of verbal rating scales. Journal of Advertising Research 6:30–37.

Bishop

G. F.

1990. Issue involvement and response effects in public opinion surveys. Public Opinion Quarterly 54:209–18.

Carp

F. M.

1974. Position effects on interview responses. Journal of Gerontology 29:581–87.

Chan

J. C.

1991. Response-order effects in Likert-type scales. Educational and Psychological Measurement 51:531–40.

Christian

L. M.

Dillman

D. A.

. 2004. The influence of graphical and symbolic language manipulations on responses to self-administered questions. Public Opinion Quarterly 68:58–81.

Christian

L. M.

Dillman

D. A.

Smyth

J. D.

. 2008. The effects of mode and format on answers to scalar questions in telephone and web surveys. In Advances in telephone survey methodology, eds. Lepkowski

J. M.

Tucker

Brick

J. M.

de Leeuw

Japec

Lavrakas

P. J.

Link

M. W.

Sangster

R. L.

, 250–75. Hoboken, NJ: John Wiley.

Christian

L. M.

Parsons

Dillman

D. A.

. 2009. Designing scalar questions for web surveys. Sociological Methods and Research 37:393–425.

Dickson

Albaum

. 1975. Effects of polarity of semantic differential scales in consumer research. Advances in Consumer Research 2:507–14.

10.

Dillman

D. A.

Smyth

J. D.

Christian

L. M.

. 2014. Internet, phone, mail, and mixed-mode surveys: The tailored design method. Hoboken, NJ: John Wiley.

11.

Friedman

H. H.

Friedman

L. W.

Gluck

. 1988. The effects of scale-checking styles on responses to semantic differential scale. Journal of the Market Research Society 30:477–81.

12.

Gaziano

2005. Comparative analysis of within-household respondent selection techniques. Public Opinion Quarterly 69:124–57.

13.

Hartley

Betts

L. R.

. 2010. Four layouts and a finding: The effects of changes in the order of the verbal labels and numerical values on Likert-type scales. International Journal of Social Research Methodology 13:17–27.

14.

Israel

G. D.

2006. Visual cues and response format effects in mail surveys. Presented at the annual meeting of the Southern Rural Sociological Association, Orlando, FL, February 7.

15.

Israel

G. D.

Taylor

C. L.

. 1990. Can response order bias evaluations? Evaluation and Program Planning 13:1–7.

16.

Johnson

D. J.

1981. Effects of the order of presentation of evaluative dimensions for bipolar scales in four societies. The Journal of Social Psychology 113:21–27.

17.

Kalton

Collins

Brook

. 1978. Experiments in wording opinion questions. Journal of the Royal Statistical Society. Series C (Applied Statistics) 27:149–61.

18.

Keusch

2012. The direction of rating scales and its influence on response behavior in web surveys. Presented at the American Association for Public Opinion Research Conference, Orlando, FL, May 17–20.

19.

Krosnick

J. A.

1991. Response strategies for coping with cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5:213–36.

20.

Krosnick

J. A.

1999. Survey research. Annual Review of Psychology 50:537–67.

21.

Liu

Keusch

. 2017. Effects of scale direction on response style of ordinal rating scales. Journal of Official Statistics 33:137–54.

22.

Mathews

C. O.

1929. The effect of the printed response words on an interest questionnaire. Journal of Educational Psychology 30:128–34.

23.

Mingay

D. J.

Greenwell

M. T.

. 1989. Memory bias and response-order effects. Journal of Official Statistics 5:253–63.

24.

Narayan

Krosnick

J. A.

. 1996. Education moderates some response effects in attitude measurement. Public Opinion Quarterly 60:58–88.

25.

Olson

Parkhurst

. 2013. Collecting paradata for measurement error evaluations. In Improving surveys with paradata: Analytic uses of process information, ed. Kreuter

, 43–72. Hoboken, NJ: John Wiley.

26.

Payne

J. D.

1972. The effects of reversing the order of verbal rating scales in a postal survey. Journal of the Market Research Society 14:30–44.

27.

Powers

E. A.

Morrow

Goudy

W. J.

Keith

P. M.

. 1977. Serial order preference in survey research. Public Opinion Research 41:80–85.

28.

Rammstedt

Krebs

. 2007. Does response scale format affect the answering of personality scales? Assessing the big five dimensions of personality with different response scales in a dependent sample. European Journal of Psychological Assessment 23:32–38.

29.

Salmon

Nichols

. 1983. The next-birthday method of respondent selection. Public Opinion Quarterly 47:270–76.

30.

SAS Institute, Inc. n.d. Software, Version 9.4. Cary, NC: SAS Institute, Inc.

31.

Schuman

Presser

. 1981. Questions and answers in attitude surveys: Experiments on question form, wording, and context. Thousand Oaks, CA: Sage.

32.

Sheluga

Jacoby

Major

. 1978. Whether to agree–disagree or disagree–agree: The effects of anchor order on item nonresponse. Advances in Consumer Research 5:109–13.

33.

Stern

M. J.

2008. The use of client-side paradata in analyzing the effects of visual layout on changing responses in web surveys. Field Methods 20:377–98.

34.

Toepoel

Das

van Soest

. 2009. Design of web questionnaires: The effects of layout in rating scales. Journal of Official Statistics 25:509–28.

35.

Tourangeau

Couper

M. P.

Conrad

. 2004. Spacing, position, and order: Interpretive heuristics for visual features of surveys questions. Public Opinion Quarterly 68:368–93.

36.

Weng

L.-J.

Cheng

C.-P.

. 2000. Effects of response order on Likert-type scales. Educational and Psychological Measurement 60:908–24.

37.

Yan

Keusch

. 2015. The effects of the direction of rating scales on survey responses in a telephone survey. Public Opinion Quarterly 79:145–65.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.33 MB

Effects of Stem and Response Order on Response Patterns in Satisfaction Ratings

Abstract

Get full access to this article

References

Supplementary Material