Sage Journals: Discover world-class research

Abstract

Questions using agree/disagree (A/D) scales are ubiquitous in survey research because they save time and space on questionnaires through display in grids, but they have also been criticized for being prone to acquiescent reports. Alternatively, questions using self-description (SD) scales (asking respondents how well a statement describes them from Completely to Not at All) can also be presented in grids or with a common question stem, and by omitting the word agree, SD scales may reduce acquiescence. However, no research has examined how response patterns may differ across A/D and SD scales. In this article, we compare survey estimates, item nonresponse and nondifferentiation across these two types of scales in a mail survey. We find that SD scales outperform A/D scales for non-socially desirable questions that ask about positive topics. For questions that ask about negative topics, we find that estimates for SD items are significantly more negative than A/D items. This may occur because the SD scale is unipolar and has only one negative response option (Not at All), whereas the bipolar A/D scale has two negative response options (Disagree and Strongly Disagree). We recommend that researchers use SD scales for non-socially desirable positive valence questions.

Keywords

agree/disagree scales data quality mail survey questionnaire design self-description scales

Get full access to this article

View all access options for this article.

References

Agency for Healthcare Research and Quality. (2017). MEPS Survey Questionnaires. https://meps.ahrq.gov/survey_comp/survey.jsp

Alwin

D. F.

Baumgartner

E. M.

Beattie

B. A.

(2018). Number of response categories and reliability in attitude measurement. Journal of Survey Statistics and Methodology, 6(2), 212–239.

Astin

A. W.

Astin

H. S.

Lindholm

J. A.

(2011). Assessing students’ spiritual and religious qualities. Journal of College Student Development, 52(1), 39–61.

Berg

I. A.

Rapaport

G. M.

(1954). Response bias in an unstructured questionnaire. The Journal of Psychology, 38(2), 475–481.

Burgard

S. A.

Brand

J. E.

House

J. S.

(2009). Perceived job insecurity and worker health in the United States. Social Science & Medicine, 69(5), 777–785.

Christian

L. M.

Dillman

D. A.

(2004). The influence of graphical and symbolic language manipulations on responses to self-administered questions. Public Opinion Quarterly, 68(1), 57–80.

Christian

L. M.

Dillman

D. A.

Smyth

J. D.

(2007). The effects of mode and format on answers to scalar questions in telephone and web surveys. In Lepkowski

J. M.

Tucker

Brick

J. M.

de Leeuw

E. D.

Japec

Lavrakas

P. J.

Link

M. W.

Sangstera

R. L.

(Eds.), Advances in telephone survey methodology (pp. 250–275). John Wiley.

Cicchetti

D. V.

(1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290.

DeCastellarnau

(2018). A classification of response scale characteristics that affect data quality: A literature review. Quality & Quantity, 52(4), 1523–1559.

10.

de Leeuw

E. D

. (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21(5), 233–255.

11.

Dillman

D. A.

Eltinge

J. L.

Groves

R. M.

Little

R. J. A.

(2002). Survey nonresponse in design, data collection, and analysis. In Groves

R. M.

Dillman

D. A.

Eltinge

J. L.

Little

R. J. A.

(Eds.), Survey nonresponse (pp. 3–26). Wiley.

12.

Dillman

D. A.

Smyth

J. D.

Christian

L. M.

(2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Wiley.

13.

Groves

R. M.

(1989). Survey errors and survey costs. Wiley.

14.

Holbrook

A. L.

(2013, October 23). Agree-disagree questions: Problems and some solutions [Paper presentation]. Webinar presented as part of the American Association for Public Opinion Research Online Continuing Education Program. http://www.srl.uic.edu/seminars/SRL_Webinar_041114.pdf

15.

Holbrook

A. L.

Green

M. C.

Krosnick

J. A.

(2003). Telephone versus face-to-face interviewing of national probability samples with long questionnaires. Public Opinion Quarterly, 67(1), 79–125.

16.

Hox

J. J.

(1994). Hierarchical regression models for interviewer and respondent effects. Sociological Methods & Research, 22(3), 300–318.

17.

Inglehart

Haerpfer

Moreno

Welzel

Kizilova

Diez-Medrano

Lagos

Norris

Ponarin

Puranen

(2014). World values survey questionnaire. http://www.worldvaluessurvey.org/WVSDocumentationWV6.jsp

18.

Kreuter

Presser

Tourangeau

(2008). Social desirability bias in CATI, IVR, and web surveys: The effects of mode and question sensitivity. Public Opinion Quarterly, 72(5), 847–865.

19.

Krosnick

J. A.

(1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236.

20.

Krosnick

J. A.

Presser

(2010). Question and questionnaire design. In Marsden

P. V.

Wright

J. D.

(Eds.), Handbook of survey research (2nd ed., pp. 263–314). Emerald Group.

21.

Kruger

Gilovich

(2004). Actions, intentions, and self-assessment: The road to self-enhancement is paved with good intentions. Personality and Social Psychology Bulletin, 30(3), 328–339.

22.

Lenski

G. E.

Leggett

J. C.

(1960). Caste, class, and deference in the research interview. American Journal of Sociology, 65(5), 463–467.

23.

Mangione

Fowler

F. J.

Louis

T. A.

(1992). Question characteristics and interviewer effects. Journal of Official Statistics, 8(3), 293–307.

24.

Morgan

N. A.

Rego

L. L.

(2006). The value of different customer satisfaction and loyalty metrics in predicting business performance. Marketing Science, 25(5), 426–439.

25.

Narayan

Krosnick

J. A.

(1996). Education moderates some response effects in attitude measurement. Public Opinion Quarterly, 60(1), 58–88.

26.

Nicholls

J. A. F.

Gilbert

G. R.

Roslow

(1998). Parsimonious measurement of customer satisfaction with personal service and the service setting. Journal of Consumer Marketing, 15(3), 239–253.

27.

Nir

(2011). Motivated reasoning and public opinion perception. Public Opinion Quarterly, 75(3), 504–532.

28.

Olson

Bilgen

(2011). The role of interviewer experience on acquiescence. Public Opinion Quarterly, 75(1), 99–114.

29.

Olson

Peytchev

(2007). Effect of interviewer experience on interview pace and interviewer attitudes. Public Opinion Quarterly, 71(2), 273–286.

30.

Olson

Smyth

J. D.

Cochran

(2018). Item location, the interviewer–respondent interaction, and responses to battery questions in telephone surveys. Sociological Methodology, 48(1), 225–268.

31.

Pintrich

P. R.

Smith

D. A. F.

Garcia

McKeachie

W. J.

(1993). Reliability and predictive validity of the Motivated Strategies for Learning Questionnaire (MSLQ). Educational and Psychological Measurement, 53(3), 801–813.

32.

Ross

C. E.

Mirowsky

(1984). Socially-desirable response and acquiescence in a cross-cultural survey of mental health. Journal of Health and Social Behavior, 25(2), 189–197.

33.

Saris

Revilla

Krosnick

J. A.

Shaeffer

E. M.

(2010). Comparing questions with agree/disagree response options to questions with item-specific response options. Survey Research Methods, 4(1), 61–79.

34.

Schnell

Kreuter

(2005). Separating interviewer and sampling-point effects. Journal of Official Statistics, 21(3), 389–410.

35.

Schuman

Presser

(1996). Questions and answers in attitude surveys: Experiments on question form, wording, and context. SAGE.

36.

Smith

Davern

Freese

Morgan

S. L.

Son

Schapiro

Chatterjee

(2018). General social surveys, 1972-2018: Cumulative codebook. National Opinion Research Center.

37.

Sudman

Bradburn

N. M.

(1974). Response effects in surveys: A review and synthesis. Aldine.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.19 MB

Are self-description scales better than agree/disagree scales?

Abstract

Keywords

Get full access to this article

References

Supplementary Material