Sage Journals: Discover world-class research

Abstract

Questionnaires are one of the most important tools in psychological assessment, yet the impact of different numbers of response options on psychometric properties of questionnaires is limited. This study extends existing research by analyzing respondents’ acceptance of and the efficiency of different numbers of response options and replicate findings on reliability and validity. We studied these questions in 540 respondents who filled out the Big Five Inventory–2 and the Emotion Regulation Questionnaire. Two response options, 11 response options and the visual analog scale showed disadvantages in acceptance compared with the original number of response options. The completion time increased by 1.7s per item when moving from 2 to 11 response options. Cronbach’s alpha (but not ordinal alpha based on polychoric covariance) was lowest for two response options. Validity was unaffected. Overall, compared with the typical choice of five or seven response options, fewer or more response options resulted in disadvantages.

Keywords

rating scales scale development psychometrics reliability acceptance

Get full access to this article

View all access options for this article.

References

Abler

Kessler

(2009). Emotion Regulation Questionnaire: Eine deutschsprachige Fassung des ERQ von Gross und John [Emotion Regulation Questionnaire: A German version of the ERQ by Gross and John]. Diagnostica, 55(3), 144-152. https://doi.org/10.1026/0012-1924.55.3.144

Bendig

A. W.

(1953). The reliability of self-ratings as a function of the amount of verbal anchoring and of the number of categories on the scale. Journal of Applied Psychology, 37(1), 38-41. https://doi.org/10.1037/h0057911

Brandstätter

(2014). Persönlichkeits-Adjektiv Skalen PASK5 [Personality-Adjective Scales PASK5]. Zusammenstellung sozialwissenschaftlicher Items und Skalen. https://doi.org/10.6102/zis19

Cronbach

L. J.

(1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. https://doi.org/10.1007/BF02310555

Danner

Rammstedt

Bluemke

Treiber

Berres

Soto

C. J.

John

O. P.

(2016). Die deutsche Version des Big Five Inventory 2 (BFI-2) [The German version of the Big Five Inventory 2 (BFI-2)]. Zusammenstellung sozialwissenschaftlicher Items und Skalen. https://doi.org/10.6102/zis247

Fahrenberg

Hampel

Selg

(2010). Das Freiburger Persönlichkeitsinventar FPI. Revidierte Fassung FPI-R [The Freiburger Personality Inventory FPI. Revised version FPI-R] (8th. ed.). Hogrefe.

Finn

J. A.

Ben-Porath

Y.S.

Tellegen

(2015). Dichotomous versus polytomous response options in psychopathology assessment: Method or meaningful variance? Psychological Assessment, 27(1), 184-193. https://doi.org/10.1037/pas0000044

Fisher

R. A.

(1921). On the probable error of a coefficient of correlation deduced from a small sample. Metron, 1, 3-32. https://digital.library.adelaide.edu.au/dspace/handle/2440/15169

Gadermann

A. M.

Guhn

Zumbo

B. D.

(2012). Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research, and Evaluation, 17(1), Article 3. https://doi.org/10.7275/n560-j767

10.

Garnefski

Kraaij

(2006). Cognitive Emotion Regulation Questionnaire: Development of a short 18-item version (CERQ-short). Personality and Individual Differences, 41(6), 1045-1053. https://doi.org/10.1016/j.paid.2006.04.010

11.

Hilbert

Küchenhoff

Sarubin

Nakagawa

T. T.

Bühner

(2016). The influence of the response format in a personality questionnaire: An analysis of a dichotomous, a Likert-type, and a visual analogue scale. TPM-Testing, Psychometrics, Methodology in Applied Psychology, 23, 3-24. https://doi.org/10.4473/TPM23.1.1

12.

Kemery

E. R.

Dunlap

W. P.

Griffeth

R. W.

(1988). Correction for variance restriction in point-biserial correlations. Journal of Applied Psychology, 73(4), 688-691. https://doi.org/10.1037/0021-9010.73.4.688

13.

Komorita

Graham

(1965). Number of scale points and the reliability of scales. Educational and Psychological Measurement, 25(4), 987-995. https://doi.org/10.1177/001316446502500404

14.

Leung

(2011). A comparison of psychometric properties and normality in 4-, 5-, 6-, and 11-Point Likert scales. Journal of Social Service Research, 37(4), 412-421. https://doi.org/10.1080/01488376.2011.580697

15.

Likert

(1932). A technique for the measurement of attitudes. Archives of Psychology, 22, 1-55.

16.

Lissitz

R. W.

Green

S. B.

(1975). Effect of the number of scale points on reliability: A Monte Carlo approach. Journal of Applied Psychology, 60(1), 10-13. https://doi.org/10.1037/h0076268

17.

Loch

Hiller

Witthöft

(2011). Der Cognitive Emotion Regulation Questionnaire (CERQ) [The Cognitive Emotion Regulation Questionnaire (CERQ)]. Zeitschrift für Klinische Psychologie und Psychotherapie, 40(2), 94-106. https://doi.org/10.1026/1616-3443/a000079

18.

Lozano

García-Cueto

Muñiz

(2008). Effect of the number of response categories on the reliability and validity of rating scales. Methodology, 4(2), 73-79. https://doi.org/10.1027/1614-2241.4.2.73

19.

Matell

M. S.

Jacoby

(1971). Is there an optimal number of alternatives for Likert scale items? Study 1: Reliability and validity. Educational and Psychological Measurement, 31(3), 657-674. https://doi.org/10.1037/h0033601

20.

McNeish

(2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412-433. https://doi.org/10.1037/met0000144

21.

Miller

M. B.

(1995). Coefficient alpha: A basic introduction from the perspectives of classical test theory and structural equation modeling. Structural Equation Modeling, 2(3), 255-273. https://doi.org/10.1080/10705519509540013

22.

Podsakoff

P. M.

MacKenzie

S. B.

Lee

J. Y.

Podsakoff

N. P.

(2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879-903. https://doi.org/10.1037/0021-9010.88.5.879

23.

Preston

C. C.

Colman

A. M.

(2000). Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104(1), 1-15. https://doi.org/10.1016/S0001-6918(99)00050-5

24.

Roberts

B. W.

(2018). Yes or No 2.0? Are Likert scales always preferable to dichotomous rating scales? [Blog post]. https://pigee.wordpress.com/2018/09/07/yes-or-no-2-0-are-likert-scales-always-preferable-to-dichotomous-rating-scales/

25.

Simms

L. J.

Zelazny

K. A.

Williams

Bernstein

(2019). Does the number of response options matter? Psychometric perspectives using personality questionnaire data. Psychological Assessment, 31(4), 557-566. https://doi.org/10.1037/pas0000648

26.

M. L.

Leung

S. O.

(2018). Effects of varying numbers of Likert scale points on factor structure of the Rosenberg Self-Esteem Scale. Asian Journal of Social Psychology, 21(3), 119-128. https://doi.org/10.1111/ajsp.12214

27.

Zumbo

B. D.

Gadermann

A. M.

Zeisser

(2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods, 6(1), 21-29. https://doi.org/10.22237/jmasm/1177992180

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.22 MB

The Agony of Choice: Acceptance,Efficiency,and Psychometric Properties of Questionnaires With Different Numbers of Response Options

Abstract

Keywords

Get full access to this article

References

Supplementary Material