We evaluated how the number of response options affects the psychometric properties of the Big Five Inventory-2 (BFI-2). Using two large samples collected from a market research company (Ns = 893 and 1,213), we tested how different response options of the BFI-2 influenced scale score distributions, internal consistency estimates, convergent validity correlations, and criterion validity correlations. Results suggest that score distributions were impacted by the number of response options such that ceiling and floor effects were more common when using two or three response options than when using more options. Estimates of Cronbach’s alpha were generally lower with fewer scale points as compared with more scale points, but these effects disappeared when ordinal alpha was used. There were no systematic effects of response options on convergent validity and criterion validity correlations. Given these results, there seems to be few psychometric reasons for deciding whether to administer personality items with five, six, or seven scale points.
AdelsonJ. L.McCoachD. B. (2010). Measuring the mathematical attitudes of elementary students: The effects of a 4-point or 5-point Likert-type scale. Educational and Psychological Measurement, 70, 796–807.
2.
AguinisH.PierceC. A.CulpepperS. A. (2009). Scale coarseness as a methodological artifact: Correcting correlation coefficients attenuated from using coarse scales. Organizational Research Methods, 12(4), 623–652.
3.
AustF.DiedenhofenB.UllrichS.MuschJ. (2013). Seriousness checks are useful to improve data validity in online research. Behavior Research Methods, 45, 527–535.
4.
BollenK. A.BarbK. H. (1981). Pearson’s R and coarsely categorized measures. American Sociological Review, 46, 232–239.
5.
CampbellD. T.FiskeD. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105. https://doi.org/10.1037/h0046016
6.
CapikC.GozumS. (2015). Psychometric features of an assessment instrument with Likert and dichotomous response formats. Public Health Nursing, 32(1), 81–86.
7.
CochranW. G. (1954). The combination of estimates from different experiments. Biometrics, 10, 101–129.
8.
CohenP.CohenJ.AikenL. S.WestS. G. (1999). The problem of units and the circumstance for POMP. Multivariate Behavioral Research, 34, 315–346.
9.
CoxA.CourrégéS. C.FederA. H.WeedN. C. (2017). Effects of augmenting response options of the MMPI-2-RF: An extension of previous findings. Cogent Psychology, 4, Article 1323988.
10.
DawesJ. (2008). Do data characteristics change according to the number of scale points used? An experiment using 5-point, 7-point and 10-point scales. International Journal of Market Research, 50, 61–104.
11.
DiedenhofenB.MuschJ. (2016). cocron: A web interface and R package for the statistical comparison of Cronbach’s alpha coefficients. International Journal of Internet Science, 11, 51–60.
12.
DienerE. D.EmmonsR. A.LarsenR. J.GriffinS. (1985). The satisfaction with life scale. Journal of Personality Assessment, 49, 71–75.
13.
DonnellanM. B.OswaldF. L.BairdB. M.LucasR. E. (2006). The Mini-IPIP scales: Tiny-yet-effective measures of the Big Five factors of personality. Psychological Assessment, 18, 192–203.
14.
DonnellanM. B.RakhshaniA. (2023). How does the number of response options impact the psychometric properties of the Rosenberg Self-Esteem Scale?Assessment, 30(6), 1737–1749. https://doi.org/10.1177/10731911221119532
15.
FinnJ. A.Ben-PorathY. S.TellegenA. (2015). Dichotomous versus polytomous response options in psychopathology assessment: Method or meaningful variance?Psychological Assessment, 27, 184–193.
16.
GadermannA. M.GuhnM.ZumboB. D. (2012). Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research, and Evaluation, 17(1), Article 3.
17.
GoldbergL. R. (1993). The structure of phenotypic personality traits. American Psychologist, 48(1), 26–34.
18.
HilbertS.KüchenhoffH.SarubinN.NakagawaT. T.BühnerM. (2016). The influence of the response format in a personality questionnaire: An analysis of a dichotomous, a Likert-type, and a visual analogue scale. TPM-Testing, Psychometrics, Methodology in Applied Psychology, 23, 3–24.
19.
JohnO. P. (2021). History, measurement, and conceptual elaboration of the Big-Five trait taxonomy: The paradigm matures. In JohnO. P.RobinsR. W. (Eds.), Handbook of personality: Theory and research (pp. 35–82). The Guilford Press.
20.
LeungS. O. (2011). A comparison of psychometric properties and normality in 4-, 5-, 6-, and 11-point Likert scales. Journal of Social Service Research, 37, 412–421.
21.
LozanoL. M.García-CuetoE.MuñizJ. (2008). Effect of the number of response categories on the reliability and validity of rating scales. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 4(2), 73–79.
22.
MillerJ. D.GaughanE. T.MaplesJ.PriceJ. (2011). A comparison of agreeableness scores from the Big Five Inventory and the NEO PI-R: Consequences for the study of narcissism and psychopathy. Assessment, 18(3), 335–339.
23.
MüssigM.KubiakJ.EgloffB. (2022). The agony of choice: Acceptance, efficiency, and psychometric properties of questionnaires with different numbers of response options. Assessment, 29(8), 1700–1713. https://doi.org/10.1177/10731911211029379
24.
PaceV. L.BrannickM. T. (2010). How similar are personality scales of the “same” construct? A meta-analytic investigation. Personality and Individual Differences, 493(7), 669–676.
25.
PrestonC. C.ColmanA. M. (2000). Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104, 1–15.
26.
RhemtullaM.Brosseau-LiardP. É.SavaleiV. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354–373.
27.
RosenbergM. (1965). Rosenberg Self-Esteem Scale (RSES). APA PsycTests.
28.
SimmsL. J.ZelaznyK.WilliamsT. F.BernsteinL. (2019). Does the number of response options matter? Psychometric perspectives using personality questionnaire data. Psychological Assessment, 31(4), 557–566. https://doi.org/10.1037/pas0000648
29.
SleepC. E.LynamD. R.MillerJ. D. (2021). A comparison of the validity of very brief measures of the Big Five/Five-factor model of personality. Assessment, 28(3), 739–758.
30.
SotoC. J.JohnO. P. (2017). The next Big Five Inventory (BFI-2): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power. Journal of Personality and Social Psychology, 113(1), 117–143. https://doi.org/10.1037/pspp0000096
31.
WirthR. J.EdwardsM. C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12(1), 58–79. https://doi.org/10.1037/1082-989X.12.1.58
32.
XuM. L.LeungS. O. (2018). Effects of varying numbers of Likert scale points on factor structure of the Rosenberg self-esteem scale. Asian Journal of Social Psychology, 21(3), 119–128.
33.
ZumboB. D.GadermannA. M.ZeisserC. (2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods, 6(1), 21–29.
