We examined the performance of coefficient alpha and its potential competitors (ordinal alpha, omega total, Revelle’s omega total [omega RT], omega hierarchical [omega h], greatest lower bound [GLB], and coefficient H) with continuous and discrete data having different types of non-normality. Results showed the estimation bias was acceptable for continuous data with varying degrees of non-normality when the scales were strong (high loadings). This bias, however, became quite large with moderate strength scales and increased with increasing non-normality. For Likert-type scales, other than omega h, most indices were acceptable with non-normal data having at least four points, and more points were better. For different exponential distributed data, omega RT and GLB were robust, whereas the bias of other indices for binomial-beta distribution was generally large. An examination of an authentic large-scale international survey suggested that its items were at worst moderately non-normal; hence, non-normality was not a big concern. We recommend (a) the demand for continuous and normally distributed data for alpha may not be necessary for less severely non-normal data; (b) for severely non-normal data, we should have at least four scale points, and more points are better; and (c) there is no single golden standard for all data types, other issues such as scale loading, model structure, or scale length are also important.
BandalosD. L.EndersC. K. (1996). The effects of nonnormality and number of response categories on reliability. Applied Measurement in Education, 9(2), 151–160. https://doi.org/10.1207/s15324818ame0902_4
2.
BayK. S. (1973). The effect of non-normality on the sampling distribution and standard error of reliability coefficient estimates under an analysis of variance model. British Journal of Mathematical & Statistical Psychology, 26, 45–57. https://doi.org/10.1111/j.2044-8317.1973.tb00505.x
BlancaM. J.ArnauJ.López-MontielD.BonoR.BendayanR. (2013). Skewness and kurtosis in real data samples. Methodology, 9, 78–84. https://doi.org/10.1027/1614-2241/a000057
5.
BonanomiA.CantaluppiG.RusconeM. N.OsmettiS. (2015). A new estimator of Zumbo’s ordinal alpha: A copula approach. Quality & Quantity: International Journal of Methodology, 49, 941–953. https://doi.org/10.1007/s11135-014-0114-8
6.
ChalmersR. P. (2018). On misconceptions and the limited usefulness of ordinal alpha. Educational and Psychological Measurement, 78(6), 1056–1071. https://doi.org/10.1177/0013164417727036
ChoE. (2022). The accuracy of reliability coefficients: A reanalysis of existing simulations. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000475
9.
CronbachL. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555
10.
CurranP. J.WestS. G.FinchJ. F. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1(1), 16–29.
11.
EdwardsA. A.JoynerK. J.SchatschneiderC. (2021). A simulation study on the performance of different reliability estimation methods. Educational and Psychological Measurement, 81(6), 1089–1117. https://doi.org/10.1177/0013164421994184
12.
FleishmanA. I. (1978). A method for simulating non-normal distributions. Psychometrika, 43(4), 521–532.
13.
FoldnesN.GrønnebergS. (2019). On identification and non-normal simulation in ordinal covariance and item response models. Psychometrika, 84(4), 1000–1017. https://doi.org/10.1007/s11336-019-09688-z
14.
FoldnesN.GrønnebergS. (2020). Pernicious polychorics: The impact and detection of underlying non-normality. Structural Equation Modeling, 27(4), 525–543. https://doi.org/10.1080/10705511.2019.1673168
15.
FosterR. C. (2021). KR-20 and KR-21 for some non-dichotomous data (It’s not just Cronbach’s alpha). Educational and Psychological Measurement, 81(6), 1172–1202. https://doi.org/10.1177/0013164421992535
16.
FuY.WenZ.WangY. (2021). A comparison of reliability estimation based on confirmatory factor analysis and exploratory structural equation models. Educational and Psychological Measurement, 82(2), 205–224. https://doi.org/10.1177/00131644211008953
17.
GadermannA. M.GuhnM.ZumboB. D. (2012). Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research & Evaluation, 17(1), 1–13.
18.
GeldhofG. J.PreacherK. J.ZyphurM. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72–91. https://doi.org/10.1037/a0032138
19.
GreenS. B.YangY. (2009). Reliability of summed item scores using structural equation modeling: An alternative to coefficient alpha. Psychometrika, 74(1), 155–167. https://doi.org/10.1007/s11336-008-9099-3
20.
GuttmanL. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10(4), 255–282.
21.
HancockG. R.MuellerR. O. (2011). The reliability paradox in assessing structural relations within covariance structure models. Educational and Psychological Measurement, 71(2), 306–324. https://doi.org/10.1177/0013164410384856
22.
HauK. T.MarshH. W. (2004). The use of item parcels in structural equation modelling: Non-normal data and small sample sizes. British Journal of Mathematical & Statistical Psychology, 57(2), 327–351. https://doi.10.1111/j.2044-8317.2004.tb00142.x
23.
HooglandJ. J.BoomsmaA. (1998). Robustness studies in covariance structure modeling an overview and a meta-analysis. Sociological Methods and Research, 26(3), 329–367. https://doi.org/10.1177/0049124198026003003
24.
JacksonP. H.AgunwambaC. C. (1977). Lower bounds for the reliability of the total score on a test composed of non-homogeneous items: I algebraic lower bounds. Psychometrika, 42(4), 567–578.
25.
KelleyK.PornprasertmanitS. (2016). Confidence intervals for population reliability coefficients: Evaluation of methods, recommendations, and software for composite measures. Psychological Methods, 21(1), 69–92. https://doi.org/10.1037/a0040086
26.
KimS.LuZ.CohenA. S. (2020). Reliability for tests with items having different numbers of ordered categories. Applied Psychological Measurement, 44(2), 137–149. https://doi.org/10.1177/0146621619835498
27.
LiC. H. (2016). The performance of ML, DWLS, and ULS estimation with robust corrections in structural equation models with ordinal variables. Psychological Methods, 21(3), 369–387. https://doi.org/10.1037/met0000093.supp
28.
LiC. H. (2021). Statistical estimation of structural equation models with a mixture of continuous and categorical observed variables. Behavior Research Methods, 53, 2191–2213. https://doi.org/10.3758/s13428-021-01547-z
29.
LissitzR. W.GreenS. B. (1975). Effect of the number of scale points on reliability: A Monte Carlo approach. Journal of Applied Psychology, 60(1), 10–13. https://doi.org/10.1037/h0076268
30.
LordF. M.NovickM. R. (1968). Statistical theories of mental test scores. Addison-Wesley.
31.
McDonaldR. P. (1999). Test theory: A unified approach. Erlbaum.
32.
McNeishD. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412–433. https://doi.org/10.1037/met0000144
33.
MicceriT. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156–166.
34.
MurrayA. L.BoothT.EisnerM.ObsuthI.RibeaudD. (2019). Quantifying the strength of general factors in psychopathology: A comparison of CFA with maximum likelihood estimation, BSEM, and ESEM/EFA bifactor approaches. Journal of Personality Assessment, 101(6), 631–643. https://doi.org/10.1080/00223891.2018.1468338
35.
NunnallyJ. C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
36.
OECD. (2019a). PISA 2018 assessment and analytical framework. OECD Publishing.
Olvera AstiviaO. L.KrocE.ZumboB. D. (2020). The role of item distributions on reliability estimation: The case of Cronbach’s coefficient alpha. Educational and Psychological Measurement, 80(5), 825–846. https://doi.org/10.1177/0013164420903770
39.
RaykovT.MarcoulidesG. A. (2016). Scale reliability evaluation under multiple assumption violations. Structural Equation Modeling, 23(1–2), 302–313.
40.
RaykovT.MarcoulidesG. A. (2019). Thanks coefficient alpha, we still need you!. Educational and Psychological Measurement, 79(1), 200–210. https://doi.org/10.1177/0013164417725127
SavaleiV. (2010). Small sample statistics for incomplete nonnormal data: Extensions of complete data formulae and a Monte Carlo comparison. Structural Equation Modeling, 17(2), 241–264. https://doi.org/10.1080/10705511003659375
43.
SavaleiV.ReiseS. P. (2019). Don’t forget the model in your model-based reliability coefficients: A reply to McNeish (2018). Collabra: Psychology, 5(1), 36. https://doi.org/10.1525/collabra.247
44.
SchmidJ.LeimanJ. M. (1957). The development of hierarchical factor solutions. Psychometrika, 22(1), 53–61.
45.
ShapiroA.Ten BergeJ. M. (2002). Statistical inference of minimum rank factor analysis. Psychometrika, 67(1), 79–94.
ShultzG. S. (1993). A Monte Carlo study of the robustness of coefficient alpha [Master’s thesis]. University of Ottawa.
48.
SijtsmaK.PfadtJ. M. (2021). Part II: On the use, the misuse, and the very limited usefulness of Cronbach’s alpha: Discussing lower bounds and correlated errors. Psychometrika, 86, 843–860. https://doi.org/10.1007/s11336-021-09789-8
49.
TrincheraL.MarieN.MarcoulidesG. A. (2018). A distribution free interval estimate for coefficient alpha. Structural Equation Modeling, 25(6), 876–887. https://doi.org/10.1080/10705511.2018.1431544
50.
Trizano-HermosillaI.AlvaradoJ. M. (2016). Best alternatives to Cronbach’s alpha reliability in realistic conditions: Congeneric and asymmetrical measurements. Frontiers in Psychology, 7(MAY), 1–8. https://doi.org/10.3389/fpsyg.2016.00769
51.
Trizano-HermosillaI.Gálvez-NietoJ. L.AlvaradoJ. M.SaizJ. L.Salvo-GarridoS. (2021). Reliability estimation in multidimensional scales: Comparing the bias of six estimators in measures with a bifactor structure. Frontiers in Psychology, 12, 508287.
XiaoL. F.HauK. T. (2021). Alternatives to Cronbach’s alpha: Bias, sensitivity to mismatched item, scale strength, scale length, and sample size [Unpublished manuscript]. Faculty of Education, The Chinese University of Hong Kong.
54.
YaziciB.YolacanS. (2007). A comparison of various tests of normality. Journal of Statistical Computation and Simulation, 77(2), 175–183. https://doi.org/10.1080/10629360600678310
55.
ZimmermanD. W.ZumboB. D.LalondeC. (1993). Coefficient alpha as an estimate of test reliability under violation of two assumptions. Educational and Psychological Measurement, 53, 33–49. https://doi.org/10.1177/07399863870092005
56.
ZinbargR.RevelleW.YovelI.WenL. (2005). Cronbach’s α, Revelle’s β, and McDonald’s ωh: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70(1), 123–133. https://doi.org/10.1007/s11336-003-0974-7
57.
ZumboB. D.GadermannA. M.ZeisserC. (2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods, 6(1), 21–29. https://doi.org/10.22237/jmasm/1177992180
58.
ZumboB. D.KrocE. (2019). A measurement is a choice and Stevens’ scales of measurement do not help make it: A response to Chalmers. Educational and Psychological Measurement, 79(6), 1184–1197. https://doi.org/10.1177/0013164419844305
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
