This article discusses the theoretical and practical contributions of Zumbo, Gadermann, and Zeisser’s family of ordinal reliability statistics. Implications, interpretation, recommendations, and practical applications regarding their ordinal measures, particularly ordinal alpha, are discussed. General misconceptions relating to this family of ordinal reliability statistics are highlighted, and arguments for interpreting ordinal alpha as a measure of hypothetical reliability, as opposed to observed reliability, are presented. It is concluded that ordinal alpha should not be used in routine reliability analyses and reports, and instead should be understood as hypothetical tool, similar to the Spearman–Brown prophecy formula, for theoretically increasing the number of ordinal categorical response options in future applied testing applications.
AgrestiA. (2002). Categorical data analysis (2nd ed.). Hoboken, NJ: John Wiley.
2.
BakerF. B.KimS. H. (2004). Item response theory: Parameter estimation techniques (2nd ed.). New York, NY: Dekker.
3.
BartholomewD.KnottM.MoustakiI. (2011). Latent variable models and factor analysis: A unified approach. New York, NY: Wiley.
4.
BentlerP. M. (2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74(1), 137-143. doi:10.1007/S11336-008-9100-1
5.
ChalmersR. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1-29. doi:10.18637/jss.v048.i06
6.
CramérH. (1946). Mathematical methods of statistics. Princeton, NJ: Princeton University Press.
7.
CrockerL.AlginaJ. (1986). Introduction to classical and modern test theory. New York, NY: Holt, Rinehart, & Winston.
8.
CronbachL. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
9.
CudeckR. (1989). Analysis of correlation matrices using covariance structure models. Psychological Bulletin, 105, 317-327.
10.
EmbretsonS. E.ReiseS. P. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.
11.
FloraD. B.CurranP. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9, 466-491.
12.
FloraD. B.LaBrishC.ChalmersR. P. (2012). Old and new ideas for data screening and assumption testing for exploratory and confirmatory factor analysis. Frontiers in Psychology, 3, 55. doi:10.3389/fpsyg.2012.00055
13.
FoxJ. (2008). Applied regression analysis and generalized linear models. Thousand Oaks, CA: SAGE.
14.
GadermannA. M.GuhnM.ZumboB. D. (2012). Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research & Evaluation, 17(3), 1-13.
15.
GreenS. B.YangY. (2009). Reliability of summed item scores using structural equation modeling: An alternative to coefficient alpha. Psychometrika, 74, 155-167. doi:10.1007/S11336-008-9099-3
16.
GuttmanL. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10, 255-282.
17.
JörsekogK. G.SörbomD. (2006). LISREL 8.8 for Windows [Computer software]. Skokie, IL: Scientific Software International.
18.
KuderG. F.RichardsonM. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2, 151-160.
19.
LiuY.WuA. D.ZumboB. D. (2010). The impact of outliers on Cronbach’s coefficient alpha estimate of reliability: Ordinal/rating scale item responses. Educational and Psychological Measurement, 70(1), 5-21. doi:10.1177/0013164409344548
20.
LordF. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
21.
LordF. M.NovickM. R. (1968). Statistical theory of mental test scores. Reading, MA: Addison-Wesley.
22.
MacCallumR.ZhangS.PreacherK. J.RuckerD. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19-40.
23.
McDonaldR. P. (1985). Factor analysis and related methods. Hillsdale, NJ: Lawrence Erlbaum.
24.
McDonaldR. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.
25.
McNeishD. (2018). Thanks Coefficient Alpha, We’ll Take It From Here. Psychological Methods, 23(3), 412-433.
26.
MuthénL. K.MuthénB. O. (2008). Mplus (Version 5.0) [Computer program]. Los Angeles, CA: Muthén & Muthén.
27.
OlidenP. E.ZumboB. D. (2008). Coefficients of feasibility for ordinal response scales. Psicothema, 20, 896-901.
28.
OlssonU. (1979). Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44, 443-460.
29.
R Core Team. (2016). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria: R Foundation. Retrieved from https://www.R-project.org/
30.
RevelleW. (1979). Hierarchical cluster-analysis and the internal structure of tests. Multivariate Behavioral Research, 14(1), 57-74.
31.
RhemtullaM.Brousseau-LaidP. E.SavaleiV. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354-373.
32.
RosseelY. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2). doi:10.18637/jss.v048.i02. Retrieved from http://www.jstatsoft.org/v48/i02
33.
SamejimaF. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monographs, (17). doi:10.1002/j.2333-8504.1968.tb00153.x
34.
SijtsmaK. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74, 107-120. doi:10.1007/s11336-008-9101-0
35.
StevensS. S. (1946). On the theory of scales of measurement. Science, 103, 677-680.
36.
TraubR. E. (1997). Classical test theory in historical perspective. Educational Measurement: Issues and Practice, 16(4), 8-14.
37.
ZumboB. D.GadermannA. M.ZeisserC. (2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods, 6, 21-29.
