This study describes a structural equation modeling (SEM) approach to reliability for tests with items having different numbers of ordered categories. A simulation study is provided to compare the performance of this reliability coefficient, coefficient alpha and population reliability for tests having items with different numbers of ordered categories, a one-factor and a bifactor structures, and different skewness distributions of test scores. Results indicated that the proposed reliability coefficient was close to the population reliability in most conditions. An empirical example was used to illustrate the performance of the different coefficients for a test of items with two or three ordered categories.
BandalosD. L.EndersC. K. (1996). The effects of nonnormality and number of response categories on reliability. Applied Measurement in Education, 9, 151-160.
2.
BentlerP. M. (2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74, 137-143.
3.
BernsteinI. H.TengG. (1989). Factoring items and factoring scales are different: Spurious evidence for multidimensionality due to item categorization. Psychological Bulletin, 105, 467-477.
4.
BollenK. A. (1989). Structural equations with latent variables. New York, NY: Wiley.
5.
ChoE.KimS. (2015). Cronbach’s coefficient alpha: Well known but poorly understood. Organizational Research Methods, 18, 207-230.
6.
CronbachL. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
7.
CronbachL. J.ShavelsonR. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64, 391-418.
8.
DeMarsC. E. (2006). Application of the bi-factor multidimensional item response theory model to testlet-based tests. Journal of Educational Measurement, 43, 145-168.
9.
DoaneD. P.SewardL. E. (2011). Measuring skewness: A forgotten statistic?Journal of Statistics Education, 19(2), 1-18.
10.
FinneyS. J.DiStefanoC. (2006). Non-normal and categorical data in structural equation modeling. In HancockG. R.MuellerR. O. (Eds.), Structural equation modeling: A second course (pp. 269-314). Greenwich, CT: Information Age.
11.
FloraD. B.CurranP. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9, 466-491.
12.
GreenS. B.LissitzR. W.MulaikS. A. (1977). Limitations of coefficient alpha as an index of test unidimensionality. Educational and Psychological Measurement, 37, 827-838.
13.
GreenS. B.YangY. (2009). Reliability of summed item scores using structural equation modeling: An alternative to coefficient alpha. Psychometrika, 74, 155-167.
14.
GreenS. B.YangY. (2015). Evaluation of dimensionality in the assessment of internal consistency reliability: Coefficient alpha and omega coefficients. Educational Measurement: Issues and Practice, 34(4), 14-20.
15.
HylandP.BoduszekD.DhingraK.ShevlinM.EganA. (2014). A bifactor approach to modelling the Rosenberg Self Esteem Scale. Personality and Individual Differences, 66, 188-192.
16.
JanusM.OffordD. R. (2007). Development and psychometric properties of the Early Development Instrument (EDI): A measure of children’s school readiness. Canadian Journal of Behavioural Science, 39, 1-22.
17.
KomoritaS. S.GrahamW. K. (1965). Number of scale points and the reliability of scales. Educational and Psychological Measurement, 25, 987-995.
18.
LiY.BoltD. M.FuJ. (2006). A comparison of alternative models for testlets. Applied Psychological Measurement, 30, 3-21.
19.
LissitzR. W.GreenS. B. (1975). Effect of the number of scale points on reliability: A Monte Carlo approach. Journal of Applied Psychology, 60, 10-13.
20.
LordF. M.NovickM. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
21.
LozanoL. M.García-CuetoE.MuñizJ. (2008). Effect of the number of response categories on the reliability and validity of rating scales. Methodology, 4(2), 73-79.
22.
McAbeeS. T.OswaldF. L.ConnellyB. S. (2014). Bifactor models of personality and college student performance: A broad versus narrow view. European Journal of Personality, 28, 604-619.
23.
McDonaldR. P. (1985). Factor analysis and related methods. Hillsdale, NJ: Lawrence Erlbaum.
24.
McDonaldR. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.
25.
MillerM. B. (1995). Coefficient alpha: A basic introduction from the perspectives of classical test theory and structural equation modeling. Structural Equation Modeling, 2, 255-273.
26.
MuthénB.du ToitS. H. C.SpisicD. (1997). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes. Retrieved from http://www.statmodel.com/bmuthen/articles/Article_075.pdf
27.
MuthénB.KaplanD. (1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology, 38, 171-189.
28.
MuthénL. K.MuthénB. O. (1998-2015). Mplus user’s guide (7th ed.). Los Angeles, CA: Author.
29.
NovickM.LewisC. (1967). Coefficient alpha and the reliability of composite measurements. Psychometrika, 32, 1-13.
RaykovT. (1997). Estimation of composite reliability for congeneric measures. Applied Psychological Measurement, 21, 173-184.
32.
RaykovT.ShroutP. E. (2002). Reliability of scales with general structure: Point and interval estimation using a structural equation modeling approach. Structural Equation Modeling, 9, 195-212.
33.
R Core Team. (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Available from http://www.R-project.org/
34.
RevelleW. R. (2017). psych: Procedures for personality and psychological research (R Package Version 1.8.4). Retrieved from https://CRAN.R-project.org/package=psych
35.
RijmenF. (2010). Formal relations and an empirical comparison among the bi-factor, the testlet, and a second-order multidimensional IRT model. Journal of Educational Measurement, 47, 361-372.
36.
SijtsmaK. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74, 107-120.
WengL. J. (2004). Impact of the number of response categories and anchor labels on coefficient alpha and test-retest reliability. Educational and Psychological Measurement, 64, 956-972.
39.
YangY.GreenS. B. (2010). A note on structural equation modeling estimates of reliability. Structural Equation Modeling, 17(1), 66-81.
40.
YangY.GreenS. B. (2011). Coefficient alpha: A reliability coefficient for the 21st century?Journal of Psychoeducational Assessment, 29, 377-392.
41.
YangY.GreenS. B. (2015). Evaluation of structural equation modeling estimates of reliability for scales with ordered categorical items. Methodology, 11(1), 23-34.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
