The fit of cognitive diagnostic models (CDMs) to response data needs to be evaluated, since CDMs might yield misleading results when they do not fit the data well. Limited-information statistic M2 and the associated root mean square error of approximation (RMSEA2) in item factor analysis were extended to evaluate the fit of CDMs. The findings suggested that the M2 statistic has proper empirical Type I error rates and good statistical power, and it could be used as a general statistical tool. More importantly, we found that there was a strong linear relationship between mean marginal misclassification rates and RMSEA2 when there was model–data misfit. The evidence demonstrated that .030 and .045 could be reasonable thresholds for excellent and good fit, respectively, under the saturated log-linear cognitive diagnosis model.
BrownT. A. (2006). Confirmatory factor analysis for applied research. New York, NY: Guilford.
4.
BrowneM. W. (1984). Asymptotically distribution-free methods for the analysis of covariance structures. British Journal of Mathematical and Statistical Psychology, 37, 62–83.
5.
BrowneM. W.CudeckR. (1993). Alternative ways of assessing model fit. In BollenK. A.LongJ. S. (Eds.), Testing structural equation models (pp. 136–162). Newbury Park, CA: Sage.
6.
CaiL. (2013). flexMIRT® version 2: Flexible multilevel multidimensional item analysis and test scoring [Computer software]. Chapel Hill, NC: Vector Psychometric Group.
7.
CaiL.HansenM. (2013). Limited-information goodness-of-fit testing of hierarchical item factor models. British Journal of Mathematical and Statistical Psychology, 66, 245–276.
8.
CaiL.Maydeu-OlivaresA.CoffmanD. L.ThissenD. (2006). Limited-information goodness-of-fit testing of item response theory models for sparse 2P tables. British Journal of Mathematical and Statistical Psychology, 59, 173–194.
9.
ChenJ.de la TorreJ.ZhangZ. (2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50, 123–140.
10.
DeCarloL. T. (2010). On the analysis of fraction subtraction data: The DINA model, classification, latent class sizes, and the Q-matrix. Applied Psychological Measurement, 35, 8–26.
11.
de la TorreJ. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115–130.
12.
de la TorreJ. (2011). The generalized DINA model framework. Psychometrika, 76, 179–199.
13.
de la TorreJ.DouglasJ. A. (2008). Model evaluation and multiple strategies in cognitive diagnosis: An analysis of fraction subtraction data. Psychometrika, 73, 595–624.
14.
de la TorreJ.LeeY.-S. (2013). Evaluating the Wald test for item-level comparison of saturated and reduced models in cognitive diagnosis. Journal of Educational Measurement, 50, 355–373.
15.
HartzS. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality (Unpublished doctoral dissertation). Department of Statistics, University of Illinois at Urbana-Champaign, Champaign.
16.
HensonR. A.TemplinJ. L.WillseJ. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191–210.
17.
HuL.BentlerP. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, 424–453.
18.
JunkerB. W.SijtsmaK. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272.
19.
JurichD. P. (2014). Assessing model fit of multidimensional item response theory and diagnostic classification models using limited-information statistics (Unpublished doctoral dissertation). Department of Graduate Psychology, James Madison University, Harrisonburg.
20.
KhatriC. G. (1966). A note on a MANOVA model applied to problems in growth curve. Annals of the Institute of Statistical Mathematics, 18, 75–86.
21.
KoehlerK. J.LarntzK. (1980). An empirical investigation of goodness-of-fit statistics for sparse multinomials. Journal of the American Statistical Association, 75, 336–344.
22.
Kunina-HabenichtO.RuppA. A.WilhelmO. (2012). The impact of model misspecification on parameter estimation and item-fit assessment in log-linear diagnostic classification models. Journal of Educational Measurement, 49, 59–81.
23.
LiuY.Maydeu-OlivaresA. (2014). Identifying the source of misfit in item response theory models. Multivariate Behavioral Research, 49, 354–371.
24.
Maydeu-OlivaresA. (2013). Goodness-of-fit assessment of item response theory models. Measurement: Interdisciplinary Research and Perspectives, 11, 71–101.
25.
Maydeu-OlivaresA.JoeH. (2005). Limited- and full-information estimation and goodness-of-fit testing in 2n contingency tables: A unified framework. Journal of the American Statistical Association, 100, 1009–1020.
26.
Maydeu-OlivaresA.JoeH. (2014). Assessing approximate fit in categorical data analysis. Multivariate Behavioral Research, 49, 305–328.
27.
MislevyR. J. (1986). Bayes modal estimation in item response models. Psychometrika, 51, 177–195.
28.
MuthénL. K.MuthénB. O. (2012). Mplus user’s guide (7th ed.). Los Angeles, CA: Author.
29.
R Core Team. (2014). R: A language and environment for statistical computing [Computer software]. Vienna, Austria: R Foundation for Statistical Computing.
30.
ReiserM. (1996). Analysis of residuals for the multionmial item response model. Psychometrika, 61, 509–528.
31.
ReiserM.LinY. (1999). A goodness-of-fit test for the latent class model when expected frequencies are small. Sociological Methodology, 29, 81–111.
RuppA. A.TemplinJ. (2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA Model. Educational and Psychological Measurement, 68, 78–96.
34.
RuppA. A.TemplinJ.HensonR. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guilford.
35.
SinharayS. (2006). Model diagnostics for Bayesian networks. Journal of Educational and Behavioral Statistics, 31, 1–33.
36.
SinharayS.AlmondR. G. (2007). Assessing fit of cognitive diagnostic models a case study. Educational and Psychological Measurement, 67, 239–257.
37.
SinharayS.PuhanG.HabermanS. J. (2011). An NCME instructional module on subscores. Educational Measurement: Issues and Practice, 30, 29–40.
38.
TatsuokaK. K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In FrederiksenN.GlaserR.LesgoldA.SaftoM. (Eds.), Monitoring Skills and Knowledge Acquisition (pp. 453–488). Hillsdale, NJ: Erlbaum.
39.
TemplinJ. L.HensonR. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological methods, 11, 287–305.
