Limited-information fit measures appear to be promising in assessing the goodness-of-fit of dichotomous response cognitive diagnosis models (CDMs), but their performance has not been examined for polytomous response CDMs. This study investigates the performance of the Mord statistic and standardized root mean square residual (SRMSR) for an ordinal response CDM—the sequential generalized deterministic inputs, noisy “and” gate model. Simulation studies showed that the Mord statistic had well-calibrated Type I error rates, but the correct detection rates were influenced by various factors such as item quality, sample size, and the number of response categories. In addition, the SRMSR was also influenced by many factors and the common practice of comparing the SRMSR against a prespecified cut-off (e.g., .05) may not be appropriate. A set of real data was analyzed as well to illustrate the use of Mord statistic and SRMSR in practice.
BishopY. M. M.FienbergS. E.HollandP. W. (1975). Discrete multivariate analysis: Theory and practice (With the collaboration of R. J. Light & F. Mosteller). Cambridge, MA: MIT Press.
2.
BradshawL.IzsákA.TemplinJ.JacobsonE. (2014). Diagnosing teachers’ understandings of rational numbers: Building a multidimensional test within the diagnostic classification framework. Educational Measurement: Issues and Practice, 33, 2-14.
3.
CaiL. (2017). flexMIRT: Flexible multilevel multidimensional item analysis and test scoring [Computer software] (Version 3.51). Chapel Hill, NC: Vector Psychometric Group.
4.
CaiL.HansenM. (2013). Limited-information goodness-of-fit testing of hierarchical item factor models. British Journal of Mathematical and Statistical Psychology, 66, 245-276.
5.
ChangH.-H.YingZ. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213-229.
6.
ChenF.LiuY.XinT.CuiY. (2018). Applying the M2 statistic to evaluate the fit of diagnostic classification models in the presence of attribute hierarchies. Frontiers in Psychology, 9, Article 1875.
7.
ChenJ.de la TorreJ.ZhangZ. (2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50, 123-140.
8.
ChiuC.-Y.DouglasJ. A.LiX. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74, 633-665.
9.
de la TorreJ. (2011). The generalized DINA model framework. Psychometrika, 76, 179-199.
10.
de la TorreJ.ChiuC.-Y. (2016). A general method of empirical Q-matrix validation. Psychometrika, 81, 253-273.
11.
de la TorreJ.LeeY. S. (2013). Evaluating the Wald test for item-level comparison of saturated and reduced models in cognitive diagnosis. Journal of Educational Measurement, 50, 355-373.
12.
GeorgeA. C.RobitzschA. (2015). Cognitive diagnosis models in R: A didactic. The Quantitative Methods for Psychology, 11, 189-205.
13.
HaertelE. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 301-321.
14.
HansenM.CaiL.MonroeS.LiZ. (2016). Limited-information goodness-of-fit testing of diagnostic classification item response models. British Journal of Mathematical and Statistical Psychology, 69, 225-252.
15.
JiangZ.MaW. (2018). Integrating differential evolution optimization to cognitive diagnostic model estimation. Frontiers in Psychology, 9, Article 2142.
16.
JurichD. P. (2014). Assessing model fit of multidimensional item response theory and diagnostic classification models using limited-information statistics (Unpublished doctoral dissertation). James Madison University, Harrisonburg, VA.
17.
LiuR.Huggins-ManleyA. C.BulutO. (2018). Retrofitting diagnostic classification models to responses from IRT-based assessment forms. Educational and Psychological Measurement, 78, 357-383.
18.
LiuR.JiangZ. (2018). Diagnostic classification models for ordinal item responses. Frontiers in Psychology, 9, Article 2512.
19.
LiuY.TianW.XinT. (2016). An application of M2 statistic to evaluate the fit of cognitive diagnostic models. Journal of Educational and Behavioral Statistics, 41, 3-26.
20.
LiuY.XinT.LiL.TianW.LiuX. (2016). An improved method for differential item functioning detection in cognitive diagnosis models: An application of Wald statistic based on observed information matrix. Acta Psychologica Sinica, 48, 588-598.
21.
MaW.de la TorreJ. (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69, 253-275.
22.
MaW.de la TorreJ. (2018). GDINA: The generalized DINA model framework [Computer software] (Version 2.1). Retrieved from https://CRAN.R-project.org/package=GDINA
23.
MaW.de la TorreJ. (2019). Category-level model selection for the sequential G-DINA model. Journal of Educational and Behavioral Statistics, 44, 45-77.
24.
MaW.de la TorreJ. (2020). An empirical Q-matrix validation method for the sequential G-DINA model. British Journal of Mathematical and Statistical Psychology, 73, 143-166.
25.
MaW.IaconangeloC.de la TorreJ. (2016). Model similarity, model selection, and attribute classification. Applied Psychological Measurement, 40, 200-217.
26.
Maydeu-OlivaresA. (2013). Goodness-of-fit assessment of item response theory models. Measurement: Interdisciplinary Research and Perspectives, 11, 71-101.
27.
Maydeu-OlivaresA.JoeH. (2005). Limited-and full-information estimation and goodness-of-fit testing in 2 n contingency tables: A unified framework. Journal of the American Statistical Association, 100, 1009-1020.
28.
Maydeu-OlivaresA.JoeH. (2006). Limited information goodness-of-fit testing in multidimensional contingency tables. Psychometrika, 71, 713-732.
29.
Maydeu-OlivaresA.JoeH. (2014). Assessing approximate fit in categorical data analysis. Multivariate Behavioral Research, 49, 305-328.
30.
ReiserM. (2008). Goodness-of-fit testing using components based on marginal frequencies of multinomial data. British Journal of Mathematical and Statistical Psychology, 61, 331-360.
31.
SamejimaF. (1997). Graded response model. In van der LindenW. J.HambletonR. K. (Eds.), Handbook of modern item response theory (pp. 85-100). New York, NY: Springer.
32.
SessomsJ.HensonR. A. (2018). Applications of diagnostic classification models: A literature review and critical commentary. Measurement: Interdisciplinary Research and Perspectives, 16, 1-17.
33.
SorrelM. A.AbadF. J.OleaJ.de la TorreJ.BarradaJ. R. (2017). Inferential item-fit evaluation in cognitive diagnosis modeling. Applied Psychological Measurement, 41, 614-631.
34.
TemplinJ. L.HensonR. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287-305.
35.
von DavierM. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287-307.
36.
WangC.ShuZ.ShangZ.XuG. (2015). Assessing item-level fit for the DINA model. Applied Psychological Measurement, 39, 525-538.
37.
XuX.ChangH.DouglasJ. (2003, April). Computerized adaptive testing strategies for cognitive diagnosis. Paper presented at the Annual Meeting of the National Council of Measurement in Education, Chicago, IL.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
