Sage Journals: Discover world-class research

Abstract

In applications of cognitive diagnostic models (CDMs), practitioners usually face the difficulty of choosing appropriate CDMs and building accurate Q-matrices. However, functions of model-fit indices that are supposed to inform model and Q-matrix choices are not well understood. This study examines the performance of several promising model-fit indices in selecting model and Q-matrix under different sample size conditions. Relative performance between Akaike information criterion and Bayesian information criterion in model and Q-matrix selection appears to depend on the complexity of data generating models, Q-matrices, and sample sizes. Among the absolute fit indices, MX2 is least sensitive to sample size under correct model and Q-matrix specifications, and performs the best in power. Sample size is found to be the most influential factor on model-fit index values. Consequences of selecting inaccurate model and Q-matrix in classification accuracy of attribute mastery are also evaluated.

Keywords

model-fit indices cognitive diagnostic models Q-matrices simulation

Get full access to this article

View all access options for this article.

References

Chen

de la Torre

Zhang

(2013). Relative and absolute fit evaluation in cognitive diagnostic modeling. Journal of Educational Measurement, 50, 123-140.

Chen

Thissen

(1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265-289.

Choi

H.-J.

Templin

J. L.

Cohen

A. S.

Atwood

C. H.

(2010, April). The impact of model misspecification on estimation accuracy in diagnostic classification models. Paper presented at the meeting of the National Council on Measurement in Education (NCME), Denver, CO.

DeCarlo

L. T.

(2012). Recognizing uncertainty in the Q-matrix via a Bayesian extension of the DINA model. Applied Psychological Measurement, 36, 447-468.

de la Torre

(2008). An empirically-based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45, 343-362.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76, 179-199.

de la Torre

Lee

Y.-S.

(2013). Evaluating the Wald test for item-level comparison of saturated and reduced models in cognitive diagnosis. Journal of Educational Measurement, 50, 353-373.

DiBello

L. V.

Roussos

L. A.

Stout

W. F.

(2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In Rao

C. R.

Sinharay

(Eds.), Handbook of statistics (Vol. 26, Psychometrics, pp. 979-1030). Amsterdam, The Netherlands: Elsevier.

Doornik

J. A.

(2003). Object-Oriented Matrix Programming Using Ox (Version 3.1) [Computer software]. London, England: Timberlake Consultants Press.

10.

Hansen

Cai

Monroe

(2014). Limited-information goodness-of-fit testing of diagnostic classification item response theory models (CRESST Report 840). Los Angeles: National Center for Research on Evaluation. Standards, and Student Testing (CRESST), University of California.

11.

Hartz

S. M.

(2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality (Unpublished doctoral dissertation). University of Illinois at Urbana-Champaign.

12.

Henson

R. A.

Templin

J. L.

Willse

J. T.

(2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191-210.

13.

Huebner

Wang

(2011). A note on comparing examinee classification methods for cognitive diagnosis models. Educational and Psychological Measurement, 71, 407-419.

14.

Jiao

(2009). Diagnostic classification models: Which one should I use? Measurement: Interdisciplinary Research & Perspective, 7(1), 65-67.

15.

Kunina-Habenicht

Rupp

A. A.

Wilhelm

(2009). A practical illustration of multidimensional diagnostic skills profiling: Comparing results from confirmatory factor analysis and diagnostic classification models. Studies in Educational Evaluation, 35, 64-70.

16.

Kunina-Habenicht

Rupp

A. A.

Wilhelm

(2012). The impact of model misspecification on parameter estimation and item-fit assessment in log-linear diagnostic classification models. Journal of Educational Measurement, 49, 59-81.

17.

Levy

Mislevy

R. J.

Sinharay

(2007, April). Posterior predictive model checking for conjunctive multidimensionality in item response theory. Paper presented at the 2007 Annual Meeting of the National Council on Measurement in Education, Chicago, IL.

18.

Suen

H. K.

(2013). Constructing and validating a Q-matrix for cognitive diagnostic analyses of a reading test. Educational Assessment, 18, 1-25.

19.

McDonald

R. P.

Mok

M. M.-C.

(1995). Goodness of fit in item response models. Multivariate Behavioral Research, 30, 23-40.

20.

Robitzsch

Kiefer

George

A. C.

Uenlue

(2013). Cognitive Diagnosis Modeling (R package version 1.5-10). Retrieved from https://sites.google.com/site/alexanderrobitzsch/software

21.

Rojas

de la Torre

Olea

(2012, April). Choosing between general and specific cognitive diagnosis models when the sample size is small. Paper presented at the meeting of the National Council on Measurement in Education, Vancouver, British Columbia, Canada.

22.

Rupp

A. A.

Templin

(2008). Effects of Q-matrix misspecification on parameter estimates and misclassification rates in the DINA model. Educational and Psychological Measurement, 68, 78-98.

23.

Templin

J. L.

Hoffman

(2013). Obtaining diagnostic classification model estimates using Mplus. Educational Measurement: Issues and Practice, 32(2), 37-50.

24.

Wang

Gierl

M. J.

(2011). Using the Attribute Hierarchy Method to make diagnostic inferences about examinees’ cognitive skills in critical reading. Journal of Educational Measurement, 48, 165-187.

25.

Yen

W. M.

(1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125-145.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.35 MB

Performance of Fit Indices in Choosing Correct Cognitive Diagnostic Models and Q-Matrices

Abstract

Keywords

Get full access to this article

References

Supplementary Material