Sage Journals: Discover world-class research

Abstract

Diagnostic classification models (DCMs) have been widely used in education, psychology, and many other disciplines. To select the most appropriate DCM for each item, the Wald test has been recommended. However, prior research has revealed that this test provides inflated Type I error rates. To address this problem, the authors propose to replace the asymptotic covariance matrix from the original version of the Wald statistic with a matrix obtained from improved computation methods. In this study, the Wald test based on the observed information matrix and the Wald test based on the sandwich-type matrix are proposed for item-level model comparisons and a simulation study is conducted to investigate their empirical behavior. Simulation results indicate that when the sample size is reasonably large ( $N \geq 1, 000$ ), the Type I error rates of the Wald test based on the sandwich-type matrix are accurate with adequate or excellent power under most of the simulation conditions.

Keywords

model comparison diagnostic classification models Wald test asymptotic covariance matrix likelihood ratio test

Get full access to this article

View all access options for this article.

References

Bradley

(1978). Robustness? British Journal of Mathematical and Statistical Psychology, 31, 144-152.

de la Torre

(2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115-130.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76, 179-199.

de la Torre

Lee

Y.-S.

(2013). Evaluating the Wald test for item-level comparison of saturated and reduced models in cognitive diagnosis. Journal of Educational Measurement, 50, 355-373.

Greeno

J. G.

(1980). Trends in the theory of knowledge for problem solving. In Tuma

D. T.

Reif

(Eds.), Problem solving and education: Issues in teaching and research (pp. 9-23). Hillsdale, NJ: Lawrence Erlbaum.

Haertel

E. H.

(1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 301-323.

Hartz

S. M.

(2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality (Unpublished doctoral dissertation), University of Illinois, Urbana–Champaign.

Henson

R. A.

Templin

J. L.

Willse

J. T.

(2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191-210.

Junker

B. W.

Sijtsma

(2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272.

10.

Kirk

R. E.

(1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.

11.

Kunina-Habenicht

Rupp

A. A.

Wilhelm

(2012). The impact of model misspecification on parameter estimation and item-fit assessment in log-linear diagnostic classification models. Journal of Educational Measurement, 49, 59-81.

12.

Leighton

Gierl

(2007). Cognitive diagnostic assessment for education: Theory and applications. Cambridge, UK: Cambridge University Press.

13.

Liu

Tian

Xin

(2016). An application of M₂ statistic to evaluate the fit of cognitive diagnostic models. Journal of Educational and Behavioral Statistics, 41, 3-26.

14.

Liu

Xin

(2017). dcminfo: Information matrix for diagnostic classification models (R package version 0.1.6). Retrieved from https://CRAN.R-project.org/package=dcminfo

15.

Liu

Xin

Andersson

Tian

(2018). Information matrix estimation procedures for cognitive diagnostic models. British Journal of Mathematical and Statistical Psychology. doi:10.1111/bmsp.12134

16.

Iaconangelo

de la Torre

(2016). Model similarity, model selection, and attribute classification. Applied Psychological Measurement, 40, 200-217.

17.

Maris

(1999). Estimating multiple classification latent class models. Psychometrika, 64, 187-212.

18.

Philipp

Strobl

de la Torre

Zeileis

(2018). On the estimation of standard errors in cognitive diagnosis models. Journal of Educational and Behavioral Statistics, 43, 88-115.

19.

R Core Team. (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/

20.

Robitzsch

Kiefer

George

A. C.

Uenlue

(2017). CDM: Cognitive diagnosis modeling (R package version 5.9-27). Retrieved from http://CRAN.R-project.org/package=CDM

21.

Rojas

de la Torre

Olea

(2012, April). Choosing between general and specific cognitive diagnosis models when the sample size is small. Paper presented at the meeting of the National Council on Measurement in Education, Vancouver, British Columbia, Canada.

22.

Rupp

A. A.

Templin

Henson

R. A.

(2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guilford Press.

23.

Sorrel

M. A.

Abad

F. J.

Olea

de la Torre

Barrada

J. R.

(2017). Inferential item-fit evaluation in cognitive diagnosis modeling. Applied Psychological Measurement, 41, 614-631.

24.

Sorrel

M. A.

de la Torre

Abad

F. J.

Olea

(2017). Two-step likelihood ratio test for item-level model comparison in cognitive diagnosis models. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 13, 39-47.

25.

Sorrel

M. A.

Olea

Abad

F. J.

de la Torre

Aguado

Lievens

(2016). Validity and reliability of situational judgement test scores: A new approach based on cognitive diagnosis models. Organizational Research Methods, 19, 506-532.

26.

Tatsuoka

K. K.

(1990). Toward an integration of item-response theory and cognitive error diagnosis. In Frederiksen

Glaser

Lesgold

Safto

(Eds.), Monitoring skills and knowledge acquisition (pp. 453-488). Hillsdale, NJ: Lawrence Erlbaum.

27.

Templin

J. L.

Henson

R. A.

(2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287-305.

28.

von Davier

. (2005). A general diagnostic model applied to language testing data (ETS Research Report RR-05-16). Princeton, NJ: Educational Testing Service.

29.

von Davier

. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287-307.

30.

Xin

Liu

Tian

(2016, April). Information matrix estimation procedures for cognitive diagnostic model. Paper presented at the meeting of the National Council on Measurement in Education, Washington, DC.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.09 MB

Improved Wald Statistics for Item-Level Model Comparison in Diagnostic Classification Models

Abstract

Keywords

Get full access to this article

References

Supplementary Material