Sage Journals: Discover world-class research

Abstract

Attribute hierarchy is a common assumption in the educational context, where the mastery of one attribute is assumed to be a prerequisite to the mastery of another one. The attribute hierarchy can be incorporated through a restricted Q matrix that implies the specified structure. The latent class–based cognitive diagnostic models (CDMs) usually do not assume a hierarchical structure among attributes, which means all profiles of attributes are possible in a population of interest. This study investigates different estimation methods to the classification accuracy for a family of CDMs when they are combined with a restricted Q-matrix design. A simulation study is used to explain the misclassification caused by an unrestricted estimation procedure. The advantages of the restricted estimation procedure utilizing attribute hierarchies for increased classification accuracy are also further illustrated through a real data analysis on a syllogistic reasoning diagnostic assessment. This research can provide guidelines for educational and psychological researchers and practitioners when they use CDMs to analyze the data with a restricted Q-matrix design and make them be aware of the potentially contaminated classification results if ignoring attribute hierarchies.

Keywords

attribute hierarchies classification accuracy cognitive diagnostic models restricted Q matrix

Get full access to this article

View all access options for this article.

References

Akaike

(1974). A new look at the statistical identification model. IEEE Transactions on Automated Control, 19, 716-723.

Chiu

C. Y.

Douglas

J. A.

(2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74, 633-665.

de la Torre

(2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115-130.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76, 179-199.

de la Torre

(2012, April). Cognitive diagnosis modeling: A general framework approach (Session5: Estimation of CDMs). Training session provided at the annual meeting of the National Council of Measurement Research, Vancouver, Canada.

de la Torre

Hong

Deng

(2010). Factors affecting the item parameter estimation and classification accuracy of the DINA model. Journal of Educational Measurement, 47, 227-249.

George

A. C.

Robitzsch

Kiefer

Groß

Ünlü

(2016). The R package CDM for cognitive diagnosis models. Journal of Statistical Software, 74(2), 1-24.

Gierl

M. J.

(2007). Making diagnostic inferences about cognitive attributes using the rule space model and attribute hierarchy method. Journal of Educational Measurement, 44, 325-340.

Gierl

M. J.

(2008). Using the attribute hierarchy method to identify and interpret cognitive skills that produce group differences. Journal of Educational Measurement, 4, 565-589.

10.

Groß

George

A. C.

(2014). On permissible attribute classes in noncompensatory Cognitive Diagnosis Models. Methodology, 3, 100-107.

11.

Haertel

E. H.

(1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333-352.

12.

Hartz

S. M.

(2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality (Unpublished doctoral dissertation). University of Illinois at Urbana–Champaign.

13.

Henson

Templin

Willse

(2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191-210.

14.

Huebner

Wang

(2011). A note on comparing examinee classification methods for cognitive diagnosis models. Educational and Psychological Measurement, 71, 407-419.

15.

Johnson-Laird

P. N.

(1983). Mental models: Towards a cognitive science of language, inference, and consciousness. Cambridge, MA: Harvard University Press.

16.

Johnson-Laird

P. N.

Bara

B. G.

(1984). Syllogistic inference. Cognition, 16, 1-61.

17.

Johnson-Laird

P. N.

Byrne

R. M. J.

(1991). Deduction. Hillsdale, NJ: Lawrence Erlbaum.

18.

Junker

B. W.

Sijtsma

(2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272.

19.

Leighton

J. P.

Gierl

M. J.

(2007). Cognitive diagnostic assessment for education: Theory and practice. Cambridge, UK: Cambridge University Press.

20.

Leighton

J. P.

Gierl

M. J.

Hunka

(2004). The attribute hierarchy model: An approach for integrating cognitive theory with assessment practice. Journal of Educational Measurement, 41, 205-236.

21.

de la Torre

(2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69, 253-275.

22.

Rupp

A. A.

Templin

J. L.

Henson

R. A.

(2010). Diagnostic assessment: Theory, methods, and applications. New York, NY: Guilford Press.

23.

Schwarz

(1978). Estimating the dimension of a model. Annals of Statistics, 6, 461-464.

24.

Sternberg

R. J.

Ben-Zeev

(Eds.) (1996): The nature of mathematical thinking. Hillsdale, NJ: Erlbaum.

25.

Choi

Lee

Choi

McAninch

(2013). Hierarchical cognitive diagnostic analysis for TIMSS2003 mathematics (CASMA Research Report No. 35). Iowa City: Center for Advanced Studies in Measurement and Assessment, University of Iowa.

26.

Tatsuoka

K. K.

(1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354.

27.

Tatsuoka

K. K.

(1985). A probabilistic model for diagnosing misconceptions by the pattern classification approach. Journal of Educational and Behavioral Statistics, 10, 55-73.

28.

Tatsuoka

K. K.

(1995). Architecture of knowledge structure and cognitive diagnosis: A statistical pattern recognition and classification approach. In Nichols

P. D.

Chipman

S. F.

Brennan

R. L.

(Eds.), Cognitively diagnostic assessment (pp. 327-361). Hillsdale, NJ: Lawrence Erlbaum.

29.

Tatsuoka

K. K.

(2009). Cognitive assessment: An introduction of the rule space method. New York, NY: Routledge.

30.

Templin

Bradshaw

(2014). Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies. Psychometrika, 79, 317-339.

31.

Templin

Henson

(2006). Measurement of psychological disorders using cognitive diagnostic models. Psychological Methods, 11, 287-305.

32.

von Davier

. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287-307.

33.

Wang

Gierl

M. J.

(2011). Using the attribute hierarchy method to make diagnostic inferences about examinees’ cognitive skills in critical reading. Journal of Educational Measurement, 48, 165-118.

34.

Wang

(2017). Two-stage maximum likelihood estimation in the misspecified restricted latent class model. British Journal of Mathematical and Statistical Psychology. Advance online publication. doi:10.1111/bmsp.12119

35.

Wang

Douglas

(2015). Consistency of nonparametric classification in cognitive diagnosis. Psychometrika, 80, 85-100.

36.

(2017). Identifiability of restricted latent class models with binary responses. The Annals of Statistics, 45, 675-707.

37.

Zhang

(2016). Identifiability of diagnostic classification models. Psychometrika, 81, 625-649.

38.

Zhang

S. S.

(2014). Statistical inference and experimental design for Q-matrix based Cognitive Diagnosis Models (Doctoral dissertation). Columbia University, New York, NY.

39.

Zhang

S. S.

DeCarlo

L. T.

Ying

(2013). Non-identifiability, equivalence classes, and attribute-specific classification in Q-matrix based Cognitive Diagnosis Models (arXiv:1303.0426). Retrieved from https://arxiv.org/pdf/1303.0426.pdf

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.15 MB

Cognitive Diagnostic Models With Attribute Hierarchies: Model Estimation With a Restricted Q-Matrix Design

Abstract

Keywords

Get full access to this article

References

Supplementary Material