Attribute hierarchies are commonly assumed to exist in many fields of psychological and educational assessment. Several theory-driven and data-driven approaches have been used to validate or explore attribute hierarchies, such as validating attribute hierarchies in the cognitive diagnostic model (CDM) through statistical hypothesis testing or even learning attribute hierarchies directly from data. A class of structural parameter standard error estimation methods for CDMs is useful for exploring attribute hierarchies, with the limitation that the information matrices of some model parameters may be unstable or singular, leading to biased hypothesis testing. An iterative method of attribute hierarchy testing was proposed to modify the original z-statistic method. The simulation study systematically compares the performance of the z-statistic and the iterative z-statistic in exploring the attribute hierarchy. The results show that the iterative z-statistic provides a better Type I error control rate and statistical power, and it partially solves the problem that the z-statistic is too conservative. In addition, the iterative z-statistic method also achieves satisfactory results on real data.
BuckG.TatsuokaK. (1998). Application of the rule-space procedure to language testing: Examining attributes of a free response listening test. Language Testing, 15(2), 119–157. https://doi.org/10.1177/026553229801500201
2.
ChenY.LiuJ.XuG.YingZ. (2015). Statistical analysis of Q-matrix based diagnostic classification models. Journal of the American Statistical Association, 110(510), 850–866. https://doi.org/10.1080/01621459.2014.934827
3.
ChenY.LiX.LiuJ.YingZ. (2017). Regularized latent class analysis with application in cognitive diagnosis. Psychometrika, 82(3), 660–692. https://doi.org/10.1007/s11336-016-9545-6
CuiY.LeightonJ. P. (2009), The Hierarchy Consistency Index: Evaluating Person Fit for Cognitive Diagnostic Assessment. Journal of Educational Measurement, 46, 429–449. https://doi.org/10.1111/j.1745-3984.2009.00091.x
6.
DeCarloT. (2011). On the analysis of fraction subtraction data: The DINA model, classification, latent class sizes, and the Q-Matrix. Applied Psychological Measurement, 35(1), 8–26.
7.
de la TorreJ. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34(1), 115–130. http://www.jstor.org/stable/40263519
de la TorreJ.DouglasJ. A. (2004). Higher order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353. https://doi.org/10.1007/BF02295640
10.
EmbretsonS. E. (2004). Focus article: The second century of ability testing: Some predictions and speculations. Measurement: Interdisciplinary Research & Perspective, 2(1), 1–32. https://doi.org/10.1207/s15366359mea0201_1
11.
GierlM. J.LeightonJ. P.HunkaS. M. (2007). Using the attribute hierarchy method to make diagnostic inferences about examinees’ cognitive skills. In LeightonJ. P.GierlM. J. (Eds.), Cognitive diagnostic assessment for education: Theory and applications; Cognitive diagnostic assessment for education: Theory and applications (pp. 242–274, 371 pages). Cambridge University Press.
12.
GuY.XuG. (2019). Learning attribute patterns in high-dimensional structured latent attribute models. Journal of Machine Learning Research, 20, 1–58. https://doi.org/10.48550/arXiv.1904.04378
13.
HaertelE. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26(4), 301–321. https://doi.org/10.1111/j.1745-3984.1989.tb00336.x
14.
HensonR. A.TemplinJ. L.WillseJ. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191–210. https://doi.org/10.1007/s11336-008-9089-5
15.
HuB.TemplinJ. (2020). Using diagnostic classification models to validate attribute hierarchies and evaluate model fit in Bayesian Networks. Multivariate Behavioral Research, 55(2), 300–311. https://doi.org/10.1080/00273171.2019.1632165
16.
LeightonJ. P.GierlM. J.HunkaS. M. (2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41(3), 205–237. https://doi.org/10.1111/j.1745-3984.2004.tb01163.x
17.
LiuR. (2018). Misspecification of attribute structure in diagnostic measurement. Educational and Psychological Measurement, 78(4), 605–634. https://doi.org/10.1177/0013164417702458
18.
LiuR.Huggins-ManleyA. C.BradshawL. (2017). The impact of Q-matrix designs on diagnostic classification accuracy in the presence of attribute hierarchies. Educational and Psychological Measurement, 77(2), 220–240. https://doi.org/10.1177/0013164416645636
19.
LiuY.TianW.XinT. (2016). An application of M2 statistic to evaluate the fit of cognitive diagnostic models. Journal of Educational Behavioral Statistics, 41(1), 3–26. https://doi.org/10.3102/1076998615621293
20.
LiuY.XinT.AnderssonB.TianW. (2019). Information matrix estimation procedures for cognitive diagnostic models. British Journal of Mathematical and Statistical Psychology, 72(1), 18–37. https://doi.org/10.1111/bmsp.12134
21.
LiuY.XinT.JiangY. (2022). Structural parameter standard error estimation method in diagnostic classification models: Estimation and application. Multivariate Behavioral Research, 57(5), 784–803. https://doi.org/10.1080/00273171.2021.1919048
22.
MaC.OuyangJ.XuG. (2022). Learning latent and hierarchical structures in cognitive diagnosis models. Psychometrika. Advance online publication. https://doi.org/10.1007/s11336-022-09867-5
23.
MaC.XuG. (2021). Hypothesis testing for hierarchical structures in cognitive diagnosis models. Journal of Data Science, 20(3), 279–302. https://doi.org/10.6339/21-JDS1024
24.
MaW.de la TorreJ. (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical Statistical Psychology, 69(3), 253–275. https://doi.org/10.1111/bmsp.12070
25.
MaW.de la TorreJ. (2020). GDINA: An R package for cognitive diagnosis modeling. Journal of Statistical Software, 93(14), 1–26. https://doi.org/10.18637/jss.v093.i14
26.
PhilippM.StroblC.delaTorreJ.ZeileisA. (2018). On the estimation of standard errors in cognitive diagnosis models. Journal of Educational and Behavioral Statistics, 43(1), 88–115. https://doi.org/10.3102/1076998617719728
27.
RuppA.TemplinJ.HensonR. A. (2010). Diagnostic measurement: Theory, methods, and applications. Guilford Press.
28.
TatsuokaK. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345–354. https://doi.org/10.1111/j.1745-3984.1983.tb00212.x
29.
TatsuokaK. K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In FrederiksenN.GlaserR.LesgoldA.ShaftoM. G. (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 453–488, 510 Pages). Lawrence Erlbaum Associates, Inc.
30.
TemplinJ.BradshawL. (2014). Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies. Psychometrika, 79(2), 317–339. https://doi.org/10.1007/s11336-013-9362-0
31.
TemplinJ.HoffmanL. (2013). Obtaining diagnostic classification model estimates using Mplus. Educational Measurement-Issues and Practice, 32(2), 37–50. https://doi.org/10.1111/emip.12010
32.
TemplinJ. L.HensonR. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287–305. https://doi.org/10.1037/1082-989x.11.3.287
33.
von DavierM. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical & Statistical Psychology, 61(2), 287–307. https://doi.org/10.1348/000711007x193957
34.
von DavierM. (2014). The Log-Linear Cognitive Diagnostic Model (LCDM) as a special case of the General Diagnostic Model (GDM). ETS Research Report Series, 2014, 1–13. https://doi.org/10.1002/ets2.12043
35.
WangC.LuJ. (2021). Learning attribute hierarchies from data: Two exploratory approaches. Journal of Educational and Behavioral Statistics, 46(1), 58–84. https://doi.org/10.3102/1076998620931094
36.
WangC. J.GierlM. J. (2011). Using the attribute hierarchy method to make diagnostic inferences about examinees’ cognitive skills in critical reading. Journal of Educational Measurement, 48(2), 165–187. https://doi.org/10.1111/j.1745-3984.2011.00142.x
37.
XuG. J.ShangZ. R. (2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association, 113(523), 1284–1295. https://doi.org/10.1080/01621459.2017.1340889
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
