Using Regularized Methods to Validate Q-Matrix in Cognitive Diagnostic Assessment

Abstract

One of the central components of cognitive diagnostic assessment is the Q-matrix, which is an essential loading indicator matrix and is typically constructed by subject matter experts. Nonetheless, to a large extent, the construction of Q-matrix remains a subjective process and might lead to misspecifications. Many researchers have recognized the importance of estimating or validating the Q-matrix, but most of them focus on the conditions of relatively large sample sizes. This article aims to explore Q-matrix validation possibilities under small sample conditions and uses regularized methods to validate the Q-matrix based on the compensatory reparametrized unified model and generalized deterministic inputs, noisy “and” gate models. Simulation studies were conducted to evaluate the viability of the modified least absolute shrinkage and selection operator (Lasso) and modified smoothly clipped absolute deviation (SCAD) methods, comparing them with existing methods. Results show that the modified Lasso and the modified SCAD methods outperform the stepwise, Hull, and MLR-B methods in general, especially under the conditions of small sample sizes. While good recovery in all small sample size conditions is not guaranteed, the modified methods demonstrate advantages across various item quality conditions. Also, a real data set is analyzed to illustrate the application of the modified methods.

Keywords

cognitive diagnosis Q-matrix attribute Lasso SCAD

Get full access to this article

View all access options for this article.

References

Akaike

(1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723.

Belzak

W. C. M.

Bauer

D. J.

(2020). Improving the assessment of measurement invariance: Using regularization to select anchor items and identify differential item functioning. Psychological Methods, 25(6), 673–690.

Breheny

Huang

(2011). Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. Annals of Applied Statistics, 5(1), 232–253.

Chen

(2017). A residual-based approach to validate Q-matrix specifications. Applied Psychological Measurement, 41(4), 277–293.

Chen

Culpepper

S. A.

Chen

Douglas

(2018). Bayesian estimation of the DINA Q matrix. Psychometrika, 83(1), 89–108.

Chen

Culpepper

Liang

(2020). A sparse latent class model for cognitive diagnosis. Psychometrika, 85, 121–153.

Chen

Liu

Ying

(2015). Statistical analysis of Q-matrix based on diagnostic classification models. Journal of the American Statistical Association, 110(510), 850–866.

Chiu

C. Y.

(2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37(8), 598–618.

Culpepper

S. A.

(2019). Estimating the cognitive diagnosis Q matrix with expert knowledge: Application to the fraction-subtraction dataset. Psychometrika, 84(2), 333–357.

10.

de la Torre

(2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343–362.

11.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199.

12.

de la Torre

Chiu

C.-Y.

(2016). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253–273.

13.

de La Torre

Douglas

J. A.

(2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333–353.

14.

DeCarlo

L. T.

(2012). Recognizing uncertainty in the Q-matrix via a Bayesian extension of the DINA model. Applied Psychological Measurement, 36(6), 447–468.

15.

Efron

Tibshirani

R. J.

(1994). An introduction to the bootstrap. CRC Press.

16.

Fan

(2001). Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96(456), 1348–1360.

17.

Friedman

Hastie

Tibshirani

(2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1), 1–22.

18.

Liu

Ying

(2018). Hypothesis testing of the Q-matrix. Psychometrika, 83(3), 515–537.

19.

(2019). The sufficient and necessary condition for the identifiability and estimability of the DINA model. Psychometrika, 84(2), 468–483.

20.

(2020). Partial identifiability of restricted latent class models. Annals of Statistics, 48(4), 2082–2107.

21.

(2021). Sufficient and necessary conditions for the identifiability of the Q-matrix. Statistica Sinica, 31(1), 449–472.

22.

Haertel

E. H.

(1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26(4), 301–321.

23.

Hartz

S. M.

(2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality [Doctoral dissertation, University of Illinois at Urbana-Champaign].

24.

Hastie

Tibshirani

Friedman

(2009). The elements of statistical learning: Data mining, inference, and prediction. Springer.

25.

Hastie

Tibshirani

R. J.

(2017). Extended comparisons of best subset selection, forward stepwise selection, and the Lasso. arXiv preprint arXiv:1707.08692 .

26.

Huang

P. H.

(2018). A penalized likelihood method for multi-group structural equation modelling. British Journal of Mathematical and Statistical Psychology, 71(3), 499–522.

27.

Leighton

Gierl

(2007). Cognitive diagnostic assessment for education. Cambridge University Press.

28.

Wan

Tan

Yang

Xie

(2020). A fast DC-based dictionary learning algorithm with the SCAD penalty. Neurocomputing, 429(12), 89–100.

29.

Liu

Ying

(2012). Data-driven learning of Q-matrix. Applied Psychological Measurement, 36(7), 548–564.

30.

Liu

Ying

(2013). Theory of self-learning Q-matrix. Bernoulli, 19(5A), 1790–1817.

31.

de la Torre

(2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69(3), 253–275.

32.

de la Torre

(2020a). An empirical Q-matrix validation method for the sequential generalized DINA model. British Journal of Mathematical and Statistical Psychology, 73(1), 142–163.

33.

de la Torre

(2020b). GDINA: An R package for cognitive diagnosis modeling. Journal of Statistical Software, 93(14), 1–26.

34.

McFadden

(1974). The measurement of urban travel demand. Journal of Public Economics, 3(4), 303–328.

35.

Nájera

Sorrel

M. A.

de la Torre

Abad

F. J.

(2020). Improving robustness in Q-matrix validation using an iterative and dynamic procedure. Applied Psychological Measurement, 44(6), 431–446.

36.

Nájera

Sorrel

M. A.

de la Torre

Abad

F. J.

(2021). Balancing fit and parsimony to improve Q-matrix validation. British Journal of Mathematical and Statistical Psychology, 74(Suppl 1), 110–130.

37.

Park

J. Y.

Lee

Y. S.

Johnson

M. S.

(2017). An efficient standard error estimator of the DINA model parameters when analysing clustered data. International Journal of Quantitative Research in Education, 4(1-2), 159–190.

38.

Pitt

M. A.

Myung

I. J.

(2002). When a good fit can be bad. Trends in Cognitive Sciences, 6(10), 421–425.

39.

Qin

C. Y.

Dong

S. H.

X. F.

(2024). Exploration of polytomous-attribute Q-matrix validation in cognitive diagnostic assessment. Knowledge-Based Systems, 292, 111577. https://doi.org/10.1016/j.knosys.2024.111577.

40.

Rupp

A. A.

Templin

(2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78–96.

41.

Rupp

A. A.

Templin

Henson

R. A.

(2010). Diagnostic measurement: Theory, methods, and applications. Guilford.

42.

Schwarz

(1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464.

43.

Tatsuoka

K. K.

(1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345–354.

44.

Templin

J. L.

Henson

R. A.

(2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287–305.

45.

Tibshirani

(1996). Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267–288.

46.

Chiu

Wang

Cai

Ouyang

(2023). A multiple logistic regression-based (MLR-B) Q-matrix validation method for cognitive diagnosis models: A confirmatory approach. Behavior Research Methods, 55, 2080–2292.

47.

von Davier

(2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61(2), 287–307.

48.

von Davier

Lee

Y. S.

(2019). Handbook of diagnostic classification models. Springer International.

49.

Wald

(1943). Tests of statistical hypotheses concerning several parameters when the number of observations is large. Transactions of the American Mathematical Society, 54(3), 426–482.

50.

Wang

Zhu

(2023). Using lasso and adaptive lasso to identify DIF in multidimensional 2PL models. Multivariate Behavioral Research, 58(2), 387–407.

51.

Wang

Song

Ding

Meng

Cao

Jie

(2018). An EM-based method for Q-matrix validation. Applied Psychological Measurement, 42(6), 446–459.

52.

Xie

Huang

(2009). SCAD-penalized regression in high-dimensional partially linear models. The Annals of Statistics, 37(2), 673–696.

53.

(2019). Identifiability and cognitive diagnosis models. In von Davier

Lee

Y. S.

(Eds.), Handbook of diagnostic classification models. (Methodology of Educational Measurement and Assessment) (pp. 333–357). Springer. https://doi.org/10.1007/978-3-030-05584-4_16

54.

Shang

(2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association, 113(523), 1284–1295.

55.

Cheng

(2020). Data-driven Q-matrix validation using a residual-based statistic in cognitive diagnostic assessment. British Journal of Mathematical and Statistical Psychology, 73(S1), 145–179.

56.

Zou

Hastie

(2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2), 301–320.