Bayesian Diagnostic Classification Models for a Partially Known Q-Matrix

Abstract

This study proposes a Bayesian method for diagnostic classification models (DCMs) for a partially known Q-matrix setting between exploratory and confirmatory DCMs. This Q-matrix setting is practical and useful because test experts have pre-knowledge of the Q-matrix but cannot readily specify it completely. The proposed method employs priors for the Bayesian variable selection to simultaneously estimate the effects of active and nonactive attributes, and the simulations lead to appropriate attribute recovery rates. Furthermore, the proposed method recovers the attribute mastery of individuals at the same as for a fully known Q-matrix. In addition, the proposed methods can be used to estimate the unknown Q-matrix part. A real data example indicates that the proposed Bayesian estimation method for the partially known Q-matrix fits better than a fully specified Q-matrix. Finally, extensions and future research directions are discussed.

Keywords

diagnostic classification models parameter estimation variable selection methods Bayesian methods

Get full access to this article

View all access options for this article.

References

Baghaei

Hohensinn

(2017). A method of Q-matrix validation for the linear logistic test model. Frontiers in Psychology, 8, 897. https://doi.org/10.3389/fpsyg.2017.00897

Balamuta

J. J.

Culpepper

S. A.

(2022). Exploratory restricted latent class models with monotonicity requirements under Pólya–Gamma data augmentation. Psychometrika, 87(3), 903–945. https://doi.org/10.1007/s11336-021-09815-9

Balamuta

J. J.

Culpepper

S. A.

(2023). slcm: Sparse latent class model for cognitive diagnosis [0.1.0]. https://doi.org/10.32614/CRAN.package.slcm

Carpenter

Gelman

Hoffman

M. D.

Lee

Goodrich

Betancourt

Brubaker

M. A.

Guo

Riddell

(2017). Stan: A probabilistic programming language. Journal of Statistical Software, 76, 1–32. https://doi.org/10.18637/jss.v076.i01

Carvalho

C. M.

Polson

N. G.

Scott

J. G.

(2010). The horseshoe estimator for sparse signals. Biometrika, 97(2), 465–480. https://doi.org/https://doi.org/10.1093/biomet/asq017

Chen

(2017). A residual-based approach to validate Q-matrix specifications. Applied Psychological Measurement, 41(4), 277–293. https://doi.org/10.1177/0146621616686021

Chen

de la Torre

(2014). A procedure for diagnostically modeling extant large-scale assessment data: The case of the programme for international student assessment in reading. Psychology, 5(18), 1967–1978. https://doi.org/10.4236/psych.2014.518200

Chen

Culpepper

S. A.

Chen

Douglas

(2018). Bayesian estimation of the DINA Q-matrix. Psychometrika, 83, 89–108. https://doi.org/10.1007/s11336-017-9579-4

Chen

Culpepper

Liang

(2020). A sparse latent class model for cognitive diagnosis. Psychometrika, 85, 121–153. https://doi.org/10.1007/s11336-019-09693-2

10.

Chen

Liu

Ying

(2017). Regularized latent class analysis with application in cognitive diagnosis. Psychometrika, 82, 660–692. https://doi.org/10.1007/s11336-016-9545-6

11.

Chen

Liu

Ying

(2015). Statistical analysis of Q-matrix based diagnostic classification models. Journal of the American Statistical Association, 110(510), 850–866. https://doi.org/10.1080/01621459.2014.934827

12.

Chiu

C.-Y.

(2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37(8), 598–618. https://doi.org/10.1177/0146621613488436

13.

Chiu

C.-Y.

Douglas

(2013). A nonparametric approach to cognitive diagnosis by proximity to ideal response patterns. Journal of Classification, 30, 225–250. https://doi.org/10.1007/s00357-013-9132-9

14.

Chung

(2019). A Gibbs sampling algorithm that estimates the Q-matrix for the DINA model. Journal of Mathematical Psychology, 93, 102275. https://doi.org/10.1016/j.jmp.2019.07.002

15.

Culpepper

S. A.

(2019a). Estimating the cognitive diagnosis Q-matrix with expert knowledge: Application to the fraction-subtraction dataset. Psychometrika, 84(2), 333–357. https://doi.org/10.1007/s11336-018-9643-8

16.

Culpepper

S. A.

(2019b). An exploratory diagnostic model for ordinal responses with binary attributes: Identifiability and estimation. Psychometrika, 84(4), 921–940. https://doi.org/10.1007/s11336-019-09683-4

17.

de la Torre

(2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343–362. https://doi.org/10.1111/j.1745-3984.2008.00069.x

18.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76, 179–199. https://doi.org/10.1007/s11336-011-9207-7

19.

de la Torre

Chiu

C.-Y.

(2016). A general method of empirical Q-matrix validation. Psychometrika, 81, 253–273. https://doi.org/10.1007/s11336-015-9467-8

20.

de la Torre

Douglas

J. A.

(2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333–353. https://doi.org/https://doi.org/10.1007/BF02295640

21.

de la Torre

Qiu

X.-L.

Santos

K. C.

(2022). An empirical Q-Matrix validation method for the polytomous G-DINA model. Psychometrika, 87(2), 693–724. https://doi.org/10.1007/s11336-021-09821-x

22.

DeCarlo

L. T.

(2012). Recognizing uncertainty in the Q-matrix via a Bayesian extension of the DINA model. Applied Psychological Measurement, 36(6), 447–468. https://doi.org/10.1177/0146621612449069

23.

DeCarlo

L. T.

(2019). Insights from reparameterized DINA and beyond. In von Davier

Lee

Y.-S.

(Eds.), Handbook of diagnostic classification models: Models and model extensions, applications, software packages (pp. 223–243). Springer International Publishing. https://doi.org/10.1007/978-3-030-05584-4_11

24.

Enders

C. K.

(2022). Applied missing data analysis (2nd ed.). The Guilford Press.

25.

Liu

Ying

(2018). Hypothesis testing of the Q-matrix. Psychometrika, 83, 515–537. https://doi.org/10.1007/s11336-018-9629-6

26.

Hartz

S. M.

(2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. University of Illinois at Urbana-Champaign. https://www.proquest.com/dissertations-theses/bayesian-framework-unified-model-assessing/docview/305590285/se-2

27.

Henson

R. A.

Templin

J. L.

Willse

J. T.

(2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191–210. https://doi.org/10.1007/s11336-008-9089-5

28.

Ishwaran

Rao

J. S.

(2005). Spike and slab variable selection: Frequentist and Bayesian strategies. The Annals of Statistics, 33(2), 730–773. https://doi.org/10.1214/009053604000001147

29.

Junker

B. W.

Sijtsma

(2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272. https://doi.org/10.1177/01466210122032064

30.

Kuo

Mallick

(1998). Variable selection for regression models. Sankhyā: The Indian Journal of Statistics, Series B, 60, 65–81. https://www.jstor.org/stable/25053023

31.

Liu

C.-W.

Andersson

Skrondal

(2020). A constrained Metropolis–Hastings Robbins–Monro algorithm for Q matrix estimation in DINA models. Psychometrika, 85(2), 322–357. https://doi.org/10.1007/s11336-020-09707-4

32.

Liu

Ying

(2012). Data-driven learning of Q-matrix. Applied Psychological Measurement, 36(7), 548–564. https://doi.org/10.1177/0146621612456591

33.

Liu

Ying

(2013). Theory of the self-learning Q-matrix. Bernoulli: Official Journal of the Bernoulli Society for Mathematical Statistics and Probability, 19(5A), 1790. https://doi.org/10.3150/12-BEJ430

34.

Lunn

D. J.

Thomas

Best

Spiegelhalter

(2000). WinBUGS – A Bayesian modelling framework: Concepts, structure, and extensibility. Statistics and Computing, 10, 325–337. https://doi.org/10.1023/A:1008929526011

35.

de la Torre

(2020a). GDINA: An R package for cognitive diagnosis modeling. Journal of Statistical Software, 93(14), 1–26. https://doi.org/10.18637/jss.v093.i14

36.

Torre

(2020b). An empirical Q-matrix validation method for the sequential generalized DINA model. British Journal of Mathematical and Statistical Psychology, 73(1), 142–163. https://doi.org/10.1111/bmsp.12156

37.

Macready

G. B.

Dayton

C. M.

(1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 2(2), 99–120. https://doi.org/10.3102/10769986002002099

38.

Madison

M. J.

Bradshaw

L. P.

(2018). Assessing growth in a diagnostic classification model framework. Psychometrika, 83, 963–990. https://doi.org/https://doi.org/10.1007/s11336-018-9638-5

39.

Madison

M. J.

Chung

Kim

Bradshaw

L. P.

(2024). Approaches to estimating longitudinal diagnostic classification models. Behaviormetrika, 51, 7–19. https://doi.org/10.1007/s41237-023-00202-5

40.

Makalic

Schmidt

D. F.

(2016). A simple sampler for the horseshoe estimator. IEEE Signal Processing Letters, 23(1), 179–182. https://doi.org/10.1109/LSP.2015.2503725

41.

Maris

(1999). Estimating multiple classification latent class models. Psychometrika, 64, 187–212. https://doi.org/10.1007/BF02294535

42.

Meng

X.-L.

(1994). Posterior predictive p-values. The Annals of Statistics, 22(3), 1142–1160. https://doi.org/10.1214/aos/1176325622

43.

Nájera

Sorrel

M. A.

Abad

F. J.

(2019). Reconsidering cutoff points in the general method of empirical Q-matrix validation. Educational and Psychological Measurement, 79(4), 727–753. https://doi.org/10.1177/0013164418822700

44.

Nájera

Sorrel

M. A.

de la Torre

Abad

F. J.

(2020). Improving robustness in Q-matrix validation using an iterative and dynamic procedure. Applied Psychological Measurement, 44(6), 431–446. https://doi.org/10.1177/0146621620909904

45.

Nájera

Sorrel

M. A.

de la Torre

Abad

F. J.

(2021). Balancing fit and parsimony to improve Q-matrix validation. British Journal of Mathematical and Statistical Psychology, 74, 110–130. https://doi.org/10.1111/bmsp.12228

46.

O’hara

R. B.

Sillanpää

M. J.

(2009). A review of Bayesian variable selection methods: What, how and which. Bayesian Analysis, 4(1), 85–117. https://doi.org/10.1214/09-BA403

47.

Oka

Okada

(2023). Scalable Bayesian approach for the DINA Q-matrix estimation combining stochastic optimization and variational inference. Psychometrika, 88(1), 302–331. https://doi.org/10.1007/s11336-022-09884-4

48.

Plummer

(2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. Proceedings of the 3rd international workshop on distributed statistical computing, Vienna, 20–22 March 2003, 1–10.

49.

Qin

Guo

(2024). Using machine learning to improve Q-matrix validation. Behavior Research Methods, 56, 1916–1935. https://doi.org/10.3758/s13428-023-02126-0

50.

Robitzsch

Kiefer

George

A. C.

Ünlü

(2022). CDM: Cognitive diagnosis modeling [R package version 8.2-6]. https://CRAN.R-project.org/package=CDM

51.

Rupp

A. A.

Templin

(2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78–96. https://doi.org/10.1177/0013164407301545

52.

Rupp

A. A.

Templin

Henson

R. A.

(2010). Diagnostic measurement: Theory, methods, and applications. Guilford Press.

53.

Sen

Cohen

A. S.

(2021). Sample size requirements for applying diagnostic classification models. Frontiers in Psychology, 11, 621251. https://doi.org/10.3389/fpsyg.2020.621251

54.

Shi

Robitzsch

Sorrel

M. A.

Man

(2021). Cognitively diagnostic analysis using the G-DINA model in R. Psych, 3(4), 812–835. https://doi.org/https://doi.org/10.3390/psych3040052

55.

Sorrel

M. A.

Abad

F. J.

Olea

de la Torre

Barrada

J. R.

(2017). Inferential item-fit evaluation in cognitive diagnosis modeling. Applied Psychological Measurement, 41(8), 614–631. https://doi.org/10.1177/0146621617707510

56.

Spiegelhalter

D. J.

Thomas

Best

Lunn

(2003). WinBUGS version 1.4 user manual. MRC Biostatistics Unit. http://www.mrc-bsu.cam.ac.uk/bugs; http://www.mrc-bsu.cam.ac.uk/bugs/winbugs/manual14.pdf

57.

Sun

(2016). Q-matrix learning and DINA model parameter estimation. 2016 International conference on behavioral, economic and socio-cultural computing (BESC) [Conference session]. pp. 1–6. https://doi.org/10.1109/BESC.2016.7804471

58.

Tadesse

M. G.

Vannucci

(2021). Handbook of Bayesian variable selection. CRC Press.

59.

Tatsuoka

K. K.

(1985). A probabilistic model for diagnosing misconceptions by the pattern classification approach. Journal of Educational Statistics, 10(1), 55–73. https://doi.org/10.3102/10769986010001055

60.

Templin

J. L.

Henson

R. A.

(2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287. https://doi.org/10.1037/1082-989X.11.3.287

61.

Tjoe

de la Torre

(2013). Designing cognitively-based proportional reasoning problems as an application of modern psychological measurement models. Journal of Mathematics Education, 6(2), 17–26. https://api.semanticscholar.org/CorpusID:54598161

62.

Tjoe

de la Torre

(2014). The identification and validation process of proportional reasoning attributes: An application of a cognitive diagnosis modeling framework. Mathematics Education Research Journal, 26, 237–255. https://doi.org/10.1007/s13394-013-0090-7

63.

Chiu

Wang

Cai

Ouyang

(2023). A multiple logistic regression-based (MLR-B) Q-matrix validation method for cognitive diagnosis models: A confirmatory approach. Behavior Research Methods, 55(4), 2080–2092. https://doi.org/10.3758/s13428-022-01880-x

64.

van Erp

Oberski

D. L.

Mulder

(2019). Shrinkage priors for Bayesian penalized regression. Journal of Mathematical Psychology, 89, 31–50. https://doi.org/10.1016/j.jmp.2018.12.004

65.

Vehtari

Gelman

Gabry

(2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. https://doi.org/10.1007/s11222-016-9696-4

66.

von Davier

(2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61(2), 287–307. https://doi.org/https://doi.org/10.1348/000711007X193957

67.

von Davier

(2014a). The DINA model as a constrained general diagnostic model: Two variants of a model equivalency. British Journal of Mathematical and Statistical Psychology, 67(1), 49–71. https://doi.org/10.1111/bmsp.12003

68.

von Davier

(2014b). The log-linear cognitive diagnostic model (LCDM) as a special case of the general diagnostic model (GDM). ETS Research Report Series, 2014(2), 1–13. https://doi.org/10.1002/ets2.12043

69.

von Davier

Lee

Y.-S.

(2019). Handbook of diagnostic classification models. Springer International Publishing.

70.

Wang

Cai

(2020). Q-Matrix estimation methods for cognitive diagnosis models: Based on partial known Q-matrix. Multivariate Behavioral Research, 1–13. https://doi.org/10.1080/00273171.2020.1746901

71.

Wang

Song

Ding

Meng

Cao

Jie

(2018). An EM-based method for Q-matrix validation. Applied Psychological Measurement, 42(6), 446–459. https://doi.org/10.1177/0146621617752991

72.

Wang

Song

Ding

Wang

Gao

Xiong

(2020). A semi-supervised learning method for Q-matrix specification under the DINA and DINO model with independent structure. Frontiers in Psychology, 11, 524203. https://doi.org/10.3389/fpsyg.2020.02120

73.

Watanabe

(2010). Asymptotic equivalence of bayes cross validation and widely applicable information criterion in singular learning theory. Journal of Machine Learning Research, 11, 3571–3594. https://www.jmlr.org/papers/volume11/watanabe10a/watanabe10a.pdf

74.

Watanabe

(2018). Mathematical theory of Bayesian statistics. CRC Press.

75.

Xiong

Luo

(2022). Data-driven Q-matrix learning based on Boolean matrix factorization in cognitive diagnostic assessment. British Journal of Mathematical and Statistical Psychology, 75(3), 638–667. https://doi.org/10.1111/bmsp.12271

76.

Shang

(2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association, 113(523), 1284–1295. https://doi.org/10.1080/01621459.2017.1340889

77.

de la Torre

Zhang

Guo

Shi

(2020). Estimating CDMs using the slice-within-Gibbs sampler. Frontiers in Psychology, 11, 2260. https://doi.org/10.3389/fpsyg.2020.02260

78.

Yamaguchi

Templin

(2022). A Gibbs sampling algorithm with monotonicity constraints for diagnostic classification models. Journal of Classification, 39(1), 24–54. https://doi.org/10.1007/s00357-021-09392-7

79.

Yamaguchi

Zhang

(2023). Fully Gibbs sampling algorithms for Bayesian variable selection in latent regression models. Journal of Educational Measurement, 60(2), 202–234. https://doi.org/10.1111/jedm.12348

80.

Cheng

(2020). Data-driven Q-matrix validation using a residual-based statistic in cognitive diagnostic assessment. British Journal of Mathematical and Statistical Psychology, 73, 145–179. https://doi.org/10.1111/bmsp.12191

81.

Zhan

Jiao

Man

Wang

(2019). Using JAGS for Bayesian cognitive diagnosis modeling: A tutorial. Journal of Educational and Behavioral Statistics, 44(4), 473–503. https://doi.org/10.3102/1076998619826040

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB