Despite the increasing popularity, cognitive diagnosis models have been criticized for limited utility for small samples. In this study, the authors proposed to use Bayes modal (BM) estimation and monotonic constraints to stabilize item parameter estimation and facilitate person classification in small samples based on the generalized deterministic input noisy “and” gate (G-DINA) model. Both simulation study and real data analysis were used to assess the utility of the BM estimation and monotonic constraints. Results showed that in small samples, (a) the G-DINA model with BM estimation is more likely to converge successfully, (b) when prior distributions are specified reasonably, and monotonicity is not violated, the BM estimation with monotonicity tends to produce more stable item parameter estimates and more accurate person classification, and (c) the G-DINA model using the BM estimation with monotonicity is less likely to overfit the data and shows higher predictive power.
BakemanR. (2005). Recommended effect size statistics for repeated measures designs. Behavior Research Methods, 37(3), 379–384.
2.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. In LordF. M.NovickM. R. (Eds.), Statistical theories of mental test scores (pp. 397–479). Addison-Wesley.
3.
ChangY.-P.ChiuC.-Y.TsaiR.-C. (2018). Nonparametric CAT for CD in educational settings with small samples. Applied Psychological Measurement, 43(7), 543–561.
4.
ChenJ. (2017). A residual-based approach to validate Q-matrix specifications. Applied Psychological Measurement, 41, 277–293.
5.
ChenJ.de la TorreJ. (2013). A general cognitive diagnosis model for expert-defined polytomous attributes. Applied Psychological Measurement, 37, 419–437.
6.
ChenJ.de la TorreJ.ZhangZ. (2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50, 123–140.
ChenY.CulpepperS. A. (2020). A multivariate probit model for learning trajectories: A fine-grained evaluation of an educational intervention. Applied Psychological Measurement, 44(7–8), 5151–530. https://doi.org/10.1177/0146621620920928
ChenY.CulpepperS. A.WangS.DouglasJ. (2018). A hidden Markov model for learning trajectories in cognitive diagnosis with application to spatial rotation skills. Applied Psychological Measurement, 42(1), 5–23.
11.
ChenY.LiuJ.XuG.YingZ. (2015). Statistical analysis of Q-matrix based diagnostic classification models. Journal of the American Statistical Association, 110(510), 850–866.
12.
ChiuC.-Y.DouglasJ. A.LiX. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74, 633–665. https://doi.org/10.1007/s11336-009-9125-0
13.
ChiuC.-Y.SunY.BianY. (2018). Cognitive diagnosis for small educational programs: The general nonparametric classification method. Psychometrika, 83(2), 355–375.
14.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Academic Press.
15.
CulpepperS. A. (2015). Bayesian estimation of the DINA model with Gibbs sampling. Journal of Educational and Behavioral Statistics, 40(5), 454–476. https://doi.org/10.3102/1076998615595403
16.
CulpepperS. A. (2019). Estimating the cognitive diagnosis Q matrix with expert knowledge: Application to the fraction-subtraction dataset. Psychometrika, 84(2), 333–357.
17.
CulpepperS. A.HudsonA. (2018). An improved strategy for Bayesian estimation of the reduced reparameterized unified model. Applied Psychological Measurement, 42(2), 99–115.
18.
da SilvaM. A.de OliveiraE. S.von DavierA. A.BazánJ. L. (2018). Estimating the DINA model parameters using the No-U-Turn sampler. Biometrical Journal, 60(2), 352–368.
19.
De AyalaR. J. (2013). The theory and practice of item response theory. Guilford Press.
20.
DeCarloL. T. (2011). On the analysis of fraction subtraction data: The DINA model, classification, latent class sizes, and the Q-matrix. Applied Psychological Measurement, 35(1), 8–26. https://doi.org/10.1177/0146621610377081
de la TorreJ.LeeY. S. (2013). Evaluating the Wald test for item-level comparison of saturated and reduced models in cognitive diagnosis. Journal of Educational Measurement, 50, 355–373. https://doi.org/10.1111/jedm.12022
24.
de la TorreJ.SorrelM. A. (2017, July18–21). Attribute classification accuracy improvement: Monotonicity constraints on the G-DINA model [Conference session]. Annual Meeting of Psychometric Society, Zurich, Switzerland.
25.
de la TorreJ.van der ArkL. A.RossiG. (2018). Analysis of clinical data from cognitive diagnosis modeling framework. Measurement and Evaluation in Counseling and Development, 51(4), 281–296. https://doi.org/10.1080/07481756.2017.1327286
26.
DempsterA. P.LairdN. M.RubinD. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39, 1–38.
27.
GarreF. G.VermuntJ. K. (2006). Avoiding boundary estimates in latent class analysis by Bayesian posterior mode estimation. Behaviormetrika, 33, 43–59. https://doi.org/10.2333/bhmk.33.43
28.
GelmanA. (2002). Prior distribution. Encyclopedia of Environmetrics, 3(4), 1634–1637.
29.
GelmanA.HwangJ.VehtariA. (2013). Understanding predictive information criteria for Bayesian models. Statistics and Computing, 24(6), 997–1016.
30.
HellerJ.WickelmaierF. (2013). Minimum discrepancy estimation in probabilistic knowledge structures. Electronic Notes in Discrete Mathematics, 42, 49–56.
31.
HensonR. A. (2009). Diagnostic classification models: Thoughts and future directions. Measurement: Interdisciplinary Research and Perspectives, 7, 34–36.
32.
HongC.-Y.ChangY.-W.TsaiR.-C. (2016). Estimation of generalized DINA model with order restrictions. Journal of Classification, 33, 460–484.
33.
JiangZ.CarterR. (2019). Using Hamiltonian Monte Carlo to estimate the log-linear cognitive diagnosis model via Stan. Behavior Research Methods, 51, 651–662.
34.
JiangZ.MaW. (2018). Integrating differential evolution optimization to cognitive diagnostic model estimation. Frontiers in Psychology, 9, Article 2142.
JurichD. P.BradshawL. P. (2013). An illustration of diagnostic classification modeling in student learning outcomes assessment. International Journal of Testing, 14(1), 49–72. https://doi.org/10.1080/15305058.2013.835728
37.
KayaY.LeiteW. L. (2017). Assessing change in latent skills across time with longitudinal cognitive diagnosis modeling: An evaluation of model performance. Educational and Psychological Measurement, 77(3), 369–388.
38.
KoshyT. (2004). Discrete mathematics with applications. Elsevier.
39.
KraftD. (1988). A software package for sequential quadratic programming (Technical report DFVLR-FB 88–28). Institut fuer Dynamik der Flugsysteme.
LiuY.DouglasJ. A.HensonR. A. (2009). Testing person fit in cognitive diagnosis. Applied Psychological Measurement, 33, 579–598.
42.
MaW.de la TorreJ. (2019). An empirical Q-matrix validation method for the sequential G-DINA model. British Journal of Mathematical and Statistical Psychology, 73(1), 142–163. https://doi.org/10.1111/bmsp.12156
43.
MaW.de la TorreJ. (2020). GDINA: An R package for cognitive diagnosis modeling. Journal of Statistical Software, 93(14). https://doi.org/10.18637/jss.v093.i14
44.
MaW.GuoW. (2019). Cognitive diagnosis models for multiple strategies. The British Journal of Mathematical and Statistical Psychology, 72(2), 370–392.
45.
MaW.IaconangeloC.de la TorreJ. (2016). Model similarity, model selection, and attribute classification. Applied Psychological Measurement, 40, 200–217. 10.1177/0146621615621717
46.
MaW.MinchenN.de la TorreJ. (2020). Choosing between cdm and unidimensional IRT: The proportional reasoning test case. Measurement: Interdisciplinary Research and Perspectives, 18, 87–96.
47.
McLachlanG.KrishnanT. (2008). The EM algorithm and extensions (2nd ed.). John Wiley & Sons.
48.
MislevyR. J. (1986). Bayes modal estimation in item response models. Psychometrika, 51(2), 177–195.
49.
NájeraP.SorrelM. A.AbadF. J. (2019). Reconsidering cutoff points in the general method of empirical Q-matrix validation. Educational and Psychological Measurement, 79(4), 727–753. https://doi.org/10.1177/0013164418822700
50.
OlejnikS.AlginaJ. (2003). Generalized eta and omega squared statistics: Measures of effect size for some common research designs. Psychological Methods, 8(4), 434–447.
51.
PhilippM.StroblC.de la TorreJ.ZeileisA. (2018). On the estimation of standard errors in cognitive diagnosis models. Journal of Educational and Behavioral Statistics, 43, 88–115.
52.
R Core Team. (2017). R: A language and environment for statistical computing [Computer software manual]. https://www.R-project.org/
SessomsJ.HensonR. A. (2018). Applications of diagnostic classification models: A literature review and critical commentary. Measurement: Interdisciplinary Research and Perspectives, 16, 1–17.
55.
SijtsmaK.van der ArkL. A. (2017). A tutorial on how to do a Mokken scale analysis on your test and questionnaire data. British Journal of Mathematical and Statistical Psychology, 70(1), 137–158.
56.
SmythP. (2000). Model selection for probabilistic clustering using cross-validated likelihood. Statistics and Computing, 10(1), 63–72. https://doi.org/10.1023/a:1008940618127
57.
SorrelM. A.AbadF. J.OleaJ.de la TorreJ.BarradaJ. R. (2017). Inferential item-fit evaluation in cognitive diagnosis modeling. Applied Psychological Measurement, 41(8), 614–631. https://doi.org/10.1177/0146621617707510
58.
SorrelM. A.OleaJ.AbadF. J.de la TorreJ.AguadoD.LievensF. (2016). Validity and reliability of situational judgement test scores: A new approach based on cognitive diagnosis models. Organizational Research Methods, 19, 506–532. https://doi.org/10.1177/1094428116630065
59.
TatsuokaC. (2002). Data analytic methods for latent partially ordered classification models. Journal of the Royal Statistical Society: Series C (Applied Statistics), 51, 337–350.
60.
TatsuokaK. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345–354. https://doi.org/10.1111/j.1745-3984.1983.tb00212.x
61.
TemplinJ.BradshawL. (2014). Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies. Psychometrika, 79, 317–339.
62.
van ErpS.MulderJ.OberskiD. L. (2018). Prior sensitivity analysis in default Bayesian structural equation modeling. Psychological Methods, 23, 363–388.
63.
WangS.YangY.CulpepperS. A.DouglasJ. A. (2018). Tracking skill acquisition with cognitive diagnosis models: A higher-order, hidden markov model with covariates. Journal of Educational and Behavioral Statistics, 43(1), 57–87.
64.
WangS.ZhangS.DouglasJ.CulpepperS. (2018). Using response times to assess learning progress: A joint model for responses and response times, 16, 45–58.
65.
WuH.-M. (2019). Online individualised tutor for improving mathematics learning: A cognitive diagnostic model approach. Educational Psychology, 39(10), 1218–1232.
66.
ZhanP. (2020). A Markov estimation strategy for longitudinal learning diagnosis: Providing timely diagnostic feedback. Educational and Psychological Measurement, 80(6), 1145–1167. https://doi.org/10.1177/0013164420912318
67.
ZhanP.JiaoH.ManK.WangL. (2019). Using JAGS for bayesian cognitive diagnosis modeling: A tutorial. Journal of Educational and Behavioral Statistics, 44(4), 473–503.
68.
ZhangS.CulpepperS. A. (2017, April). Bayesian estimation of a general class of identified restricted latent class models[Conference session]. Annual Meeting of the National Council on Measurement in Education, San Antonio, TX, United States.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
