The demand for diagnostic feedback has triggered extensive research on cognitive diagnostic models (CDMs), such as the deterministic input, noisy output “and” gate (DINA) model. This study explored two Q-matrix specifications with the DINA model in a statewide large-scale mathematics assessment. The first Q-matrix was developed based on five predefined content reporting categories, and the second was based on the post hoc coding of 15 attributes by test-development experts. Total raw scores correlated strongly with the number of skills mastered, using both Q-matrices. Correlations between the DINA-model item statistics and those from the item response theory analyses were moderate to strong, but were always lower for the 15-skill model. Results highlighted the trade-off between finer-grained modeling and less precise model estimation.
AkaikeH. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716-723.
2.
BarnesT. (2010). Novel derivation and application of skill matrices: The Q-matrix method. In RomeroC.VenturaS.PechenizkiyM.BakerR. S. J. d. (Eds.), Handbook on educational data mining (pp. 159-172). Boca Raton, FL: Chapman & Hall.
3.
BasokcuT. O. (2014). Classification accuracy effects of Q-matrix validation and sample size in DINA and G-DINA models. Journal of Education and Practice, 5, 220-230.
4.
BoxG. E.DraperN. R. (1987). Empirical model-building and response surfaces. New York, NY: John Wiley.
5.
ChiuC.-Y. (2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37, 598-618.
6.
DeCarloL. T. (2012). Recognizing uncertainty in the Q-matrix via a Bayesian extension of the DINA model. Applied Psychological Measurement, 36, 447-468.
7.
de la TorreJ. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45, 343-362.
8.
de la TorreJ. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115-130.
9.
de la TorreJ.ChiuC.-Y. (2016). A general method of empirical Q-matrix validation. Psychometrika, 81, 253-273.
10.
DoornikJ. A. (2002). Object-oriented matrix programming using Ox (Version 3.1). London, England: Timberlake Consultants Press.
11.
GorinJ. (2007). Test construction and diagnostic testing. In LeightonJ. P.GierlM. J. (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 173-201). New York, NY: Cambridge University Press.
12.
HaertelE. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 301-321.
13.
HuebnerA.WangB.LeeS. (2009, June3). Practical issues concerning the application of the DINA model to CAT data. Paper presented at the Proceedings of the 2009 GMAC Conference on Computerized Adaptive Testing. Retrieved from http://iacat.org/sites/default/files/biblio/cat09huebner.pdf
14.
HuffK.GoodmanD. P. (2007). The demand for cognitive diagnostic assessment. In LeightonJ. P.GierlM. J. (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 19-60). Cambridge, UK: Cambridge University Press.
15.
JunkerB. W.SijtsmaK. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272.
16.
LeeY.-S.de la TorreJ.ParkY. S. (2012). Relationships between cognitive diagnosis, CTT, and IRT indices: An empirical investigation. Asia Pacific Education Review, 13, 333-345.
17.
LeeY.-W.SawakiY. (2009). Application of three cognitive diagnosis models to ESL reading and listening assessments. Language Assessment Quarterly, 6, 239-263.
18.
LiH.HunterC. V.LeiP.-W. (2016). The selection of cognitive diagnostic models for a reading comprehension test. Language Testing, 33, 391-409.
19.
LiH.SuenH. K. (2013). Constructing and validating a Q-matrix for cognitive diagnostic analyses of a reading test. Educational Assessment, 18, 1-25.
LordF. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
22.
MacreadyG. B.DaytonC. M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 2, 99-120.
23.
MadisonM. J.BradshawL. P. (2015). The effects of Q-matrix design on classification accuracy in the log-linear cognitive diagnosis model. Educational and Psychological Measurement, 75, 491-511.
24.
MislevyR. J. (1994). Evidence and inference in educational assessment. Psychometrika, 59, 439-483.
25.
No Child Left Behind (NCLB) Act of 2001, Pub. L. No. 107-110 C.F.R. § 115 (2002).
26.
RavandH. (2016). Application of a cognitive diagnostic model to a high-stakes reading comprehension test. Journal of Psychoeducational Assessment, 34, 782-799.
27.
RavandH.RobitzschA. (2018). Cognitive diagnostic model of best choice: A study of reading comprehension. Educational Psychology, 38, 1255-1277.
28.
RoussosL. A.TemplinJ. L.HensonR. A. (2007). Skills diagnosis using IRT-based latent class models. Journal of Educational Measurement, 44, 293-311.
29.
RuppA. A.TemplinJ. (2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68, 78-96.
30.
SchwarzG. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461-464.
31.
SpearmanC. (1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72-101.
32.
TatsuokaK. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354.
33.
TemplinJ.HensonR. A. (2006, April). A Bayesian method for incorporating uncertainty into Q-matrix estimation in skills assessment. Paper presented at the Annual Meeting of the National Council for Measurement in Education, San Francisco, CA.
VriezeS. I. (2012). Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychological Methods, 17, 228-243.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
