Diagnostic classification models are psychometric models that aim to classify examinees according to their mastery or non-mastery of specified latent characteristics. These models are well-suited for providing diagnostic feedback on educational assessments because of their practical efficiency and increased reliability when compared with other multidimensional measurement models. A priori specifications of which latent characteristics or attributes are measured by each item are a core element of the diagnostic assessment design. This item–attribute alignment, expressed in a Q-matrix, precedes and supports any inference resulting from the application of the diagnostic classification model. This study investigates the effects of Q-matrix design on classification accuracy for the log-linear cognitive diagnosis model. Results indicate that classification accuracy, reliability, and convergence rates improve when the Q-matrix contains isolated information from each measured attribute.
BoorsboomD.MellenbergG. D. (2007). Test validity in cognitive assessment. In LeightonJ. P.GierlM. J. (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 19-60). Cambridge, England: Cambridge University Press.
2.
BradshawL.IzsákA.TemplinJ.JacobsonE. (2014). Diagnosing teachers’ understandings of rational number: Building a multidimensional test within the diagnostic classification framework. Educational Measurement: Issues and Practice, 33(1), 2-14.
3.
BradshawL.TemplinJ. (2013). Combining scaling and classification: A psychometric model for scaling ability and diagnosing misconceptions. Psychometrika. Advance online publication. doi:10.1007/s11336-013-9350-4
4.
Center for K-12 Assessment and Performance Management at ETS. (2014, March). Coming together to raise achievement: New assessments for the common core state standards. Retrieved from http://www.k12center.org
5.
ChiuC.-Y.DouglasJ.LiX. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74, 633-665.
6.
CuiY.GierlM. J.ChangH. (2012). Estimating classification consistency and accuracy for cognitive diagnostic assessment. Journal of Educational Measurement, 49, 19-38.
7.
de la TorreJ. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115-130.
8.
DeCarloL. T. (2011). On the analysis of fraction subtraction data: The DINA model, classification, latent class sizes, and the Q-matrix. Applied Psychological Measurement, 35, 8-26.
9.
HaertelE. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333-352.
10.
HensonR.DouglasJ. (2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29, 262-277.
11.
HensonR.TemplinJ.WillseJ. (2009). Defining a family of cognitive diagnosis models using log linear models with latent variables. Psychometrika, 74, 191-210.
12.
HuffK.GoodmanD. P. (2007). The demand for cognitive diagnostic assessment. In LeightonJ. P.GierlM. J. (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 19-60). Cambridge, England: Cambridge University Press.
13.
JunkerB. W.SijtsmaK. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272.
14.
Kunina-HabenichtO.RuppA. A.WilhemO. (2012). The impact of model misspecification on parameter estimation and item-fit assessment in log-linear diagnostic classification models. Journal of Educational Measurement, 49, 59-81.
15.
McDonaldR. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.
16.
MuthénL. K.MuthénB. O. (1998-2012). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén.
17.
NunnallyJ. C.BernsteinI. R. (1994). Psychometric theory. New York, NY: McGraw-Hill.
18.
RuppA. A.TemplinJ. (2008). Effects of Q-matrix misspecification on parameter estimates and misclassification rates in the DINA model. Educational and Psychological Measurement, 68, 78-98.
19.
RuppA. A.TemplinJ.HensonR. (2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guilford Press.
20.
SinharayS.PuhanG.HabermanS. J. (2011). An NCME instructional module on subscores. Educational Measurement: Issues and Practice, 30(3), 29-40.
21.
TatsuokaK. K. (1983). Rule-space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354.
22.
TatsuokaK. K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In FrederiksenN.GlaserR.LesgoldA.SaftoM. (Eds.), Monitoring skills and knowledge acquisition (pp. 453-488). Hillsdale, NJ: Lawrence Erlbaum.
23.
TemplinJ.BradshawL. (2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30, 251-275.
24.
TemplinJ. L.HensonR. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287-305.
