Attribute structure is an explicit way of presenting the relationship between attributes in diagnostic measurement. The specification of attribute structures directly affects the classification accuracy resulted from psychometric modeling. This study provides a conceptual framework for understanding misspecifications of attribute structures. Under the framework, each attribute structure can be represented through an external shape and an internal organization. A simulation study and an application example were used to investigate how misspecification of external shapes and internal organizations affects model fit and item fit assessments, and respondent classification. The proposed framework and simulation results aim to support using attribute structures to (a) develop better diagnostic assessments and (b) inform theories of constructs.
AkaikeH. (1987). Factor analysis and AIC. Psychometrika, 52, 317-332.
2.
AlmondR.MislevyR. J.SteinbergL. S.WilliamsonD. M.YanD. (2015). Bayesian networks in educational assessment. New York, NY: Springer.
3.
AyersE.NugentR.DeanN. (2009, July1-3). A comparison of student skill knowledge estimates. Paper presented at the Second International Conference on Educational Data Mining (EDM), Cordoba, Spain.
4.
BongersI. L.KootH. M.Van Der EndeJ.VerhulstF. C. (2004). Developmental trajectories of externalizing behaviors in childhood and adolescence. Child Development, 75, 1523-1537.
5.
BradshawL.IzsákA.TemplinJ.JacobsonE. (2014). Diagnosing teachers’ understandings of rational numbers: Building a multidimensional test within the diagnostic classification framework. Educational Measurement: Issues and Practice, 33(1), 2-14.
6.
BroidyL. M.NaginD. S.TremblayR. E.BatesJ. E.BrameB.DodgeK. A.LynamD. R. (2003). Developmental trajectories of childhood disruptive behaviors and adolescent delinquency: A six-site, cross-national study. Developmental Psychology, 39, 222.
7.
ChenJ.de la TorreJ. (2013). A general cognitive diagnosis model for expert-defined polytomous attributes. Applied Psychological Measurement, 37, 419-437.
8.
ChenJ.de la TorreJ.ZhangZ. (2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50, 123-140.
9.
ClarkA.KingstonN.TemplinJ.PardosZ. (2014). Summary of results from the fall 2013 pilot administration of the Dynamic Learning Maps™ Alternate Assessment System (Technical Report No. 14-01). Lawrence: University of Kansas Center for Educational Testing and Evaluation.
10.
de la TorreJ. (2011). The generalized DINA model framework. Psychometrika, 76, 179-199.
11.
de la TorreJ.HongY.DengW. (2010). Factors affecting the item parameter estimation and classification accuracy of the DINA model. Journal of Educational Measurement, 47, 227-249.
12.
Darling-HammondL.BarronB.PearsonP. D.SchoenfeldA. H.StageE. K.ZimmermanT. D.… TilsonJ. L. (2015). Powerful learning: What we know about teaching for understanding. New York, NY: Wiley.
13.
DiBelloL. V.RoussosL. A.StoutW. F. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In RaoC. R.SinharayS. (Eds.), Handbook of statistics (Vol. 26, pp. 979-1030). Amsterdam, Netherlands: Elsevier.
14.
EntwistleN.RamsdenP. (2015). Understanding student learning (Routledge revivals). New York, NY: Routledge.
GierlM. J. (2007). Making diagnostic inferences about cognitive attributes using the rule-space model and attribute hierarchy method. Journal of Educational Measurement, 44, 325-340.
17.
GierlM. J.AlvesC.MajeauR. T. (2010). Using the attribute hierarchy method to make diagnostic inferences about examinees’ knowledge and skills in mathematics: An operational implementation of cognitive diagnostic assessment. International Journal of Testing, 10, 318-341.
18.
GierlM. J.LeightonJ. P.HunkaS. M. (2000). An NCME instructional module on exploring the logic of Tatsuoka’s rule-space model for test development and analysis. Educational Measurement: Issues and Practice, 19(3), 34-44.
19.
HensonR.TemplinJ.WillseJ. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191-210.
20.
ImS.CorterJ. E. (2011). Statistical consequences of attribute misspecification in the rule space method. Educational and Psychological Measurement, 71, 712-731.
21.
Kunina-HabenichtO.RuppA. A.WilhelmO. (2009). A practical illustration of multidimensional diagnostic skills profiling: Comparing results from confirmatory factor analysis and diagnostic classification models. Studies in Educational Evaluation, 35, 64-70.
22.
Kunina-HabenichtO.RuppA. A.WilhelmO. (2012). The impact of model misspecification on parameter estimation and item-fit assessment in log-linear diagnostic classification models. Journal of Educational Measurement, 49, 59-81.
23.
LeightonJ. P.GierlM. J.HunkaS. M. (2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41, 205-237.
24.
LiuR.Huggins-ManleyA. C. (2016). The specification of attribute structures and its effects on classification accuracy in diagnostic test design. In van der ArkL. A.BoltD. M.WangW.-C.DouglasJ. A.WibergM. (Eds.), Quantitative psychology research (pp. 243-254). New York, NY: Springer.
25.
LiuR.Huggins-ManleyA. C.BradshawL. (2016). The impact of Q-matrix designs on diagnostic classification accuracy in the presence of attribute hierarchies. Educational and Psychological Measurement. doi:10.1177/0013164416645636
26.
MadisonM.BradshawL. (2015). The effects of Q-matrix design on classification accuracy in the log-linear cognitive diagnosis model. Educational and Psychological Measurement, 75, 491-511.
27.
Maydeu-OlivaresA. (2013). Goodness-of-fit assessment of item response theory models (with discussion). Measurement: Interdisciplinary Research and Perspectives, 11, 71-137.
28.
Maydeu-OlivaresA.JoeH. (2005). Limited-and full-information estimation and goodness-of-fit testing in 2n contingency tables: A unified framework. Journal of the American Statistical Association, 100, 1009-1020.
29.
Maydeu-OlivaresA.JoeH. (2014). Assessing approximate fit in categorical data analysis. Multivariate Behavioral Research, 49, 305-328.
30.
McDonaldR. P.MokM. M.-C. (1995). Goodness of fit in item response models. Multivariate Behavioral Research, 30, 23-40.
31.
R Core Team. (2016). R (Version 3.3) [Computer Software]. Vienna, Austria: R Foundation for Statistical Computing.
32.
ReckaseM. (2009). Multidimensional item response theory. New York, NY: Springer.
33.
RobitzschA.KieferT.GeorgeA. C.UenlueA. (2017). CDM: Cognitive diagnosis modeling (R package Version 5.4.0) [Computer software manual]. Retrieved from http://CRAN.R-project.org/package=CDM/
34.
RuppA. A.TemplinJ. (2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68, 78-96.
35.
RuppA. A.TemplinJ.HensonR. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guilford Press.
36.
SchwarzG. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461-464.
37.
TatsuokaK. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354.
38.
TemplinJ. (2007, October). Assessing cognitive diagnosis model fit using limited information methods. Paper presented at the International Conference on Advances in Interdisciplinary Statistics and Combinatorics in Greensboro, NC.
39.
TemplinJ.BradshawL. (2014). Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies. Psychometrika, 79, 317-339.
40.
TemplinJ. L.HensonR. A.TemplinS. E.RoussosL. (2008). Robustness of hierarchical modeling of skill association in cognitive diagnosis models. Applied Psychological Measurement, 32, 559-574.
41.
TemplinJ.HoffmanL. (2013). Obtaining diagnostic classification model estimates using Mplus. Educational Measurement: Issues and Practice, 32, 37-50.
42.
von DavierM. (2005). A general diagnostic model applied to language testing data (ETS Research Report RR-05-16). Princeton, NJ: ETS.
43.
WuH. (2013). A comparison of general diagnostic models (GDM) and Bayesian networks using a middle school mathematics test (Unpublished doctoral dissertation). Florida State University, Tallahassee, FL.
44.
YenW. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125-145.
