The present study investigates the statistical consequences of attribute misspecification in the rule space method for cognitively diagnostic measurement. The two types of attribute misspecifications examined in the present study are exclusion of an essential attribute (which affects problem-solving performance) and inclusion of a superfluous attribute (which does not). Results of a simulation study show that exclusion of an essential attribute tends to lead to underestimation of examinees’ mastery probabilities for the remaining attributes, whereas inclusion of a superfluous attribute generally leads to overestimation of attribute mastery probabilities for the other attributes. In addition, order relations among attributes induced by superset/subset relationships affect the biases in the estimated attribute mastery probabilities in systematic ways. These results underscore the importance of correct attribute specification in cognitively diagnostic assessment and delineate some specific effects of using incorrect attribute sets.
BakerF. B. (1993). Sensitivity of the linear logistic test model to misspecification of the weight matrix. Applied Psychological Measurement, 17, 201-210.
2.
BirenbaumM.KellyA. E.TatsuokaK. (1993). Diagnosing knowledge states in algebra using the rule space model. Journal for Research in Mathematics Education, 24, 442-459.
3.
BirenbaumM.TatsuokaC.YamadaY. (2004). Diagnostic assessment in TIMMS-R: Between countries and within country comparisons of eight graders’ mathematics performance. Studies in Educational Evaluation, 30, 151-173.
4.
BuckG.TatsuokaK. (1998). Application of the rule-space procedure to language testing: Examining attributes of a free response listening test. Language Testing, 15, 119-157.
5.
BuckG.TatsuokaK.KostinI. (1997). The sub-skills of reading: Rule-space analysis of a multiple-choice test of second language reading comprehension. Language Learning, 47, 423-466.
6.
de la TorreJ. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45, 343-362.
7.
de la TorreJ.DouglasJ. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333-353.
8.
DiBelloL.StoutW.RoussosL. (1995). Unified cognitive/psychometric diagnostic assessment: Likelihood-based classification techniques. In NicholsP. D.ChipmanS. F.BrennanR. L. (Eds.), Cognitively diagnostic assessment (pp. 361-389). Hillsdale, NJ: Lawrence Erlbaum.
9.
DoganE.TatsuokaK. (2007). An international comparison using a diagnostic testing model: Turkish students’ profile of mathematical skills on TIMSS-R. Educational Studies in Mathematics, 68, 263-272.
10.
FischerG. (1973). The linear logistic test model as an instrument of educational research. Acta Psychologica, 37, 359-374.
11.
HartzS. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality (Unpublished doctoral dissertation). University of Illinois, Urbana-Champaign.
12.
ImS.YinY. (2009). Diagnosing skills of statistical hypothesis testing using the rule space method. Studies in Educational Evaluation, 35, 193-199.
13.
JunkerB. W.SijtsmaK. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272.
14.
KatzI. R.MartinezM. E.SheehanK. M.TatsuokaK. (1998). Extending the rule space methodology to a semantically-rich domain: Diagnostic assessment in architecture. Journal of Educational and Behavioral Statistics, 24, 254-278.
15.
LeightonJ. P.GierlM. J.HunkaS. (2004). The attribute hierarchy model: An approach for integrating cognitive theory with assessment practice. Journal of Educational Measurement, 41, 205-236.
16.
RuppA.TemplinJ. (2007). The effects of Q-matrix misspecification on parameter estimates and misclassification rates in the DINA model. Educational and Psychological Measurement, 68, 78-96.
17.
TatsuokaC. (2002). Data analytic methods for latent partially ordered classification models. Journal of the Royal Statistical Society: Series C (Applied Statistics), 51, 337-350.
TatsuokaK. (1984). Analysis of errors in fraction addition and subtraction problems. Technical report NIE-G-81-0002. Computer-based education research laboratory, University of Illinois at Urbana-Champaign.
20.
TatsuokaK. (1985). A probabilistic model for diagnosing misconceptions by the pattern classification approach. Journal of Educational Statistics, 10, 55-73.
21.
TatsuokaK. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In FrederiksenN.GlaserR.LesgoldA.ShaftoM. (Eds.), Diagnostic monitoring of skill and knowledge acquisition. (pp. 453-488). Hillsdale, NJ: Lawrence Erlbaum.
22.
TatsuokaK. (1991). Boolean algebra applied to determination of universal set of knowledge states (Research Report RR-91-44). Princeton, NJ: Educational Testing Service.
23.
TatsuokaK. (1996). Use of generalized person–fit indexes, zetas for statistical pattern classification. Applied Measurement in Education, 9, 65-75.
24.
TatsuokaK. (2009). Cognitive diagnostic assessment: An introduction to the rule space method. New York, NY: Routledge, Taylor & Francis.
25.
TatsuokaK.CorterJ. E.TatsuokaC. (2004). Patterns of diagnosed mathematical content and process skills in TIMSS-R across a sample of 20 countries. American Educational Research Journal, 41, 901-926.
26.
TatsuokaK.GuerreroA.CorterJ.TatsuokaC.YamadaT.XinT.. . . ImS. (2006). International comparison of mathematical thinking skills in the TIMSS-R. Japanese Journal for Research on Testing, 2, 3-4.
27.
TatsuokaK.TatsuokaM. (1987). Bug distribution and statistical pattern classification. Psychometrika, 52, 193-206.
28.
von DavierM. (2005, September). A general diagnostic model applied to language testing data (ETS Research Report RR-05-16). Princeton, NJ: Educational Testing Service.
29.
Yepes-BarayaM.AllenN. L. (2003, April). An attribute-based study to obtain diagnostic assessment information on the attainment of cognitive dimensions relevant in science education. Paper presented at the 2003 American Educational Research Association meeting, Chicago, IL.
