Cognitive diagnostic models (CDMs) attempt to uncover latent skills or attributes that examinees must possess in order to answer test items correctly. The DINA (deterministic input, noisy ‘‘and’’) model is a popular CDM that has been widely used. It is shown here that a logistic version of the model can easily be fit with standard software for latent class analysis. A partly Bayesian approach to estimation, posterior mode estimation, is used as a simple alternative to a fully Bayesian approach via Markov chain Monte Carlo methods. A latent-class analysis of a widely analyzed data set, the fraction subtraction data of K. K. Tatsuoka, reveals some neglected problems with respect to the classification of examinees; for example, examinees who get all of the items incorrect are classified as having most of the skills. It is also noted that obtaining large estimates of the latent class sizes can indicate misspecification of the Q-matrix, such as the inclusion of an irrelevant skill. It is shown, analytically and via simulations, that the problems are largely associated with the structure of the Q-matrix.
Agresti, A. ( 2002). Categorical data analysis (2nd ed.). Hoboken, NJ: Wiley.
2.
Barnes, T., Bitzer, D., & Vouk, M. ( 2005, May). Experimental analysis of the Q-matrix method in knowledge discovery. Paper presented at the 15th International Symposium on Methodologies for Intelligent Systems 2005, Saratoga Springs, NY.
3.
Cen, H., Koedinger, K., Junker, B. ( 2005). Learning factors analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. Ashley, & T. Chan (Eds.), Intelligent Tutoring Systems: 8th International Conference (pp. 164-175). Berlin, Germany: Springer.
4.
Clogg, C.C. ( 1995). Latent class models. In G. Arminger, C. C. Clogg, & M. E. Sobel (Eds.), Handbook of statistical modeling for the social and behavioral sciences (pp. 311-359). New York, NY: Plenum.
5.
Clogg, C.C., & Eliason, S.R. ( 1987). Some common problems in log-linear analysis. Sociological Methods and Research, 16, 8-44.
DeCarlo, L.T. ( 2002). A latent class extension of signal detection theory, with applications. Multivariate Behavioral Research, 37, 423-451.
8.
DeCarlo, L.T. ( 2005). A model of rater behavior in essay grading based on signal detection theory. Journal of Educational Measurement, 42, 53-76.
9.
DeCarlo, L.T. ( 2008). Studies of a latent class signal detection model for constructed response scoring (ETS Technical Report No. RR-08-63). Princeton NJ: Educational Testing Service.
10.
DeCarlo, L.T. ( 2010). Studies of a latent class signal detection model for constructed response scoring II: Incomplete and hierarchical designs ( ETS Technical Report No.RR-10-08). Princeton NJ: Educational Testing Service.
11.
de la Torre, J. ( 2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115-130.
12.
de la Torre, J., & Douglas, J.A. ( 2004). Higher-order latent trait models for cognitive diagnosis . Psychometrika, 69, 333-353.
13.
de la Torre, J., & Douglas, J.A. ( 2008). Model evaluation and multiple strategies in cognitive diagnosis: An analysis of fraction subtraction data. Psychometrika , 73, 595-624.
14.
DiBello, L.V., Roussos, L.A., & Stout, W.F. ( 2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics Vol. 26. Psychometrics (pp. 979-1030). Amsterdam, Netherlands: Elsevier.
15.
Fahrmeir, L., & Tutz, G. ( 2001). Multivariate statistical modelling based on generalized linear models (2nd ed.). New York, NY: Springer-Verlag.
16.
Fu, J., & Li, Y. ( 2007, April). An integrative review of cognitively diagnostic psychometric models. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.
17.
Galindo-Garre, F., & Vermunt, J.K. ( 2006). Avoiding boundary estimates in latent class analysis by Bayesian posterior mode estimation. Behaviormetrika, 33, 43-59.
18.
Galindo-Garre, F., Vermunt, J.K., & Bergsma, W.P. ( 2004). Bayesian posterior estimation of logit parameters with small samples. Sociological Methods & Research, 33, 88-117.
19.
Gelman, A., Carlin, J.B., Stern, H.S., & Rubin, D.B. ( 1995). Bayesian data analysis. New York, NY: Chapman & Hall.
20.
Gierl, M.J., & Cui, Y. ( 2008). Defining characteristics of diagnostic classification models and the problem of retrofitting in cognitive diagnostic assessment . Measurement, 6, 263-268.
21.
Haertel, E.H. ( 1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement , 26, 301-321.
Henson, R.A., Templin, J.L., & Willse, J.T. ( 2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191-210.
24.
Im, S. ( 2007). Statistical consequences of attribute misspecification in the rule space model (Unpublished doctoral dissertation). Teachers College, Columbia University, New York, NY .
25.
Junker, B.W., & Sijtsma, K. ( 2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272.
26.
Kang, T., & Cohen, A.S. ( 2007). Model selection methods for dichotomous items. Applied Psychological Measurement, 31, 331-358.
27.
Leighton, J.P. ( 2008). Where’s the psychology? A commentary on ‘‘Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art.’’ Measurement , 6, 272-275.
28.
Li, F., Cohen, A.S., Kim, S.-H., & Cho, S.-J. (2009). Model selection methods for dichotomous IRT models. Applied Psychological Measurement , 33, 353-373.
29.
Macmillan, N.A., & Creelman, C.D. (2005). Detection theory: A user’s guide (2nd ed.). New York, NY : Cambridge University Press.
30.
Macready, G.B., & Dayton, C.M. ( 1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 2, 99-120.
31.
Maris, E. ( 1999). Estimating multiple classification latent class models . Psychometrika, 64, 187-212.
Rupp, A.A., & Templin, J.L. ( 2008a). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement, 6, 219-262.
34.
Rupp, A.A., & Templin, J.L. ( 2008b). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68, 78-96.
35.
Schafer, J.L. ( 1997). Analysis of incomplete multivariate data. New York, NY: Chapman & Hall.
36.
Tatsuoka, C. ( 2002). Data analytic methods for latent partially ordered classification models. Journal of the Royal Statistical Society, Series C (Applied Statistics), 51, 337-350.
37.
Tatsuoka, K.K. ( 1990). Toward an integration of item-response theory and cognitive error diagnosis. In N. Frederiksen, R. Glaser, A. Lesgold, & M. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 453-488). Hillsdale, NJ: Erlbaum.
38.
Templin, J., Henson, R., & Douglas, J. ( 2006, April). General theory and estimation of cognitive diagnosis models: Using Mplus to derive model estimates. Paper presented at the 2007 National Council on Measurement in Education training session, Chicago, IL.
39.
Templin, J.L., Henson, R.A., Templin, S.E., & Roussos, L. ( 2008). Robustness of hierarchical modeling of skill association in cognitive diagnosis models. Applied Psychological Measurement , 32, 559-574.
40.
Templin, J.L., & Hoffman, L. ( 2010). Obtaining diagnostic classification model estimates using Mplus. Manuscript submitted for publication.
41.
Vermunt, J.K. ( 1997). LEM: A general program for the analysis of categorical data. Tilburg, Netherlands: Tilburg University.
42.
Vermunt, J.K., & Magidson, J. ( 2005). Technical guide for Latent Gold 4.0: Basic and advanced . Belmont, MA: Statistical Innovations Inc.
von Davier, M. ( 2005). A general diagnostic model applied to language testing data (ETS Technical Report No. RR-05-16). Princeton NJ: Educational Testing Service.
45.
von Davier, M. ( 2009). Some notes on the reinvention of latent structure models as diagnostic classification models. Measurement, 7, 67-74.
