Reducing the Misclassification Costs of Cognitive Diagnosis Computerized Adaptive Testing: Item Selection With Minimum Expected Risk

Abstract

Cognitive diagnosis computerized adaptive testing (CD-CAT) aims to identify each examinee’s strengths and weaknesses on latent attributes for appropriate classification into an attribute profile. As the cost of a CD-CAT misclassification differs across user needs (e.g., remedial program vs. scholarship eligibilities), item selection can incorporate such costs to improve measurement efficiency. This study proposes such a method, minimum expected risk (MER), based on Bayesian decision theory. According to simulations, using MER to identify examinees with no mastery (MER-U0) or full mastery (MER-U1) showed greater classification accuracy and efficiency than other methods for these attribute profiles, especially for shorter tests or low quality item banks. For other attribute profiles, regardless of item quality or termination criterion, MER methods, modified posterior-weighted Kullback–Leibler information (MPWKL), posterior-weighted CDM discrimination index (PWCDI), and Shannon entropy (SHE) performed similarly and outperformed posterior-weighted attribute-level CDM discrimination index (PWACDI) in classification accuracy and test efficiency, especially on short tests. MER with a zero-one loss function, MER-U0, MER-U1, and PWACDI utilized item banks more effectively than the other methods. Overall, these results show the feasibility of using MER in CD-CAT to increase the accuracy for specific attribute profiles to address different user needs.

Keywords

cognitive diagnosis computerized adaptive testing item selection bayesian decision theory minimum expected cost

Get full access to this article

View all access options for this article.

References

Chang

H. H.

Ying

Z. L.

(1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20(3), 213–229. https://doi.org/10.1177/014662169602000303

Cheng

(2009). When cognitive diagnosis meets computerized adaptive testing: CD-CAT. Psychometrika, 74(4), 619–632. https://doi.org/10.1007/s11336-009-9123-2

Cover

Thomas

(1991). Elements of information theory. Wiley.

Finkelman

M. D.

Kim

Weissman

Cook

R. J.

(2014). Cognitive diagnostic models and computerized adaptive testing: Two new item-selection methods that incorporate response times. Journal of Computerized Adaptive Testing, 2(4), 59–76. https://doi.org/10.7333/1412-0204059

Haertel

E. H.

(1989). Using restricted latent class models to map skill structure of achievement items. Journal of Educational Measurement, 26(4), 301–323. http://www.jstor.org/stable/1434756

Hartz

(2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practice. [Unpublished doctoral thesis, University of Illinois at Urbana-Champaign].

Henson

Douglas

(2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29(4), 262–277. https://doi.org/10.1177/0146621604272623

Henson

Roussos

Douglas

(2008). Cognitive diagnostic attribute-level discrimination indices. Applied Psychological Measurement, 32(4), 275–288. https://doi.org/10.1177/0146621607302478

Hsu

C.-L.

Wang

W.-C.

Chen

S.-Y.

(2013). Variable-length computerized adaptive testing based on cognitive diagnosis models. Applied Psychological Measurement, 37(7), 563–582. https://doi.org/10.1177/0146621613488642

10.

Junker

B. W.

Sijtsma

(2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272. https://doi.org/10.1177/01466210122032064

11.

Kaplan

de la Torre

Barrada

J. R.

(2015). New item selection methods for computerized adaptive testing. Applied Psychological Measurement, 39(3), 167–188. https://doi.org/10.1177/0146621614554650

12.

de la Torre

(2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69, 253–275. https://doi.org/10.1111/bmsp.12070

13.

Rudner

L. M.

(2009). Scoring and classification examinees using measurement decision theory. Practical Assessment Research and Evolution, 14(8), 1–11. http://pareonline.net/getvn.asp?v=14&n=8

14.

Rupp

A. A.

Templin

Henson

R. A.

(2010). Diagnostic measurement: Theory, methods, and applications (The statistical structure of core DCMs). Guilford.

15.

Tatsouka

(2002). Data analytic methods for latent partially ordered classification models. Journal of Royal Statistical Society: Series C (Apply Statistics), 51, 337–350. https://doi.org/10.1111/1467-9876.00272

16.

Tatsouka

Ferguson

(2003). Sequential classification on partially ordered sets. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 65(1), 143–157. http://www.jstor.org/stable/3088831

17.

Tatsuoka

K. K.

(1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345–354. http://www.jstor.org/stable/1434951

18.

van der Linden

W. J.

Glas

C.A.W.

(Eds.), (2000).Computerized adaptive testing: Theory and practice. Kluwer.

19.

Wainer

(Ed.), (2000) Computerized adaptive testing: A primer (2nd Ed.). Erlbaum.

20.

Wang

(2013). Mutual information item selection method in cognitive diagnostic computerized adaptive testing with short test length. Educational and Psychological Measurement, 73(6), 1017–1035. https://doi.org/10.1177/0013164413498256

21.

Weiss

D. J.

(1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6(4), 473–492. https://doi.org/10.1177/014662168200600408

22.

Chang

Douglas

(2003). A simulation study to compare CAT strategies for cognitive diagnosis. Annual Meeting of the American Education Research Association.

23.

Wang

Shang

(2016). On initial item selection in cognitive diagnostic computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 69(3), 291–315. https://doi.org/10.1111/bmsp.12072

24.

Zheng

Chang

H. H.

(2016). High-efficiency response distribution-based item selection algorithms for short-length cognitive diagnostic computerized adaptive testing. Applied Psychological Measurement, 40(8), 608–624. https://doi.org/10.1177/0146621616665196

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.33 MB