Sage Journals: Discover world-class research

Abstract

This article introduces a blocked-design procedure for cognitive diagnosis computerized adaptive testing (CD-CAT), which allows examinees to review items and change their answers during test administration. Four blocking versions of the new procedure were proposed. In addition, the impact of several factors, namely, item quality, generating model, block size, and test length, on the classification rates was investigated. Three popular item selection indices in CD-CAT were used and their efficiency compared using the new procedure. An additional study was carried out to examine the potential benefit of item review. The results showed that the new procedure is promising in that allowing item review resulted only in a small loss in attribute classification accuracy under some conditions. Moreover, using a blocked-design CD-CAT is beneficial to the extent that it alleviates the negative impact of test anxiety on examinees’ true performance.

Keywords

item review CD-CAT block design

Get full access to this article

View all access options for this article.

References

Benjamin

L. T.

Cavell

T. A.

Shallenberger

W. R.

III . (1984). Staying with initial answers on objective tests: Is it a myth? Teaching of Psychology, 11, 133-141.

Cassady

J. C.

Johnson

R. E.

(2002). Cognitive test anxiety and academic performance. Contemporary Educational Psychology, 27, 270-295.

Cheng

(2009). When cognitive diagnosis meets computerized adaptive testing: CD-CAT. Psychometrika, 74, 619-632.

de la Torre

(2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115-130.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76, 179-199.

de la Torre

Douglas

A. J.

(2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333-353.

de la Torre

Hong

Deng

(2010). Factors affecting the item parameter estimation and classification accuracy of the DINA model. Journal of Educational Measurement, 47, 227-249.

Han

K. T.

(2013). Item pocket method to allow response review and change in computerized adaptive testing. Applied Psychological Measurement, 37, 259-275.

Henson

R. A.

Templin

J. L.

Willse

J. T.

(2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191-210.

10.

Kaplan

de la Torre

Barrada

J. R.

(2015). New item selection methods for cognitive diagnosis computerized adaptive testing. Applied Psychological Measurement, 39, 167-188.

11.

Lord

F. M.

(1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.

12.

Meijer

R. R.

Nering

M. L.

(1999). Computerized adaptive testing: Overview and introduction. Applied Psychological Measurement, 23, 187-194.

13.

Stocking

M. L.

(1997). Revising item responses in computerized adaptive tests: A comparison of three models. Applied Psychological Measurement, 21, 129-142.

14.

Tatsuoka

(2002). Data analytic methods for latent partially ordered classification models. Journal of the Royal Statistical Society: Series C (Applied Statistics), 51, 337-350.

15.

Tatsuoka

Ferguson

(2003). Sequential classification on partially ordered sets. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 65, 143-157.

16.

Tatsuoka

(1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354.

17.

Templin

Henson

(2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287-305.

18.

van der Linden

W. J.

Pashley

P. J

. (2010). Item selection and ability estimation in adaptive testing. In van der Linden

W. J.

Glas

C. A. W.

(Eds.), Elements of adaptive testing (pp. 3-30). Boston, MA: Kluwer.

19.

Vispoel

W. P.

(1998). Reviewing and changing answers on computer-adaptive and self-adaptive vocabulary tests. Journal of Educational Measurement, 35, 328-345.

20.

Vispoel

W. P.

(2000). Reviewing and changing answers on computerized fixed-item vocabulary tests. Educational and Psychological Measurement, 60, 371-384.

21.

Vispoel

W. P.

Clough

S. J.

Bleiler

(2005). A closer look at using judgments of item difficulty to change answers on computerized adaptive tests. Journal of Educational Measurement, 42, 331-350.

22.

von Davier

. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287-307.

23.

von Davier

Cheng

. (2014). Multistage testing using diagnostic models. In Duanli

von Davier

A. A.

Lewis

(Eds.), Computerized multistage testing: Theory and applications (pp. 219-227). Boca Raton, FL: CRC Press.

24.

Wang

(2013). Mutual information item selection method in cognitive diagnostic computerized adaptive testing with short test length. Educational and Psychological Measurement, 73, 1017-1035.

25.

Wise

S. L.

(1996, April). A critical analysis of the arguments for and against item review in computerized adaptive testing. Paper presented at the annual meeting of the National Council on Measurement in Education, New York, NY.

26.

Wang

Shang

(2016). On initial item selection in cognitive diagnostic computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 69, 291-315.

27.

Chang

H.-H.

Douglas

(2003, April). A simulation study to compare CAT strategies for cognitive diagnosis. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.

28.

Yen

Y.-C.

R.-G.

Liao

W.-W.

Chen

L.-J.

(2012). Reducing the impact of inappropriate items on reviewable computerized adaptive testing. Journal of Educational Technology & Society, 15, 231-243.

29.

Zheng

Chang

H.-H.

(2015). On-the-fly assembled multistage adaptive testing. Applied Psychological Measurement, 39, 104-118.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.92 MB

A Blocked-CAT Procedure for CD-CAT

Abstract

Keywords

Get full access to this article

References

Supplementary Material