Flexible Computerized Adaptive Tests to Detect Misconceptions and Estimate Ability Simultaneously

Abstract

The Scaling Individuals and Classifying Misconceptions (SICM) model is an advanced psychometric model that can provide feedback to examinees’ misconceptions and a general ability simultaneously. These two types of feedback are represented by a discrete and a continuous latent variable, respectively, in the SICM model. The complex structure of the SICM model brings difficulties in estimating both misconception profile and ability efficiently in a linear test. To overcome this challenge, this study proposes a flexible computerized adaptive test (FCAT) design as a new test delivery method to increase test efficiency by administering an individualized test to examinees. We propose three item selection methods and two transition criteria to determine adaptive steps based on the needs of estimating one or two latent variables. Through two simulation studies, we demonstrate how to select an appropriate item selection method for an adaptive step and what transition criterion should be used between two adaptive steps. Results reveal the combination of the item selection method and the transition criterion could improve the estimation accuracy of a specific latent variable to a different extent and thus provide further guidance in designing an FCAT.

Keywords

diagnostic classification model flexible computerized adaptive test misconceptions adaptive design dual-purpose assessment

Get full access to this article

View all access options for this article.

References

Bradshaw

Templin

(2014). Combining item response theory and diagnostic classification models: A psychometric model for scaling ability and diagnosing misconceptions. Psychometrika, 79(3), 403–425.

Cheng

(2009). When cognitive diagnosis meets computerized adaptive testing: CD-CAT. Psychometrika, 74(4), 619–632.

Cheng

Chang

(2007, April). Dual information method in cognitive diagnostic computerized adaptive testing. Annual Meeting of National Council on Measurement in Education, Chicago, IL.

Cheng

Chang

H.-H.

(2009). The maximum priority index method for severely constrained item selection in computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 62(2), 369–383.

Dai

Zhang

(2016). Exploration of item selection in dual-purpose cognitive diagnostic computerized adaptive testing: Based on the RRUM. Applied Psychological Measurement, 40(8), 625–640.

de la Torre

Douglas

J. A.

(2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333–353.

Every Student Succeeds Act (ESSA) of 2015, Public Law No. 114-95, S.1177, 114th Cong. (2015). https://www.congress.gov/114/plaws/publ95/PLAW-114publ95.pdf

Hambleton

R. K.

Swaminathan

Rogers

H. J.

(1991). Measurement methods for the social sciences series: Vol. 2. Fundamentals of item response theory. SAGE.

Hestenes

Wells

Swackhamer

(1992). Force Concept Inventory. The Physics Teacher, 30(3), 141–158.

10.

Kang

H.-A.

Zhang

Chang

H.-H.

(2017). Dual-objective item selection criteria in cognitive diagnostic computerized adaptive testing. Journal of Educational Measurement, 54(2), 165–183.

11.

Kaplan

de la Torre

Barrada

J. R.

(2015). New item selection methods for cognitive diagnosis computerized adaptive testing. Applied Psychological Measurement, 39(3), 167–188.

12.

Luecht

R. M.

Sireci

S. G.

(2011). A review of models for computer-based testing (Research report 2011-12). College Board.

13.

McGlohen

Chang

H.-H.

(2008). Combining computer adaptive testing technology with cognitively diagnostic assessment. Behavior Research Methods, 40(3), 808–821.

14.

Muthén

Kaplan

(1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology, 38(2), 171–189.

15.

Roussos

L. A.

DiBello

L. V.

Stout

Hartz

S. M.

Henson

R. A.

Templin

(2007). The fusion model skills diagnosis system. In Leighton

Gierl

(Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 275–318). Cambridge University Press.

16.

Roussos

L. A.

Hartz

Stout

(2003, March). A description of the fusion model skill diagnosis system using real data examples. Annual Meeting of the National Council on Measurement in Education, Chicago, IL.

17.

Rupp

Templin

Henson

R. A.

(2010). Diagnostic measurement: Theory, methods, and applications. Guilford Press.

18.

Tatsuoka

K. K.

(1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345–354.

19.

Wang

Chang

H.-H.

Douglas

(2012). Combining cat with cognitive diagnosis: A weighted item selection approach. Behavior Research Methods, 44(1), 95–109.

20.

Wang

Zheng

Chang

H.-H.

(2014). An enhanced approach to combine item response theory with cognitive diagnosis in adaptive testing. Journal of Educational Measurement, 51(4), 358–380.

21.

Wang

Lin

Chang

H.-H.

Douglas

(2016). Hybrid computerized adaptive testing: From group sequential design to fully sequential design. Journal of Educational Measurement, 53(1), 45–62.

22.

Wang

Zhang

Shen

(2019). A joint modeling framework of responses and response times to assess learning outcomes. Multivariate Behavioral Research, 55, 49–68.

23.

Chang

Douglas

(2003, April). Computerized adaptive testing strategies for cognitive diagnosis. Annual Meeting of National Council on Measurement in Education, Montreal, Canada.

24.

Zheng

Chang

H.-H.

(2016). High-efficiency response distribution–based item selection algorithms for short-length cognitive diagnostic computerized adaptive testing. Applied Psychological Measurement, 40(8), 608–624.

25.

Zheng

Chang

H.-H.

(2015). On-the-fly assembled multistage adaptive testing. Applied Psychological Measurement, 39(2), 104–118.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.34 MB