Development of a High-Accuracy and Effective Online Calibration Method in CD-CAT Based on Gini Index

Abstract

To improve the calibration accuracy and calibration efficiency of cognitive diagnostic computerized adaptive testing (CD-CAT) for new items and, ultimately, contribute to the widespread application of CD-CAT in practice, the current article proposed a Gini-based online calibration method that can simultaneously calibrate the Q-matrix and item parameters of new items. Three simulation studies with simulated and real item banks were conducted to investigate the performance of the proposed method and compare it with the joint estimation algorithm (JEA) and the single-item estimation (SIE) methods. The results indicated that the proposed Gini-based online calibration method yielded higher calibration efficiency than those of the SIE method and outperformed the JEA method on item calibration tasks in terms of both accuracy and efficiency under most experimental conditions.

Keywords

cognitive diagnostic computerized adaptive testing online calibration Q-matrix cognitive diagnosis models Gini index

Get full access to this article

View all access options for this article.

References

Ban

J.-C.

Hanson

B.H.

Wang

Harris

D.J.

(2001). A comparative study of online pretest item-calibration/scaling methods in computerized adaptive testing. Journal of Educational Measurement, 38, 191–212. https://doi.org/10.1111/j.1745-3984.2001.tb01123.x

Breiman

Friedman

Stone

C. J.

Olshen

R. A.

(1984). Classification and regression trees. Chapman & Hall.

Chandra

Paul Varghese

(2009). Fuzzifying Gini Index based decision trees. Expert Systems With Applications, 36(4), 8549–8559. https://doi.org/10.1016/j.eswa.2008.10.053

Chen

(2017). A comparative study of online item calibration methods in multidimensional computerized adaptive testing. Journal of Educational and Behavioral Statistics, 42(5), 559–590. https://doi.org/10.3102/1076998617695098

Chen

Xin

(2011, April). Item replenishing in cognitive diagnostic computerized adaptive testing. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.

Chen

Xin

Wang

Chang

(2012). Online calibration methods for the DINA model with independent attributes in CD-CAT. Psychometrika, 77(2), 201–222. https://doi.org/10.1007/s11336-012-9255-7

Chen

Liu

Ying

(2015). Online item calibration for Q-matrix in CD-CAT. Applied Psychological Measurement, 39(1), 5–15. https://doi.org/10.1177/0146621613513065

Cheng

(2009). When cognitive diagnosis meets computerized adaptive testing: CD-CAT. Psychometrika, 74, 619–632. https://doi.org/10.1007/s11336-009-9123-2

de la Torre

(2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343–362. https://doi.org/10.1111/j.1745-3984.2008.00069.x

10.

de la Torre

(2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115–130. https://doi.org/10.3102/1076998607309474

11.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76, 179–199. https://doi.org/10.1007/s11336-011-9207-7

12.

de la Torre

Chiu

C.-Y.

(2016). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253–273. https://doi.org/10.1007/s11336-015-9467-8

13.

Demaree

R. G

. (1950). An investigation of homogeneity in the interpretation and scaling of test results. University of Illinois at Urbana-Champaign.

14.

DiBello

L. V.

Stout

(2007). Guest editors introduction and overview: IRT-based cognitive diagnostic models and related methods. Journal of Educational Measurement, 44, 285–291. https://doi.org/10.1111/j.1745-3984.2007.00039.x

15.

Gibbons

R. D.

Weiss

D. J.

Pilkonis

P. A.

Frank

Moore

Kim

J. B.

Kupfer

D. J.

(2012). Development of a computerized adaptive test for depression. Archives of General Psychiatry, 69(11), 1104–1112. https://doi.org/10.1001/archgenpsychiatry.2012.14

16.

Corter

J. E.

(2011). Statistical consequences of attribute misspecification in the rule space method. Educational and Psychological Measurement, 71, 712–731. https://doi.org/10.1177/0013164410384855

17.

Junker

B. W.

Sijtsma

(2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272. https://doi.org/10.1177/01466210122032064

18.

Karad

S. M.

Patil

D. D.

Halgaonkar

P. S.

Wadhai

V. M.

Kharat

M. U.

(2012). Two phase iterative clustering for educational data. International Journal of Applied Information Systems, 1(5), 11–15. https://doi.org/10.5120/ijais12-450173

19.

Liu

You

Wang

Ding

Chang

(2013). The development of computerized adaptive testing with cognitive diagnosis for an English achievement test in China. Journal of Classification, 30(2), 152–172. https://link.springer.com/article/10.1007/s00357-013-9128-5

20.

Lord

F. M.

(1986). Maximum likelihood and Bayesian parameter estimation in item response theory. Journal of Educational Measurement, 23(2), 157–162. https://doi.org/10.1111/j.1745-3984.1986.tb00241.x

21.

de la Torre

(2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69(3), 253–275. https://doi.org/10.1111/bmsp.12070

22.

Mehta

Agrawal

Riassnen

(1996). SLIQ: A fast scalable classifier for data mining, in extending database technology (pp. 18–32). Springer. https://doi.org/10.1007/BFb0014141

23.

Nájera

Sorrel

M. A.

Abad

F. J.

(2019). Reconsidering cutoff points in the general method of empirical Q-matrix validation. Educational and psychological measurement, 79(4), 727–753. https://doi.org/10.1177/0013164418822700

24.

No Child Left Behind Act of 2001, Pub. L. No. 1-7-110 (2001).

25.

Rupp

A. A.

Templin

J. L.

(2008). The effect of Q-matrix misspecification on parameter estimates and misclassification rates in the DINA model. Educational and Psychological Measurement, 68, 78–96. https://doi.org/10.1177/0013164407301545

26.

Stocking

M. L.

(1988). Scale drift in online calibration (Tech. Rep. No. RR-88-28-ONR). Educational Testing Service. https://doi.org/10.1002/j.2330-8516.1988.tb00284.x

27.

Strobl

Boulesteix

A.-L.

Augustin

(2007). Unbiased split selection for classification trees based on the Gini Index. Computational Statistics & Data Analysis, 52(1), 483–501. https://doi.org/10.1016/j.csda.2006.12.030

28.

Tan

P. N.

Steinbach

Kumar

. (2016). Introduction to data mining. Pearson Education India.

29.

Tatsuoka

K. K.

(1990). Toward an integration of item-response theory and cognitive error diagnosis. In Frederiksen

Glaser

Lesgold

Safto

(Eds.), Diagnostic monitoring skills and knowledge acquisition (pp. 453–488). Erlbaum.

30.

Tatsuoka

K. K.

Tatsuoka

M. M.

(1997). Computerized cognitive diagnostic adaptive testing: Effect on remedial instruction as empirical validation. Journal of Educational Measurement, 34(1), 3–20. https://doi.org/10.1111/j.1745-3984.1997.tb00504.x

31.

Wainer

Mislevy

R. J.

(1990). Item response theory, item calibration, and proficiency estimation. Computerized adaptive testing: A primer, 4, 65–102.

32.

Wang

Song

Ding

Meng

Cao

Jie

(2018). An EM-based method for Q-matrix validation. Applied Psychological Measurement, 42(6), 446–459. https://doi.org/10.1177/0146621617752991

33.

Cheng

(2020). Data-driven Q-matrix validation using a residual-based statistic in cognitive diagnostic assessment. British Journal of Mathematical and Statistical Psychology, 73, 145–179. https://doi.org/10.1111/bmsp.12191

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.12 MB