Sage Journals: Discover world-class research

Abstract

Online calibration is a key technology for calibrating new items in computerized adaptive testing (CAT). As multidimensional polytomous data become popular, online calibration methods applicable to multidimensional CAT with polytomously scored items (P-MCAT) have been proposed. However, the existing methods are mainly based on marginal MLE with an expectation-maximization algorithm (MMLE/EM), making it difficult to accurately estimate parameters in high-dimensional scenarios without sufficient calibration sample size or suitable initial values. To conquer these challenges, a neural network (NN)-based online calibration framework was put forward. The new NN-based methods differ profoundly from the traditional ones in that the parameter estimates of new items are obtained by learning the patterns between input and output data instead of finding solutions to the log-marginal likelihood. Moreover, an alternative solution was proposed for traditional methods to obtain appropriate initial values. Simulation studies were conducted to compare the NN- and MMLE/EM-based methods under various conditions, and further explore the properties of the NN-based methods. Results showed that both the NN-based methods and the alternative solution found their strengths in recovering the item parameters of new items, while the MMLE/EM-based methods struggled to converge when more than three dimensions were involved in the test.

Keywords

online calibration neural network item bank construction and maintenance P-MCAT

Get full access to this article

View all access options for this article.

References

Alpaydin

(2004). Introduction to machine learning. MIT Press.

Ban

J.-C.

Hanson

B. A.

Wang

T. Y.

Harris

D. J.

(2001). A comparative study of on-line pretest item-calibration/scaling methods in computerized adaptive testing. Journal of Educational Measurement, 38(3), 191–212. https://doi.org/10.1111/j.1745-3984.2001.tb01123.x

Bergmeir

Benítez

J. M.

(2012). Neural Networks in R Using the Stuttgart Neural Network Simulator: RSNNS. Journal of Statistical Software, 46(7), 1–26. https://doi.org/10.18637/jss.v046.i07

Bishop

C. M.

(2006). Pattern recognition and machine learning. Springer-Verlag.

Cai

(2008). SEM of another flavour: Two new applications of the supplemented EM algorithm. British Journal of Mathematical and Statistical Psychology, 61(2), 309–329. https://doi.org/10.1348/000711007x249603

Cai

(2010). Metropolis-Hastings Robbins-Monro algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35(3), 307–335. https://doi.org/10.3102/1076998609353115

Cattell

H. E. P.

Mead

A. D.

(2008). The sixteen personality factor questionnaire (16PF). In Boyle

G. J.

Matthews

Saklofske

D. H.

(Eds.), The SAGE handbook of personality theory and assessment, Vol. 2. Personality measurement and testing (pp. 135–159). Sage Publications, Inc.

Chang

H. H.

(2015). Psychometrics behind computerized adaptive testing. Psychometrika, 80(1), 1–20. https://doi.org/10.1007/s11336-014-9401-5

Chen

(2017). A comparative study of online item calibration methods in multidimensional computerized adaptive testing. Journal of Educational and Behavioral Statistics, 42(5), 559–590. https://doi.org/10.3102/1076998617695098

10.

Chen

Wang

(2016). A new online calibration method for multidimensional computerized adaptive testing. Psychometrika, 81(3), 674–701. https://doi.org/10.1007/s11336-015-9482-9

11.

Chen

Wang

Xin

Chang

H. H.

(2017). Developing new online calibration methods for multidimensional computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 70(1), 81–117. https://doi.org/10.1111/bmsp.12083

12.

Chen

Xin

Wang

Chang

H. H.

(2012). Online calibration methods for the DINA model with independent attributes in CD-CAT. Psychometrika, 77(2), 201–222. https://doi.org/10.1007/S11336-012-9255-7

13.

Converse

Curi

Oliveira

Templin

(2021) Estimation of multidimensional item response theory models with correlated latent variables using variational autoencoders. Machine Learning, 110(6), 1463–1480. https://doi.org/10.1007/s10994-021-06005-7

14.

De Ayala

R. J

. (1994). The influence of multidimensionality on the graded response model. Applied Psychological Measurement, 18(2), 155–170. https://doi.org/10.1177/014662169401800205

15.

Goodfellow

Bengio

Courville

(2016). Deep learning. MIT Press.

16.

Kang

Zheng

Chang

(2020). Online calibration of a joint model of item responses and response times in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 45(2), 175–208. https://doi.org/10.3102/1076998619879040

17.

LeCun

Bengio

Hinton

(2015). Deep learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539

18.

Likert

(1932). A technique for the measurement of attitudes. Archives of Psychology, 22(140), 5–55.

19.

Liu

Wang

(2022). Estimating three- and four-parameter MIRT models with importance-weighted sampling enhanced variational auto-encoder. Frontiers in Psychology, 13, 1–19. https://doi.org/10.3389/fpsyg.2022.935419

20.

Parshall

C. G.

(1998). Item development and pretesting in a computer-based testing environment. Paper presented at the colloquium Computer-Based Testing: Building the Foundation for Future Assessments, September, Philadelphia, PA.

21.

Ripley

B. D.

(1996). Pattern recognition and neural networks. Cambridge University Press.

22.

Ruder

(2016). An overview of gradient descent optimization algorithms. arXiv:1609.04747.

23.

Segall

D. O.

(1996). Multidimensional adaptive testing. Psychometrika, 61(2), 331–354. https://doi.org/10.1007/BF02294343

24.

Stocking

M. L.

(1988). Scale drift in on-line calibration (Research Report 88-28). ETS.

25.

Tang

Wang

Liu

Ying

(2020). Latent feature extraction for process data via multidimensional scaling. Psychometrika, 85(2), 378–397. https://doi.org/10.1007/s11336-020-09708-3

26.

Urban

C. J.

Bauer

D. J.

(2021). A deep learning algorithm for high-dimensional exploratory item factor analysis. Psychometrika, 86(1), 1–29. https://doi.org/10.1007/s11336-021-09748-3

27.

Wainer

Mislevy

R. J.

(1990). Item response theory, item calibration, and proficiency estimation. In Wainer

Dorans

N. J.

Flaugher

Green

B. F.

Mislevy

R. J.

Steinberg

Thissen

(Eds.), Computerized adaptive testing: A primer (Chap. 4, pp. 65–102). Erlbaum.

28.

Wang

Weiss

D. J.

(2018). Robustness of parameter estimation to assumptions of normality in the multidimensional graded response model. Multivariate Behavioral Research, 53(3), 403–418. https://doi.org/10.1080/00273171.2018.1455572

29.

Xiong

Ding

Luo

(2020). Online calibration of polytomous items under the graded response model. Frontiers in Psychology, 10, 3085. https://doi.org/10.3389/fpsyg.2019.03085

30.

Yuan

Huang

Chen

(2023). Online calibration in multidimensional computerized adaptive testing with polytomously scored items. Journal of Educational Measurement, 60(3), 476–500. https://doi.org/10.1111/jedm.12353

31.

Zheng

(2014). New methods of online calibration for item bank replenishment [Unpublished doctoral thesis]. University of Illinois at Urbana-Champaign.

32.

Zheng

(2016). Online calibration of polytomous items under the generalized partial credit model. Applied Psychological Measurement, 40(6), 434–450. https://doi.org/10.1177/0146621616650406

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB

Online Calibration for Multidimensional CAT With Polytomously Scored Items: A Neural Network–Based Approach

Abstract

Keywords

Get full access to this article

References

Supplementary Material