In multidimensional computerized adaptive testing (MCAT), the existing online calibration designs do not account for the time spent on items, potentially resulting in reduced calibration efficiency. To improve the calibration efficiency for MCAT, response time was used as auxiliary information for the online calibration of items. Drawing upon the D-optimal design for MCAT (M-D design) and A-optimal design for MCAT (M-A design), this study introduced the D-optimal per expectation time unit design (M-D-ET design) and A-optimal per expectation time unit design (M-A-ET design) for MCAT. These two new designs prioritized the selection of new items that yield the greatest increment in calibration information per time unit for examinees. Simulation studies were conducted to compare the performances of the M-D, M-A, M-D-ET, and M-A-ET designs across various scenarios. The results indicate that, prior to the calibration of approximately 70% of the new items, the M-D-ET/M-A-ET designs outperformed the M-D/M-A designs in terms of calibration information per time unit and average response time. In the continuous testing scenario, the M-D-ET/M-A-ET designs outperformed the M-D/M-A designs by generating more information per time unit, reducing response time, and maintaining approximately equivalent calibration accuracy. A study based on real response data yielded comparable findings.
AliU. S.ChangH. H. (2014). An item-driven adaptive design for calibrating pretest items. ETS Research Report Series, 2014(2), 1–12.
2.
BergerM. P. (1991). On the efficiency of IRT models when applied to different sampling designs. Applied Psychological Measurement, 15(3), 293–306.
3.
ChalmersP.PritikinJ.RobitzschA.ZoltakM.KimK.FalkC. F.MeadeA. (2015). mirt: Multidimensional item response theory. R package version 1.14. http://CRAN.R-project.org/package=mirt
4.
ChenP.WangC. (2016). A new online calibration method for multidimensional computerized adaptive testing. Psychometrika, 81(3), 674–701.
5.
ChenP.WangC.XinT.ChangH. H. (2017). Developing new online calibration methods for multidimensional computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 70(1), 81–117.
6.
ChenP.ZhangJ.XinT. (2013). Application of online calibration technique in computerized adaptive testing. Advances in Psychological Science, 21(10), 1883.
7.
ChengY.YuanK. H. (2010). The impact of fallible item parameter estimates on latent trait recovery. Psychometrika, 75(2), 280–291.
8.
ChoeE. M.KernJ. L.ChangH. H. (2018). Optimizing the use of response times for item selection in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 43(2), 135–158.
9.
EntinkR. K.van der LindenW. J.FoxJ. P. (2009). A Box–Cox normal model for response times. British Journal of Mathematical and Statistical Psychology, 62(3), 621–640.
10.
FanZ.WangC.ChangH. H.DouglasJ. (2012). Utilizing response time distributions for item selection in CAT. Journal of Educational and Behavioral Statistics, 37(5), 655–670.
11.
GulliksenH. (2013). Theory of mental tests. Routledge.
12.
GuoF.WangL. (2003). Online calibration and scale stability of a CAT program. Annual meeting of the National Council on Measurement in Education, Chicago, IL.
13.
GuoL.ShangP.XiaL. (2017). Advantages and illustrations of application of response time model in psychological and educational testing. Advances in Psychological Science, 25(4), 701–712.
HeY.ChenP.LiY. (2020). New efficient and practicable adaptive designs for calibrating items online. Applied Psychological Measurement, 44(1), 3–16.
16.
HeY.ChenP.LiY. (2021). Maximum information per time unit designs for continuous online item calibration. British Journal of Mathematical and Statistical Psychology, 74, 24–51.
17.
HeY.QiY. Y. (2023). Using response time in multidimensional computerized adaptive testing. Journal of Educational Measurement, 60(4), 697–738.
KangH. A.ZhengY.ChangH. H. (2020). Online calibration of a joint model of item responses and response times in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 45(2), 175–208.
20.
KernJ. L.ChoeE. (2021). Using a response time–based expected a posteriori estimator to control for differential speededness in computerized adaptive test. Applied Psychological Measurement, 45(5), 361–385.
21.
LeeY. H.IpE. H.FuhC. D. (2008). A strategy for controlling item exposure in multidimensional computerized adaptive testing. Educational and Psychological Measurement, 68(2), 215–232.
22.
ManK.HarringJ. R.JiaoH.ZhanP. (2019). Joint modeling of compensatory multidimensional item responses and response times. Applied Psychological Measurement, 43(8), 639–654.
23.
RangerJ.KuhnJ. T. (2012). A flexible latent trait model for response times in tests. Psychometrika, 77(1), 31–47.
24.
ReckaseM. D. (1985). The difficulty of test items that measure more than one ability. Applied Psychological Measurement, 9(4), 401–412.
25.
ReckaseM. D. (2009). Multidimensional item response theory models. In ReckaseM. D. (Ed.), Multidimensional item response theory (pp. 79–112). Springer.
26.
RenH.van der LindenW. J.DiaoQ. (2017). Continuous online item calibration: Parameter recovery and item utilization. Psychometrika, 82(2), 498–522.
SamejimaF. (1974). Normal ogive model on the continuous response level in the multidimensional latent space. Psychometrika, 39(1), 111–121.
29.
SegallD. O. (1996). Multidimensional adaptive testing. Psychometrika, 61(2), 331–354.
30.
StockingM. L. (1990). Specifying optimum examinees for item parameter estimation in item response theory. Psychometrika, 55(3), 461–475.
31.
van der LindenW. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204.
32.
van der LindenW. J. (2008). Using response times for item selection in adaptive testing. Journal of Educational and Behavioral Statistics, 33(1), 5–20.
33.
van der LindenW. J.GuoF. (2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73(3), 365–384.
34.
van der LindenW. J.RenH. (2015). Optimal Bayesian adaptive design for test-item calibration. Psychometrika, 80(2), 263–288.
35.
van der LindenW. J.van Krimpen-StoopE. M. (2003). Using response times to detect aberrant responses in computerized adaptive testing. Psychometrika, 68(2), 251–265.
36.
VeldkampB. P. (2016). On the issue of item selection in computerized adaptive testing with response times. Journal of Educational Measurement, 53(2), 212–228.
37.
WainerH.MislevyR. J. (1990). Item response theory, item c3alibration, and proficiency estimation. Computerized Adaptive Testing: A Primer, 4, 65–102.
38.
WangC. (2015). On latent trait estimation in multidimensional compensatory item response models. Psychometrika, 80, 428–449.
39.
WangC.ChangH. H.BoughtonK. A. (2011). Kullback–Leibler information and its applications in multi-dimensional adaptive testing. Psychometrika, 76(1), 13–39.
ZhengY. (2014). New methods of online calibration for item bank replenishment. University of Illinois at Urbana-Champaign.
42.
ZhengY.ChangH. H. (2017). A comparison of five methods for pretest item selection in online calibration. International Journal of Quantitative Research in Education, 4(1/2), 133–158.
