Under the theory of sequential design, compound optimal design with two optimality criteria can be used to solve the problem of efficient calibration of item parameters of item response theory model. In order to efficiently calibrate item parameters in computerized testing, a compound optimal design is proposed for the simultaneous estimation of item difficulty and discrimination parameters under the two-parameter logistic model, which adaptively focuses on optimizing the parameter which is difficult to estimate. The compound optimal design using the acceptance probability can provide ability design points to optimize the item difficulty and discrimination parameters, respectively. Simulation and real data analysis studies showed that the compound optimal design outperformed than the D-optimal and random design in terms of the recovery of both discrimination and difficulty parameters.
BabcockB.WeissD. J. (2012). Termination criteria in computerized adaptive tests: Do variable-length CATs provide efficient and effective measurement. Journal of Computerized Adaptive Testing, 1(1), 1–18. https://doi.org/10.7333/1212-0101001
2.
BanJ.-C.HansonB. A.WangT.YiQ.HarrisD. J. (2001). A comparative study of on-line pretest item: Calibration/scaling methods in computerized adaptive testing. Journal of Educational Measurement, 38(3), 191–212. https://doi.org/10.1111/j.1745-3984.2001.tb01123.x
3.
BanJ.-C.HansonB. A.YiQ.HarrisD. J. (2002). Data sparseness and on-line pretest item calibration-scaling methods in CAT. Journal of Educational Measurement, 39(3), 207–218. https://doi.org/10.1111/j.1745-3984.2002.tb01174.x
4.
BengsD.KroehneU.BrefeldU. (2021). Simultaneous constrained adaptive item selection for group-based testing. Journal of Educational Measurement, 58(2), 236–261. https://doi.org/10.1111/jedm.12285
5.
BergerM. P. F. (1992). Sequential sampling designs for the two-parameter item response theory model. Psychometrika, 57(4), 521–538. https://doi.org/10.1007/bf02294418
6.
BergerS.VerschoorA. J.EggenT. J. H. M.MoserU. (2019). Efficiency of targeted multistage calibration designs under practical constraints: A simulation study. Journal of Educational Measurement, 56(1), 121–146. https://doi.org/10.1111/jedm.12203
7.
CaoX.ChenX.BieglerL. T. (2024). Sequential design of experiments for parameter estimation with Markov chain Monte Carlo. In ManentiF.ReklaitisG. V. (Eds.), Computer aided chemical engineering (Vol. 53, pp. 3199–3204). Elsevier. https://doi.org/10.1016/b978-0-443-28824-1.50534-2
8.
ChangH.-H. (2004). Understanding computerized adaptive testing: From Robbins-Monro to Lord and beyond. In KaplanD. W. (Ed.), The Sage handbook of quantitative methodology for the social sciences. Sage Publications, Inc.
9.
ChangH.-H.WangC.YingZ. (2016). Information theory and its application to testing. In LindenW. J. v. d. (Ed.), Handbook of item response theory, volume two, statistical tools (pp. 105–123). CRC Press, Taylor & Francis Group.
10.
ChangH.-H.YingZ. (2007). Computerized adaptive testing. In SalkindN. J.RasmussenK. (Eds.), Encyclopedia of measurement and statistics (Vol. 1, pp. 170–173). Sage Publications, Inc.
ChenP. (2017). A comparative study of online item calibration methods in multidimensional computerized adaptive testing. Journal of Educational and Behavioral Statistics, 42(5), 559–590. https://doi.org/10.3102/1076998617695098
14.
ChenP.WangC. (2016). A new online calibration method for multidimensional computerized adaptive testing. Psychometrika, 81(3), 674–701. https://doi.org/10.1007/s11336-015-9482-9
15.
ChenP.WangC.XinT.ChangH.-H. (2017). Developing new online calibration methods for multidimensional computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 70(1), 81–117. https://doi.org/10.1111/bmsp.12083
16.
ChenP.XinT. (2014). Online calibration with cognitive diagnostic assessment. In ChengY.ChangH.-H. (Eds.), Advancing methodologies to support both summative and formative assessments (pp. 287–313). Information Age.
17.
ChoiS. W.GradyM. W.DoddB. G. (2011). A new stopping rule for computerized adaptive testing. Educational and Psychological Measurement, 71(1), 37–53. https://doi.org/10.1177/0013164410387338
18.
Du ToitM. (Ed.), (2003). IRT from SSI: Bilog-MG, multilog, parscale, testfact. Scientific Software International, Inc.
19.
EcclestonJ. A.WhitakerD. (1999). On the design of optimal change-over experiments through multi-objective simulated annealing. Statistics and Computing, 9(1), 37–42. https://doi.org/10.1023/a:1008810109585
FinchH. (2008). Estimation of item response theory parameters in the presence of missing data. Journal of Educational Measurement, 45(3), 225–245. https://doi.org/10.1111/j.1745-3984.2008.00062.x
22.
FoyP.AroraA.StancoG. M. (Eds.). (2013). TIMSS 2011 user guide for the international database. TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College, and International Association for the Evaluation of Educational Achievement (IEA).
23.
GivensG. H.HoetingJ. A. (2013). Computational statistics. John Wiley & Sons, Inc.
24.
HassanM. U.MillerF. (2024). Optimal calibration of items for multidimensional achievement tests. Journal of Educational Measurement, 61(2), 274–302. https://doi.org/10.1111/jedm.12386
25.
HeW.ReckaseM. D. (2014). Item pool design for an operational variable-length computerized adaptive test. Educational and Psychological Measurement, 74(3), 473–494. https://doi.org/10.1177/0013164413509629
26.
HeY.ChenP. (2020). Optimal online calibration designs for item replenishment in adaptive testing. Psychometrika, 85(1), 35–55. https://doi.org/10.1007/s11336-019-09687-0
27.
HeY.ChenP.LiY. (2020). New efficient and practicable adaptive designs for calibrating items online. Applied Psychological Measurement, 44(1), 3–16. https://doi.org/10.1177/0146621618824854
28.
HeY.ChenP.LiY. (2021). Maximum information per time unit designs for continuous online item calibration. British Journal of Mathematical and Statistical Psychology, 74(Suppl 1), 24–51. https://doi.org/10.1111/bmsp.12221
29.
HeY.ChenP.LiY.ZhangS. (2017). A new online calibration method based on Lord’s bias-correction. Applied Psychological Measurement, 41(6), 456–471. https://doi.org/10.1177/0146621617697958
30.
HuebnerA. (2010). An overview of recent developments in cognitive diagnostic computer adaptive assessments. Practical Assessment, Research and Evaluation, 15(3), 1–7. https://doi.org/10.7275/7fdd-6897
KangH.-A.ZhengY.ChangH.-H. (2020). Online calibration of a joint model of item responses and response times in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 45(2), 175–208. https://doi.org/10.3102/1076998619879040
33.
KitsosC. P. (2013). Optimal experimental design for non-linear models: Theory and applications. Springer.
34.
LiJ.KuangT. (2023). The new item selection strategies in CAT in combination with item response time and item discrimination. Journal of Jiangxi Normal University (Nature Science), 47(4), 377–382. https://doi.org/10.16357/j.cnki.issn1000-5862.2023.04.07
MagisD.YanD.von DavierA. A. (2017). Computerized adaptive and multistage testing with R: Using packages catR and mstR. Springer.
37.
McGreeJ. M.EcclestonJ. A.DuffullS. B. (2008). Compound optimal design criteria for nonlinear models. Journal of Biopharmaceutical Statistics, 18(4), 646–661. https://doi.org/10.1080/10543400802071352
38.
MislevyR. J.WuP. (1996). Missing responses and IRT ability estimation: Omits, choice, time limits, and adaptive testing (ETS Research Reports Series No. RR-96-30-ONR). Educational Testing Service.
39.
OleaJ.BarradaJ. R.AbadF. J.PonsodaV.CuevasL. (2012). Computerized adaptive testing: The capitalization on chance problem. Spanish Journal of Psychology, 15(1), 424–441. https://doi.org/10.5209/rev_sjop.2012.v15.n1.37348
40.
PattonJ. M.ChengY.YuanK. H.DiaoQ. (2013). The influence of item calibration error on variable-length computerized adaptive testing. Applied Psychological Measurement, 37(1), 24–40. https://doi.org/10.1177/0146621612461727
van der LindenW. J.GlasC. A. W. (2000). Capitalization on item calibration error in adaptive testing. Applied Measurement in Education, 13(1), 35–53. https://doi.org/10.1207/s15324818ame1301_2
50.
van der LindenW. J.RenH. (2015). Optimal bayesian adaptive design for test-item calibration. Psychometrika, 80(2), 263–288. https://doi.org/10.1007/s11336-013-9391-8
51.
WagnerI.LoescheP.BißantzS. (2022). Low-stakes performance testing in Germany by the VERA assessment: Analysis of the mode effects between computer-based testing and paper-pencil testing. European Journal of Psychology of Education, 37(2), 531–549. https://doi.org/10.1007/s10212-021-00532-6
52.
WainerH.DoransN. J.EignorD.FlaugherR.GreenB. F.MislevyR. J.ThissenD. (2000). Computerized adaptive testing: A primer. Lawrence Erlbaum Associates.
53.
WainerH.MislevyR. J. (1990). Item response theory, item calibration, and proficiency estimation. In WainerH. (Ed.), Computerized adaptive testing: A primer (pp. 65–102). Erlbaum.
54.
WeissD. J.KingsburyG. G. (1984). Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21(4), 361–375. https://doi.org/10.1111/j.1745-3984.1984.tb01040.x
55.
YanD.von DavierA. A.LewisC. (2014). Computerized multistage testing: Theory and applications. CRC Press.
56.
YouX.-F.DingS.-L.LiuH.-Y. (2010). Parameter estimation of the raw item in computerized adaptive testing. Acta Psychologica Sinica, 42(7), 813–820. https://doi.org/10.3724/sp.j.1041.2010.00813
57.
ZhengY. (2016). Online calibration of polytomous items under the generalized partial credit model. Applied Psychological Measurement, 40(6), 434–450. https://doi.org/10.1177/0146621616650406
