Compound Optimal Design for Online Item Calibration Under the Two-Parameter Logistic Model

Abstract

Under the theory of sequential design, compound optimal design with two optimality criteria can be used to solve the problem of efficient calibration of item parameters of item response theory model. In order to efficiently calibrate item parameters in computerized testing, a compound optimal design is proposed for the simultaneous estimation of item difficulty and discrimination parameters under the two-parameter logistic model, which adaptively focuses on optimizing the parameter which is difficult to estimate. The compound optimal design using the acceptance probability can provide ability design points to optimize the item difficulty and discrimination parameters, respectively. Simulation and real data analysis studies showed that the compound optimal design outperformed than the D-optimal and random design in terms of the recovery of both discrimination and difficulty parameters.

Keywords

computerized adaptive testing two-parameter logistic model D-optimal design compound optimal design online calibration

Get full access to this article

View all access options for this article.

References

Babcock

Weiss

D. J.

(2012). Termination criteria in computerized adaptive tests: Do variable-length CATs provide efficient and effective measurement. Journal of Computerized Adaptive Testing, 1(1), 1–18. https://doi.org/10.7333/1212-0101001

Ban

J.-C.

Hanson

B. A.

Wang

Harris

D. J.

(2001). A comparative study of on-line pretest item: Calibration/scaling methods in computerized adaptive testing. Journal of Educational Measurement, 38(3), 191–212. https://doi.org/10.1111/j.1745-3984.2001.tb01123.x

Ban

J.-C.

Hanson

B. A.

Harris

D. J.

(2002). Data sparseness and on-line pretest item calibration-scaling methods in CAT. Journal of Educational Measurement, 39(3), 207–218. https://doi.org/10.1111/j.1745-3984.2002.tb01174.x

Bengs

Kroehne

Brefeld

(2021). Simultaneous constrained adaptive item selection for group-based testing. Journal of Educational Measurement, 58(2), 236–261. https://doi.org/10.1111/jedm.12285

Berger

M. P. F.

(1992). Sequential sampling designs for the two-parameter item response theory model. Psychometrika, 57(4), 521–538. https://doi.org/10.1007/bf02294418

Berger

Verschoor

A. J.

Eggen

T. J. H. M.

Moser

(2019). Efficiency of targeted multistage calibration designs under practical constraints: A simulation study. Journal of Educational Measurement, 56(1), 121–146. https://doi.org/10.1111/jedm.12203

Cao

Chen

Biegler

L. T.

(2024). Sequential design of experiments for parameter estimation with Markov chain Monte Carlo. In Manenti

Reklaitis

G. V.

(Eds.), Computer aided chemical engineering (Vol. 53, pp. 3199–3204). Elsevier. https://doi.org/10.1016/b978-0-443-28824-1.50534-2

Chang

H.-H.

(2004). Understanding computerized adaptive testing: From Robbins-Monro to Lord and beyond. In Kaplan

D. W.

(Ed.), The Sage handbook of quantitative methodology for the social sciences. Sage Publications, Inc.

Chang

H.-H.

Wang

Ying

(2016). Information theory and its application to testing. In Linden

W. J. v. d.

(Ed.), Handbook of item response theory, volume two, statistical tools (pp. 105–123). CRC Press, Taylor & Francis Group.

10.

Chang

H.-H.

Ying

(2007). Computerized adaptive testing. In Salkind

N. J.

Rasmussen

(Eds.), Encyclopedia of measurement and statistics (Vol. 1, pp. 170–173). Sage Publications, Inc.

11.

Chang

Y.-C. I.

H.-Y.

(2010). Online calibration via variable length computerized adaptive testing. Psychometrika, 75(1), 140–157. https://doi.org/10.1007/s11336-009-9133-0

12.

Chen

(2016). Two new online calibration methods for computerized adaptive testing. Acta Psychologica Sinica, 48(9), 1184. https://doi.org/10.3724/sp.j.1041.2016.01184

13.

Chen

(2017). A comparative study of online item calibration methods in multidimensional computerized adaptive testing. Journal of Educational and Behavioral Statistics, 42(5), 559–590. https://doi.org/10.3102/1076998617695098

14.

Chen

Wang

(2016). A new online calibration method for multidimensional computerized adaptive testing. Psychometrika, 81(3), 674–701. https://doi.org/10.1007/s11336-015-9482-9

15.

Chen

Wang

Xin

Chang

H.-H.

(2017). Developing new online calibration methods for multidimensional computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 70(1), 81–117. https://doi.org/10.1111/bmsp.12083

16.

Chen

Xin

(2014). Online calibration with cognitive diagnostic assessment. In Cheng

Chang

H.-H.

(Eds.), Advancing methodologies to support both summative and formative assessments (pp. 287–313). Information Age.

17.

Choi

S. W.

Grady

M. W.

Dodd

B. G.

(2011). A new stopping rule for computerized adaptive testing. Educational and Psychological Measurement, 71(1), 37–53. https://doi.org/10.1177/0013164410387338

18.

Du Toit

(Ed.), (2003). IRT from SSI: Bilog-MG, multilog, parscale, testfact. Scientific Software International, Inc.

19.

Eccleston

J. A.

Whitaker

(1999). On the design of optimal change-over experiments through multi-objective simulated annealing. Statistics and Computing, 9(1), 37–42. https://doi.org/10.1023/a:1008810109585

20.

Eggen

T. J. H. M.

Verhelst

N. D.

(2011). Item calibration in incomplete testing designs. Psicológica, 32(1), 107–132. https://www.uv.es/psicologica/articulos1.11/7EGGEN.pdf

21.

Finch

(2008). Estimation of item response theory parameters in the presence of missing data. Journal of Educational Measurement, 45(3), 225–245. https://doi.org/10.1111/j.1745-3984.2008.00062.x

22.

Foy

Arora

Stanco

G. M.

(Eds.). (2013). TIMSS 2011 user guide for the international database. TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College, and International Association for the Evaluation of Educational Achievement (IEA).

23.

Givens

G. H.

Hoeting

J. A.

(2013). Computational statistics. John Wiley & Sons, Inc.

24.

Hassan

M. U.

Miller

(2024). Optimal calibration of items for multidimensional achievement tests. Journal of Educational Measurement, 61(2), 274–302. https://doi.org/10.1111/jedm.12386

25.

Reckase

M. D.

(2014). Item pool design for an operational variable-length computerized adaptive test. Educational and Psychological Measurement, 74(3), 473–494. https://doi.org/10.1177/0013164413509629

26.

Chen

(2020). Optimal online calibration designs for item replenishment in adaptive testing. Psychometrika, 85(1), 35–55. https://doi.org/10.1007/s11336-019-09687-0

27.

Chen

(2020). New efficient and practicable adaptive designs for calibrating items online. Applied Psychological Measurement, 44(1), 3–16. https://doi.org/10.1177/0146621618824854

28.

Chen

(2021). Maximum information per time unit designs for continuous online item calibration. British Journal of Mathematical and Statistical Psychology, 74(Suppl 1), 24–51. https://doi.org/10.1111/bmsp.12221

29.

Chen

Zhang

(2017). A new online calibration method based on Lord’s bias-correction. Applied Psychological Measurement, 41(6), 456–471. https://doi.org/10.1177/0146621617697958

30.

Huebner

(2010). An overview of recent developments in cognitive diagnostic computer adaptive assessments. Practical Assessment, Research and Evaluation, 15(3), 1–7. https://doi.org/10.7275/7fdd-6897

31.

Jones

D. H.

Jin

(1994). Optimal sequential designs for on-line item estimation. Psychometrika, 59(1), 59–75. https://doi.org/10.1007/bf02294265

32.

Kang

H.-A.

Zheng

Chang

H.-H.

(2020). Online calibration of a joint model of item responses and response times in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 45(2), 175–208. https://doi.org/10.3102/1076998619879040

33.

Kitsos

C. P.

(2013). Optimal experimental design for non-linear models: Theory and applications. Springer.

34.

Kuang

(2023). The new item selection strategies in CAT in combination with item response time and item discrimination. Journal of Jiangxi Normal University (Nature Science), 47(4), 377–382. https://doi.org/10.16357/j.cnki.issn1000-5862.2023.04.07

35.

Lord

F. M.

(1971). Robbins-Monro procedures for tailored testing. Educational and Psychological Measurement, 31(1), 3–31. https://doi.org/10.1177/001316447103100101

36.

Magis

Yan

von Davier

A. A.

(2017). Computerized adaptive and multistage testing with R: Using packages catR and mstR. Springer.

37.

McGree

J. M.

Eccleston

J. A.

Duffull

S. B.

(2008). Compound optimal design criteria for nonlinear models. Journal of Biopharmaceutical Statistics, 18(4), 646–661. https://doi.org/10.1080/10543400802071352

38.

Mislevy

R. J.

(1996). Missing responses and IRT ability estimation: Omits, choice, time limits, and adaptive testing (ETS Research Reports Series No. RR-96-30-ONR). Educational Testing Service.

39.

Olea

Barrada

J. R.

Abad

F. J.

Ponsoda

Cuevas

(2012). Computerized adaptive testing: The capitalization on chance problem. Spanish Journal of Psychology, 15(1), 424–441. https://doi.org/10.5209/rev_sjop.2012.v15.n1.37348

40.

Patton

J. M.

Cheng

Yuan

K. H.

Diao

(2013). The influence of item calibration error on variable-length computerized adaptive testing. Applied Psychological Measurement, 37(1), 24–40. https://doi.org/10.1177/0146621612461727

41.

Quellmalz

E. S.

Pellegrino

J. W.

(2009). Technology and testing. Science, 323(5910), 75–79. https://doi.org/10.1126/science.1168046

42.

Reckase

M. D.

(2010). Designing item pools to optimize the functioning of a computerized adaptive test. Psychological Test and Assessment Modeling, 52(2), 127–141. https://ptam-journal.com/wp-content/uploads/2025/01/01_Reckase.pdf

43.

Ren

van der Linden

W. J.

Diao

(2017). Continuous online item calibration: Parameter recovery and item utilization. Psychometrika, 82(2), 498–522. https://doi.org/10.1007/s11336-017-9553-1

44.

Robitzsch

Kiefer

George

A. C.

Uenlue

(2022). CDM: Cognitive Diagnosis Modeling. R package version 8.2-6. https://cran.r-project.org/web/packages/CDM/index.html

45.

Silvey

S. D.

(1980). Optimal design. Chapman and Hall Ltd.

46.

Stocking

M. L.

(1988). Scale drift in on-line calibration (RR-88-28). Educational Testing Service.

47.

Stocking

M. L.

(1994). Three practical issues for modern adaptive testing item pools. Educational Testing Service.

48.

Ul Hassan

Miller

(2019). Optimal item calibration for computerized achievement tests. Psychometrika, 84(4), 1101–1128. https://doi.org/10.1007/s11336-019-09673-6

49.

van der Linden

W. J.

Glas

C. A. W.

(2000). Capitalization on item calibration error in adaptive testing. Applied Measurement in Education, 13(1), 35–53. https://doi.org/10.1207/s15324818ame1301_2

50.

van der Linden

W. J.

Ren

(2015). Optimal bayesian adaptive design for test-item calibration. Psychometrika, 80(2), 263–288. https://doi.org/10.1007/s11336-013-9391-8

51.

Wagner

Loesche

Bißantz

(2022). Low-stakes performance testing in Germany by the VERA assessment: Analysis of the mode effects between computer-based testing and paper-pencil testing. European Journal of Psychology of Education, 37(2), 531–549. https://doi.org/10.1007/s10212-021-00532-6

52.

Wainer

Dorans

N. J.

Eignor

Flaugher

Green

B. F.

Mislevy

R. J.

Thissen

(2000). Computerized adaptive testing: A primer. Lawrence Erlbaum Associates.

53.

Wainer

Mislevy

R. J.

(1990). Item response theory, item calibration, and proficiency estimation. In Wainer

(Ed.), Computerized adaptive testing: A primer (pp. 65–102). Erlbaum.

54.

Weiss

D. J.

Kingsbury

G. G.

(1984). Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21(4), 361–375. https://doi.org/10.1111/j.1745-3984.1984.tb01040.x

55.

Yan

von Davier

A. A.

Lewis

(2014). Computerized multistage testing: Theory and applications. CRC Press.

56.

You

X.-F.

Ding

S.-L.

Liu

H.-Y.

(2010). Parameter estimation of the raw item in computerized adaptive testing. Acta Psychologica Sinica, 42(7), 813–820. https://doi.org/10.3724/sp.j.1041.2010.00813

57.

Zheng

(2016). Online calibration of polytomous items under the generalized partial credit model. Applied Psychological Measurement, 40(6), 434–450. https://doi.org/10.1177/0146621616650406