Sage Journals: Discover world-class research

Abstract

When calibrating new items online, it is practicable to first compare all new items according to some criterion and then assign the most suitable one to the current examinee who reaches a seeding location. The modified D-optimal design proposed by van der Linden and Ren (denoted as D-VR design) works within this practicable framework with the aim of directly optimizing the estimation of item parameters. However, the optimal design point for a given new item should be obtained by comparing all examinees in a static examinee pool. Thus, D-VR design still has room for improvement in calibration efficiency from the view of traditional optimal design. To this end, this article incorporates the idea of traditional optimal design into D-VR design and proposes a new online calibration design criterion, namely, excellence degree (ED) criterion. Four different schemes are developed to measure the information provided by the current examinee when implementing this new criterion, and four new ED designs equipped with them are put forward accordingly. Simulation studies were conducted under a variety of conditions to compare the D-VR design and the four proposed ED designs in terms of calibration efficiency. Results showed that the four ED designs outperformed D-VR design in almost all simulation conditions.

Keywords

item response theory computerized adaptive testing online calibration adaptive design optimal design sequential design

Get full access to this article

View all access options for this article.

References

Anderson

T. W.

(1984). An introduction to multivariate statistical analysis (2nd ed.). New York, NY: John Wiley.

Baker

F. B.

(2001). The basics of item response theory (2nd ed.). Washington, DC: ERIC Clearinghouse on Assessment and Evaluation.

Baker

F. B.

Kim

S.-H.

(2004). Item response theory: Parameter estimation techniques (2nd ed.). New York, NY: Marcel Dekker.

Ban

J. C.

Hanson

B. A.

Wang

T. Y.

Harris

D. J.

(2001). A comparative study of on-line pretest item—Calibration/scaling methods in computerized adaptive testing. Journal of Educational Measurement, 38, 191-212.

Berger

M. P. F.

King

C. Y. J.

Wong

W. K.

(2000). Minimax D-optimal designs for item response theory models. Psychometrika, 65, 377-390.

Berger

M. P. F.

Tan

F. E. S.

(2004). Robust designs for linear mixed effects models. Applied Statistics, 53, 569-581.

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 379-479). Reading, MA: Addison-Wesley.

Chang

H.-H.

Qian

J. H.

Ying

Z. L.

(2001). a-stratified multistage computerized adaptive testing with b blocking. Applied Psychological Measurement, 25, 333-341.

Chang

H.-H.

Ying

Z. L.

(2009). Nonlinear sequential designs for logistic item response theory models with applications to computerized adaptive tests. The Annals of Statistics, 37, 1466-1488.

10.

Chang

Y. C. I.

H. Y.

(2010). Online calibration via variable length computerized adaptive testing. Psychometrika, 75, 140-157.

11.

Chen

(2017). A comparative study of online item calibration methods in multidimensional computerized adaptive testing. Journal of Educational and Behavioral Statistics, 42, 559-590.

12.

Chen

Wang

(2016). A new online calibration method for multidimensional computerized adaptive testing. Psychometrika, 81, 674-701.

13.

Chen

Wang

Xin

Chang

H.-H.

(2017). Developing new online calibration methods for multidimensional computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 70, 81-117.

14.

Chen

Xin

Wang

Chang

H.-H.

(2012). Online calibration methods for the DINA model with independent attributes in CD-CAT. Psychometrika, 77, 201-222.

15.

Cheng

Yuan

K. H.

(2010). The impact of fallible item parameter estimates on latent trait recovery. Psychometrika, 75, 280-291.

16.

Firth

(1993). Bias reduction of maximum likelihood estimates. Biometrika, 80, 27-38.

17.

Jones

D. H.

Jin

Z. Y.

(1994). Optimal sequential designs for on-line item estimation. Psychometrika, 59, 59-75.

18.

Kim

(2006). A comparative study of IRT fixed parameter calibration methods. Journal of Educational Measurement, 43, 355-381.

19.

King

Wong

W. K.

(2000). Minimax D-optimal designs for the logistic model. Biometrics, 56, 1263-1267.

20.

Kingsbury

G. G.

(2009). Adaptive item calibration: A process for estimating item parameters within a computerized adaptive test. In Weiss

D. J.

(Ed.), Proceedings of the 2009 GMAC conference on computerized adaptive testing. Retrieved from http://iacat.org/sites/default/files/biblio/cat09kingsbury.pdf

21.

Lord

F. M.

(1968). Some test theory for tailored testing. ETS Research Bulletin RB-68-38. Princeton, NJ: Educational Testing Service.

22.

Lord

F. M.

(1980). Applications of item response theory to practical testing problems. Mahwah, NJ: Lawrence Erlbaum.

23.

Lord

F. M.

Wingersky

M. S.

(1984). An investigation of methods for reducing sampling error in certain IRT procedures. Applied Psychological Measurement, 8, 347-364.

24.

H. Y.

(2014). Application of optimal designs to item calibration. PLoS ONE, 9(9), e106747.

25.

Mislevy

R. J.

Wingersky

M. S.

Sheehan

K. M.

(1994). Dealing with uncertainty about item parameters: Expected response functions, ETS Research Report RR-94-28-ONR. Princeton, NJ: Educational Testing Service.

26.

Silvey

S. D.

(1980). Optimal design. London, England: Chapman & Hall.

27.

Stocking

M. L.

(1988). Scale drift in on-line calibration (Research Report No. 88-28). Princeton, NJ: Educational Testing Service.

28.

Tsutakawa

R. K.

Johnson

J. C.

(1990). The effect of uncertainty of item parameter estimation on ability estimates. Psychometrika, 55, 371-390.

29.

van der Linden

W. J.

Ren

. (2015). Optimal Bayesian adaptive design for test-item calibration. Psychometrika, 80, 263-288.

30.

Wainer

Mislevy

R. J.

(1990). Item response theory, item calibration and proficiency estimation. In Wainer

(Ed.), Computerized adaptive testing: A primer (Vol. 4, pp. 65-102). Hillsdale, NJ: Lawrence Erlbaum.

31.

Wang

Chang

H.-H.

(2011). Item selection in multidimensional computerized adaptive testing—Gaining information from different angles. Psychometrika, 76, 363-384.

32.

Warm

T. A.

(1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427-450.

33.

Weiss

D. J.

(1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6, 473-492.

34.

Yang

J. S.

Hansen

Cai

(2012). Characterizing sources of uncertainty in IRT scale scores. Educational and Psychological Measurement, 72, 264-290.

35.

Zhang

J. M.

Xie

M. G.

Song

X. L.

(2011). Investigating the impact of uncertainty about item parameters on ability estimation. Psychometrika, 76, 97-118.

36.

Zheng

(2014). New methods of online calibration for item bank replenishment (Unpublished doctoral dissertation). University of Illinois at Urbana–Champaign.

37.

Zhu

Stein

M. L.

(2005). Spatial sampling design for parameter estimation of the covariance function. Journal of Statistical Planning and Inference, 134, 583-603.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB

New Efficient and Practicable Adaptive Designs for Calibrating Items Online

Abstract

Keywords

Get full access to this article

References

Supplementary Material