Online calibration technique has been widely employed to calibrate new items due to its advantages. Method A is the simplest online calibration method and has attracted many attentions from researchers recently. However, a key assumption of Method A is that it treats person-parameter estimates (obtained by maximum likelihood estimation [MLE]) as their true values , thus the deviation of the estimated from their true values might yield inaccurate item calibration when the deviation is nonignorable. To improve the performance of Method A, a new method, MLE-LBCI-Method A, is proposed. This new method combines a modified Lord’s bias-correction method (named as maximum likelihood estimation-Lord’s bias-correction with iteration [MLE-LBCI]) with the original Method A in an effort to correct the deviation of which may adversely affect the item calibration precision. Two simulation studies were carried out to explore the performance of both MLE-LBCI and MLE-LBCI-Method A under several scenarios. Simulation results showed that MLE-LBCI could make a significant improvement over the ML ability estimates, and MLE-LBCI-Method A did outperform Method A in almost all experimental conditions.
BakerF. B.KimS.-H. (2004). Item response theory: Parameter estimation techniques (2nd ed.). New York, NY: Marcel Dekker.
2.
BanJ. C.HansonB. A.WangT. Y.YiQ.HarrisD. J. (2001). A comparative study of on-line pretest item—Calibration/scaling methods in computerized adaptive testing. Journal of Educational Measurement, 38, 191-212.
3.
BergerM. P. F. (1994). D-optimal sequential sampling designs for item response theory models. Journal of Educational Statistics, 19, 43-56.
4.
BergerM. P. F.KingJ.WongW. K. (2000). Minimax D-optimal designs for item response theory models. Psychometrika, 65, 377-390.
5.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. In LordF. M.NovickM. R. (Eds.), Statistical theories of mental test scores (pp. 379-479). Reading, MA: Addison-Wesley.
6.
CarrollR. J.RuppertD.StefanskiL. A.CrainiceanuC. M. (2006). Measurement error in nonlinear models: A modern perspective (2nd ed.). London, England: Chapman & Hall.
7.
ChangH.-H. (2012). Making computerized adaptive testing diagnostic tools for schools. In LissitzR. W.JiaoH. (Ed.), Computers and their impact on state assessments: Recent history and predictions for the future (pp. 195-226), Information Age.
ChangH.-H.QianJ. H.YingZ. L. (2001). a-stratified multistage computerized adaptive testing with b blocking. Applied Psychological Measurement, 25, 333-341.
10.
ChangH.-H.ZhangJ. M. (2002). Hypergeometric family and item overlap rates in computerized adaptive testing. Psychometrika, 67, 387-398.
11.
ChangY. C. I.LuH. Y. (2010). Online calibration via variable length computerized adaptive testing. Psychometrika, 75, 140-157.
12.
ChenP.WangC. (2016). A new online calibration method for multidimensional computerized adaptive testing. Psychometrika, 81, 674-701.
13.
ChenP.WangC.XinT.ChangH.-H. (2017). Developing new online calibration methods for multidimensional computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 70, 81-117.
14.
ChenP.XinT. (2014). Online calibration with cognitive diagnostic assessment. In ChengY.ChangH.-H. (Eds.), Advancing methodologies to support both summative and formative assessments (pp. 287-313). Charlotte, NC: Information Age.
15.
ChenP.XinT.WangC.ChangH.-H. (2012). Online calibration methods for the DINA model with independent attributes in CD-CAT. Psychometrika, 77, 201-222.
16.
ChenY.LiuJ. C.YingZ. L. (2015). Online item calibration for Q-matrix in CD-CAT. Applied Psychological Measurement, 39, 5-15.
17.
ChengY. (2008). Computerized adaptive testing—New developments and applications (Unpublished doctoral thesis). University of Illinois at Urbana-Champaign.
18.
ChengY.ChangH.-H. (2009). The maximum priority index method for severely constrained item selection in computerized adaptive testing. British Journal of Mathematical & Statistical Psychology, 62, 369-383.
19.
FirthD. (1993). Bias reduction of maximum likelihood estimates. Biometrika, 80, 27-38.
20.
HanK. T. (2016). Maximum likelihood score estimation method with fences for short-length tests and computerized adaptive tests. Applied Psychological Measurement, 40, 289-301.
21.
HechtM.WeirichS.SiegleT.FreyA. (2015). Modeling booklet effects for nonequivalent group designs in large-scale assessment. Educational and Psychological Measurement, 75, 568-584.
22.
JonesD. H.JinZ. Y. (1994). Optimal sequential designs for on-line item estimation. Psychometrika, 59, 59-75.
23.
KangH.-A. (2016). Likelihood estimation for jointly analyzing item responses and response times (Unpublished doctoral thesis). University of Illinois at Urbana-Champaign.
24.
KimS. (2006). A comparative study of IRT fixed parameter calibration methods. Journal of Educational Measurement, 43, 355-381.
25.
LienD.-H. D. (1985). Moments of truncated bivariate log-normal distributions. Economics Letters, 19, 243-247.
26.
LordF. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
27.
LordF. M. (1983). Unbiased estimators of ability parameters, of their variance, and of their Parallel-Forms reliability. Psychometrika, 48, 233-245.
28.
MakranskyG. (2009, June). An automatic online calibration design in adaptive testing. Paper presented at the GMAC conference on Computerized Adaptive Testing, McLean, VA.
29.
MurakiE. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176.
30.
ParshallC. G. (1998, September). Item development and pretesting in a computer-based testing environment. Paper presented at the colloquium Computer-Based Testing: Building the Foundation for Future Assessments, Philadelphia, PA.
31.
PattonJ. M.ChengY.YuanK. H.DiaoQ. (2013). The influence of item calibration error on variable-length computerized adaptive testing. Applied Psychological Measurement, 37, 24-40.
32.
ReyD.NeuhäuserM. (2011). Wilcoxon Signed-Rank Test. Berlin, Germany: Springer Berlin Heidelberg.
33.
RosnerB.GlynnR. J.LeeM. L. T. (2006). The Wilcoxon signed rank test for paired comparisons of clustered data. Biometrics, 62, 185-192.
34.
StockingM. L. (1988). Scale drift in on-line calibration (Research Report No. 88-28). Princeton, NJ: Educational Testing Service.
35.
van der LindenW. J.GlasC. A. W. (2000). Capitalization on item calibration error in adaptive testing. Applied Measurement in Education, 13, 35-53.
36.
van der LindenW. J.Klein EntinkR. H.FoxJ. P. (2010). IRT parameter estimation with response times as collateral information. Applied Psychological Measurement, 3, 327-347.
37.
van der LindenW. J.RenH. (2015). Optimal Bayesian adaptive design for test-item calibration. Psychometrika, 80, 263-288.
38.
WainerH.MislevyR. J. (1990). Item response theory, item calibration, and proficiency estimation. In WainerH. (Ed.), Computerized adaptive testing: A primer (pp. 65-102). Hillsdale, NJ: Lawrence Erlbaum.
39.
WangC.ChangH.-H.BoughtonK. A. (2013). Deriving stopping rules for multidimensional computerized adaptive testing. Applied Psychological Measurement, 37, 99-122.
40.
WangC.XuG. (2015). A mixture hierarchical model for response times and response accuracy. British Journal of Mathematical and Statistical Psychology, 68, 456-477.
41.
WangT.HansonB. A. (2005). Development and calibration of an item response model that incorporates response time. Applied Psychological Measurement, 29, 323-339.
42.
WangT.HansonB. A.LauC. M. A. (1999). Reducing bias in CAT trait estimation: A comparison of approaches. Applied Psychological Measurement, 23, 263-278.
43.
WarmT. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427-450.
44.
WeissD. J. (1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6, 473-492.
45.
ZhangJ. M.XieM. G.SongX. L.LuT. (2011). Investigating the impact of uncertainty about item parameters on ability estimation. Psychometrika, 76, 97-118.
46.
ZhengY. (2014). New methods of online calibration for item bank replenishment (Unpublished doctoral thesis). University of Illinois at Urbana–Champaign.
47.
ZhengY. (2016). Online calibration of polytomous items under the generalized partial credit model. Applied Psychological Measurement, 40, 434-450.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
