With the widespread use of computers in modern assessment, online calibration has become increasingly popular as a way of replenishing an item pool. The present study discusses online calibration strategies for a joint model of responses and response times. The study proposes likelihood inference methods for item paramter estimation and evaluates their performance along with optimal sampling procedures. An extensive simulation study indicates that the proposed online calibration strategies perform well with relatively small samples (e.g., 500∼800 examinees). The analysis of estimated parameters suggests that response time information can be used to improve the recovery of the response model parameters. Among a number of sampling methods investigated, A-optimal sampling was found most advantageous when the item parameters were weakly correlated. When the parameters were strongly correlated, D-optimal sampling tended to achieve the most accurate parameter recovery. The study provides guidelines for deciding sampling design under a specific goal of online calibration given the characteristics of field-testing items.
BanJ.-C.HansonB. A.WangT.YiQ.HarrisD. J. (2001). A comparative study of on-line pretest item-calibration/scaling methods in computerized adaptive testing. Journal of Educational Measurement, 38, 191–212.
2.
BanJ.-C.HansonB. A.YiQ.HarrisD. J. (2002). Data sparseness and on-line pretest item calibration-scaling methods in CAT. Journal of Educational Measurement, 39, 207–218.
3.
BergerM. P. F. (1991). On the efficiency of IRT models when applied to different sampling designs. Applied Psychological Measurement, 15, 293–306.
4.
BergerM. P. F. (1992). Sequential sampling designs for the two-parameter item response theory model. Psychometrika, 57, 521–538.
5.
BergerM. P. F. (1994). D-optimal sequential sampling designs for item response theory models. Journal of Educational Statistics, 19, 43–56.
6.
BirnbaumA. (1968). Theories of mental test scores. In LordF. M.NovickM. R. (Eds.), Some latent trait models and their use in inferring an examinee’s ability (pp. 397–479). Reading, MA: Addison-Wesley.
7.
BockR. D.AitkinM. (1981). Marginal maximum likelihood estimation of item parameters: An application of an em-algorithm. Psychometrika, 46, 443–459.
8.
BolsinovaM.De BoeckP. E.TijmstraJ. (2018). Modelling conditional dependence between response time and accuracy. Psychometrika, 82, 1126–1148.
9.
BuyskeS. (2005). Optimal design in educational testing. In BergerM. P. F.WongW. K. (Eds.), Applied optimal designs (pp. 1–19). New York, NY: John Wiley.
ChenP. (2017). A comparative study of online item calibration methods in multidimensional computerized adaptive testing. Journal of Educational and Behavioral Statistics, 42, 559–590.
12.
ChenP.WangC. (2016). A new online calibration method for multidimensional computerized adaptive testing. Psychometrika, 81, 674–701.
13.
ChenP.WangC.XinT.ChangH.-H. (2017). Developing new online calibration methods for multidimensional computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 70, 81–117.
14.
ChenP.XinT.WangC.ChangH.-H. (2012). Online calibration methods for the DINA model with independent attributes in CD-CAT. Psychometrika, 77, 201–222.
15.
ChenW.-H.ThissenD. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265–289.
16.
DempsterA. P.LairdN. M.RubinD. B. (1977). Maximum likelihood from incomplete data via the EM algorithm (with discussion). Journal of the Royal Statistical Society, Series B, 39, 1–38.
17.
FanZ.WangC.ChangH.-H.DouglasJ. (2012). Utilizing response time distributions for item selection in CAT. Journal of Educational and Behavioral Statistics, 37, 655–670.
18.
FoxJ.-P.Klein EntinkR. H.van der LindenW. J. (2007). Modeling of responses and response times with the package CIRT. Journal of Statistical Software, 20, 1–14.
19.
FoxJ.-P.MariantiS. (2017). Person-fit statistics for joint models for accuracy and speed. Journal of Educational Measurement, 54, 243–262.
20.
GlasC. A. W.Suárez-FalcónJ. C. (2003). A comparison of item-fit statistics for the three parameter logistic model. Applied Psychological Measurement, 27, 87–106.
21.
GlasC. A. W.van der LindenW. J. (2010). Marginal likelihood inference for a model for item responses and response times. British Journal of Mathematical and Statistical Psychology, 63, 603–626.
22.
HoutsC. R.EdwardsM. C. (2013). The performance of local dependence measures with psychological data. Applied Psychological Measurement, 37, 541–562.
KangH.-A. (2016). Likelihood estimation for jointly analyzing item responses and response times (Unpublished doctoral dissertation). University of Illinois at Urbana-Champaign, Champaign, IL.
25.
KangH.-A. (2017). Penalized partial likelihood inference of proportional hazards latent trait models. British Journal of Mathematical and Statistical Psychology, 70, 187–208.
26.
Klein EntinkR. H.FoxJ.-P.van der LindenW. J. (2009). A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika, 74, 21–48.
27.
Klein EntinkR. H.KuhnJ.-T.HornkeL. F.FoxJ.-P. (2009). Evaluating cognitive theory: A joint modeling approach using responses and response times. Psychological Methods, 14, 54–75.
28.
Klein EntinkR. H.van der LindenW. J.FoxJ.-P. (2009). A Box-Cox normal model for response times. British Journal of Mathematical and Statistical Psychology, 62, 621–640.
29.
LiuY.Maydeu-OlivaresA. (2012). Local dependence diagnostics in IRT modeling of binary data. Educational Psychological Measurement, 73, 254–274.
30.
MariantiS.FoxJ.-P.AvetisyanM.VeldkampB. P.TijmstraJ. (2014). Testing for aberrant behavior in response time modeling. Journal of Educational and Behavioral Statistics, 39, 426–451.
31.
MislevyR. J.StockingM. L. (1989). A consumers guide to LOGIST and BILOG. Applied Psychological Measurement, 13, 57–75.
32.
MulderJ.van der LindenW. J. (2009). Multidimensional adaptive testing with optimal design criteria for item selection. Psychometrika, 74, 273–296.
33.
RangerJ. (2013). A note on the hierarchical model for responses and response times in tests of van der Linden (2007). Psychometrika, 78, 538–544.
34.
RangerJ.KuhnJ.-T. (2012). A flexible latent trait model for response times in tests. Psychometrika, 77, 31–47.
35.
RenH.van der LindenW. J.DiaoQ. (2017). Continuous online item calibration: Parameter recovery and item utilization. Psychometrika, 82, 498–522.
36.
SegallD. O. (2003, April). Calibrating CAT pools and online pretest items using MCMC methods. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.
37.
SilveyS. D. (1980). Optimal design. London, England: Chapman & Hall.
38.
SteinC. M. (1981). Estimation of the mean of a multivariate normal distribution. The Annals of Statistics, 9, 1135–1151.
39.
StockingM. L. (1988). Scale drift in on-line calibration(ETS Research Report Nos. 88–28). Princeton, NJ: ETS, 12.
40.
van der LindenW. J. (1988). Optimizing incomplete sampling designs for item response model parameters(Research Report No. 88-5). Enschede, the Netherlands: University of Twente.
41.
van der LindenW. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31, 181–204.
42.
van der LindenW. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287–308.
43.
van der LindenW. J. (2008). Using response times for item selection in adaptive testing. Journal of Educational and Behavioral Statistics, 33, 5–20.
44.
van der LindenW. J. (2009). Predictive control of speededness in adaptive testing. Applied Psychological Measurement, 33, 24–41.
45.
van der LindenW. J. (2011). Test design and speededness. Journal of Educational Measurement, 48, 44–60.
46.
van der LindenW. J.GlasC. A. W. (2010). Statistical tests of conditional independence between responses and/or response times on test items. Psychometrika, 75, 120–139.
47.
van der LindenW. J.GuoF. (2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73, 365–384.
48.
van der LindenW. J.Klein EntinkR. H.FoxJ.-P. (2010). IRT parameter estimation with response times as collateral information. Applied Psychological Measurement, 34, 327–347.
49.
van der LindenW. J.RenH. (2014). Optimal Bayesian adaptive design for test-item calibration. Psychometrika, 80, 263–288.
50.
van der LindenW. J.ScramsD. J.SchnipkeD. L. (1999). Using response-time constraints to control for differential speededness in computerized adaptive testing. Applied Psychological Measurement, 23, 195–210.
51.
van der LindenW. J.van Krimpen-StoopE. M. L. A. (2003). Using response times to detect aberrant response patterns in computerized adaptive testing. Psychometrika, 68, 251–265.
52.
WangC.FanZ.ChangH.-H.DouglasJ. (2013). A semiparametric model for jointly analyzing response times and accuracy in computerized testing. Journal of Educational and Behavioral Statistics, 38, 381–417.
53.
YenW. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125–145.
54.
ZhengY. (2016). Online calibration of polytomous items under the generalized partial credit model. Applied Psychological Measurement, 40, 434–450.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
