An informational distance/divergence-based approach is proposed to detect the presence of parameter drift in multidimensional computerized adaptive testing (MCAT). The study presents significance testing procedures for identifying changes in multidimensional item response functions (MIRFs) over time based on informational distance/divergence measures that capture the discrepancy between two probability functions. To approximate the MIRFs from the observed response data, the k-nearest neighbors algorithm is used with the random search method. A simulation study suggests that the distance/divergence-based drift measures perform effectively in identifying the instances of parameter drift in MCAT. They showed moderate power with small samples of 500 examinees and excellent power when the sample size was as large as 1,000. The proposed drift measures also adequately controlled for Type I error at the nominal level under the null hypothesis.
AckermanT. A. (1992). A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. Journal of Educational Measurement, 29, 67-91.
2.
BergstraJ.BengioY. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13, 281-305.
3.
BhattacharyyaA. (1943). On a measure of divergence between two statistical populations defined by their probability distributions. Bulletin of Calcutta Mathematical Society, 35, 99-109.
4.
BloxomB. M.ValeC. D. (1987, June). Multidimensional adaptive testing: A procedure for sequential estimation of the posterior centroid and dispersion of theta. Paper presented at annual meeting of the Psychometric Society, Montreal, Quebec, Canada.
5.
BockR. D.MurakiE.PfeiffenbergerW. (1988). Item pool maintenance in the presence of item parameter drift. Journal of Educational Measurement, 25, 275-285.
6.
BoltD. M. (2002). A Monte Carlo comparison of parametric and nonparametric polytomous DIF detection methods. Applied Measurement in Education, 15, 113-141.
7.
ChanK.-Y.DrasgowF.SawinL. L. (1999). What is the shelf life of a test? The effect of time on the psychometrics of a Cognitive Ability Test battery. Journal of Educational Measurement, 84, 610-619.
ChangH.-H.YingZ. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213-229.
10.
ClarkA. K. (2014, April). Parameter drift methodology and operational testing application. Poster presented at the annual meeting of the National Council on Measurement in Education, Philadelphia, PA.
11.
CohenJ. (1992). A power primer. Psychological Bulletin, 112, 155-159.
12.
CookL. L.EignorD. R.TaftH. L. (1988). A comparative study of the effects of recency of instruction on the stability of IRT and conventional item parameter estimates. Journal of Educational Measurement, 25, 31-45.
13.
DeMarsC. E. (2004). Detection of item parameter drift over multiple test administrations. Applied Measurement in Education, 17, 265-300.
14.
DonoghueJ. R.IshamS. P. (1998). A comparison of procedures to detect item parameter drift. Applied Psychological Measurement, 22, 33-51.
15.
GlasC. A. W. (2000). Item calibration and parameter drift. In van der LindenW. J.GlasC. A. W. (Eds.), Computerized adaptive testing: Theory and practice (pp. 183-199). Norwell, MA: Kluwer Academic.
16.
GoldsteinH. (1983). Measuring changes in educational attainment over time: Problems and possibilities. Journal of Educational Measurement, 20, 369-377.
17.
GuoR.ZhengY.ChangH.-H. (2015). A stepwise test characteristic curve method to detect item parameter drift. Journal of Educational Measurement, 52, 280-300.
18.
KimS.-H.CohenA. S. (1991). A comparison of two area measures for detecting differential item functioning. Applied Psychological Measurement, 15, 269-278.
19.
KimS.-H.CohenA. S.ParkT. H. (1995). Detection of differential item functioning in multiple groups. Journal of Educational Measurement, 32, 261-276.
20.
KullbackS.LeiblerR. A. (1951). On information and sufficiency. The Annals of Mathematical Statistics, 22, 79-86.
21.
LiX. (2008). Multidimensionality and item parameter drift: An investigation of linking items in a large-scale certification test (Unpublished doctoral dissertation). Michigan State University, East Lansing.
22.
LordF. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
23.
RajuN. S. (1988). The area between two item characteristic curves. Psychometrika, 53, 495-502.
24.
RaoC. R. (1982). Diversity and dissimilarity coefficients: A unified approach. Theoretical Population Biology, 21, 24-43.
25.
ReckaseM. D. (1997). The past and future of multidimensional item response theory. Applied Psychological Measurement, 21, 25-36.
26.
SegallD. O. (1996). Multidimensional adaptive testing. Psychometrika, 61, 331-354.
27.
SykesR. C.ItoK. (1993, April). Item parameter drift in IRT-based licensure examinations. Paper presented at the annual meeting of the National Council on Measurement in Education, Atlanta, GA.
28.
TamS. S. (1992). A comparison of methods for adaptive estimation of a multidimensional trait (Unpublished doctoral dissertation). Columbia University, New York, NY.
29.
VeerkampW. J. J.GlasC. A. W. (2000). Detection of known items in adaptive testing with a statistical quality control method. Journal of Educational and Behavioral Statistics, 25, 373-389.
30.
WollackJ. A.CohenA. S.WellsC. S. (2003). A method for maintaining scale stability in the presence of test speededness. Journal of Educational Measurement, 40, 307-330.
31.
ZhangJ. (2014). A sequential procedure for detecting compromised items in the item pool of a CAT system. Applied Psychological Measurement, 38, 87-104.
32.
ZhangJ.StoutW. (1999). The theoretical DETECT index of dimensionality and its application to approximate simple structure. Psychometrika, 64, 213-249.
