IRT Residual-Based Approach to Detecting Item Parameter Drift in CAT

Abstract

This study proposes a residual-based item parameter drift (RIPD) detection framework for computerized adaptive testing (CAT). Introducing three statistics (RIPD _R , RIPD _S , and RIPD _RS ), the framework detects drift by comparing focal group responses to synthetic drift-free reference responses generated without item recalibration. Three simulation studies evaluated the RIPD framework under curriculum-related drift and item compromise scenarios. Results indicated that RIPD _R and RIPD _RS consistently outperformed the pseudo-count D² method, demonstrating robust false positive control and detection power, particularly with sufficiently large reference groups and a longer test length. While RIPD _S was critical for detecting nonuniform drift, it exhibited inflated false positive rates under severe contamination. Overall, the findings support RIPD as a practical and scalable method for post-administration IPD detection in CAT.

Keywords

item parameter drift computerized adaptive test item response theory differential item functioning

Get full access to this article

View all access options for this article.

References

Billingsley

(1995). Probability and measure (3rd ed.). Wiley.

Bock

R. D.

Muraki

Pfeiffenberger

(1988). Item pool maintenance in the presence of item parameter drift. Journal of Educational Measurement, 25(4), 275–285.

Cappaert

K. J.

Wen

Chang

Y. F.

(2018). Evaluating CAT-adjusted approaches for suspected item parameter drift detection. Measurement: Interdisciplinary Research and Perspectives, 16(4), 226–238.

Cohen

A. S.

Kim

S. H.

(1993). A comparison of Lord’s χ² and Raju’s area measures in detection of DIF. Applied Psychological Measurement, 17(1), 39–52.

Cook

L. L.

Eignor

D. R.

Taft

H. L.

(1988). A comparative study of the effects of recency of instruction on the stability of IRT and conventional item parameter estimates. Journal of Educational Measurement, 25, 31–45.

Efron

Tibshirani

R. J.

(1993). An introduction to the bootstrap. Chapman & Hall/CRC.

Glas

C. A.

(2010). Item parameter estimation and item fit analysis. In van der Linden

W. J.

Glas

C. A.

(Eds.), Elements of adaptive testing (pp. 269–288). Springer.

Han

K. T.

(2012). An efficiency balanced information criterion for item selection in computerized adaptive testing. Journal of Educational Measurement, 49(3), 225–246.

Han

K. T.

Guo

(2011). Potential impact of item parameter drift due to practice and curriculum change on item calibration in computerized adaptive testing (GMAC Research Reports, RR-11-02).

10.

Holland

P. W.

Thayer

D. T.

(1988). Differential item performance and the Mantel–Haenszel procedure. In Wainer

Braun

H. I.

(Eds.), Test validity (pp. 129–145). Lawrence Erlbaum Associates.

11.

Ito

Sykes

R. C.

(1994). The effect of restricting ability distributions in the estimation of item difficulties: Implications for a CAT implementation [Paper presentation]. Annual conference of the National Council on Measurement in Education (NCME).

12.

Kang

H. A.

Chang

H. H.

(2016). Parameter drift detection in multidimensional computerized adaptive testing based on informational distance/divergence measures. Applied Psychological Measurement, 40(7), 534–550.

13.

Lee

Qian

(2022). Hybrid threshold-based sequential procedures for detecting compromised items in a computerized adaptive testing licensure exam. Educational and Psychological Measurement, 82(4), 782–810.

14.

Lim

Choe

E. M.

(2023). Detecting differential item functioning in CAT using IRT residual DIF approach. Journal of Educational Measurement, 60(4), 626–650.

15.

Lim

Choe

E. M.

Han

(2022). A residual-based differential item functioning detection framework in item response theory. Journal of Educational Measurement, 59(1), 80–104.

16.

Lim

Han

(2022, July). Factors affecting item calibration using adaptively administered test data [Poster presentation]. Annual International Meeting of the Psychometric Society (IMPS).

17.

Lim

Malatesta

(2024, April). Application of the IRT residual DIF framework to ordinally scored polytomous items [Paper presentation]. Annual conference of the National Council on Measurement in Education (NCME).

18.

Lord

F. M.

(1980). Applications of item response theory to practical testing problems. Lawrence Erlbaum Associates.

19.

Nevitt

Hancock

G. R.

(2001). Performance of bootstrapping approaches to model test statistics and parameter standard error estimation in structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 8(3), 353–377.

20.

R Core Team. (2025). R: A language and environment for statistical computing. R Foundation for Statistical Computing.

21.

Rosa

Kocianova

(2023). Accuracy of critical gap estimates. Alexandria Engineering Journal, 75, 565–576.

22.

Shealy

Stout

(1993). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58(2), 159–194.

23.

Sinharay

(2017). Detection of item preknowledge using likelihood ratio test and score test. Journal of Educational and Behavioral Statistics, 42(1), 46–68.

24.

Stone

C. A.

(2000). Monte Carlo based null distribution for an alternative goodness-of-fit test statistic in IRT models. Journal of Educational Measurement, 37(1), 58–75.

25.

Veerkamp

W. J.

Glas

C. A.

(2000). Detection of known items in adaptive testing with a statistical quality control method. Journal of Educational and Behavioral Statistics, 25(4), 373–389.

26.

Wells

C. S.

Hambleton

R. K.

Kirkpatrick

Meng

(2014). An examination of two procedures for identifying consequential item parameter drift. Applied Measurement in Education, 27(3), 214–231.

27.

Zhang

(2014). A sequential procedure for detecting compromised items in the item pool of a CAT system. Applied Psychological Measurement, 38(2), 87–104.

28.

Zhang

(2016). Monitoring items in real time to enhance CAT security. Journal of Educational Measurement, 53(2), 131–151.

29.

Zwick

Thayer

D. T.

Wingersky

(1994). A simulation study of methods for assessing differential item functioning in computerized adaptive tests. Applied Psychological Measurement, 18(2), 121–140.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.13 MB