Using Item Scores and Response Times to Detect Item Compromise in Computerized Adaptive Testing

Abstract

Sequential procedures have been shown to be effective methods for real-time detection of compromised items in computerized adaptive testing. In this study, we propose three item response theory-based sequential procedures that involve the use of item scores and response times (RTs). The first procedure requires that either the score-based statistic or the RT-based statistic be extreme, the second procedure requires that both the score-based statistic and the RT-based statistic be extreme, and the third procedure requires that a combined score and RT-based statistic be extreme. Results suggest that the third procedure is the most promising, providing a reasonable balance between the false-positive rate and the true-positive rate while also producing relatively short lag times across a wide range of simulation conditions.

Keywords

computerized adaptive testing item compromise item response theory response time test security

Get full access to this article

View all access options for this article.

References

Belov

D. I.

(2014). Detecting item preknowledge in computerized adaptive testing using information theory and combinatorial optimization. Journal of Computerized Adaptive Testing, 2(3), 37–58.

Choe

E. M.

Zhang

Chang

H.-H.

(2018). Sequential detection of compromised items using response times in computerized adaptive testing. Psychometrika, 83(3), 650–673. https://doi.org/10.1007/s11336-017-9596-3

Zhang

(2025). Detecting compromised items with response times using a Bayesian change-point approach. Journal of Educational and Behavioral Statistics, 50(2), 296–330. https://doi.org/10.3102/10769986241290713

Zhang

Chang

H.-H.

(2023). Compromised item detection: A Bayesian change-point perspective. British Journal of Mathematical and Statistical Psychology, 76(1), 131–153. https://doi.org/10.1111/bmsp.12286

Gorney

Lee

Chen

(2025). A score-based method for detecting item compromise and preknowledge in computerized adaptive testing. Journal of Computerized Adaptive Testing, 12(2), 123–136. https://doi.org/10.7333/2506-1202123

Gorney

Wollack

J. A.

(2025). Using response times in answer similarity analysis. Journal of Educational and Behavioral Statistics, 50(3), 449–470. https://doi.org/10.3102/10769986241248770

Kang

H.-A.

(2023). Sequential generalized likelihood ratio tests for online item monitoring. Psychometrika, 88(2), 672–696. https://doi.org/10.1007/s11336-022-09871-9

Lee

Y.-H.

Lewis

(2021). Monitoring item performance with CUSUM statistics in continuous testing. Journal of Educational and Behavioral Statistics, 46(5), 611–648. https://doi.org/10.3102/1076998621994563

Lee

Y.-H.

Lewis

von Davier

A. A.

(2014). Monitoring the quality and security of multistage tests. In Yan

von Davier

A. A.

Lewis

(Eds.), Computerized multistage testing: Theory and applications (pp. 285–300). CRC Press.

10.

Pan

Sinharay

Livne

Wollack

J. A.

(2022). A machine learning approach for detecting item compromise and preknowledge in computerized adaptive testing. Psychological Test and Assessment Modeling, 64(4), 385–424.

11.

Sinharay

Johnson

M. S.

(2020). The use of item scores and response times to detect examinees who may have benefited from item preknowledge. British Journal of Mathematical and Statistical Psychology, 73(3), 397–419. https://doi.org/10.1111/bmsp.12187

12.

Toton

S. L.

Maynes

D. D.

(2019). Detecting examinees with pre-knowledge in experimental data using conditional scaling of response times. Frontiers in Education, 4, Article 49. https://doi.org/10.3389/feduc.2019.00049

13.

van der Linden

W. J

. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204. https://doi.org/10.3102/10769986031002181

14.

van der Linden

W. J

. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308. https://doi.org/10.1007/s11336-006-1478-z

15.

van der Linden

W. J

. (2022). Two statistical tests for the detection of item compromise. Journal of Educational and Behavioral Statistics, 47(4), 485–504. https://doi.org/10.3102/10769986221094789

16.

van der Linden

W. J.

Belov

D. I

. (2023). A statistical test for the detection of item compromise combining responses and response times. Journal of Educational Measurement, 60(2), 235–254. https://doi.org/10.1111/jedm.12346

17.

Veerkamp

W. J. J.

Glas

C. A. W.

(2000). Detection of known items in adaptive testing with a statistical quality control method. Journal of Educational and Behavioral Statistics, 25(4), 373–389. https://doi.org/10.3102/10769986025004373

18.

Wang

Liu

(2020). Detecting compromised items using information from secure items. Journal of Educational and Behavioral Statistics, 45(6), 667–689. https://doi.org/10.3102/1076998620912549

19.

Zhang

(2014). A sequential procedure for detecting compromised items in the item pool of a CAT system. Applied Psychological Measurement, 38(2), 87–104. https://doi.org/10.1177/0146621613510062

20.

Zhang

(2016). Monitoring items in real time to enhance CAT security. Journal of Educational Measurement, 53(2), 131–151. https://doi.org/10.1111/jedm.12104

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.61 MB