Detecting Compromised Items Using Information From Secure Items

Abstract

In continuous testing programs, some items are repeatedly used across test administrations, and statistical methods are often used to evaluate whether items become compromised due to examinees’ preknowledge. In this study, we proposed a residual method to detect compromised items when a test can be partitioned into two subsets of items: secure items and possibly compromised items. We derived the standard error of the residual statistic by taking the sampling error in both ability and item parameter estimate into account. The simulation results suggest that the Type I error is close to the nominal level when both sources of error are adjusted, and item parameter error can be ignored only when the item calibration sample size is much larger than the evaluation sample size. We also investigated the performance of the residual method when not using information from secure items in both simulation and real data analyses.