Sage Journals: Discover world-class research

Abstract

In many educational assessments, items are reused in different administrations throughout the life of the assessments. Ideally, a reused item should perform relatively similarly over time. In reality, an item may become easier with exposure, especially when item preknowledge has occurred. This article presents a novel cumulative sum procedure for detecting item preknowledge in continuous testing where data for each reused item may be obtained from small and varying sample sizes across administrations. Its performance is evaluated with simulations and analytical work. The approach is effective in detecting item preknowledge quickly with group size at least 10 and is easy to implement with varying item parameters. In addition, it is robust to the ability estimation error introduced in the simulations.

Keywords

test security item preknowledge item residual item response theory quality assurance change point

Get full access to this article

View all access options for this article.

References

Armstrong

R. D.

Shi

(2009). A parametric cumulative sum statistic for person fit. Applied Psychological Measurement, 33, 391–410.

Chen

(2019). Compound sequential change point detection in multiple data streams. arXiv preprint arXiv:1909.05903.

Choe

E. M.

Zhang

Chang

H. H.

(2018). Sequential detection of compromised items using response times in computerized adaptive testing. Psychometrika, 83(3), 650–673.

Donoghue

J. R.

Isham

S. P.

(1998). A comparison of procedures to detect item parameter drift. Applied Psychological Measurement, 22(1), 33–51.

Glas

C. A. W.

(2010). Item parameter estimation and item fit analysis. In van der Linden

W. J.

Glas

C. A. W.

(Eds.), Elements of adaptive testing (pp. 269–288). New York, NY: Springer.

Guo

Robin

Dorans

(2017). Detecting item drift in large-scale testing. Journal of Educational Measurement, 54(3), 265–284.

Haberman

S. J.

Sinharay

Chon

K. H.

(2013). Assessing item fit for unidimensional item response theory models using residuals from estimated item response functions. Psychometrika, 78, 417–440.

Hambleton

R. K.

Swaminathan

Rogers

H. J.

(1991). Fundamentals of item response theory. Newbury Park, CA: Sage.

Hawkins

D. M.

Olwell

P. H.

(1998). Cumulative sum charts and charting for quality improvement. New York, NY: Springer Verlag.

10.

Holland

Thayer

(1988). Differential item functioning and the Mantel-Haenszel procedure. In Wainer

Braun

(Eds.), Test validity (pp. 129–145). Hillsdale, NJ: Lawrence Erlbaum.

11.

Knoth

(2016). R package “spc”: Statistical process control—Collection of some useful functions (Version 0.5.3). Available from https://cran.r-project.org/web/packages/spc/.

12.

Lai

T. L.

(2001). Sequential analysis: Some classical problems and new challenges. Statistica Sinica, 11, 303–351.

13.

Lee

Y.-H.

Lewis

von Davier

A. A.

(2014). Monitoring the quality and security of multistage tests. In Yan

von Davier

A. A.

Lewis

(Eds.), Computerized multistage testing: Theory and applications (pp. 285–300). Boca Raton, FL: CRC Press.

14.

Lee

Y.-H.

von Davier

A. A.

(2013). Monitoring scale scores over time via quality control charts, model-based approaches, and time series techniques. Psychometrika, 78(3), 557–575. https://doi.org/10.1007/s11336-013-9317-5

15.

Lord

F. M.

Novick

M. R.

(1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

16.

Meijer

R. R.

(2002). Outlier detection in high-stakes certification testing. Journal of Educational Measurement, 39, 219–233.

17.

Montgomery

D. C.

(2009). Introduction to statistical quality control (6th ed.). Hoboken, NJ: John Wiley & Sons.

18.

Moustakides

G. V.

(1986). Optimal stopping for detecting changes in distribution. Annals of Statistics, 14, 1379–1387.

19.

Page

E. S.

(1954). Continuous inspection schemes. Biometrika, 41, 100–115.

20.

Robin

Steffen

Liang

(2014). GRE® revised General Test (Admission Test). In Yan

von Davier

A. A.

Lewis

(Eds.), Computerized multistage testing: Theory and applications (pp. 325–342). Boca Raton, FL: CRC Press.

21.

Siegmund

(1985). Sequential analysis: Tests and confidence intervals. New York, NY: Springer-Verlag.

22.

Sinharay

(2018). Detecting fraudulent erasures at an aggregate level. Journal of Educational and Behavioral Statistics, 43, 286–315.

23.

Van Krimpen-Stoop

E. M. L. A.

Meijer

R. R.

(2001). CUSUM-based person-fit statistics for adaptive testing. Journal of Educational and Behavioral Statistics, 26, 199–218.

24.

Veerkamp

W. J. J.

Glas

C. A. W.

(2000). Detection of known items in adaptive testing with a statistical quality control method. Journal of Educational and Behavioral Statistics, 25, 373–389.

25.

Wald

(1945). Sequential tests of statistical hypotheses. Annals of Mathematical Statistics, 16, 117–186.

26.

Fellouris

Culpepper

S. A.

Douglas

(2016). Sequential detection of learning in cognitive diagnosis. British Journal of Mathematical and Statistical Psychology, 69, 139–158.

27.

Zhang

(2014). A sequential procedure for detecting compromised items in the item pool of a CAT system. Applied Psychological Measurement, 38, 87–104.

28.

Zhang

(2016). Monitoring items in real time to enhance CAT security. Journal of Educational Measurement, 53, 131–151.

29.

Zieky

(2011). The origins of procedures for using differential item functioning statistics at Educational Testing Service. In Dorans

N. J.

Sinharay

(Eds.), Looking Back: Proceedings of a Conference in Honor of Paul W. Holland, Lecture Notes in Statistics 202 (pp. 115–127). New York, NY: Springer.

30.

Zwick

(2012). A review of ETS differential item functioning assessment procedures: Flagging rules, minimum sample size requirements, and criterion refinement (Research Report No. RR-12-08). Princeton, NJ: Educational Testing Service.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.79 MB

Monitoring Item Performance With CUSUM Statistics in Continuous Testing

Abstract

Keywords

Get full access to this article

References

Supplementary Material