This article develops a new cumulative sum (CUSUM) statistic to detect aberrant item response behavior. Shifts in behavior are modeled with quadratic functions and a series of likelihood ratio tests are used to detect aberrancy. The new CUSUM statistic is compared against another CUSUM approach as well as traditional person-fit statistics. A simulation study demonstrates the advantage of the proposed method. Also, the person-fit methods are applied to real response data from the administration of a high-stakes exam. The use of CUSUM charts to help visually identify types of aberrant behavior is demonstrated.
Armstrong, R.D., Stoumbos, Z.G., Kung, M.T., & Shi, M. (2007). On the performance of lz statistic in personfit measurement. Practical Assessment, Research & Evaluation , 12. Retrieved December 14, 2007 , from http://pareonline.net/getvn.asp?v=12&n=15
2.
Armstrong, R.D., & Shi, M. (in press). Model-free CUSUM methods for person fit. Journal of Educational Measurement.
3.
Bradlow, E.T., Weiss, R.E., & Cho, M. (1998). Bayesian identification of outliers in computerized adaptive tests. Journal of the American Statistical Association , 93, 910-919.
4.
Drasgow, F., Levine, M.V., & McLaughlin, M.E. (1987). Detecting inappropriate test scores with optimal and practical appropriateness indices. Applied Psychological Measurement, 11, 59-79.
5.
Drasgow, F., Levine, M.V., & McLaughlin, M.E. (1991). Appropriateness measurement for some multidimensional test batteries. Applied Psychological Measurement, 15, 171-191.
6.
Drasgow, F., Levine, M.V., & Williams, E.A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67-86.
7.
Drasgow, F., Levine, M.V., & Zickar, M.J. (1996). Optimal identification of mismeasured individuals . Applied Measurement in Education, 9, 47-64.
8.
Gan, F.F. (1993). An optimal design of CUSUM control charts for binomial counts. Journal of Applied Statistics, 20, 445-460.
9.
Karabatsos, G. (2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education, 16, 277-298.
10.
Klauer, K.C. (1991). An exact and optimal standardized person fit test for assessing consistency with the Rasch model. Psychometrika , 54, 213-228.
11.
Lehmann, E.L. (1986). Testing statistical hypotheses (2nd ed.). New York: John Wiley.
Levine, M.V., & Rubin, D.B. (1979). Measuring the appropriateness of multiple-choice test scores. Journal of Educational Statistics, 4, 269-290.
14.
Lord, F. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
15.
Meijer, R.R. (2002). Outlier detection in high-stakes certification testing. Journal of Educational Measurement, 39, 219-233.
16.
Meijer, R.R. (2003). Diagnosing item score patterns on a test using item response theory-based person-fit statistics. Psychological Methods, 8, 72-87.
17.
Meijer, R.R., Egberink, I.J.L., Emons, W.H.M., & Sijtsma, K. (2008). Detection and validation of unscalable item score patterns using item response theory: An illustration with Harter's self-perception profile for children. Journal of Personality Assessment, 90, 227-238.
18.
Meijer, R.R., Muijtjens, A.M.M., & van der Vleuten, C.P.M. (1996). Nonparametric person-fit research: Some theoretical issues and an empirical example. Applied Measurement in Education, 9, 77-89.
19.
Meijer, R.R., & Sijtsma, K. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107-135.
20.
Montgomery, D.C. (2005). Introduction to statistical quality control (5th ed.). New Jersey: John Wiley.
21.
Rudner, L.M. (2005). PARAM-3PL calibration software for the 3 parameter logistic IRT Model (freeware). Available from http://edres.org/irt/param
22.
Rudner, L.M., Bracey, G., & Skaggs, G. (1996). The use of a person-fit statistic with one high quality achievement test. Applied Measurement in Education , 9, 91-109.
23.
Smith, R.M. (1985). A comparison of Rasch person analysis and robust estimators. Educational and Psychological Measurement, 45, 433-444.
24.
Snijders, T. (2001). Asymptotic distribution of person-fit statistics with estimated person parameter. Psychometrika, 66, 331-342.
25.
van Krimpen-Stoop, E.M.L.A., & Meijer, R.R. (1999). Simulating the null distribution of person-fit statistics for conventional and adaptive tests. Applied Psychological Measurement, 23, 327-345.
26.
van Krimpen-Stoop, E.M.L.A., & Meijer, R.R. (2000). Detection of person misfit in adaptive testing using statistical process control techniques. In W. J. van der Linden and C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 201-219). Boston: Kluwer-Nijhoff.
27.
van Krimpen-Stoop, E.M.L.A., & Meijer, R.R. (2001). CUSUM-based person-fit statistics for adaptive testing. Journal of Educational and Behavioral Statistics , 26(2), 199-218.
28.
van der Linden, W.J. (2005). Linear models for optimal test design. New York: Springer.
29.
Wright, B.D. (1980). Afterword. In G. Rasch (Ed.), Probabilistic models for some intelligence and attainment tests: With foreword and afterword by Benjamin D. Wright. Chicago: MESA Press.
30.
Wright, B.D., & Stone, M.H. (1979). Best test design: Rasch measurement. Chicago: MESA Press.
