Meijer and van Krimpen-Stoop noted that the number of person-fit statistics (PFSs) that have been designed for computerized adaptive tests (CATs) is relatively modest. This article partially addresses that concern by suggesting three new PFSs for CATs. The statistics are based on tests for a change point and can be used to detect an abrupt change in test performance of examinees during a CAT. The Type I error rate and power of the statistics are computed from a detailed simulation study. The performances of the new statistics are compared with those of four existing PFSs using receiver operating characteristics curves. The new statistics are then computed using data from an operational and high-stakes CAT. The new PFSs appear promising for assessment of person fit for CATs.
American Educational Research Association, American Psychological Association, & National Council for Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
2.
AndrewsD. (1993). Tests for parameter instability and structural change with unknown change point. Econometrica, 61, 821–856.
3.
ArmstrongR.ShiM. (2009a). A parametric cumulative sum statistic for person fit. Applied Psychological Measurement, 33, 391–410.
4.
ArmstrongR.ShiM. (2009b). Model-free CUSUM methods for person fit. Journal of Educational Measurement, 46, 408–428.
5.
BakerF. B.KimH. S. (2004). Item response theory: Parameter estimation techniques (2nd ed.). New York, NY: Marcel Dekker.
6.
BoltD. M.CohenA. S.WollackJ. A. (2002). Item parameter estimation under conditions of test speededness: Application of a mixture Rasch model with ordinal constraints. Journal of Educational Measurement, 39, 331–348.
7.
BradlowE.WeissR. E. (2001). Outlier measures and norming methods for computerized adaptive tests. Journal of Educational and Behavioral Statistics, 26, 85–104.
8.
BradlowE.WeissR. E.ChoM. (1998). Bayesian detection of outliers in computerized adaptive tests. Journal of the American Statistical Association, 93, 910–919.
9.
BreuschT. S. (1979). Conflict among criteria for testing hypotheses: Extensions and comments. Econometrica, 47, 203–207.
10.
ChenJ.GuptaA. K. (2012). Parametric statistical change point analysis (2nd ed.). Boston, MA: Birkhauser.
11.
ConijnJ. M.EmonsW. H. M.SijtsmaK. (2014). Statistic -based person-fit methods for noncognitive multiscale measures. Applied Psychological Measurement, 38, 122–136.
12.
ConradK. J.BezruczkoN.ChanY.RileyB.DiamondG.DennisM. L. (2010). Screening for atypical suicide risk with person fit statistics among people presenting to alcohol and other drug treatment. Drug and Alcohol Dependence, 106, 92–100.
13.
CoxD. R.HinkleyD. V. (1974). Theoretical statistics. London, England: Chapman and Hall.
14.
CsorgoM.HorvathL. (1997). Limit theorems in change-point analysis. New York, NY: Wiley.
15.
EstrellaA. (2003). Critical values and p values of Bessel process distributions: Computation and application to structural break tests. Econometric Theory, 19, 1128–1143.
16.
FerrandoP. J. (2012). Assessing inconsistent responding in E and N measures: An application of person-fit analysis in personality. Personality and Individual Differences, 52, 718–722.
17.
FerrandoP. J. (2015). Assessing person-fit in typical-response measures. In ReiseS. P.RevickiD. (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 128–156). New York, NY: Routledge.
18.
FerrandoP. J.Anguiano-CarrascoC. (2013). A structural model-based optimal person-fit procedure for identifying faking. Educational and Psychological Measurement, 73, 173–190.
19.
FinkelmanM.WeissD. J.Kim-KangG. (2010). Item selection and hypothesis testing for the adaptive measurement of change. Applied Psychological Measurement, 34, 238–254.
20.
GlasC. A. W.DagohoyA. V. T. (2007). A person fit test for IRT models for polytomous items. Psychometrika, 72, 159–180.
21.
GlasC. A. W.MeijerR. R.van Krimpen-StoopE. M. L. A. (1998). Statistical tests for person misfit in computerized adaptive testing (Research Rep. No. 98-01). Enschede, the Netherlands: University of Twente.
22.
GombayE.HorvathL. (1996). On the rate of approximations for maximum likelihood tests in change-point models. Journal of Multivariate Analysis, 56, 120–152.
23.
HansonB. A.HarrisD. J.BrennanR. L. (1994). A comparison of several statistical methods for examining allegations of copying (ACT research report series no. 87-15). Iowa City, IA: American College Testing.
24.
HawkinsD.QiuP.KangC. (2003). The changepoint model for statistical process control. Journal of Quality Technology, 35, 355–366.
25.
KingsburyG. G.ZaraA. R. (1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2, 359–375.
26.
KlauerK. C.RettigK. (1990). An approximately standardized person test for assessing consistency with a latent trait model. British Journal of Mathematical and Statistical Psychology, 43, 193–206.
27.
LeeY.-H.von DavierA. A. (2013). Monitoring scale scores over time via quality control charts, model-based approaches, and time series techniques. Psychometrika, 78, 557–575.
28.
LehmannE. L.RomanoJ. P. (2005). Testing statistical hypotheses (3rd ed.). New York, NY: Springer-Verlag.
29.
McLeodL. D.LewisC. (1999). Detecting item memorization in the CAT environment. Applied Psycholgoical Measurement, 23, 147–160.
30.
McLeodL. D.LewisC.ThissenD. (2003). A Bayesian method for the detection of item preknowledge in computerized adaptive testing. Applied Psychological Measurement, 27, 121–137.
31.
MeijerR. R. (2002). Outlier detection in high-stakes certification testing. Journal of Educational Measurement, 39, 219–233.
32.
MeijerR. R. (2004). Using patterns of summed scores in paper-and-pencil tests and computer-adaptive tests to detect misfitting item score patterns. Journal of Educational Measurement, 41, 119–136.
33.
MeijerR. R.EgberinkI. J.EmonsW. H.SijtsmaK. (2008). Detection and validation of unscalable item score patterns using item response theory: An illustration with Harters self-perception profile for children. Journal of Personality Assessment, 90, 227–238.
34.
MeijerR. R.SijtsmaK. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107–135.
35.
MeijerR. R.TendeiroJ. N. (2014). The use of person-fit scores in high-stakes educational testing: How to use them and what they tell us (LSAC Research Report Series). Newtown, PA: Law School Admission Council.
36.
MeijerR. R.van Krimpen-StoopE. M. L. A. (2010). Detecting person misfit in adaptive testing. In van der LindenW. J.GlasC. A. (Eds.), Elements of adaptive testing (pp. 315–329). Dordrecht, the Netherlands: Springer.
37.
MislevyR. J.ChangH. H. (2000). Does adaptive testing violate local independence?Psychometrika, 65, 149–156.
38.
MontgomeryD. C. (2013). Introduction to statistical quality control. New York, NY: Wiley.
39.
NeringM. L. (1997). The distribution of indexes of person fit within the computerized adaptive testing environment. Applied Psychological Measurement, 21, 115–127.
40.
OlsonJ. F.FremerJ. (2013). TILSA test security guidebook: Preventing, detecting, and investigating test securities irregularities. Washington, DC: Council of Chief State School Officers.
41.
RaoC. R. (1973). Linear statistical inference and its applications (2nd ed.). New York, NY: John Wiley.
42.
RomeroM.RiascosA.JaraD. (2015). On the optimality of answer-copying indices: Theory and practice. Journal of Educational and Behavioral Statistics, 40, 435–453.
43.
SenP. K. (1981). Sequential nonparametrics: Invariance principles and statistical inference. New York, NY: Wiley.
44.
ShaoC.LiJ.ChengY. (2015). A change point based method for test speededness detection. Psychometrika. doi:10.1007/s11336-015-9476-7
45.
SiegmundD. (1985). Sequential analysis: Tests and confidence intervals. New York, NY: Springer.
46.
SinharayS. (2016). Assessment of person fit using resampling-based approaches. Journal of Educational Measurement, 53, 63–85.
47.
SnijdersT. (2001). Asymptotic distribution of person-fit statistics with estimated person parameter. Psychometrika, 66, 331–342.
48.
TendeiroJ. N.MeijerR. R. (2012). A CUSUM to detect person misfit: A discussion and some alternatives for existing procedures. Applied Psychological Measurement, 36, 420–442.
49.
TendeiroJ. N.MeijerR. R. (2014). Detection of invalid test scores: The usefulness of simple nonparametric statistics. Journal of Educational Measurement, 51, 239–259.
50.
ThissenD.OrlandoM. (2001). Item response theory for items scored in two categories. In ThissenD.WainerH. (Eds.), Test scoring (pp. 73–140). Hillsdale, NJ: Lawrence Erlbaum.
51.
van der LindenW. J. (1998). Bayesian item selection criteria for adaptive testing. Psychometrika, 63, 201–216.
52.
van Krimpen-StoopE. M. L. A.MeijerR. R. (1999). The null distribution of person-fit statistics for conventional and adaptive tests. Applied Psychological Measurement, 23, 327–345.
53.
van Krimpen-StoopE. M. L. A.MeijerR. R. (2000). Detecting person misfit in adaptive testing using statistical process control techniques. In van der LindenW. J.GlasC. A. (Eds.), Computerized adaptive testing: Theory and practice (pp. 201–219). Dordrecht, the Netherlands: Springer.
54.
van Krimpen-StoopE. M. L. A.MeijerR. R. (2001). CUSUM-based person-fit statistics for adaptive testing. Journal of Educational and Behavioral Statistics, 26, 199–217.
55.
van Krimpen-StoopE. M. L. A.MeijerR. R. (2002). Detection of person misfit in computerized adaptive tests with polytomous items. Applied Psychological Measurement, 26, 164–180.
56.
WarmT. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427–450.
