Unproctored Internet Testing (UIT) is becoming more popular in personnel recruitment and selection. A drawback of UIT is that cheating is easy and, therefore, a proctored test is often administered after an UIT procedure. To detect inconsistent test scores from UIT, a cumulative sum procedure (CUSUM) is proposed. This procedure is applied to empirical data from an adaptive computer-based test in a real personnel selection context. The usefulness of the CUSUM is illustrated, and the unique contribution of the CUSUM to existing procedures is discussed.
ArmstrongR. D.ShiM. (2009). A parametric cumulative sum statistic for person fit. Applied Psychological Measurement, 33, 391-410.
2.
BelovD. I. (2011). Detection of answer copying based on the structure of a high-stakes test. Applied Psychological Measurement, 35, 495-517.
3.
BelovD. I.ArmstrongR. D. (2010). Automatic detection of answer copying via Kullback-Leibler divergence and K-index. Applied Psychological Measurement, 34, 379-392.
4.
BelovD. I.ArmstrongR. D. (2011). Distributions of the Kullback-Leibler divergence with applications. British Journal of Mathematical and Statistical Psychology, 64, 291-309.
5.
BelovD. I.PashleyP. J.LewisC.ArmstrongR. D. (2007). Detecting aberrant responses with Kullback-Leibler distance. In ShigemasuK.OkadaA.ImaizumiT.HoshinoT. (Eds.), New trends in psychometrics (pp. 7-14). Tokyo, Japan: Universal Academy Press.
6.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. In LordF. M.NovickM. R. (Eds.), Statistical theories of mental test scores (pp. 397-479). Reading, MA: Addison-Wesley.
7.
BradlowE. T.WeissR. E.ChoM. (1998). Bayesian identification of outliers in computerized adaptive tests. Journal of the American Statistical Association, 93, 910-919.
8.
BuchananT.SmithJ. L. (1999). Using the Internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, 90, 125-144.
9.
BurkeE. (2009). Preserving the integrity of online testing. Industrial and Organizational Psychology: Perspectives on Science and Practice, 2, 35-38.
10.
DrasgowF.LevineM. V.WilliamsE. A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67-86.
11.
EfronB.TibshiraniR. J. (1993). An introduction to the bootstrap. New York, NY: Chapman Hall/CRC Press.
12.
EggenT. J. H. M. (1999). Item selection in adaptive testing with the sequential probability ratio test. Applied Psychological Measurement, 23, 249-261.
13.
EggenT. J. H. M.StraetmansG. J. J. M. (2000). Computerized adaptive testing for classifying examinees into three categories. Educational and Psychological Measurement, 60, 713-734.
14.
EmbretsonS. E.ReiseS. P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
15.
FinkelmanM. (2008). On using stochastic curtailment to shorten the SPRT in sequential mastery testing. Journal of Educational and Behavioral Statistics, 33, 442-463.
16.
GuoJ.DrasgowF. (2010). Identifying cheating on unproctored Internet tests: The z-test and the likelihood ratio test. International Journal of Selection and Assessment, 18, 351-364.
17.
GuoJ.TayL.DrasgowF. (2009). Conspiracies and test compromise: An evaluation of the resistance of test systems to small-scale cheating. International Journal of Testing, 9, 283-309.
18.
HawkinsD. M.OlwellD. H. (1998). Cumulative sum charts and charting for quality improvement. New York, NY: Springer-Verlag.
19.
HofkinsD. (1995, June16). Cheating “rife” in national tests. Times Educational Supplement, p. 1.
20.
ImparaJ. C.FosterD. (2006). Item and test development strategies to minimize test fraud. In DowningS. M.HaladynaT. M. (Eds.), Handbook of test development (pp. 91-114). Mahwah, NJ: Lawrence Erlbaum.
21.
JacobB. A.LevittS. D. (2003). Rotten apples: An investigation of the prevalence and predictors of teacher teaching. Quarterly Journal of Economics, 118, 843-877.
22.
KarabatsosG. (2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education, 16, 277-298.
LievensF.BurkeE. (2010). Dealing with the threats inherent in unproctored Internet testing of cognitive ability: Results from a large-scale operational test program. Journal of Occupational and Organizational Psychology, 84, 817-824.
MakranskyG.GlasC. A. W. (2011). Unproctored internet test verification: Using adaptive confirmation testing. Organizational Research Methods, 14, 608-630.
27.
MeijerR. R. (1996). Person-fit research: An introduction. Applied Measurement in Education, 9, 3-8.
28.
MeijerR. R.SijtsmaK. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107-135.
29.
MeijerR. R.van Krimpen-StoopE. M. L. A. (2010). Detecting person misfit in adaptive testing. In van der LindenW. J.GlasC. A. W. (Eds.), Elements of adaptive testing (pp. 315-329). New York, NY: Springer.
30.
MolenaarI. W.HoijtinkH. (1990). The many null distributions of person fit indices. Psychometrika, 55, 75-106.
31.
NaglieriJ. A.DrasgowF.SchmitM.HandlerL.PrifiteraA.MargolisA.VelasquezR. (2004). Psychological testing on the Internet: New problems, old issues. American Psychologist, 59, 150-162.
32.
NeymanJ.PearsonE. S. (1933). On the problems of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, 231, 289-337.
33.
NyeC. D.DoB.-R.DrasgowF.FineS. (2008). Two-step testing in employee selection: Is score inflation a problem?International Journal of Selection and Assessment, 16, 112-120.
34.
PageE. S. (1954). Continuous inspection schemes. Biometrika, 41, 100-115.
35.
PearlmanK. (2009). Unproctored Internet Testing: Practical, legal, and ethical concerns. Industrial and Organizational Psychology: Perspectives on Science and Practice, 2, 14-19.
36.
R Development Core Team. (2009). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org
37.
SotaridonaL. S.MeijerR. R. (2003). Two new statistics to detect answer copying. Journal of Educational Measurement, 40, 53-69.
38.
TendeiroJ. N.MeijerR. R. (2012). A CUSUM to detect person misfit: A discussion and some alternatives for existing procedures. Applied Psychological Measurement, 36, 420-442.
39.
TippinsN. T. (2009). Internet alternatives to traditional proctored testing: Where are we now?Industrial and Organizational Psychology: Perspectives on Science and Practice, 2, 2-10.
40.
TippinsN. T.BeatyJ.DrasgowF.GibsonW. M.PearlmanK.SegallD. O.ShepherdW. (2006). Unproctored Internet testing in employment settings. Personnel Psychology, 59, 189-225.
41.
van Krimpen-StoopE. M. L. A.MeijerR. R. (2000). Detection of person misfit in adaptive testing using statistical process control techniques. In van der LindenW. J.GlasC. A. W. (Eds.), Computerized adaptive testing: Theory and practice (pp. 201-219). Boston, MA: Kluwer-Nijhoff.
42.
van Krimpen-StoopE. M. L. A.MeijerR. R. (2001). CUSUM-based person-fit statistics for adaptive testing. Journal of Educational and Behavioral Statistics, 26, 199-218.
43.
WaldA. (1947). Sequential analysis. New York, NY: Wiley.
