Examinees with item preknowledge tend to obtain inflated test scores that undermine test score validity. With the availability of process data collected in computer-based assessments, the research on detecting item preknowledge has progressed on using both item scores and response times. Item revisit patterns of examinees can also be utilized as an additional source of information. This study proposes a new statistic for detecting item preknowledge when compromised items are known by utilizing the hierarchical speed–accuracy revisits model. By simultaneously evaluating abnormal changes in the latent abilities, speeds, and revisit propensities of examinees, the procedure was found to provide greater statistical power and stronger substantive evidence that an examinee had indeed benefited from item preknowledge.
American Educational Research Association, American Psychological Association, & National Council for Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
3.
BelovD. I. (2016). Comparing the performance of eight item preknowledge detection statistics. Applied Psychological Measurement, 40, 83–97.
4.
BelovD. I. (2017). On the optimality of the detection of examinees with aberrant answer changes. Applied Psychological Measurement, 41(5), 338–352.
5.
BezirhanU.von DavierM.GrabovskyI. (2020). Modeling item revisit behavior: The hierarchical speed-accuracy-revisits model. Educational and Psychological Measurement, 81, 1–25. https://doi.org/10.1177/0013164420950556
6.
CandellG. L.DrasgowF. (1988). An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement, 12, 253–260.
7.
ChangH.ZhangJ. (2002). Hypergeometric family and test overlap rates in computerized adaptive testing. Psychometrika, 67, 387–398.
8.
ChoeM. E. (2017). Advancements in test security: Preventive test assembly methods and change-point detection of compromised items in adaptive testing[Doctoral dissertation]. http://hdl.handle.net/2142/98091
9.
ChoeM. E.ZhangJ.ChangH. (2018). Sequential detection of compromised items using response times in computerized adaptive testing. Psychometrika, 83(3), 650–673. https://doi.org/10.1007/s11336-017-9596-3
10.
CochranW. G. (1952) The χ2 test of goodness of fit. Annals of Mathematical Statistics, 23, 315–345.
11.
DykstraR. (1991). Asymptotic normality for chi-bar-square distributions. Canadian Journal of Statistics, 19, 297–306. https://doi.org/10.2307/3315395
12.
FoxJ.-P.MariantiS. (2017). Person-fit statistics for joint models for accuracy and speed. Journal of Educational Measurement, 54, 243–262. https://doi.org/10.1111/jedm.12143
13.
FriedmanS. J.CookG. L. (1995). Is an examinee’s cognitive style related to the impact of answer changing on multiple-choice tests?The Journal of Experimental Education, 63(3), 199–213.
14.
JeonM.de BoeckP.van der LindenW. (2017). Modeling answer change behavior: An application of a generalized item response tree model. Journal of Educational and Behavioral Statistics, 42(4), 467–490.
LeeS. Y.WollackJ. (2017). A mixture model to detect item preknowledge using item responses and response times. Paper presented at the Conference on Test Security, Madison, WI.
17.
ManKHarringJ. R. (2021). Assessing preknowledge cheating via innovative measures: A multiple-group analysis of jointly modeling item responses, response times, and visual fixation counts. Educational Psychological Measurement, 81(3), 441–465. https://doi.org/10.1177/0013164420968630
18.
McLeodL. D.LewisC.ThissenD. (2003). A Bayesian method for the detection of item preknowledge in computerized adaptive testing. Applied Psychological Measurement, 27, 121–137. https://doi.org/10.1177/0146621602250534
19.
McMorrisR. F.DeMersL. P.SchwarzS. P. (1987). Attitudes, behaviors, and reasons for changing responses following answer-changing instruction. Journal of Educational Measurement, 24, 131–143.
20.
QianH.StaniewskaD.ReckaseM.WooA. (2016). Using response time to detect item preknowledge in computer-based licensure examinations. Educational Measurement: Issues and Practice, 35, 38–47.
21.
RaschG. (1960). Studies in mathematical psychology: Vol. 1. Probabilistic models for some intelligence and attainment tests. Danish Institute for Educational Research.
22.
RaschG. (1980). Probabilistic models for some intelligence and attainment tests (Expanded ed.). University of Chicago Press.
23.
ShaoC. (2017). Aberrant response detection using change-point analysis [Doctoral dissertation, University of Notre Dame]. ProQuest Dissertations and Theses database. (UMI No. 10308190).
24.
SinharayS. (2017). Detection of item preknowledge using likelihood ratio test and score test. Journal of Educational and Behavioral Statistics, 42, 46–68. https://doi.org/10.3102/1076998616673872
25.
SinharayS. (2018). Detecting fraudulent erasures at an aggregate level. Journal of Educational and Behavioral Statistics, 43(3), 286–315. https://doi.org/10.3102/1076998617739626
26.
SinharayS. (2020). Detection of item preknowledge using response times. Applied Psychological Measurement, 44, 376–392. https://doi.org/10.1177/0146621620909893
27.
SinharayS.DuongM. Q.WoodS. W. (2017). A new statistic for detection of aberrant answer changes. Journal of Educational Measurement, 54, 200–217. https://doi.org/10.1111/jedm.12141
28.
SinharayS.JohnsonM. S. (2017). Three new methods for analysis of answer changes. Educational and Psychological Measurement, 77(1), 54–81. https://doi.org/10.1177/0013164416632287
29.
SinharayS.JohnsonM. S. (2019). The use of item scores and response times to detect examinees who may have benefited from item preknowledge. British Journal of Mathematical and Statistical Psychology. Advance online publication. https://doi.org/10.1111/bmsp.12187
30.
SmithR. W.Davis-BeckerS. L. (2011, April). Detecting suspect examinees: An application of differential person functioning analysis [Paper presentation]. Annual Meeting of the National Council of Measurement in Education, New Orleans, LA, United States.
31.
van der LindenW. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31, 181–204.
32.
van der LindenW. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287–308. https://doi.org/10.1007/s11336-006-1478-z
33.
van der LindenW. J.GuoF. (2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73, 365–384. https://doi.org/10.1007/s11336-007-9046-8
34.
VispoelW. P. (1998). Reviewing and changing answers on computer-adaptive and self-adaptive vocabulary tests. Journal of Educational Measurement, 35(4), 328–345.
35.
WangC.XuG.ShangZ.KuncelN. (2018). Detecting aberrant behavior and item preknowledge: A comparison of mixture modeling method and residual method. Journal of Educational and Behavioral Statistics, 43, 469–501. https://doi.org/10.3102/1076998618767123
36.
WangC.ZhengY.ChangH.-H. (2014). Does standard deviation matter? Using “standard deviation” to quantify security of multistage testing. Psychometrika, 79, 154–174.
37.
WangX.LiuY.HambletonR. K. (2017). Detecting item preknowledge using a predictive checking method. Applied Psychological Measurement, 41, 243–263. https://doi.org/10.1177/0146621616687285
38.
WollackJ. A.CohenA. S.EckerlyC. A. (2015). Detecting test tampering using item response theory. Educational and Psychological Measurement, 75(6), 931–953. https://doi.org/10.1177/0013164414568716
39.
WollackJ. A.EckerlyC. (2017). Detecting test tampering at the group level. In CizekG. J.WollackJ. A. (Eds.), Handbook of detecting cheating on tests (pp. 214–231). Routledge.
40.
WollackJ. A.SchoenigR. W. (2018). Cheating. In FreyB. B. (Ed.), The Sage encyclopedia of educational research, measurement, and evaluation (pp. 260–265). SAGE.
41.
YiQ.ZhangJ.ChangH. (2008). Severity of organized item theft in computerized adaptive testing: A simulation study. Applied Psychological Measurement, 32, 543–558.
42.
ZhangJ. (2014). A sequential procedure for detecting compromised items in the item pool of a CAT system. Applied Psychological Measurement, 38(2), 87–104.
43.
ZhangJ.LiJ. (2016). Monitoring items in real time to enhance CAT security. Journal of Educational Measurement, 53(2), 131–151. https://doi.org/10.1111/jedm.12104
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
