Which Statistic Should Be Used to Detect Item Preknowledge When the Set of Compromised Items Is Known?

Abstract

Benefiting from item preknowledge is a major type of fraudulent behavior during educational assessments. Belov suggested the posterior shift statistic for detection of item preknowledge and showed its performance to be better on average than that of seven other statistics for detection of item preknowledge for a known set of compromised items. Sinharay suggested a statistic based on the likelihood ratio test for detection of item preknowledge; the advantage of the statistic is that its null distribution is known. Results from simulated and real data and adaptive and nonadaptive tests are used to demonstrate that the Type I error rate and power of the statistic based on the likelihood ratio test are very similar to those of the posterior shift statistic. Thus, the statistic based on the likelihood ratio test appears promising in detecting item preknowledge when the set of compromised items is known.

Keywords

cheating fraud test security

Get full access to this article

View all access options for this article.

References

American Educational Research Association, American Psychological Association, & National Council for Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Belov

D. I.

(2015). Robust detection of examinees with aberrant answer changes. Journal of Educational Measurement, 52, 437-456.

Belov

D. I.

(2016). Comparing the performance of eight item preknowledge detection statistics. Applied Psychological Measurement, 40, 83-97.

Box

G. E. P.

(1979). Some problems of statistics and everyday life. Journal of the American Statistical Association, 74, 1-4.

Chang

H. H.

Stout

(1993). The asymptotic posterior normality of the latent trait in an IRT model. Psychometrika, 58, 37-52.

Cizek

G. J.

Wollack

J. A.

(2017). Handbook of detecting cheating on tests. Washington, DC: Routledge.

Cox

D. R.

Hinkley

D. V.

(1974). Theoretical statistics. London, England: Chapman & Hall.

Drasgow

Levine

M. V.

Zickar

M. J.

(1996). Optimal identification of mismeasured individuals. Applied Measurement in Education, 9, 47-64.

Eckerly

C. A.

Babcock

B. G.

Wollack

J. A.

(2015, April). Preknowledge detection using a scale-purified deterministic gated IRT model. Paper presented at the Annual meeting of the National Council of Measurement in Education, Chicago, IL.

10.

Hanson

B. A.

Harris

D. J.

Brennan

R. L.

(1994). A comparison of several statistical methods for examining allegations of copying (ACT Research Report Series No. 87-15). Iowa City, IA: American College Testing.

11.

Karabatsos

(2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education, 16, 277-298.

12.

Kyle

(2002). Cheating scandal rocks GRE, ETS. Retrieved from http://thedartmouth.com/2002/08/09/cheating-scandal-rocks-gre-ets/

13.

Lehmann

E. L.

Romano

J. P.

(2005). Testing statistical hypotheses (3rd ed.). New York, NY: Springer-Verlag.

14.

Levine

M. V.

Drasgow

(1988). Optimal appropriateness index. Psychometrika, 53, 161-176.

15.

McLeod

L. D.

Lewis

Thissen

(2003). A Bayesian method for the detection of item preknowledge in computerized adaptive testing. Applied Psychological Measurement, 27, 121-137.

16.

Qian

Staniewska

Reckase

Woo

(2016). Using response time to detect item preknowledge in computer-based licensure examinations. Educational Measurement: Issues and Practice, 35, 38-47.

17.

R Core Team. (2015). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.

18.

Rao

C. R.

(1973). Linear statistical inference and its applications (2nd ed.). New York, NY: John Wiley.

19.

Segall

D. O.

(2002). An item response model for characterizing test compromise. Journal of Educational and Behavioral Statistics, 27, 163-179.

20.

Sinharay

(2016). Asymptotic corrections of standardized extended caution indices. Applied Psychological Measurement, 40, 418-433.

21.

Sinharay

(2017a). Detection of item preknowledge using likelihood ratio test and score test. Journal of Educational and Behavioral Statistics, 42, 46-68. doi:10.3102/1076998616673872

22.

Sinharay

(2017b). On the equivalence of a likelihood ratio of Drasgow, Levine, and Zickar (1996) and the statistic based on the Neyman-Pearson lemma of Belov (2016). Applied Psychological Measurement, 41, 145-149. doi:10.1177/0146621616673597

23.

Sinharay

Duong

M. Q.

Wood

S. W.

(in press). A new statistic for detection of aberrant answer changes. Journal of Educational Measurement.

24.

Tendeiro

J. N.

Meijer

R. R.

(2014). Detection of invalid test scores: The usefulness of simple nonparametric statistics. Journal of Educational Measurement, 51, 239-259.

25.

van der Linden

W. J

. (2006). A lognormal model for response times on test forms. Journal of Educational and Behavioral Statistics, 31, 181-204.

26.

van der Linden

W. J.

van Krimpen-Stoop

E. M. L. A

. (2003). Using response times to detect aberrant responses in computerized adaptive testing. Psychometrika, 68, 251-265.

27.

Veerkamp

W. J. J.

Glas

C. A. W.

(2000). Detection of known items in adaptive testing with a statistical quality control method. Journal of Educational and Behavioral Statistics, 25, 373-389.

28.

Warm

T. A.

(1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427-450.