A Randomization P-Value Test for Detecting Copying on Multiple-Choice Exams

Abstract

This article is concerned with the statistical detection of copying on multiple-choice exams. As an alternative to existing permutation- and model-based copy-detection approaches, a simple randomization p-value (RP) test is proposed. The RP test, which is based on an intuitive match-score statistic, makes no assumptions about the distribution of examinees’ answer vectors and hence is broadly applicable. Especially important in this copy-detection setting, the RP test is shown to be exact in that its size is guaranteed to be no larger than a nominal α value. Additionally, simulation results suggest that the RP test is typically more powerful for copy detection than the existing approximate tests. The development of the RP test is based on the idea that the copy-detection problem can be recast as a causal inference and missing data problem. In particular, the observed data are viewed as a subset of a larger collection of potential values, or counterfactuals, and the null hypothesis of “no copying” is viewed as a “no causal effect” hypothesis and formally expressed in terms of constraints on potential variables.

Keywords

answer-copying indexes causal inference cheating copying on multiple-choice exams exact versus approximate tests model-,permutation-,and randomization-based tests nominal item response model potential variables

Get full access to this article

View all access options for this article.

References

Bock

R. D.

(1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29–51.

Casella

Berger

R. L.

(2002). Statistical inference (2nd ed.). Duxbury.

Cizek

G. J.

(1999). Cheating on tests: How to do it, detect it, and prevent it. Lawrence Erlbaum.

Eden

Yates

(1933). On the validity of Fisher’s z test when applied to an actual example of non-normal data. The Journal of Agricultural Science, 23(01), 6–17.

Edgington

Onghena

. (2007). Randomization tests. CRC Press.

Ernst

M. D.

(2004). Permutation methods: A basis for exact inference. Statistical Science, 19(4), 676–685.

Fisher

R. A.

(1935). The design of experiments. Oliver & Boyd.

Holland

P. W.

(1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–968.

Holland

P. W.

(1996). Assessing unusual agreement between the incorrect answers of two examinees using the K-index: Statistical theory and empirical support (ETS Technical Rep. No. 96-4). Educational Testing.

10.

Kempthorne

(1955). The randomization theory of experimental inference. Journal of the American Statistical Association, 50(271), 946–967.

11.

Lang

J. B.

(2015). A closer look at testing the “no-treatment-effect” hypothesis in a comparative experiment. Statistical Science, 30(3), 352–371.

12.

Lehmann

E. L.

(1998). Nonparametrics. Prentice Hall.

13.

Maynes

D. D.

(2017). Detecting potential collusion among individual examinees using similarity analysis. In Cizek

G. J.

Wollack

J. A.

(eds.) Handbook of quantitative methods for detecting cheating on tests (Chapter 3, pp. 47–69). Taylor & Francis.

14.

Neyman

(1923). On the application of probability theory to agricultural experiments. Essay on principles. Section 9, Roczniki Nauk Rolniczych Tom X [in Polish]; English translation of excerpts by D.M. Dabrowska and T.P. Speed (1990). Statistical Science, 5(4), 463–472.

15.

Pearl

(2010a). The foundations of causal inference. Sociological Methodology, 40(1), 75–149.

16.

Pearl

(2010b). On the consistency rule in causal inference: Axiom, definition, assumption, or theorem? Epidemiology, 21(6), 872–875.

17.

Pitman

E. J. G.

(1937a). Significance tests which may be applied to samples from any populations. Journal of the Royal Statistical Society, 4, 119–130.

18.

Pitman

E. J. G.

(1937b). Significance tests which may be applied to samples from any populations. II. The correlation coefficient test. Journal of the Royal Statistical Society, 4, 225–232.

19.

Romero

Riascos

Á.

Jara

(2015). On the optimality of answer-copying indices: Theory and practice. Journal of the Educational and Behavioral Statistics, 40(5), 435–453.

20.

Rosenbaum

P. R.

Rubin

D. B.

(1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.

21.

Rubin

D. B.

(1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of the Educational Psychology, 66, 688–701.

22.

Rubin

D. B.

(2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469), 322–331.

23.

Sotaridona

L. S.

Meijer

R. R

. (2002). Statistical properties of the K-index for detecting answer copying. Journal of the Educational Measurement, 39, 115–132.

24.

Sotaridona

L. S.

Meijer

R. R

. (2003). Two new statistics to detect answer copying. Journal of the Educational Measurement, 40, 53–69.

25.

van der Linden

W. J.

Sotaridona

L. S.

(2006). Detecting answer copying when the regular response process follows a known response model. Journal of the Educational and Behavioral Statistics, 31, 283–304.

26.

Wollack

J. A

. (1996). Detection of answer copying using item response theory. Dissertation Abstracts International, 57/05, 2015.

27.

Wollack

J. A.

(1997). A nominal response model approach for detecting answer copying. Applied Psychological Measurement, 21(4), 307–320.

28.

Wollack

J. A.

Cohen

A. S.

(1998). Detection of answer copying with unknown item and trait parameters. Applied Psychological Measurement, 22, 144–152.

29.

Zopluoglu

(2016, updated 2018). “Computing Statistical Indices to Detect Answer Copying on Multiple-Choice Tests,” R Package “CopyDetect.” ver. 1.2, 1.3.

30.

Zopluoglu

Davenport

E. C.

Jr (2012). The empirical power and type I error rates of the GBT and

ω

indices in detecting answer copying on multiple-choice tests. Educational and Psychological Measurement, 72(6), 975–1000.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.49 MB