The Lack of Robustness of a Statistic Based on the Neyman–Pearson Lemma to Violations of Its Underlying Assumptions

Abstract

Drasgow, Levine, and Zickar (1996) suggested a statistic based on the Neyman–Pearson lemma (NPL; e.g., Lehmann & Romano, 2005, p. 60) for detecting preknowledge on a known set of items. The statistic is a special case of the optimal appropriateness indices (OAIs) of Levine and Drasgow (1988) and is the most powerful statistic for detecting item preknowledge when the assumptions underlying the statistic hold for the data (e.g., Belov, 2016Belov, 2016; Drasgow et al., 1996). This paper demonstrated using real data analysis that one assumption underlying the statistic of Drasgow et al. (1996) is often likely to be violated in practice. This paper also demonstrated, using simulated data, that the statistic is not robust to realistic violations of its underlying assumptions. Together, the results from the real data and the simulations demonstrate that the statistic of Drasgow et al. (1996) may not always be the optimum statistic in practice and occasionally has smaller power than another statistic for detecting preknowledge on a known set of items, especially when the assumptions underlying the former statistic do not hold. The findings of this paper demonstrate the importance of keeping in mind the assumptions underlying and the limitations of any statistic or method.

Keywords

item preknowledge optimal appropriateness index signed likelihood ratio test

Get full access to this article

View all access options for this article.

References

Belov

D. I.

(2016). Comparing the performance of eight item preknowledge detection statistics. Applied Psychological Measurement, 40(2), 83–97. https://doi.org/10.1177/0146621615603327

Belov

D. I.

Toton

S. L.

(2020, September). Clique-based detection of examinees with pre-knowledge on real, marked data. Paper presented at the Virtual Annual meeting of the National Council of Measurement in Education, September 9‐11.

Boughton

Smith

Ren

(2017). Using response time data to detect compromised items and/or people. In Cizek

G. J.

Wollack

J. A.

(Eds.), Handbook of detecting cheating on tests (pp. 177–190). Routledge.

Box

G. E. P.

(1979). Some problems of statistics and everyday life. Journal of the American Statistical Association, 74(365), 1–4. https://doi.org/10.1080/01621459.1979.10481600

Chalmers

R. P.

(2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06

Chen

W.-H.

Thissen

(1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22(3), 265–289. https://doi.org/10.2307/1165285

Cizek

G. J.

Wollack

J. A.

(2017). Handbook of detecting cheating on tests. Routledge.

Drasgow

Levine

M. V.

Williams

E. A.

(1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38(1), 67–86. https://doi.org/10.1111/j.2044-8317.1985.tb00817.x

Drasgow

Levine

M. V.

Zickar

M. J.

(1996). Optimal identification of mismeasured individuals. Applied Measurement in Education, 9(1), 47–64. https://doi.org/10.1207/s15324818ame0901_5

10.

Eckerly

Smith

Lee

(2018, October). An introduction to item preknowledge detection with real data applications. Paper presented at the Conference on Test Security, Park City, UT, October 10‐12.

11.

Glas

C. A. W.

Dagohoy

A. V. T.

(2007). A person fit test for IRT models for polytomous items. Psychometrika, 72(2), 159–180. https://doi.org/10.1007/s11336-003-1081-5

12.

Guo

Drasgow

(2010). Identifying cheating on unproctored internet tests: The Z-test and the likelihood ratio test. International Journal of Selection and Assessment, 18(4), 351–364. https://doi.org/10.1111/j.1468-2389.2010.00518.x

13.

Hanley

J. A.

McNeil

B. J.

(1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29–36. https://doi.org/10.1148/radiology.143.1.7063747

14.

Lehmann

E. L.

Romano

J. P.

(2005). Testing statistical hypotheses (3rd ed.). Springer-Verlag.

15.

Levine

M. V.

Drasgow

(1988). Optimal appropriateness index. Psychometrika, 53(2), 161–176. https://doi.org/10.1007/BF02294130

16.

Cai

(2017). Summed score likelihood–based indices for testing latent variable distribution fit in item response theory. Educational and Psychological Measurement, 78(5), 857–886. https://doi.org/10.1177/0013164417717024

17.

Montgomery

D. C.

(2013). Design and analysis of experiments (8th ed.). Wiley.

18.

Orlando

Thissen

(2000). Likelihood-based item-fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24(1), 50–64. https://doi.org/10.1177/01466216000241003

19.

Sinharay

(2017a). On the equivalence of a likelihood ratio of Drasgow, Levine, and Zickar (1996) and the statistic based on the Neyman-Pearson lemma of Belov (2016). Applied Psychological Measurement, 41(2), 145–149. https://doi.org/10.1177/0146621616673597

20.

Sinharay

(2017b). Are the nonparametric person-fit statistics more powerful than their parametric counterparts? Revisiting the simulations in Karabatsos (2003). Applied Measurement in Education, 30(4), 314–328. https://doi.org/10.1080/08957347.2017.1353990

21.

Sinharay

(2017c). Detection of item preknowledge using likelihood ratio test and score test. Journal of Educational and Behavioral Statistics, 42(1), 46–68. https://doi.org/10.3102/1076998616673872

22.

Sinharay

(2017d). Which statistic should be used to detect item preknowledge when the set of compromised items is known? Applied Psychological Measurement, 41(6), 403–421. https://doi.org/10.1177/0146621617698453

23.

Sinharay

Jensen

J. L.

(2019). Higher-order asymptotics and its application to testing the equality of the examinee ability over two sets of items. Psychometrika, 84(2), 484–510. https://doi.org/10.1007/s11336-018-9627-8

24.

Sinharay

Johnson

M. S.

(2020a). Detecting test fraud using Bayes factors. Behaviormetrika, 47(2), 339–354. https://doi.org/10.1007/s41237-020-00113-9

25.

Sinharay

Johnson

M. S.

(2020b). The use of the posterior probability in score differencing. Journal of Educational and Behavioral Statistics, 46(4), 403–429. https://doi.org/10.3102/1076998620957423.

26.

Smith

R. W.

Davis-Becker

S. L.

(2011, April). Detecting suspect examinees: An application of differential person functioning analysis. Paper presented at the annual meeting of the National Council of Measurement in Education, New Orleans, LA, April 9‐11.

27.

Toton

S. L.

Maynes

D. D.

(2019). Detecting examinees with pre-knowledge in experimental data using conditional scaling of response times. Frontiers in Education, 4(49), 1–18. https://doi.org/10.3389/feduc.2019.00049

28.

Wang

Liu

Hambleton

R. K.

(2017). Detecting item preknowledge using a predictive checking method. Applied Psychological Measurement, 41(4), 243–263. https://doi.org/10.1177/0146621616687285

29.

Wang

Liu

Robin

Guo

(2019). A comparison of methods for detecting examinee preknowledge of items. International Journal of Testing, 19(3), 207–226. https://doi.org/10.1080/15305058.2019.1610886

30.

Wollack

J. A.

Cohen

A. S.

Eckerly

C. A.

(2015). Detecting test tampering using item response theory. Educational and Psychological Measurement, 75(3), 931–953. https://doi.org/10.1177/0013164414568716

31.

Woods

C. M.

Thissen

(2006). Item response theory with estimation of the latent population distribution using spline-based densities. Psychometrika, 71(2), 281–301. https://doi.org/10.1007/s11336-004-1175-8

32.

Zickar

M. J.

Drasgow

(1996). Detecting faking on a personality instrument using appropriateness measurement. Applied Psychological Measurement, 20(1), 71–87. https://doi.org/10.1177/014662169602000107.

33.

Zopluoglu

(2017). Similarity, answer copying, and aberrance: Understanding the status quo. In Cizek

G. J.

Wollack

J. A.

(Eds.), Handbook of detecting cheating on tests (pp. 25–46). Routledge.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.12 MB