Sage Journals: Discover world-class research

Abstract

To evaluate preknowledge detection methods, researchers often conduct simulation studies in which they use models to generate the data. In this article, we propose two new models to represent item preknowledge. Contrary to existing models, we allow the impact of preknowledge to vary across persons and items in order to better represent situations that are encountered in practice. We use three real data sets to evaluate the fit of the new models with respect to two types of preknowledge: items only, and items and the correct answer key. Results show that the two new models provide the best fit compared to several other existing preknowledge models. Furthermore, model parameter estimates were found to vary substantially depending on the type of preknowledge being considered, indicating that answer key disclosure has a profound impact on testing behavior.

Keywords

item preknowledge item compromise cheating test security aberrant behavior

Get full access to this article

View all access options for this article.

References

Becker

Debeer

Weirich

Goldhammer

. (2021). On the speed sensitivity parameter in the lognormal model for response times and implications for high-stakes measurement practice. Applied Psychological Measurement, 45(6), 407–422. https://doi.org/10.1177/01466216211008530

Bolt

D. M.

Kim

Wollack

Pan

Eckerly

Sowles

. (2020). A psychometric model for discrete-option multiple-choice items. Applied Psychological Measurement, 44(1), 33–48. https://doi.org/10.1177/0146621619835499

Cizek

G. J.

Wollack

J. A.

(Eds.). (2017). Handbook of quantitative methods for detecting cheating on tests. Routledge. https://doi.org/10.4324/9781315743097

Drasgow

Levine

M. V.

Williams

E. A

. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38(1), 67–86. https://doi.org/10.1111/j.2044-8317.1985.tb00817.x

Eckerly

C. A

. (2017). Detecting preknowledge and item compromise: Understanding the status quo. In Cizek

G. J.

Wollack

J. A.

(Eds.), Handbook of quantitative methods for detecting cheating on tests (pp. 101–123). Routledge.

Gelman

Rubin

D. B

. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457–472. https://doi.org/10.1214/ss/1177011136

Gorney

Wollack

J. A

. (2021, June). Does item format affect test security? [Paper presentation]. National Council on Measurement in Education Annual Meeting, Virtual.

Gorney

Wollack

J. A

. (2022). Generating models for item preknowledge. Journal of Educational Measurement, 59(1), 22–42. https://doi.org/10.1111/jedm.12309

Guo

Drasgow

. (2010). Identifying cheating on unproctored internet tests: The Z-test and the likelihood ratio test. International Journal of Selection and Assessment, 18(4), 351–364. https://doi.org/10.1111/j.1468-2389.2010.00518.x

10.

Klein Entink

R. H.

Fox

J.-P.

van der Linden

W. J

. (2009). A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika, 74(1), 21–48. https://doi.org/10.1007/s11336-008-9075-y

11.

Lee

Bolt

D. M

. (2018). Asymmetric item characteristic curves and item complexity: Insights from simulation and real data analyses. Psychometrika, 83(2), 453–475. https://doi.org/10.1007/s11336-017-9586-5

12.

Plummer

. (2019). rjags: Bayesian graphical models using MCMC (Version 4.10) [Computer software]. https://CRAN.R-project.org/package=rjags

13.

R Core Team. (2021). R: A language and environment for statistical computing (Version 4.1.0) [Computer software]. https://www.R-project.org/

14.

Ranger

Schmidt

Wolgast

. (2020). The detection of cheating on e-exams in higher education – The performance of several old and some new indicators. Frontiers in Psychology, 11, Article 568825. https://doi.org/10.3389/fpsyg.2020.568825

15.

Samejima

. (2000). Logistic positive exponent family of models: Virtue of asymmetric item characteristic curves. Psychometrika, 65(3), 319–335. https://doi.org/10.1007/BF02296149

16.

San Martín

del Pino

De Boeck

. (2006). IRT models for ability-based guessing. Applied Psychological Measurement, 30(3), 183–203. https://doi.org/10.1177/0146621605282773

17.

Shu

Henson

Luecht

. (2013). Using deterministic, gated item response theory model to detect test cheating due to item compromise. Psychometrika, 78(3), 481–497. https://doi.org/10.1007/s11336-012-9311-3

18.

Sinharay

. (2021). Latent-variable approaches utilizing both item scores and response times to detect test fraud. Open Education Studies, 3(1), 1–16. https://doi.org/10.1515/edu-2020-0137

19.

Sinharay

. (2022). The lack of robustness of a statistic based on the Neyman-Pearson Lemma to violations of its underlying assumptions. Applied Psychological Measurement, 46(1), 19–39. https://doi.org/10.1177/01466216211049209

20.

Spiegelhalter

D. J.

Best

N. G.

Carlin

B. P.

van der Linde

. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64(4), 583–616. https://doi.org/10.1111/1467-9868.00353

21.

Tendeiro

J. N.

Meijer

R. R

. (2012). A CUSUM to detect person misfit: A discussion and some alternatives for existing procedures. Applied Psychological Measurement, 36(5), 420–442. https://doi.org/10.1177/0146621612446305

22.

van der Linden

W. J

. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204. https://doi.org/10.3102/10769986031002181

23.

van der Linden

W. J

. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308. https://doi.org/10.1007/s11336-006-1478-z

24.

van der Linden

W. J

van Krimpen-Stoop

E. M. L. A

. (2003). Using response times to detect aberrant responses in computerized adaptive testing. Psychometrika, 68(2), 251–265. https://doi.org/10.1007/BF02294800

25.

Wang

Shang

. (2018). A two-stage approach to differentiating normal and aberrant behavior in computer based testing. Psychometrika, 83(1), 223–254. https://doi.org/10.1007/s11336-016-9525-x

26.

Zopluoglu

Kasli

Toton

S. L

. (2021). The effect of item preknowledge on response time: Analysis of two datasets using the multiple-group lognormal response time model with a gating mechanism. Educational Measurement: Issues and Practice, 40(3), 42–51. https://doi.org/10.1111/emip.12428

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.85 MB

Two New Models for Item Preknowledge

Abstract

Keywords

Get full access to this article

References

Supplementary Material