Detecting Aberrant Behavior and Item Preknowledge: A Comparison of Mixture Modeling Method and Residual Method

Abstract

The modern web-based technology greatly popularizes computer-administered testing, also known as online testing. When these online tests are administered continuously within a certain “testing window,” many items are likely to be exposed and compromised, posing a type of test security concern. In addition, if the testing time is limited, another recognized aberrant behavior is rapid guessing, which refers to quickly answering an item without processing its meaning. Both cheating behavior and rapid guessing result in extremely short response times. This article introduces a mixture hierarchical item response theory model, using both response accuracy and response time information, to help differentiate aberrant behavior from normal behavior. The model-based approach is compared to the Bayesian residual-based fit statistic in both simulation study and two real data examples. Results show that the mixture model approach consistently outperforms the residual method in terms of correct detection rate and false positive error rate, in particular when the proportion of aberrance is high. Moreover, the model-based approach is also able to correctly identify compromised items better than residual method.

Keywords

response time aberrant behavior item preknowledge person-fit mixture model item response theory

Get full access to this article

View all access options for this article.

References

Bolt

D. M.

Cohen

A. S.

Wollack

J. A

. (2002). Item parameter estimation under conditions of test speededness: Application of a mixture Rasch model with ordinal constraints. Journal of Educational Measurement, 39, 331–348.

Boughton

K. A.

Yamamoto

. (2007). A HYBRID model for test speededness. In von Davier

Carstensen

C. H.

. (Eds.), Multivariate and mixture distribution Rasch models (pp. 147–156). New York, NY: Springer.

Chang

Y. W.

Tsai

R. C.

Hsu

N. J.

(2014). A speeded item response model: Leave the harder till later. Psychometrika, 79, 255–274.

Cizek

G. J.

(1999). Cheating on tests: How to do it, detect it, and prevent it. New York, NY: Routledge.

Dean

Martineau

(2012). A state perspective on enhancing assessment and accountability systems through systematic implementation of technology. In Lissitz

R. W.

Jiao

(Eds.), Computers and their impact on state assessment: Recent history and predictions for the future (pp. 55–77). Charlotte, NC: Information Age.

Depaoli

(2013). Mixture class recovery in GMM under varying degrees of class separation: Frequentist versus Bayesian estimation. Psychological Methods, 18, 186–219.

Fan

Wang

Chang

H.-H.

Douglas

(2012). Utilizing response time distributions for item selection in CAT. Journal of Educational and Behavioral Statistics, 37, 655–670.

Karabatsos

(2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education, 16, 277–298.

Lievens

Chapman

D. S.

(2009). Recruitment and selection. In Wilkinson

Redman

Snell

Bacon

(Eds.), The SAGE handbook of human resource management (pp. 133–154). London, England: Sage.

10.

Marianti

Fox

G. J. A.

Avetisyan

Veldkamp

B. P.

Tijmstra

. (2014). Testing for aberrant behavior in response time modeling. Journal of Educational and Behavioral Statistics, 39, 426–451. doi:10.3102/1076998614559412

11.

McLeod

L. D.

Lewis

(1999). Detecting item memorization in the CAT environment. Applied Psychological Measurement, 23, 147–160.

12.

Meijer

R. R.

Sijtsma

(1995). Detection of aberrant item score patterns: A review of recent developments. Applied Measurement in Education, 8, 261–272.

13.

Meijer

R. R.

Sijtsma

(2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107–135.

14.

Meyer

J. P.

(2010). A mixture Rasch model with item response time components. Applied Psychological Measurement, 34, 521–538.

15.

Qian

Staniewska

Reckase

Woo

(2016). Using response time to detect item preknowledge in computer-based licensure examinations. Educational Measurement: Issues and Practice, 35, 38–47.

16.

Ranger

(2013). A note on the hierarchical model for responses and response times in tests of van der Linden (2007). Psychometrika, 78, 538–544.

17.

Rouder

J. N.

Sun

Speckman

P. L.

Zhou

(2003). A hierarchical Bayesian statistical framework for response time distributions. Psychometrika, 68, 589–606.

18.

Sackett

P. R.

Lievens

(2008). Personnel selection. Annual Review of Psychology, 59, 419–450.

19.

Schnipke

D. L.

Scrams

D. J.

(1997). Modeling response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34, 213–232.

20.

Tolvanen

(2008). Latent growth mixture modeling: A simulation study (Unpublished doctoral dissertation). University of Jyväskylä, Jyväskylä, Finland.

21.

van der Linden

W. J.

(2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287–308.

22.

van der Linden

W. J.

Entink

R. H. K.

Fox

J.-P.

(2010). IRT parameter estimation with response times as collateral information. Applied Psychological Measurement, 34, 327–347.

23.

van der Linden

W. J.

Guo

(2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73, 365–384.

24.

van der Linden

W. J.

van Krimpen-Stoop

E. M.

(2003). Using response times to detect aberrant responses in computerized adaptive testing. Psychometrika, 68, 251–265.

25.

van Krimpen-Stoop

E. M.

Meijer

R. R

. (2000). Detecting person misfit in adaptive testing using statistical process control techniques. In van der Linden

W. J.

Glas

C. A. W.

(Eds.), Computerized adaptive testing: Theory and practice (pp. 211–219). Norwell, MA: Kluwer Academic.

26.

Wang

Chang

H. H.

Douglas

J. A.

(2013a). The linear transformation model with frailties for the analysis of item response times. British Journal of Mathematical and Statistical Psychology, 66, 144–168.

27.

Wang

Fan

Chang

H.-H.

Douglas

J. A.

(2013b). A semiparametric model for jointly analyzing response times and accuracy in computerized testing. Journal of Educational and Behavioral Statistics, 38, 381–417.

28.

Wang

Shu

Shang

(2015). Assessing item-level fit for the DINA model. Applied Psychological Measurement, 39, 525–538.

29.

Wang

(2015). A mixture hierarchical model for response times and response accuracy. British Journal of Mathematical and Statistical Psychology, 68, 456–477.

30.

Wang

Zheng

Chang

H.-H.

(2014). Does standard deviation matter? Using “standard deviation” to quantify security of multistage testing. Psychometrika, 79, 154–174.

31.

Wise

S. L

. (2017). Rapid-guessing behavior: Its identification, interpretation, and implications. Educational Measurement: Issues and Practice. doi:10.1111/emip.12165

32.

Wise

S. L.

DeMars

C. E.

(2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43, 19–38.

33.

Wise

S. L.

Kong

(2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18, 163–183.

34.

Yang

(2007). Methods of identifying individual guessers from item response data. Educational and Psychological Measurement, 67, 745–764.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.19 MB