Multimodal Data Fusion to Detect Preknowledge Test-Taking Behavior Using Machine Learning

Abstract

In various fields, including college admission, medical board certifications, and military recruitment, high-stakes decisions are frequently made based on scores obtained from large-scale assessments. These decisions necessitate precise and reliable scores that enable valid inferences to be drawn about test-takers. However, the ability of such tests to provide reliable, accurate inference on a test-taker’s performance could be jeopardized by aberrant test-taking practices, for instance, practicing real items prior to the test. As a result, it is crucial for administrators of such assessments to develop strategies that detect potential aberrant test-takers after data collection. The aim of this study is to explore the implementation of machine learning methods in combination with multimodal data fusion strategies that integrate bio-information technology, such as eye-tracking, and psychometric measures, including response times and item responses, to detect aberrant test-taking behaviors in technology-assisted remote testing settings.

Keywords

technology enhanced assessment cheating detection multimodal data fusion item response theory response times eye-tracking

Get full access to this article

View all access options for this article.

References

Agard

von Davier

(2018). The virtual world and reality of testing: Building virtual assessments. In Hong

Lissitz

(Eds.), Technology enhanced innovative assessment development, modeling, and scoring from an interdisciplinary perspective (pp. 1–30). Information Age Publishing.

Alpaydin

(2010). Introduction to machine learning (2nd ed.). MIT Press.

Altman

N. S.

(1992). An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician, 46(3), 175–185.

Baltrušaitis

Ahuja

Morency

L. P.

(2018). Multimodal machine learning: A survey and taxonomy. IEEE transactions on Pattern Analysis and Machine Intelligence, 41(2), 423–443.

Bayoudh

Knani

Hamdaoui

Mtibaa

(2021). A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets. The Visual Computer, 1 -32.

Biau

Scornet

(2016). A random forest guided tour. Test, 25(2), 197–227.

Bishop

C. M.

(2006). Pattern recognition and machine learning. Springer.

Bott

N. T.

Lange

Rentz

Buffalo

Clopton

Zola

(2017). Web camera based eye tracking to assess visual memory on a visual paired comparison task. Frontiers in Neuroscience, 11, Article 370. https://doi.org/10.3389/fnins.2017.00370

Breiman

(2001). Random forests. Machine Learning, 45(1), 5–32.

10.

Chen

(2012). Nearest neighbor classifiers: The effect of dataset size, dimensionality and the choice of distance measures. Journal of Applied Statistics, 39(1), 65–78.

11.

Cover

Hart

(1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1), 21–27.

12.

Cutler

D. R.

Edwards

T. C.

Jr Beard

K. H.

Cutler

Hess

K. T.

Gibson

Lawler

J. J.

(2007). Random forests for classification in ecology. Ecology, 88(11), 2783–2792.

13.

Drasgow

Levine

M. V.

Williams

E. A.

(1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38(1), 67–86.

14.

Driscoll

(2007). Westside Test Anxiety Scale validation. Education Resources Information Center.

15.

Fix

Hodges

J. L.

Jr. (1951). Discriminatory analysis–nonparametric discrimination: Consistency properties (Technical Report No. 4). U.S. Air Force School of Aviation Medicine.

16.

Forgy

E. W.

(1965). Cluster analysis of multivariate data: Efficiency versus interpretability of classifications. Biometrics, 21(3), 768–769.

17.

Fox

J.-P.

Marianti

(2017). Person-fit statistics for joint models for accuracy and speed. Journal of Educational Measurement, 54(2), 243–262.

18.

Gao

Chen

Zhang

(2020). A survey on deep learning for multimodal data fusion. Neural Computation, 32(5), 829–864.

19.

Gaw

Yousefi

Gahrooei

M. R.

(2022). Multimodal data fusion for systems improvement: A review. IISE Transactions, 54(11), 1098–1116.

20.

Gonzalez

T. F.

(1985). Clustering to minimize the maximum intercluster distance. Theoretical Computer Science, 38 (January), 293–306.

21.

Gosling

S. D.

Rentfrow

P. J.

Swann

W. B.

(2003). A very brief measure of the big-five personality domains. Journal of Research in Personality, 37(6), 504–528.

22.

Guttman

(1944). A basis for scaling qualitative data. American Sociological Review, 9, 139–150.

23.

Han

Kamber

Pei

(2011). Data mining: Concepts and techniques (3rd ed.). Morgan Kaufmann.

24.

Hastie

Tibshirani

Friedman

(2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer.

25.

Heidari

Mirniaharikandehei

Almasi

Rouhani

(2020). An efficient cheating detection method based on random forest algorithm for large scale online assessments. International Journal of Distance Education Technologies (IJDET), 18(1), 1–19.

26.

T. K.

(1995, August). Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition (Vol. 1, pp. 278–282). IEEE.

27.

T. K.

(1998). The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(8), 832–844.

28.

Huang

S. C.

Pareek

Seyyedi

Banerjee

Lungren

M. P.

(2020). Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Digital Medicine, 3(1), 136.

29.

Lahat

Adali

Jutten

(2015). Multimodal data fusion: An overview of methods, challenges, and prospects. Proceedings of the IEEE, 103(9), 1449–1477.

30.

Liaw

Wiener

(2002). Classification and regression by randomforest. R News, 2(3), 18–22.

31.

Lloyd

(1957). Least squares quantization in PCM [Paper presentation]. Bell Telephone Laboratories Paper.

32.

MacQueen

(1967, June). Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability (Vol. 1, No. 14, pp. 281–297). Statistics, University of California Press.

33.

Man

Harring

J. R.

(2019). Negative binomial models for visual fixation counts on test items. Educational and Psychological Measurement, 79(4), 617–635. https://doi.org/10.1177/0013164418824148

34.

Man

Harring

J. R.

Ouyang

Thomas

S. L.

(2018). Response time based nonparametric Kullback-Leibler divergence measure for detecting aberrant test-taking behavior. International Journal of Testing, 18, 155–177. https://doi.org/10.1080/15305058.2018.1429446

35.

Man

Harring

J. R.

Sinharay

(2019). Use of data mining methods to detect test fraud. Journal of Educational Measurement, 56(2), 251–279.

36.

Marianti

Fox

J.-P.

Avetisyan

Veldkamp

B. P.

Tijmstra

(2014). Testing for aberrant behavior in response time modeling. Journal of Educational and Behavioral Statistics, 39(6), 426–451.

37.

Pan

Wollack

J. A.

(2023). A machine learning approach for the simultaneous detection of preknowledge in examinees and items when both are unknown. Educational Measurement: Issues and Practice, 24, 76–78.

38.

Qian

Staniewska

Reckase

Woo

(2016). Using response time to detect item preknowledge in computer-based licensure examinations. Educational Measurement: Issues and Practice, 35(1), 38–47.

39.

Rasch

(1960). Studies in mathematical psychology: I. probabilistic models for some intelligence and attainment tests. Nielsen & Lydiche.

40.

Shchur

Oseledets

Khorunzhina

(2018). Pitfalls of oversampling in classification problems and a nested approach to overcome them. In Advances in intelligent systems and computing (Vol. 679, pp. 408–420).

41.

Sijtsma

Meijer

R. R.

(1992). A method for investigating the intersection of item response functions in Mokken’s nonparametric IRT model. Applied Psychological Measurement, 16(2), 149–157.

42.

Sinharay

(2018). A new person-fit statistic for the lognormal model for response times. Journal of Educational Measurement, 55(4), 457–476.

43.

Sinharay

(2020). Detection of item preknowledge using response times. Applied Psychological Measurement, 45(5), 376–392.

44.

Stone

(1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society. Series B (Methodological), 36(2), 111–147.

45.

Toton

S. L.

Maynes

D. D.

(2018). The use of data mining techniques to detect cheating. In Jiao

Lissitz

Van Wie

(Eds.), Data analytics and psychometrics: Informing assessment practices (pp. 77–108). Information Age Publishing.

46.

Toton

S. L.

Maynes

D. D.

(2019). Detecting examinees with pre-knowledge in experimental data using conditional scaling of response times. Frontiers in Education, 4, Article 49. https://www.frontiersin.org/article/10.3389/feduc.2019.00049

47.

van der Linden

W. J.

Guo

(2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73, 365–384.

48.

van der Linden

W. J.

Scrams

Schnipke

D. L.

(1999). Using response-time constraints to control speedness in computerized adaptive testing. Applied Psychological Measurement, 23, 195–210.

49.

Witten

I. H.

Frank

Hall

M. A.

(2016). Data mining: Practical machine learning tools and techniques (4th ed.). Morgan Kaufmann.

50.

Zhou

Jiao

(2022). Exploration of the stacking ensemble machine learning algorithm for cheating detection in large-scale assessment. Educational and Psychological Measurement, 83(4), 831–854. https://doi.org/10.1177/00131644221117193

51.

Zopluoglu

(2019). Detecting examinees with item preknowledge in large-scale testing using extreme gradient boosting (XGBoost). Educational and Psychological Measurement, 79(5), 931–961.