In various fields, including college admission, medical board certifications, and military recruitment, high-stakes decisions are frequently made based on scores obtained from large-scale assessments. These decisions necessitate precise and reliable scores that enable valid inferences to be drawn about test-takers. However, the ability of such tests to provide reliable, accurate inference on a test-taker’s performance could be jeopardized by aberrant test-taking practices, for instance, practicing real items prior to the test. As a result, it is crucial for administrators of such assessments to develop strategies that detect potential aberrant test-takers after data collection. The aim of this study is to explore the implementation of machine learning methods in combination with multimodal data fusion strategies that integrate bio-information technology, such as eye-tracking, and psychometric measures, including response times and item responses, to detect aberrant test-taking behaviors in technology-assisted remote testing settings.
AgardC.von DavierA. (2018). The virtual world and reality of testing: Building virtual assessments. In HongJ.LissitzR. (Eds.), Technology enhanced innovative assessment development, modeling, and scoring from an interdisciplinary perspective (pp. 1–30). Information Age Publishing.
2.
AlpaydinE. (2010). Introduction to machine learning (2nd ed.). MIT Press.
3.
AltmanN. S. (1992). An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician, 46(3), 175–185.
4.
BaltrušaitisT.AhujaC.MorencyL. P. (2018). Multimodal machine learning: A survey and taxonomy. IEEE transactions on Pattern Analysis and Machine Intelligence, 41(2), 423–443.
5.
BayoudhK.KnaniR.HamdaouiF.MtibaaA. (2021). A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets. The Visual Computer, 1 -32.
6.
BiauG.ScornetE. (2016). A random forest guided tour. Test, 25(2), 197–227.
7.
BishopC. M. (2006). Pattern recognition and machine learning. Springer.
8.
BottN. T.LangeA.RentzD.BuffaloE.CloptonP.ZolaS. (2017). Web camera based eye tracking to assess visual memory on a visual paired comparison task. Frontiers in Neuroscience, 11, Article 370. https://doi.org/10.3389/fnins.2017.00370
9.
BreimanL. (2001). Random forests. Machine Learning, 45(1), 5–32.
10.
ChenJ.LiX. (2012). Nearest neighbor classifiers: The effect of dataset size, dimensionality and the choice of distance measures. Journal of Applied Statistics, 39(1), 65–78.
11.
CoverT.HartP. (1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1), 21–27.
12.
CutlerD. R.EdwardsT. C.JrBeardK. H.CutlerA.HessK. T.GibsonJ.LawlerJ. J. (2007). Random forests for classification in ecology. Ecology, 88(11), 2783–2792.
13.
DrasgowF.LevineM. V.WilliamsE. A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38(1), 67–86.
14.
DriscollR. (2007). Westside Test Anxiety Scale validation. Education Resources Information Center.
15.
FixE.HodgesJ. L.Jr. (1951). Discriminatory analysis–nonparametric discrimination: Consistency properties (Technical Report No. 4). U.S. Air Force School of Aviation Medicine.
16.
ForgyE. W. (1965). Cluster analysis of multivariate data: Efficiency versus interpretability of classifications. Biometrics, 21(3), 768–769.
17.
FoxJ.-P.MariantiS. (2017). Person-fit statistics for joint models for accuracy and speed. Journal of Educational Measurement, 54(2), 243–262.
18.
GaoJ.LiP.ChenZ.ZhangJ. (2020). A survey on deep learning for multimodal data fusion. Neural Computation, 32(5), 829–864.
19.
GawN.YousefiS.GahrooeiM. R. (2022). Multimodal data fusion for systems improvement: A review. IISE Transactions, 54(11), 1098–1116.
20.
GonzalezT. F. (1985). Clustering to minimize the maximum intercluster distance. Theoretical Computer Science, 38 (January), 293–306.
21.
GoslingS. D.RentfrowP. J.SwannW. B. (2003). A very brief measure of the big-five personality domains. Journal of Research in Personality, 37(6), 504–528.
22.
GuttmanL. (1944). A basis for scaling qualitative data. American Sociological Review, 9, 139–150.
23.
HanJ.KamberM.PeiJ. (2011). Data mining: Concepts and techniques (3rd ed.). Morgan Kaufmann.
24.
HastieT.TibshiraniR.FriedmanJ. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer.
25.
HeidariA.MirniaharikandeheiS.AlmasiS.RouhaniS. (2020). An efficient cheating detection method based on random forest algorithm for large scale online assessments. International Journal of Distance Education Technologies (IJDET), 18(1), 1–19.
26.
HoT. K. (1995, August). Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition (Vol. 1, pp. 278–282). IEEE.
27.
HoT. K. (1998). The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(8), 832–844.
28.
HuangS. C.PareekA.SeyyediS.BanerjeeI.LungrenM. P. (2020). Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Digital Medicine, 3(1), 136.
29.
LahatD.AdaliT.JuttenC. (2015). Multimodal data fusion: An overview of methods, challenges, and prospects. Proceedings of the IEEE, 103(9), 1449–1477.
30.
LiawA.WienerM. (2002). Classification and regression by randomforest. R News, 2(3), 18–22.
31.
LloydS. (1957). Least squares quantization in PCM [Paper presentation]. Bell Telephone Laboratories Paper.
32.
MacQueenJ. (1967, June). Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability (Vol. 1, No. 14, pp. 281–297). Statistics, University of California Press.
33.
ManK.HarringJ. R. (2019). Negative binomial models for visual fixation counts on test items. Educational and Psychological Measurement, 79(4), 617–635. https://doi.org/10.1177/0013164418824148
34.
ManK.HarringJ. R.OuyangY.ThomasS. L. (2018). Response time based nonparametric Kullback-Leibler divergence measure for detecting aberrant test-taking behavior. International Journal of Testing, 18, 155–177. https://doi.org/10.1080/15305058.2018.1429446
35.
ManK.HarringJ. R.SinharayS. (2019). Use of data mining methods to detect test fraud. Journal of Educational Measurement, 56(2), 251–279.
36.
MariantiS.FoxJ.-P.AvetisyanM.VeldkampB. P.TijmstraJ. (2014). Testing for aberrant behavior in response time modeling. Journal of Educational and Behavioral Statistics, 39(6), 426–451.
37.
PanY.WollackJ. A. (2023). A machine learning approach for the simultaneous detection of preknowledge in examinees and items when both are unknown. Educational Measurement: Issues and Practice, 24, 76–78.
38.
QianH.StaniewskaD.ReckaseM.WooA. (2016). Using response time to detect item preknowledge in computer-based licensure examinations. Educational Measurement: Issues and Practice, 35(1), 38–47.
39.
RaschG. (1960). Studies in mathematical psychology: I. probabilistic models for some intelligence and attainment tests. Nielsen & Lydiche.
40.
ShchurV.OseledetsI.KhorunzhinaN. (2018). Pitfalls of oversampling in classification problems and a nested approach to overcome them. In Advances in intelligent systems and computing (Vol. 679, pp. 408–420).
41.
SijtsmaK.MeijerR. R. (1992). A method for investigating the intersection of item response functions in Mokken’s nonparametric IRT model. Applied Psychological Measurement, 16(2), 149–157.
42.
SinharayS. (2018). A new person-fit statistic for the lognormal model for response times. Journal of Educational Measurement, 55(4), 457–476.
43.
SinharayS. (2020). Detection of item preknowledge using response times. Applied Psychological Measurement, 45(5), 376–392.
44.
StoneM. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society. Series B (Methodological), 36(2), 111–147.
45.
TotonS. L.MaynesD. D. (2018). The use of data mining techniques to detect cheating. In JiaoH.LissitzR.Van WieA. (Eds.), Data analytics and psychometrics: Informing assessment practices (pp. 77–108). Information Age Publishing.
van der LindenW. J.GuoF. (2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73, 365–384.
48.
van der LindenW. J.ScramsD.SchnipkeD. L. (1999). Using response-time constraints to control speedness in computerized adaptive testing. Applied Psychological Measurement, 23, 195–210.
49.
WittenI. H.FrankE.HallM. A. (2016). Data mining: Practical machine learning tools and techniques (4th ed.). Morgan Kaufmann.
50.
ZhouT.JiaoH. (2022). Exploration of the stacking ensemble machine learning algorithm for cheating detection in large-scale assessment. Educational and Psychological Measurement, 83(4), 831–854. https://doi.org/10.1177/00131644221117193
51.
ZopluogluC. (2019). Detecting examinees with item preknowledge in large-scale testing using extreme gradient boosting (XGBoost). Educational and Psychological Measurement, 79(5), 931–961.
