This research aimed to investigate the relationship between gaze behaviour dynamics and operator performance.
Background
Individuals differ in their approach when learning a new task often resulting in performance disparity. During training some individuals learn the structure and dynamics of the task and develop a systematic approach, whereas others may achieve the same result albeit with increased perceived workload, or indeed some may fail to achieve superior performance levels. Previous research has shown that comparing gaze of experts with novices can provide unique insights into cognitive functioning of superior performers.
Methods
Twenty-five individuals participated in a computer-based simulation task. The concept of coefficient of variation (CoV) of task scores was used to compute the participants’ consistency of performance. Based on CoV, the cohort was split into two performance categories. The temporal patterns in participants gaze data were transformed using autocorrelation, and recurrence quantification analysis (RQA) was employed to analyse and quantify the patterns.
Results
A Mann–Whitney U analysis demonstrated significantly (p < .01) higher determinism, entropy and laminarity in the superior group compared to the moderate group. Pearson’s correlation revealed a significant (p < .01) negative correlation between the consistency of task performance (CoV) and the RQA measures.
Conclusion
The results demonstrated that eye gaze dynamics can be used as an objective measure of performance. Participants classified as superior performers consistently demonstrated a systematic gaze activity which were in line with the task structure.
Application
The methods presented here are applicable to observe and evaluate operators’ strategic distribution of gaze. Specifically, for tactical monitoring and decision making in task environments where spatial locations of elements-of-interest vary continuously.
AcharyaU. R.SreeS. V.ChattopadhyayS.YuW.AngP. C. A. (2011). Application of recurrence quantification analysis for the automated identification of epileptic EEG signals. International journal of neural systems, 21(03), 199–211. https://doi.org/10.1142/S0129065711002808
2.
AlbertsD. S.GarstkaJ. J.SteinF. P. (2000). Network centric warfare: Developing and leveraging information superiority. Technical report, Assistant Secretary of Defense (C3I/Command Control Research Program
3.
AndersonN. C.BischofW. F.LaidlawK. E.RiskoE. F.KingstoneA. (2013). Recurrence quantification analysis of eye movements. Behavior research methods, 45(3), 842–856. https://doi.org/10.3758/s13428-012-0299-5
4.
AndràC.ArzarelloF.FerraraF.HolmqvistK.LindströmP.RobuttiO.SabenaC. (2009). How students read mathematical representations: An eye tracking study. Proceedings of the 33rd Conference of the International Group for the Psychology of Mathematics Education, 19–24 July, 2009, Thessaloniki, Greece (vol. 2, pp. 49–56). PME Thessaloniki.
5.
AndreasenN. C.ArndtS.CizadloT.O'LearyD. S.WatkinsG. L.PontoL. L. B.HichwaR. D. (1996). Sample Size and Statistical Power in [15O]H2O Studies of Human Cognition. Journal of Cerebral Blood Flow & Metabolism, 16(5), 804–816. https://doi.org/10.1097/00004647-199609000-00005
6.
ArceneauxK.HuberG. A. (2007). What to do (and not do) with multicollinearity in state politics research. State Politics & Policy Quarterly, 7(1), 81-101. https://doi.org/10.1177/153244000700700105.
7.
BoehmU.MatzkeD.GrettonM.CastroS.CooperJ.SkinnerM.StrayerD.HeathcoteA. (2021). Real-time prediction of short-timescale fluctuations in cognitive workload. Cognitive Research: Principles and Implications, 6(1), 1–29. https://doi.org/10.1186/s41235-021-00289-y
8.
BramsS.ZivG.LevinO.SpitzJ.WagemansJ.WilliamsA. M.HelsenW. F. (2019). The relationship between gaze behavior, expertise, and performance: A systematic review. Psychological bulletin, 145(10), 980–1027. https://doi.org/10.1037/bul0000207
9.
BrockmoleJ. R.HambrickD. Z.WindischD. J.HendersonJ. M. (2008). The role of meaning in contextual cueing: Evidence from chess expertise. Quarterly Journal of Experimental Psychology, 61(12), 1886–1896. https://doi.org/10.1080/17470210701781155
10.
BrownT. A. (2015). Confirmatory factor analysis for applied research. Guilford publications.
11.
CocoM. I.DaleR. (2014). Cross-recurrence quantification analysis of categorical and continuous time series: an r package. Frontiers in Psychology, 5, 510. https://doi.org/10.3389/fpsyg.2014.00510
12.
CookeN. J.GormanJ. C.MyersC. W.DuranJ. L. (2013). Interactive team cognition. Cognitive science, 37(2), 255–285. https://doi.org/10.1111/cogs.12009
13.
CummingsM. L.GuerlainS. (2007). Developing operator capacity estimates for supervisory control of autonomous vehicles. Human Factors: The Journal of the Human Factors and Ergonomics Society, 49(1), 1–15. https://doi.org/10.1518/001872007779598109
14.
EckmannJ.KamphorstS. O.RuelleD. (1995). Recurrence plots of dynamical systems. Turbulence, Strange Attractors and Chaos, 16, 441-445. https://doi.org/10.1142/9789812833709_0030.
15.
EndsleyM. R.GarlandD. J. (2000). Theoretical underpinnings of situation awareness: A critical review. Situation awareness analysis and measurement, 1, 24. https://doi.org/10.1201/b12461.
16.
EricssonK. A. (2018). The differential influence of experience, practice, and deliberate practice on the development of superior individual performance of experts.
17.
FongA.HoffmanD.RatwaniR. M. (2016). Making sense of mobile eye-tracking data in the real-world. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 60, pp. 1569–1573). SAGE Publications Sage CA.
18.
GegenfurtnerA.LehtinenE.JarodzkaH.SäljöR. (2017). Effects of eye movement modeling examples on adaptive expertise in medical image diagnosis. Computers & Education, 113, 212-225. https://doi.org/10.1016/j.compedu.2017.06.001.
19.
GegenfurtnerA.LehtinenE.SäljöR. (2011). Expertise differences in the comprehension of visualizations: A meta-analysis of eye-tracking research in professional domains. Educational psychology review, 23(4), 523–552. https://doi.org/10.1007/s10648-011-9174-7
20.
GontarP.MulliganJ. (2016). Cross recurrence analysis as a measure of pilots’ coordination strategy. 32nd Conference of the European Association for Aviation Psychology, 26–30 September, 2016, Cascais, Portugal.
21.
gp3 (2009a). Gazepoint.
22.
gp3 (2009b). Gazepoint gp3 user manual.
23.
GriffinM. M.SteinbrecherT. D. (2013). Large-scale datasets in special education research. International Review of Research in Developmental Disabilities, 45, 155–183. Elsevier.
HarmanW. H. (1989). Tcas- a system for preventing midair collisions. The Lincoln Laboratory Journal, 2(3), 437–457.
26.
HaulandG. (2008). Measuring individual and team situation awareness during planning tasks in training of en route air traffic control. The International Journal of Aviation Psychology, 18(3), 290-304. https://doi.org/10.1080/10508410802168333
27.
HettiarachchiI. T.HanounS.VeerabhadrappaR.JiaD.HoskingS.BhattiA. (2021). Performance quantification and heart rate analysis in a repeated-trial simulation-based training task. In 2021 IEEE International Systems Conference (SysCon), 15 April–5 May, 2021 (pp. 1–7). IEEE.
28.
IskanderJ.HanounS.HettiarachchiI.HossnyM.SalehK.ZhouH.NahavandiS.BhattiA. (2018). Eye behaviour as a hazard perception measure. In 2018 Annual IEEE International Systems Conference (SysCon), Vancouver, BC, 23–26 April, 2018 (pp. 1–6). IEEE.
29.
JacobR. J.KarnK. S. (2003). Eye Tracking in Human-Computer Interaction and Usability Research. In The mind’s eye (pp. 573–605). Elsevier.
30.
JanelleC. M.HillmanC. (2003a). Expert performance in sport. Expert Performance in sports. Advances in Research on Sport Expertise, 21–23.
31.
JanelleC. M.HillmanC. (2003b). Expert performance in sport. Expert Performance in Sports: Advances in Research on Sport Expertise, 25.
32.
KokE. M.JarodzkaH.de BruinA. B.BinAmirH. A.RobbenS. G.van MerriënboerJ. J. (2016). Systematic viewing in radiology: seeing more, missing less?Advances in Health Sciences Education, 21(1), 189–205. https://doi.org/10.1007/s10459-015-9624-y
33.
LaveryM. R.AcharyaP.SivoS. A.XuL. (2019). Number of predictors and multicollinearity: What are their effects on error and bias in regression?Communications in Statistics - Simulation and Computation, 48(1), 27–38. https://doi.org/10.1080/03610918.2017.1371750
MarwanN.KurthsJ. (2002). Nonlinear analysis of bivariate data with cross recurrence plots. Physics Letters, Section A: General, Atomic and Solid State Physics, 302(5), 299–307. https://doi.org/10.1016/S0375-9601(02)01170-2
McKnightP. E.NajabJ. (2010). Mann-Whitney U Test (pp. 1). American Cancer Society.
38.
NehmeC. E. (2009). Modeling human supervisory control in heterogeneous unmanned vehicle systems. Technical report, Massachusetts Inst of tech Cambridge dept of aeronautics and astronautics.
39.
NewellA.RosenbloomP. S. (1981). Mechanisms of skill acquisition and the law of practice. Cognitive skills and their acquisition, 1, 1–55.
40.
OuyangG.LiX.DangC.RichardsD. A. (2008). Using recurrence plot for determinism analysis of EEG recordings in genetic absence epilepsy rats. Clinical neurophysiology, 119(8), 1747–1755. https://doi.org/10.1016/j.clinph.2008.04.005
41.
RussellS. M.FunkeG. J.FlachJ. M.WatamaniukS. N.StrangA. J.MillerB. T.DukesA.MenkeL.BrownR. (2014). Alternative indices of performance: An exploration of eye gaze metrics in a visual puzzle task. Technical report, Air Force Research Lab Wright-Patterson Afb Oh Human Performance Wing (711TH
42.
ShechtmanO. (2013). The coefficient of variation as an index of measurement reliability. In Methods of clinical epidemiology (pp. 39–49). Springer.
43.
TienT.PucherP. H.SodergrenM. H.SriskandarajahK.YangG.-Z.DarziA. (2014). Eye tracking for skills assessment and training: a systematic review. Journal of Surgical Research, 191(1), 169–178. https://doi.org/10.1016/j.jss.2014.04.032
44.
ToddP. M.HillsT. T.RobbinsT. W. (Eds.) (2012). Visual Search. The MIT Press.
45.
UttJ.McCalmontJ.DeschenesM. (2005). Development of a sense and avoid system. Infotech@ Aerospace, 26–29 September, 2022, Arlington, Virginia (pp. 7177).
46.
VeerabhadrappaR.HettiarachchiI. T.BhattiA. (2021). Using recurrence quantification analysis to quantify the physiological synchrony in dyadic EC data. 2021 IEEE International Systems Conference (SysCon), 25–28 April, 2022, Montreal, QC (pp. 1–8. IEEE.
47.
VeerabhadrappaR.HettiarachchiI. T.BhattiA. (2022a). Gaze convergence based collaborative performance prediction in a 3-member joint activity setting. 2022 IEEE International Systems Conference (SysCon), 25–28 April, 2022, Montreal, QC (pp. 1–7. IEEE.
48.
VeerabhadrappaR.HettiarachchiI. T.BhattiA. (2022b). Using eye-tracking to investigate the effect of gaze co-occurrence and distribution on collaborative performance. 2022 IEEE International Systems Conference (SysCon), 25–28 April, 2022, Montreal, QC (pp. 1–8). IEEE.
ZbilutJ. P.ThomassonN.WebberC. L. (2002). Recurrence quantification analysis as a tool for nonlinear exploration of nonstationary cardiac signals. Medical engineering & physics, 24(1), 53–60. https://doi.org/10.1016/s1350-4533(01)00112-6
51.
ZhengK.ChenG.HerschelM.NgiamK. Y.OoiB. C.GaoJ. (2021). Pace: Learning effective task decomposition for human-in-the-loop healthcare delivery. Proceedings of the 2021 International Conference on Management of Data, 20–25 June 2014, Xi'an, China (pp. 2156–2168).
52.
ZugalS.PinggeraJ. (2014). Low-cost eye-trackers: Useful for information systems research?International Conference on Advanced Information Systems Engineering, 16–20 June 2014, Thessaloniki, Greece (pp. 159–170). Springer.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
