Evaluating Operator Training Performance Using Recurrence Quantification Analysis of Autocorrelation Transformed Eye Gaze Data

Abstract

Objective

This research aimed to investigate the relationship between gaze behaviour dynamics and operator performance.

Background

Individuals differ in their approach when learning a new task often resulting in performance disparity. During training some individuals learn the structure and dynamics of the task and develop a systematic approach, whereas others may achieve the same result albeit with increased perceived workload, or indeed some may fail to achieve superior performance levels. Previous research has shown that comparing gaze of experts with novices can provide unique insights into cognitive functioning of superior performers.

Methods

Twenty-five individuals participated in a computer-based simulation task. The concept of coefficient of variation (CoV) of task scores was used to compute the participants’ consistency of performance. Based on CoV, the cohort was split into two performance categories. The temporal patterns in participants gaze data were transformed using autocorrelation, and recurrence quantification analysis (RQA) was employed to analyse and quantify the patterns.

Results

A Mann–Whitney U analysis demonstrated significantly (p < .01) higher determinism, entropy and laminarity in the superior group compared to the moderate group. Pearson’s correlation revealed a significant (p < .01) negative correlation between the consistency of task performance (CoV) and the RQA measures.

Conclusion

The results demonstrated that eye gaze dynamics can be used as an objective measure of performance. Participants classified as superior performers consistently demonstrated a systematic gaze activity which were in line with the task structure.

Application

The methods presented here are applicable to observe and evaluate operators’ strategic distribution of gaze. Specifically, for tactical monitoring and decision making in task environments where spatial locations of elements-of-interest vary continuously.

Keywords

visual strategies recurrence quantification analysis autocorrelation performance eye-tracking

Get full access to this article

View all access options for this article.

References

Acharya

U. R.

Sree

S. V.

Chattopadhyay

Ang

P. C. A.

(2011). Application of recurrence quantification analysis for the automated identification of epileptic EEG signals. International journal of neural systems, 21(03), 199–211. https://doi.org/10.1142/S0129065711002808

Alberts

D. S.

Garstka

J. J.

Stein

F. P.

(2000). Network centric warfare: Developing and leveraging information superiority. Technical report, Assistant Secretary of Defense (C3I/Command Control Research Program

Anderson

N. C.

Bischof

W. F.

Laidlaw

K. E.

Risko

E. F.

Kingstone

(2013). Recurrence quantification analysis of eye movements. Behavior research methods, 45(3), 842–856. https://doi.org/10.3758/s13428-012-0299-5

Andrà

Arzarello

Ferrara

Holmqvist

Lindström

Robutti

Sabena

(2009). How students read mathematical representations: An eye tracking study. Proceedings of the 33rd Conference of the International Group for the Psychology of Mathematics Education, 19–24 July, 2009, Thessaloniki, Greece (vol. 2, pp. 49–56). PME Thessaloniki.

Andreasen

N. C.

Arndt

Cizadlo

O'Leary

D. S.

Watkins

G. L.

Ponto

L. L. B.

Hichwa

R. D.

(1996). Sample Size and Statistical Power in [15O]H2O Studies of Human Cognition. Journal of Cerebral Blood Flow & Metabolism, 16(5), 804–816. https://doi.org/10.1097/00004647-199609000-00005

Arceneaux

Huber

G. A.

(2007). What to do (and not do) with multicollinearity in state politics research. State Politics & Policy Quarterly, 7(1), 81-101. https://doi.org/10.1177/153244000700700105.

Boehm

Matzke

Gretton

Castro

Cooper

Skinner

Strayer

Heathcote

(2021). Real-time prediction of short-timescale fluctuations in cognitive workload. Cognitive Research: Principles and Implications, 6(1), 1–29. https://doi.org/10.1186/s41235-021-00289-y

Brams

Ziv

Levin

Spitz

Wagemans

Williams

A. M.

Helsen

W. F.

(2019). The relationship between gaze behavior, expertise, and performance: A systematic review. Psychological bulletin, 145(10), 980–1027. https://doi.org/10.1037/bul0000207

Brockmole

J. R.

Hambrick

D. Z.

Windisch

D. J.

Henderson

J. M.

(2008). The role of meaning in contextual cueing: Evidence from chess expertise. Quarterly Journal of Experimental Psychology, 61(12), 1886–1896. https://doi.org/10.1080/17470210701781155

10.

Brown

T. A.

(2015). Confirmatory factor analysis for applied research. Guilford publications.

11.

Coco

M. I.

Dale

(2014). Cross-recurrence quantification analysis of categorical and continuous time series: an r package. Frontiers in Psychology, 5, 510. https://doi.org/10.3389/fpsyg.2014.00510

12.

Cooke

N. J.

Gorman

J. C.

Myers

C. W.

Duran

J. L.

(2013). Interactive team cognition. Cognitive science, 37(2), 255–285. https://doi.org/10.1111/cogs.12009

13.

Cummings

M. L.

Guerlain

(2007). Developing operator capacity estimates for supervisory control of autonomous vehicles. Human Factors: The Journal of the Human Factors and Ergonomics Society, 49(1), 1–15. https://doi.org/10.1518/001872007779598109

14.

Eckmann

Kamphorst

S. O.

Ruelle

(1995). Recurrence plots of dynamical systems. Turbulence, Strange Attractors and Chaos, 16, 441-445. https://doi.org/10.1142/9789812833709_0030.

15.

Endsley

M. R.

Garland

D. J.

(2000). Theoretical underpinnings of situation awareness: A critical review. Situation awareness analysis and measurement, 1, 24. https://doi.org/10.1201/b12461.

16.

Ericsson

K. A.

(2018). The differential influence of experience, practice, and deliberate practice on the development of superior individual performance of experts.

17.

Fong

Hoffman

Ratwani

R. M.

(2016). Making sense of mobile eye-tracking data in the real-world. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 60, pp. 1569–1573). SAGE Publications Sage CA.

18.

Gegenfurtner

Lehtinen

Jarodzka

Säljö

(2017). Effects of eye movement modeling examples on adaptive expertise in medical image diagnosis. Computers & Education, 113, 212-225. https://doi.org/10.1016/j.compedu.2017.06.001.

19.

Gegenfurtner

Lehtinen

Säljö

(2011). Expertise differences in the comprehension of visualizations: A meta-analysis of eye-tracking research in professional domains. Educational psychology review, 23(4), 523–552. https://doi.org/10.1007/s10648-011-9174-7

20.

Gontar

Mulligan

(2016). Cross recurrence analysis as a measure of pilots’ coordination strategy. 32nd Conference of the European Association for Aviation Psychology, 26–30 September, 2016, Cascais, Portugal.

21.

gp3 (2009a). Gazepoint.

22.

gp3 (2009b). Gazepoint gp3 user manual.

23.

Griffin

M. M.

Steinbrecher

T. D.

(2013). Large-scale datasets in special education research. International Review of Research in Developmental Disabilities, 45, 155–183. Elsevier.

24.

Jin

Chang

Zhang

(2020). Predicting webpage aesthetics with heatmap entropy. Behaviour & Information Technology, 40(7), 1–15. https://doi.org/10.1080/0144929X.2020.1717626

25.

Harman

W. H.

(1989). Tcas- a system for preventing midair collisions. The Lincoln Laboratory Journal, 2(3), 437–457.

26.

Hauland

(2008). Measuring individual and team situation awareness during planning tasks in training of en route air traffic control. The International Journal of Aviation Psychology, 18(3), 290-304. https://doi.org/10.1080/10508410802168333

27.

Hettiarachchi

I. T.

Hanoun

Veerabhadrappa

Jia

Hosking

Bhatti

(2021). Performance quantification and heart rate analysis in a repeated-trial simulation-based training task. In 2021 IEEE International Systems Conference (SysCon), 15 April–5 May, 2021 (pp. 1–7). IEEE.

28.

Iskander

Hanoun

Hettiarachchi

Hossny

Saleh

Zhou

Nahavandi

Bhatti

(2018). Eye behaviour as a hazard perception measure. In 2018 Annual IEEE International Systems Conference (SysCon), Vancouver, BC, 23–26 April, 2018 (pp. 1–6). IEEE.

29.

Jacob

R. J.

Karn

K. S.

(2003). Eye Tracking in Human-Computer Interaction and Usability Research. In The mind’s eye (pp. 573–605). Elsevier.

30.

Janelle

C. M.

Hillman

(2003a). Expert performance in sport. Expert Performance in sports. Advances in Research on Sport Expertise, 21–23.

31.

Janelle

C. M.

Hillman

(2003b). Expert performance in sport. Expert Performance in Sports: Advances in Research on Sport Expertise, 25.

32.

Kok

E. M.

Jarodzka

de Bruin

A. B.

BinAmir

H. A.

Robben

S. G.

van Merriënboer

J. J.

(2016). Systematic viewing in radiology: seeing more, missing less? Advances in Health Sciences Education, 21(1), 189–205. https://doi.org/10.1007/s10459-015-9624-y

33.

Lavery

M. R.

Acharya

Sivo

S. A.

(2019). Number of predictors and multicollinearity: What are their effects on error and bias in regression? Communications in Statistics - Simulation and Computation, 48(1), 27–38. https://doi.org/10.1080/03610918.2017.1371750

34.

Marsden

(1974). Cubic spline interpolation of continuous functions. Journal of Approximation Theory, 10(2), 103–111. https://doi.org/10.1016/0021-9045(74)90109-9

35.

Marwan

Kurths

(2002). Nonlinear analysis of bivariate data with cross recurrence plots. Physics Letters, Section A: General, Atomic and Solid State Physics, 302(5), 299–307. https://doi.org/10.1016/S0375-9601(02)01170-2

36.

Marwan

Romano

M. C.

Thiel

Kurths

(2007). Recurrence plots for the analysis of complex systems. Physics reports, 438(5–6):237–329. https://doi.org/10.1016/j.physrep.2006.11.001

37.

McKnight

P. E.

Najab

(2010). Mann-Whitney U Test (pp. 1). American Cancer Society.

38.

Nehme

C. E.

(2009). Modeling human supervisory control in heterogeneous unmanned vehicle systems. Technical report, Massachusetts Inst of tech Cambridge dept of aeronautics and astronautics.

39.

Newell

Rosenbloom

P. S.

(1981). Mechanisms of skill acquisition and the law of practice. Cognitive skills and their acquisition, 1, 1–55.

40.

Ouyang

Dang

Richards

D. A.

(2008). Using recurrence plot for determinism analysis of EEG recordings in genetic absence epilepsy rats. Clinical neurophysiology, 119(8), 1747–1755. https://doi.org/10.1016/j.clinph.2008.04.005

41.

Russell

S. M.

Funke

G. J.

Flach

J. M.

Watamaniuk

S. N.

Strang

A. J.

Miller

B. T.

Dukes

Menke

Brown

(2014). Alternative indices of performance: An exploration of eye gaze metrics in a visual puzzle task. Technical report, Air Force Research Lab Wright-Patterson Afb Oh Human Performance Wing (711TH

42.

Shechtman

(2013). The coefficient of variation as an index of measurement reliability. In Methods of clinical epidemiology (pp. 39–49). Springer.

43.

Tien

Pucher

P. H.

Sodergren

M. H.

Sriskandarajah

Yang

G.-Z.

Darzi

(2014). Eye tracking for skills assessment and training: a systematic review. Journal of Surgical Research, 191(1), 169–178. https://doi.org/10.1016/j.jss.2014.04.032

44.

Todd

P. M.

Hills

T. T.

Robbins

T. W.

(Eds.) (2012). Visual Search. The MIT Press.

45.

Utt

McCalmont

Deschenes

(2005). Development of a sense and avoid system. Infotech@ Aerospace, 26–29 September, 2022, Arlington, Virginia (pp. 7177).

46.

Veerabhadrappa

Hettiarachchi

I. T.

Bhatti

(2021). Using recurrence quantification analysis to quantify the physiological synchrony in dyadic EC data. 2021 IEEE International Systems Conference (SysCon), 25–28 April, 2022, Montreal, QC (pp. 1–8. IEEE.

47.

Veerabhadrappa

Hettiarachchi

I. T.

Bhatti

(2022a). Gaze convergence based collaborative performance prediction in a 3-member joint activity setting. 2022 IEEE International Systems Conference (SysCon), 25–28 April, 2022, Montreal, QC (pp. 1–7. IEEE.

48.

Veerabhadrappa

Hettiarachchi

I. T.

Bhatti

(2022b). Using eye-tracking to investigate the effect of gaze co-occurrence and distribution on collaborative performance. 2022 IEEE International Systems Conference (SysCon), 25–28 April, 2022, Montreal, QC (pp. 1–8). IEEE.

49.

Veerabhadrappa

Lim

Nguyen

Berk

Tye

Monaghan

Nahavandi

Bhatti

(2016). Unified selective sorting approach to analyse multi-electrode extracellular data. Scientific reports, 6(1), 28533. https://doi.org/10.1038/srep28533

50.

Zbilut

J. P.

Thomasson

Webber

C. L.

(2002). Recurrence quantification analysis as a tool for nonlinear exploration of nonstationary cardiac signals. Medical engineering & physics, 24(1), 53–60. https://doi.org/10.1016/s1350-4533(01)00112-6

51.

Zheng

Chen

Herschel

Ngiam

K. Y.

Ooi

B. C.

Gao

(2021). Pace: Learning effective task decomposition for human-in-the-loop healthcare delivery. Proceedings of the 2021 International Conference on Management of Data, 20–25 June 2014, Xi'an, China (pp. 2156–2168).

52.

Zugal

Pinggera

(2014). Low-cost eye-trackers: Useful for information systems research? International Conference on Advanced Information Systems Engineering, 16–20 June 2014, Thessaloniki, Greece (pp. 159–170). Springer.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.27 MB