In Physical Human–Robot Interaction (pHRI), the need to learn the robot’s motor-control dynamics is associated with increased cognitive load. Eye-tracking metrics can help understand the dynamics of fluctuating mental workload over the course of learning.
Objective
The aim of this study was to test eye-tracking measures’ sensitivity and reliability to variations in task difficulty, as well as their performance-prediction capability, in physical human–robot collaboration tasks involving an industrial robot for object comanipulation.
Methods
Participants (9M, 9F) learned to coperform a virtual pick-and-place task with a bimanual robot over multiple trials. Joint stiffness of the robot was manipulated to increase motor-coordination demands. The psychometric properties of eye-tracking measures and their ability to predict performance was investigated.
Results
Stationary Gaze Entropy and pupil diameter were the most reliable and sensitive measures of workload associated with changes in task difficulty and learning. Increased task difficulty was more likely to result in a robot-monitoring strategy. Eye-tracking measures were able to predict the occurrence of success or failure in each trial with 70% sensitivity and 71% accuracy.
Conclusion
The sensitivity and reliability of eye-tracking measures was acceptable, although values were lower than those observed in cognitive domains. Measures of gaze behaviors indicative of visual monitoring strategies were most sensitive to task difficulty manipulations, and should be explored further for the pHRI domain where motor-control and internal-model formation will likely be strong contributors to workload.
Application
Future collaborative robots can adapt to human cognitive state and skill-level measured using eye-tracking measures of workload and visual attention.
AronsonR. M.AdmoniH. (2018 June 26-30). Gaze for error detection during human-robot shared manipulation. In Proceedings of RSS Workshop: Towards a Framework for Joint Action,Pittsburgh, PA, UnitedStates. Retrieved September 27, 2023 fromhttp://packages.personalrobotics.ri.cmu.edu/assets/pubs/fja_rss2018_aronson.pdf
BecserN.SandT.ZwartJ.-A. (1998). Reliability of cephalic thermal thresholds in healthy subjects. Cephalalgia: An International Journal of Headache, 18(8), 574–582. https://doi.org/10.1046/j.1468-2982.1998.1808574.x
4.
BommerS. C.FendleyM. (2018). A theoretical framework for evaluating mental workload resources in human systems design for manufacturing operations. International Journal of Industrial Ergonomics, 63, 7–17. https://doi.org/10.1016/j.ergon.2016.10.007
5.
BrownD. A.LeeT. D.ReinkensmeyerD. J.DuarteJ. E. (2016). Designing robots that challenge to optimize motor learning. In ReinkensmeyerD. J.DietzV. (Eds.), Neurorehabilitation technology, (pp. 39–58). Springer International Publishing. https://doi.org/10.1007/978-3-319-28603-7_3
6.
ByrnesJ. P.MillerD. C.SchaferW. D. (1999). Gender differences in risk taking: A meta-analysis. Psychological Bulletin, 125(3), 367–383. https://doi.org/10.1037/0033-2909.125.3.367
7.
ChadwellA.KenneyL.ThiesS.GalpinA.HeadJ. (2016). The reality of myoelectric prostheses: Understanding what makes these devices difficult for some users to control. Frontiers in Neurorobotics, 10, 7. https://doi.org/10.3389/fnbot.2016.00007
8.
ClayV.KönigP.KönigS. U. (2019). Eye tracking in virtual reality. Journal of Eye Movement Research, 12(1). Article 1.https://doi.org/10.16910/jemr.12.1.3
9.
CognolatoM.AtzoriM.MüllerH. (2018). Head-mounted eye gaze tracking devices: An overview of modern devices and recent advances. Journal of Rehabilitation and Assistive Technologies Engineering, 5, 2055668318773991. https://doi.org/10.1177/2055668318773991
De RivecourtM.KuperusM. N.PostW. J.MulderL. J. M. (2008). Cardiovascular and eye activity measures as indices for momentary changes in mental effort during simulated flight. Ergonomics, 51(9), 1295–1319. https://doi.org/10.1080/00140130802120267
13.
EttingerU.KumariV.CrawfordT. J.DavisR. E.SharmaT.CorrP. J. (2003). Reliability of smooth pursuit, fixation, and saccadic eye movements. Psychophysiology, 40(4), 620–628. https://doi.org/10.1111/1469-8986.00063
14.
HaddadinS.CroftE. (2016). Physical human–robot interaction. In SicilianoB.KhatibO. (Eds.), Springer handbook of robotics, (pp. 1835–1874). Springer International Publishing. https://doi.org/10.1007/978-3-319-32552-1_69
15.
HesselsR. S.NiehorsterD. C.KemnerC.HoogeI. T. C. (2017). Noise-robust fixation detection in eye movement data: Identification by two-means clustering (I2MC). Behavior Research Methods, 49(5), 1802–1823. https://doi.org/10.3758/s13428-016-0822-1
16.
HolmqvistK.NyströmM.AnderssonR.DewhurstR.HalszkaJ.van de WeijerJ. (2011). Eye tracking: A comprehensive guide to methods and measures. Oxford University Press. http://lup.lub.lu.se/record/1852359
17.
JustM. A.CarpenterP. A.MiyakeA. (2003). Neuroindices of cognitive workload: Neuroimaging, pupillometric and event-related potential studies of brain work. Theoretical Issues in Ergonomics Science, 4(1–2), 56–88. https://doi.org/10.1080/14639220210159735
18.
KaoP.-C. (2009). Principles of motor adaptation when walking with a powered exoskeleton (p. 120).
19.
KomogortsevO. V.KarpovA. (2013). Automated classification and scoring of smooth pursuit eye movements in the presence of fixations and saccades. Behavior Research Methods, 45(1), 203–215. https://doi.org/10.3758/s13428-012-0234-9
20.
KrejtzK.DuchowskiA. T.NiedzielskaA.BieleC.KrejtzI. (2018). Eye tracking cognitive load using pupil diameter and microsaccades with fixed gaze. PLoS One, 13(9), Article e0203629. https://doi.org/10.1371/journal.pone.0203629
21.
LakensD. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4. 863. https://doi.org/10.3389/fpsyg.2013.00863
LawB.AtkinsM. S.KirkpatrickA. E.LomaxA. J. (2004, March 22-24). Eye gaze patterns differentiate novice and experts in a virtual laparoscopic surgery training environment. Proceedings of the Eye Tracking Research & Applications Symposium on Eye Tracking Research & Applications, ETRA’2004, 41–48. https://doi.org/10.1145/968363.968370
24.
Llanes-JuradoJ.Marín-MoralesJ.GuixeresJ.AlcañizM. (2020). Development and calibration of an eye-tracking fixation identification algorithm for immersive virtual reality. Sensors, 20(17), 4956. Article 17.https://doi.org/10.3390/s20174956
25.
LoftS.SandersonP.NealA.MooijM. (2007). Modeling and predicting mental workload in en route air traffic control: Critical review and broader implications. Human Factors, 49(3), 376–399. https://doi.org/10.1518/001872007X197017
26.
MarchandC.De GraafJ. B.JarrasséN. (2021). Measuring mental workload in assistive wearable devices: A review. Journal of NeuroEngineering and Rehabilitation, 18(1), 160. https://doi.org/10.1186/s12984-021-00953-w
27.
MatthewsG.Reinerman-JonesL. E.BarberD. J.AbichJ. (2015). The psychometrics of mental workload: Multiple measures are sensitive but divergent. Human Factors, 57(1), 125–143. https://doi.org/10.1177/0018720814539505
28.
NovakD.BeyelerB.OmlinX.RienerR. (2015). Workload estimation in physical human–robot interaction using physiological measurements. Interacting with Computers, 27(6), 616–629. https://doi.org/10.1093/iwc/iwu021
29.
PflegingB.FeketyD. K.SchmidtA.KunA. L. (2016). A model relating pupil diameter to mental workload and lighting conditions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 5776–5788). https://doi.org/10.1145/2858036.2858117
30.
SailerU.FlanaganJ. R.JohanssonR. S. (2005). Eye-hand coordination during learning of a novel visuomotor task. Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 25(39), 8833–8842. https://doi.org/10.1523/JNEUROSCI.2658-05.2005
31.
SearleS. R.CasellaG.McCullochC. E. (2009). Variance components. John Wiley & Sons.
32.
ShiferawB.DowneyL.CrewtherD. (2019). A review of gaze entropy as a measure of visual scanning efficiency. Neuroscience & Biobehavioral Reviews, 96, 353–366. https://doi.org/10.1016/j.neubiorev.2018.12.007
33.
ShroutP. E.FleissJ. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420
34.
SobuhM. M. D.KenneyL. P. J.GalpinA. J.ThiesS. B.McLaughlinJ.KulkarniJ.KyberdP. (2014). Visuomotor behaviours when using a myoelectric prosthesis. Journal of NeuroEngineering and Rehabilitation, 11(1), 72. https://doi.org/10.1186/1743-0003-11-72
35.
SrinivasanD.MartinB. J. (2010). Eye–hand coordination of symmetric bimanual reaching tasks: Temporal aspects. Experimental Brain Research, 203(2), 391–405. https://doi.org/10.1007/s00221-010-2241-3
36.
SteinfeldA.FongT.KaberD.LewisM.ScholtzJ.SchultzA.GoodrichM. (2006). Common metrics for human-robot interaction. In Proceeding of the 1st ACM SIGCHI/SIGART Conference on Human-Robot Interaction - HRI ’06 (p. 33). https://doi.org/10.1145/1121241.1121249
37.
StirlingL.Kelty-StephenD.FinemanR.JonesM. L. H.Daniel ParkB.-K.ReedM. P.ParhamJ.ChoiH. J. (2020). Static, dynamic, and cognitive fit of exosystems for the human operator. Human Factors, 62(3), 424–440. https://doi.org/10.1177/0018720819896898
38.
ThomasJ. R.FrenchK. E. (1985). Gender differences across age in motor performance: A meta-analysis. Psychological Bulletin, 98(2), 260–282. https://doi.org/10.1037/0033-2909.98.2.260
39.
TsangP. S.VidulichM. A. (2006). Mental workload and situation awareness. In Handbook of human factors and ergonomics (3rd ed., pp. 243–268). John Wiley & Sons, Inc. https://doi.org/10.1002/0470048204.ch9
40.
Van AckerB. B.BombekeK.DurnezW.ParmentierD. D.MateusJ. C.BiondiA.SaldienJ.VlerickP. (2020). Mobile pupillometry in manual assembly: A pilot study exploring the wearability and external validity of a renowned mental workload lab measure. International Journal of Industrial Ergonomics, 75, 102891. https://doi.org/10.1016/j.ergon.2019.102891
41.
WebbG. I. (2010). Model evaluation. In Encyclopedia of machine learning (pp. 683–683). Springer.
42.
WhiteM. M.ZhangW.WinslowA. T.ZahabiM.ZhangF.HuangH.KaberD. B. (2017). Usability comparison of conventional direct control versus pattern recognition control of transradial prostheses. IEEE Transactions on Human-Machine Systems, 47(6), 1146–1157. https://doi.org/10.1109/THMS.2017.2759762
WilsonM.McGrathJ.VineS.BrewerJ.DefriendD.MastersR. (2010). Psychomotor control in a virtual laparoscopic surgery training environment: Gaze control parameters differentiate novices from experts. Surgical Endoscopy, 24(10), 2458–2464. https://doi.org/10.1007/s00464-010-0986-1
45.
WolpertD. M.DiedrichsenJ.FlanaganJ. R. (2011). Principles of sensorimotor learning. Nature Reviews Neuroscience, 12(12), 739–751. Article 12.https://doi.org/10.1038/nrn3112
46.
WuC.ChaJ.SulekJ.SundaramC. P.WachsJ.ProctorR. W.YuD. (2021). Sensor-based indicators of performance changes between sessions during robotic surgery training. Applied Ergonomics, 90, 103251. https://doi.org/10.1016/j.apergo.2020.103251
47.
WuC.ChaJ.SulekJ.ZhouT.SundaramC. P.WachsJ.YuD. (2020). Eye-tracking metrics predict perceived workload in robotic surgical skills training. Human Factors, 62(8), 1365–1386. https://doi.org/10.1177/0018720819874544
48.
Zargari MarandiR.MadeleineP.OmlandØ.VuillermeN.SamaniA. (2018). Reliability of oculometrics during a mentally demanding task in young and old adults. IEEE Access, 6, 17500–17517. https://doi.org/10.1109/ACCESS.2018.2819211
49.
ZénonA.SidibéM.OlivierE. (2014). Pupil size variations correlate with physical effort perception. Frontiers in Behavioral Neuroscience, 8, 286. https://doi.org/10.3389/fnbeh.2014.00286
50.
ZhangY.HopkoS.YA.MehtaR. K. (2022). Capturing dynamic trust metrics during shared space human robot collaboration: An eye-tracking approach. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 66(1), 536–536. https://doi.org/10.1177/1071181322661296
51.
ZhengB.JiangX.BednarikR.AtkinsM. S. (2021). Action-related eye measures to assess surgical expertise. BJS Open, 5(5), zrab068. https://doi.org/10.1093/bjsopen/zrab068
52.
ZhouT.ZhuQ.DuJ. (2020). Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction. Advanced Engineering Informatics, 46, 101170. https://doi.org/10.1016/j.aei.2020.101170
