We conducted a literature review investigating the validity of eye tracking metrics appropriately representing trust in automation.
Background
As researchers grow interested in measuring trust in automation, there has been a need to find a reliable and accurate measurement tool. Many articles have measured automation trust using eye tracking, assuming that as trust increases, visual attention from eye tracking metrics decreases. Eye tracking is an attractive potential measure for its nonintrusive and objective nature.
Method
In this systematic literature review, we looked at studies that have tested the relationship between eye tracking and trust to determine its validity and reliability.
Results
Across 22 articles that investigated the relationship between trust and eye tracking, only about half found a negative significant relationship, whereas the other half found no relationship at all.
Conclusion
The relationship between automation trust and eye tracking is inconsistent and unreliable. A wide variety of trust and eye tracking metrics were used, but only about half of the papers found any kind of relationship. The relationship did not appear robust enough to prevail when different eye tracking and trust metrics were applied in various study designs.
Application
An effective eye tracking-trust relationship would be useful in various applications (e.g., autonomous driving). Further, this relationship is crucial when there is a clear distinction between attention allocated to automated components of a system (e.g., car display) and unrelated displays to allow for an easy separation of a location associated with high trust versus low trust.
^Azevedo-SaH.JayaramanS. K.EsterwoodC. T.YangJ. X.RobertL. P.TilburyD. M. (2021). Real-time estimation of drivers’ trust in automated driving systems. International Journal of Social Robotics, 13(8), 1911–1927. https://doi.org/10.1007/s12369-020-00694-1
2.
Azevedo-SaH.YangX. J.RobertL. P.TilburyD. M. (2021). Using trust in automation to enhance driver-(semi)autonomous vehicle interaction and improve team performance. In 2021 International Symposium on Transportation Data and Modelling. https://arxiv.org/abs/2106.02136
3.
BahnerJ. E.HüperA. D.ManzeyD. (2008). Misuse of automated decision aids: Complacency, automation bias and the impact of training experience. International Journal of Human-Computer Studies, 66(9), 688–699. https://doi.org/10.1016/j.ijhcs.2008.06.001
4.
^BeeN.WagnerJ.AndréE.CharlesF.PizziD.CavazzaM. (2010). Interacting with a gaze-aware virtual character. In Proceedings of the 2010 workshop on Eye gaze in intelligent human machine interaction (pp. 71–77). https://doi.org/10.1145/2002333.2002345
5.
BertramI.HongJ.HuangY.WeimerW.SharafiZ. (2020). Trustworthiness perceptions in code review: An eye-tracking Study. In Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) (pp. 1–6). https://doi.org/10.1145/3382494.3422164
6.
BischoffS.UlrichC.DangelmaierM.WidlroitherH.DiederichsF. (2017). Emotion recognition in user-centered design for automotive interior and automated driving. In Stuttgarter Symposium Für Produktentwicklung.
7.
^BodalaI. P.KokB. C.SngW.SohH. (2020). Modeling the interplay of trust and attention in HRI: An autonomous vehicle study. In Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction (pp. 145–147). https://doi.org/10.1145/3371382.3378262
8.
ChanceyE. T.BlissJ. P.YamaniY.HandleyH. A. H. (2017). Trust and the compliance-reliance paradigm: The effects of risk, error bias, and reliability on trust and dependence. Human Factors, 59(3), 333–345. https://doi.org/10.1177/001872081668264
de VisserE. J.BeattyP. J.EsteppJ. R.KohnS.AbubshaitA.FedotaJ. R.McDonaldC. G. (2018). Learning from the slips of others: Neural correlates of trust in automated agents. Frontiers in Human Neuroscience, 12(309), 309. https://doi.org/10.3389/fnhum.2018.00309
11.
DijkstraJ. J. (1999). User agreement with incorrect expert system advice. Behaviour & Information Technology, 18(6), 399–411. https://doi.org/10.1080/014492999118832
12.
EbnaliM.FathiR.LambR.PourfalatounS.MotamediS. (2020). Using augmented holographic UIs to communicate automation reliability in partially automated driving. In Proceedings of the automation experience across domains workshop, CHI ‘2020.
13.
^FeierleA.BellerD.BenglerK. (2019). Head-up displays in urban partially automated driving: Effects of using augmented reality. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC) (pp. 1877–1882). https://doi.org/10.1109/ITSC.2019.8917472
14.
FeitA. M.WilliamsS.ToledoA.ParadisoA.KulkarniH.KaneS.MorrisM. R. (2017). Toward everyday Gaze input: Accuracy and precision of eye tracking and implications for design. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 1118–1130). https://doi.org/10.1145/3025453.3025599
15.
^ForoughiC. K.DevlinS.PakR.BrownN. L.SibleyC.CoyneJ. T. (2023). Near-Perfect automation: Investigating performance, trust, and visual attention allocation. Human Factors, 65(4), 546–561. https://doi.org/10.1177/00187208211032889
16.
FunkeG.GreenleeE.CarterM.DukesA.BrownR.MenkeL. (2016). Which eye tracker is right for your research? Performance evaluation of several cost variant eye trackers. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 60, (1), 1239–1243. https://doi.org/10.1177/1541931213601289
17.
GoldC.KörberM.HohenbergerC.LechnerD.BenglerK. (2015). Trust in automation – before and after the experience of take-over scenarios in a highly automated vehicle. Procedia Manufacturing, 3(1), 3025–3032. https://doi.org/10.1016/j.promfg.2015.07.847
18.
GutzwillerR. S.ChiouE. K.CraigS. D.LewisC. M.LemattaG. J.HsiungC. (2019). Positive bias in the ‘Trust in Automated Systems Survey’? An examination of the Jian et al. (2000) scale. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 63(1), 217–221. https://doi.org/10.1177/1071181319631201
19.
^HagenziekerM. P.van der KintS.VissersL.van SchagenI. N. L. G.de BruinJ.van GentP.CommandeurJ. J. F. (2019). Interactions between cyclists and automation vehicles: Results of a photo experiment. Journal of Transportation Safety & Security, 12(1), 94–115. https://doi.org/10.1080/19439962.2019.1591556
^HechtT.KratzertS.BenglerK. (2018). The effects of a predictive HMI and different transition frequencies on acceptance, workload, usability, and gaze behavior during urban automated driving. Information, 11(2), 73. https://doi.org/10.3390/info11020073
23.
HergethS.LorenzL.VilimekR.KremsJ. F. (2016). Keep your scanners peeled: Gaze behavior as a measure of automation trust during highly automated driving. Human Factors, 58(3), 509–519. https://doi.org/10.1177/0018720815625744
24.
HoffK. A.BashirM. (2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407–434. https://doi.org/10.1177/0018720814547570
25.
HolmqvistK.NyströmM.AnderssonR.DewhurstR.JarodzkaH.Van de WeijerJ. (2011). Eye tracking: A comprehensive guide to methods and measures. OUP.
26.
HorreyW. J.WickensC. D. (2006). Examining the impact of cell phone conversations on driving using meta-analytic techniques. Human Factors, 48(1), 196–205. https://doi.org/10.1518/001872006776412135
27.
JayaramanS. K.CreechC.TilburyD. M.YangX. J.PradhanA. K.TsuiK. M.RobertL. P. (2019). Pedestrian trust in automated vehicles: Role of traffic signal and AV driving behavior. In Frontiers in Robotics and AI (Vol. 6, pp. 117). https://doi.org/10.3389/frobt.2019.00117
28.
JianJ.-Y.BisantzA. M.DruryC. G. (2000). Foundations for an empirically determined scale of trust in automated systems. International Journal of Cognitive Ergonomics, 4(1), 53–71. https://doi.org/10.1207/S15327566IJCE0401_04
29.
^JinM.LuG.ChenF.ShiX. (2020). How driving experience affect trust in automation from level 3 automated vehicles? An experimental analysis. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC) (pp. 11–16). IEEE. https://doi.org/10.1109/ITSC45102.2020.9294247
30.
KalbL.StreitL.BenglerK. (2018). What drivers make of directional maneuver information in a take-over scenario. (pp. 3859–3864). IEEE. https://doi.org/10.1109/SMC.2018.00654
31.
KarpinskyN. D.ChanceyE. T.PalmerD. B.YamaniY. (2018). Automation trust and attention allocation in multitasking workspace. Applied Ergonomics, 70(1), 194–201. https://doi.org/10.1016/j.apergo.2018.03.008
32.
KohnS. C.de VisserE. J.WieseE.LeeY.-C.ShawT. H. (2021). Measurement of trust in automation: A narrative review and reference guide. Frontiers in Psychology, 12(604977), Article 604977. https://doi.org/10.3389/fpsyg.2021.604977
33.
KörberM. (2018). Theoretical considerations and development of a questionnaire to measure trust in automation. In Proceedings of the 20th triennial congress of the international ergonomics association (pp. 13–30). Springer Nature. https://doi.org/10.1007/978-3-319-96074-6_2
34.
KörberM.BaselerE.BenglerK. (2018). Introduction matters: Manipulating trust in automation and reliance in automated driving. Applied Ergonomics, 66(1), 18–31. https://doi.org/10.1016/j.apergo.2017.07.006
35.
^KraftA. K.MaagC.CruzM. I.BaumannM.NeukumA. (2020). Effects of explaining system failures during maneuver coordination while driving manual or automated. Accident Analysis & Prevention, 148(1), Article 105839. https://doi.org/10.1016/j.aap.2020.105839
36.
^KraftA. K.NaujoksF.WörleJ.NeukumA. (2018). The impact of an in-vehicle display on glance distribution in partially automated driving in an on-road experiment. Transportation Research Part F: Traffic Psychology and Behaviour, 52(1), 40–50. https://doi.org/10.1016/j.trf.2017.11.012
37.
KunzeA.SummerskillS. J.MarshallR.FiltnessA. J. (2019). Automation transparency: Implications of uncertainty communication for human-automation interaction and interfaces. Ergonomics, 62(3), 345–360. https://doi.org/10.1080/00140139.2018.1547842
38.
LeeJ. D.MorayN. (1994). Trust, self-confidence, and operators’ adaptation to automation. International Journal of Human-Computer Studies, 40(1), 153–184. https://doi.org/10.1006/ijhc.1994.1007
LuY.SarterN. (2018). Eye tracking: A promising method for inferring trust in real time. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 62(1), 175–176. https://doi.org/10.1177/1541931218621042
41.
LuY.SarterN. (2019). Eye tracking: A process-oriented method for inferring trust in automation as a function of priming and system reliability. IEEE Transactions on Human-Machine Systems, 49(6), 560–568. https://doi.org/10.1109/THMS.2019.2930980
42.
LuY.SarterN. (2020). Modeling and inferring human trust in automation based on real- time eye tracking data. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 64(1), 344–348. https://doi.org/10.1177/1071181320641078
43.
MarinescuA. D.SharplesS.RitchieA. C.LópezT. S.McDowellM.MorvanH. P. (2017). Physiological parameter response to variation of mental workload. Human Factors, 60(1), 31–56. https://doi.org/10.1177/0018720817733101
44.
MarquartG.CabrallC.de WinterJ. (2015). Review of eye-related measures of drivers’ mental workload. Procedia Manufacturing, 3(1), 2854–2861. https://doi.org/10.1016/j.promfg.2015.07.783
45.
^MetzgerU.ParasuramanR. (2006). Effects of automated conflict cuing and traffic density on air traffic controller performance and visual attention in a datalink environment. The International Journal of Aviation Psychology, 16(4), 343–362. https://doi.org/10.1207/s15327108ijap1604_1
46.
MorayN. (1986). Monitoring behavior and supervisory control. In BoffK. R.KaufmanL.ThomasJ. P. (Eds.), Handbook of perception and human performance, Vol. 2. Cognitive processes and performance. John Wiley & Sons.
47.
MorrisD. M.ErnoJ. M.PilcherJ. J. (2017). Electrodermal response and automation trust during simulated self-driving car use. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 61(1), 1759–1763. https://doi.org/10.1177/1541931213601921
NeubauerC.GremillionG.PerelmanB. S.La FleurC.MetcalfeJ. S.SchaeferK. E. (2020). Analysis of facial expressions explain affective state and trust-based decisions during interaction with autonomy. In Advances in Intelligent Systems and Computing, 1131 AISC (pp. 999–1006). https://doi.org/10.1007/978-3-030-39512-4_152
50.
NyströmM.AnderssonR.HolmqvistK.van de WeijerJ. (2013). The influence of calibration method and eye physiology on eyetracking data quality. Behavior Research Methods, 45(1), 272–288. https://doi.org/10.3758/s13428-012-0247-4
51.
OnnaschL.RuffS.ManzeyD. (2014). Operators' adaptation to imperfect automation - Impact of miss-prone alarm systems on attention allocation and performance. International Journal of Human-Computer Studies, 72(10-11), 772–782. https://doi.org/10.1016/j.ijhcs.2014.05.001
52.
^OnnaschL.HildebrandtC. L. (2021). Impact of anthropomorphic robot design on trust and attention in industrial human-robot interaction. ACM Transactions on Human-Robot Interaction, 11(1), 1–24. https://doi.org/10.1145/3472224
53.
OomsK.DupontL.LaponL.PopelkaS. (2015). Accuracy and precision of fixation locations recorded with the low-cost Eye Tribe tracker in different experimental set-ups. Journal of Eye Movement Research, 8(1), 1–20. https://doi.org/10.16910/jemr.8.1.5
54.
OrquinJ.AsbyN. J. S.ClarkeA. D. F. (2015). Areas of interest as a signal detection problem in behavior eye-tracking research. Journal of Behavioral Decision Making, 29(2-3), 103–115. https://doi.org/10.1002/bdm.1867
55.
PetersenL.RobertL.YangX.TilburyD. (2019). Situational awareness, driver’s trust in automated driving systems and secondary task performance. SAE International Journal of Connected and Automated Vehicles, 2(2), 129–141. https://doi.org/10.4271/12-02-02-0009
56.
SatoT.YamaniY.LiechtyM.ChanceyE. T. (2019). Automation trust increases under high - workload multitasking scenarios involving risk. Cognition, Technology & Work, 22(2), 399–407. https://doi.org/10.1007/s10111-019-00580-5
57.
^SchmidtC.HartwichF.KremsJ. F. (2021). Looking at driving automation from a passenger’s perspective: Driving experience and gaze behavior during fully automated vs. human vehicle control. In AhramT.TaiarR.LangloisK.ChoplinA. (Eds.), Human interaction, emerging technologies and future applications III (IHIET 2020): Advances in intelligent systems and computing (Vol. 1253, pp. 3–9). Springer. https://doi.org/10.1007/978-3-030-55307-4_1
58.
SchoormanD.MayerR. C.DavisJ. H. (2007). An integrative model of organizational trust: Past, present, and future. Academy of Management Review, 32(2), 344–354. https://doi.org/10.5465/amr.2007.24348410
59.
SchwarzC.GasparJ.BrownT. (2019). The effect of reliability on drivers’ trust and behavior in conditional automation. Cognition, Technology & Work, 21(1), 41–54. https://doi.org/10.1007/s10111-018-0522-y
60.
^ShengS.PakdamanianE.HanK.KimB.TiwariP.KimI.FengL. (2019). A case study of trust on autonomous driving. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC) (pp. 4368–4373). IEEE. https://doi.org/10.1109/ITSC.2019.8917251
61.
StrauchC.MühlK.PatroK.GrabmaierC.ReithingerS.BaumannM.HuckaufA. (2019). Real autonomous driving from a passenger’s perspective: Two experimental investigations using gaze behaviour and trust ratings in field and simulator. Transportation Research Part F: Traffic Psychology and Behaviour, 66(1), 15–28. https://doi.org/10.1016/j.trf.2019.08.013
62.
^TenhundfeldN. L.de VisserE. J.HaringK. S.RiesA. J.FinomoreV. S.TossellC. C. (2019). Calibrating trust in automation through familiarity with the autoparking feature of a Tesla Model X. Journal of Cognitive Engineering and Decision Making, 13(4), 279–294. https://doi.org/10.1177/1555343419869083
63.
TitzJ.ScholzA.SedlmeierP. (2018). Comparing eye trackers by correlating their eye-metric data. Behavior Research Methods, 50(5), 1853–1863. https://doi.org/10.3758/s13428-017-0954-y
64.
UlahannanA.BirrellS.ThomsonS.SkyrpchukL.MouzakitisA.JenningsP. (2019). The interface challenge for semi-automated vehicles: How driver behavior and trust influence information requirements over time. In Proceedings in 2019 IEEE Intelligent Vehicles Symposium (IV) (pp. 96–101). IEEE. https://doi.org/10.0/Linux-x86_64
65.
UlahannanA.JenningsP.ThompsonS.BirrellS. (2020). The interface challenge for partially automated vehicles: How driver characteristics affect information usage over time. In Olaverri-MonrealC.Garcia-FernandezF.RossettiR. J. F. (Eds.), Human factors in intelligent vehicles (1st ed.). River Publishers.
66.
VlakveldW.van NesN.de BruinJ.VissersL.van der KroftM. (2018). Situation awareness increases when drivers have more time to take over the wheel in a Level 3 automated car: A simulator study. Transportation Research Part F: Traffic Psychology and Behaviour, 58(1), 917–929. https://doi.org/10.1016/j.trf.2018.07.025
WalkerF.WangJ.MartensM. H.VerweyW. B. (2019). Gaze behaviour and electrodermal activity: Objective measures of drivers’ trust in automated vehicles. Transportation Research Part F: Traffic Psychology and Behaviour, 64(1), 401–412. https://doi.org/10.1016/j.trf.2019.05.021
69.
WaytzA.HeafnerJ.EpleyN. (2014). The mind in the machine: Anthropomorphism increases trust in an autonomous vehicle. Journal of Experimental Social Psychology, 52, 113–117. https://doi.org/10.1016/j.jesp.2014.01.005
70.
WickensC. D.GohJ.HellebergJ.HorreyW. J.TalleurD. A. (2003). Attentional models of multitask pilot performance using advanced display technology. Human Factors, 45(3), 360–380. https://doi.org/10.1518/hfes.45.3.360.27250
YinW.ChaiC.ZhouZ.LiC.LuY.ShiX. (2021). Effects of trust in human-automation shared control: A human-in-the-loop driving simulation study. In 2021 IEEE international intelligent transportation systems conference (ITSC) (pp. 1147–1154). IEEE. https://doi.org/10.1109/ITSC48978.2021.9565080
73.
^ZalakeM.WoodwardJ.KapoorA.LokB. (2018). Assessing the impact of virtual human’s appearance on users’ trust levels. In Proceedings of the 18th international conference on intelligent virtual agents (pp. 329–330). https://doi.org/10.1145/3267851.3267863
74.
ZhangQ.YangJ.RobertL. P. (2021). Individual differences and expectations of automated vehicles. International Journal of Human-Computer Interaction, 38(9), 825–836. https://doi.org/10.1080/10447318.2021.1970431
75.
ZhangY.MaJ.PanC.ChangR. (2021). Effects of automation trust in drivers’ visual distraction during automation. PLoS ONE, 16(9), Article e0257201. https://doi.org/10.1371/journal.pone.0257201
