The study aimed to enhance transparency in autonomous systems by automatically generating and visualizing confidence and explanations and assessing their impacts on performance, trust, preference, and eye-tracking behaviors in human–automation interaction.
Background
System transparency is vital to maintaining appropriate levels of trust and mission success. Previous studies presented mixed results regarding the impact of displaying likelihood information and explanations, and often relied on hand-created information, limiting scalability and failing to address real-world dynamics.
Method
We conducted a dual-task experiment involving 42 university students who operated a simulated surveillance testbed with assistance from intelligent detectors. The study used a 2 (confidence visualization: yes vs. no) × 3 (visual explanations: none, bounding boxes, bounding boxes and keypoints) mixed design. Task performance, human trust, preference for intelligent detectors, and eye-tracking behaviors were evaluated.
Results
Visual explanations using bounding boxes and keypoints improved detection task performance when confidence was not displayed. Meanwhile, visual explanations enhanced trust and preference for the intelligent detector, regardless of the explanation type. Confidence visualization did not influence human trust in and preference for the intelligent detector. Moreover, both visual information slowed saccade velocities.
Conclusion
The study demonstrated that visual explanations could improve performance, trust, and preference in human–automation interaction without confidence visualization partially by changing the search strategies. However, excessive information might cause adverse effects.
Application
These findings provide guidance for the design of transparent automation, emphasizing the importance of context-appropriate and user-centered explanations to foster effective human–machine collaboration.
BellerJ.HeesenM.VollrathM. (2013). Improving the driver–automation interaction: An approach using automation uncertainty. Human Factors, 55(6), 1130–1141. https://doi.org/10.1177/0018720813482327
2.
ChanceyE. T.BlissJ. P.YamaniY.HandleyH. A. (2017). Trust and the compliance–reliance paradigm: The effects of risk, error bias, and reliability on trust and dependence. Human Factors, 59(3), 333–345. https://doi.org/10.1177/0018720816682648
3.
CohenJ. (2013). Statistical power analysis for the behavioral sciences. Academic Press.
4.
DuN.HuangK. Y.YangX. J. (2020). Not all information is equal: Effects of disclosing different types of likelihood information on trust, compliance and reliance, and task performance in human-automation teaming. Human Factors, 62(6), 987–1001. https://doi.org/10.1177/0018720819862916
5.
EibandM.BuschekD.KremerA.HussmannH. (2019). The impact of placebic explanations on trust in intelligent systems. Extended abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1–6). Association for Computing Machinery.
6.
EndsleyM. R. (2017). From here to autonomy: Lessons learned from human–automation research. Human Factors, 59(1), 5–27. https://doi.org/10.1177/0018720816681350
7.
FaulF.ErdfelderE.BuchnerA.LangA.-G. (2009). Statistical power analyses using g* power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 1149–1160. https://doi.org/10.3758/BRM.41.4.1149
GoldbergJ. H.WichanskyA. M. (2003). Eye tracking in usability evaluation: A practitioner’s guide. In the mind’s eye (pp. 493–516). Elsevier.
10.
HardmanD.MacchiL. (2003). Thinking: Psychological perspectives on reasoning, judgment and decision making. Wiley Online Library.
11.
HeK.GkioxariG.DollárP.GirshickR. (2017). Mask r-cnn. Proceedings of the IEEE International Conference on Computer Vision (pp. 2961–2969), Cambridge, MA, USA, 20–23 June 1995.
12.
JacobR. J.KarnK. S. (2003). Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. The mind’s eye (pp. 573–605). Elsevier.
13.
JanssenC. P.DonkerS. F.BrumbyD. P.KunA. L. (2019). History and future of human-automation interaction. International Journal of Human-Computer Studies, 131, 99–107. https://doi.org/10.1016/j.ijhcs.2019.05.006
14.
KaberD. B. (2018). A conceptual framework of autonomous and automated agents. Theoretical Issues in Ergonomics Science, 19(4), 406–430. https://doi.org/10.1080/1463922x.2017.1363314
15.
KooJ.KwacJ.JuW.SteinertM.LeiferL.NassC. (2015). Why did my car just do that? Explaining semi-autonomous driving actions to improve driver understanding, trust, and performance. International Journal on Interactive Design and Manufacturing, 9(4), 269–275. https://doi.org/10.1007/s12008-014-0227-2
16.
KrausJ.ScholzD.StiegemeierD.BaumannM. (2020). The more you know: Trust dynamics and calibration in highly automated driving and the effects of take-overs, system malfunction, and system transparency. Human Factors, 62(5), 718–736. https://doi.org/10.1177/0018720819853686
17.
KunkelJ.DonkersT.MichaelL.BarbuC.-M.ZieglerJ. (2019). Let me explain: Impact of personal and impersonal explanations on trust in recommender systems. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1–12). Association for Computing Machinery.
18.
LageI.ChenE.HeJ.NarayananM.KimB.GershmanS. J.Doshi-VelezF. (2019). Human evaluation of models built for interpretability. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 7(1), 59–67. https://doi.org/10.1609/hcomp.v7i1.5280
19.
LakshminarayananB.PritzelA.BlundellC. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17), Long Beach, California, USA (pp. 6405–6416). Curran Associates Inc.
LiebermanK.SarterN. (2021). A comparison of auditory and visual representations of system confidence to support trust specificity, attention management, and joint performance in human-machine teams. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 65(1), 67–71. https://doi.org/10.1177/1071181321651198
22.
LouwT.MeratN. (2017). Are you in the loop? Using gaze dispersion to understand driver visual attention during vehicle automation. Transportation Research Part C: Emerging Technologies, 76, 35–50. https://doi.org/10.1016/j.trc.2017.01.001
23.
LyonsJ. B.HavigP. R. (2014). Transparency in a human-machine context: Approaches for fostering shared awareness/intent. In ShumakerR.LackeyS. (Eds.), International Conference on Virtual, Augmented and Mixed Reality (pp. 181–190). Springer.
24.
LyonsJ. B.VoT.WynneK. T.MahoneyS.NamC. S.GallimoreD. (2021). Trusting autonomous security robots: The role of reliability and stated social intent. Human Factors, 63(4), 603–618. https://doi.org/10.1177/0018720820901629
25.
McGuirlJ. M.SarterN. B. (2006). Supporting trust calibration and the effective use of decision aids by presenting dynamic system confidence information. Human Factors, 48(4), 656–665. https://doi.org/10.1518/001872006779166334
26.
MuellerS. T.PiperB. J. (2014). The psychology experiment building language (pebl) and pebl test battery. Journal of Neuroscience Methods, 222, 250–259. https://doi.org/10.1016/j.jneumeth.2013.10.024
27.
MuirB. M. (1994). Trust in automation: Part I. Theoretical issues in the study of trust and human intervention in automated systems. Ergonomics, 37(11), 1905–1922. https://doi.org/10.1080/00140139408964957
28.
MuirB. M.MorayN. (1996). Trust in automation. Part II. Experimental studies of trust and human intervention in a process control simulation. Ergonomics, 39(3), 429–460. https://doi.org/10.1080/00140139608964474
29.
NassC.JonssonI.-M.HarrisH.ReavesB.EndoJ.BraveS.TakayamaL. (2005). Improving automotive safety by pairing driver emotion and car voice emotion. Chi’05 extended abstracts on human factors in computing systems (pp. 1973–1976). Association for Computing Machinery.
30.
NeyedliH. F.HollandsJ. G.JamiesonG. A. (2011). Beyond identity: Incorporating system reliability information into an automated combat identification system. Human Factors, 53(4), 338–355. https://doi.org/10.1177/0018720811413767
31.
NouraniM.KabirS.MohseniS.RaganE. D. (2019). Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (Vol. 7, pp. 97–105). The AAAI Press.
32.
RenS.HeK.GirshickR.SunJ. (2017). Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031
33.
ReyesM. L.LeeJ. D. (2008). Effects of cognitive load presence and duration on driver eye movements and event detection performance. Transportation Research Part F: Traffic Psychology and Behaviour, 11(6), 391–402. https://doi.org/10.1016/j.trf.2008.03.004
RudinC. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206–215. https://doi.org/10.1038/s42256-019-0048-x
36.
SelvarajuR. R.CogswellM.DasA.VedantamR.ParikhD.BatraD. (2017). Grad-CAM: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision (pp. 618–626). IEEE.
37.
SkraaningG.JamiesonG. A. (2021). Human performance benefits of the automation transparency design principle: Validation and variation. Human Factors, 63(3), 379–401. https://doi.org/10.1177/0018720819887252
38.
SorkinR. D.KantowitzB. H.KantowitzS. C. (1988). Likelihood alarm displays. Human Factors: The Journal of the Human Factors and Ergonomics Society, 30(4), 445–459. https://doi.org/10.1177/001872088803000406
39.
StangeS.HassanT.SchröderF.KonkolJ.KoppS. (2022). Self-explaining social robots: An explainable behavior generation architecture for human-robot interaction. Frontiers in Artificial Intelligence, 5, 866920. https://doi.org/10.3389/frai.2022.866920
40.
van de MerweK.MallamS.NazirS. (2022). Agent transparency, situation awareness, mental workload, and operator performance: A systematic literature review (pp. 00187208221077804). Human Factors.
41.
WalliserJ. C.de VisserE. J.ShawT. H. (2016). Application of a system-wide trust strategy when supervising multiple autonomous agents. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 60(1), 133–137. https://doi.org/10.1177/1541931213601031
42.
WangY.ReimerB.DobresJ.MehlerB. (2014). The sensitivity of different methodologies for characterizing drivers’ gaze concentration under increased cognitive demand. Transportation Research Part F: Traffic Psychology and Behaviour, 26, 227–237. https://doi.org/10.1016/j.trf.2014.08.003
43.
WeitzK.SchillerD.SchlagowskiR.HuberT.AndréE. (2019). “Do you trust me?” increasing user-trust by integrating virtual agents in explainable AI interaction design. Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents (pp. 7–9). Association for Computing Machinery.
44.
WilliamsA. (2015). Defining autonomy in systems: Challenges and solutions (p. 27). Issues for Defence Policymakers.
45.
YangX. J.UnhelkarV. V.LiK.ShahJ. A. (2017). Evaluating effects of user experience and system transparency on trust in automation. Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction (pp. 408–416). Association for Computing Machinery.
46.
ZagermannJ.PfeilU.ReitererH. (2016). Measuring cognitive load using eye tracking technology in visual computing. Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization (pp. 78–85). Association for Computing Machinery.
ZhangQ.CaoR.ShiF.WuY. N.ZhuS. (2018). Interpreting CNN knowledge via an explanatory graph. In Proceedings of the AAAI Conference on Artificial Intelligence32(1). https://doi.org/10.1609/aaai.v32i1.11819
49.
ZhangQ.ZhuS. (2018). Visual interpretability for deep learning: A survey. Frontiers of Information Technology and Electronic Engineering, 19(1), 27–39. https://doi.org/10.1631/fitee.1700808
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
