To investigate how AI-provided explanations impact efficiency, diagnostic accuracy, user perceptions, and workflow integration in ophthalmologists’ clinical diagnostic and treatment workflows, this study explores the challenges in human-AI interaction with transparency features in time-sensitive environments.
Background
While explainable AI (XAI) aims to foster trust and understanding, its introduction into complex work domains can unintentionally increase cognitive load and disrupt workflows, especially in high-stakes medical settings, potentially impairing system performance.
Method
The multi-phase, mixed-methods study included two parts. Study 1 (N = 32) was a between-subjects experiment in which ophthalmologists diagnosed diabetic retinopathy with AI support, with or without visual explanations (e.g., highlighting lesions). Measures included diagnostic accuracy, diagnostic time, trust, and usefulness. Study 2 (N = 11) employed qualitative methods, including think-aloud protocols and interviews, to explore clinicians’ experiences with AI in daily (treatment) workflows.
Results
In Study 1, explanations did not improve accuracy but increased decision time, reducing efficiency. Trends suggested lower perceived usefulness and trust in the explanation condition. Qualitative data from Study 2 supported these findings; clinicians found explanations time-consuming and disruptive, questioning their practical value, especially for routine cases.
Conclusion
A critical trade-off exists between pursuing AI transparency and the operational demand for efficiency. Explanations, while well-intentioned, can function as efficiency pitfalls in time-pressured clinical practice, highlighting the boundary conditions and challenges in designing effective human-AI systems.
Application
These insights inform future AI system design, favoring adaptable, on-demand explanations tailored to user needs. Such a user-centric approach supports complex cases without impeding routine task efficiency.
AbràmoffM. D.GarvinM. K.SonkaM. (2010). Retinal imaging and image analysis. IEEE Reviews in Biomedical Engineering, 3, 169–208. https://doi.org/10.1109/RBME.2010.2084567
2.
AmannJ.VetterD.BlombergS. N.ChristensenH. C.CoffeeM.GerkeS.GilbertT. K.HagendorffT.HolmS.LivneM.SpezzattiA.StrümkeI.ZicariR. V.MadaiV. I., on behalf of the Z-Inspection Initiative. (2022). To explain or not to explain?—Artificial intelligence explainability in clinical decision support systems. PLOS Digital Health, 1(2), Article e0000016. https://doi.org/10.1371/journal.pdig.0000016
3.
AntoniadiA. M.DuY.GuendouzY.WeiL.MazoC.BeckerB. A.MooneyC. (2021). Current challenges and future opportunities for XAI in machine learning-based clinical decision support systems: A systematic review. Applied Sciences, 11(11), Article 5088. https://doi.org/10.3390/app11115088
4.
BansalG.WuT.ZhouJ.FokR.NushiB.KamarE.RibeiroM. T.WeldD. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, Yokohama Japan, 8–13 May, 2021. https://doi.org/10.1145/3411764.3445717
BraunV.ClarkeV. (2019). Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health, 11(4), 589–597. https://doi.org/10.1080/2159676X.2019.1628806
7.
BrookeJ. (1996). SUS: A “quick and dirty” usability scale. In JordanP. W.ThomasB.McClellandI. L.WeerdmeesterB. (Eds.), Usability evaluation in industry (p. 6). CRC Press.
8.
BuçincaZ.MalayaM. B.GajosK. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), 1–21. https://doi.org/10.1145/3449287
9.
BussoneA.StumpfS.O’SullivanD. (2015). The role of explanations on trust and reliance in clinical decision support systems. In 2015 International Conference on Healthcare Informatics, Dallas, TX, USA, 21–23 October, 2015, pp. 160–169. https://doi.org/10.1109/ICHI.2015.26
10.
CalhounG. (2022). Adaptable (not adaptive) automation: Forefront of human–automation teaming. Human Factors, 64(2), 269–277. https://doi.org/10.1177/00187208211037457
11.
CarriganA. J.CharltonA.WigginsM. W.GeorgiouA.PalmeriT.CurbyK. M. (2022). Cue utilisation reduces the impact of response bias in histopathology. Applied Ergonomics, 98, Article 103590. https://doi.org/10.1016/j.apergo.2021.103590
12.
CarriganA. J.MagnussenJ.GeorgiouA.CurbyK. M.PalmeriT. J.WigginsM. W. (2021). Differentiating experience from cue utilization in radiological assessments. Human Factors: The Journal of the Human Factors and Ergonomics Society, 63(4), 635–646. https://doi.org/10.1177/0018720820902576
13.
ChandaT.HauserK.HobelsbergerS.BucherT.-C.GarciaC. N.WiesC.KittlerH.TschandlP.Navarrete-DechentC.PodlipnikS.ChousakosE.CrnaricI.MajstorovicJ.AlhajwanL.ForemanT.PeternelS.SarapS.Özdemirİ.BarnhillR. L.BrinkerT. J. (2024). Dermatologist-like explainable AI enhances trust and confidence in diagnosing melanoma. Nature Communications, 15(1), Article 524. https://doi.org/10.1038/s41467-023-43095-4
14.
CombiC.AmicoB.BellazziR.HolzingerA.MooreJ. H.ZitnikM.HolmesJ. H. (2022). A manifesto on explainability for artificial intelligence in medicine. Artificial Intelligence in Medicine, 133, Article 102423. https://doi.org/10.1016/j.artmed.2022.102423
15.
DrewT.VõM. L.-H.WolfeJ. M. (2013). The invisible gorilla strikes again: Sustained inattentional blindness in expert observers. Psychological Science, 24(9), 1848–1853. https://doi.org/10.1177/0956797613479386
16.
ElhaddadM.HamamS. (2024). AI-driven clinical decision support systems: An ongoing pursuit of potential. Cureus, 16(4), Article e57728. https://doi.org/10.7759/cureus.57728
17.
FokR.WeldD. S. (2024). In search of verifiability: Explanations rarely enable complementary performance in AI-advised decision making. AI Magazine, 45(3), 317–332. https://doi.org/10.1002/aaai.12182
18.
FrankeT.AttigC.WesselD. (2019). A personal resource for technology interaction: Development and validation of the Affinity for Technology Interaction (ATI) scale. International Journal of Human–Computer Interaction, 35(6), 456–467. https://doi.org/10.1080/10447318.2018.1456150
GaubeS.SureshH.RaueM.LermerE.KochT. K.HudecekM. F. C.AckeryA. D.GroverS. C.CoughlinJ. F.FreyD.KitamuraF. C.GhassemiM.ColakE. (2023). Non-task expert physicians benefit from correct explainable AI advice when reviewing X-rays. Scientific Reports, 13(1), Article 1383. https://doi.org/10.1038/s41598-023-28633-w
21.
GaubeS.SureshH.RaueM.MerrittA.BerkowitzS. J.LermerE.CoughlinJ. F.GuttagJ. V.ColakE.GhassemiM. (2021). Do as AI say: Susceptibility in deployment of clinical decision-aids. Npj Digital Medicine, 4, 1–8. https://doi.org/10.1038/s41746-021-00385-9
22.
GuestG.BunceA.JohnsonL. (2006). How many interviews are enough?: An experiment with data saturation and variability. Field Methods, 18(1), 59–82. https://doi.org/10.1177/1525822X05279903
23.
HeJ.BaxterS. L.XuJ.XuJ.ZhouX.ZhangK. (2019). The practical implementation of artificial intelligence technologies in medicine. Nature Medicine, 25(1), 30–36. https://doi.org/10.1038/s41591-018-0307-0
24.
HermL.-V. (2023). Impact of explainable AI on cognitive load: Insights from an empirical study. ECIS 2023 Research Papers. Article 269.
25.
HolzingerA.LangsG.DenkH.ZatloukalK.MüllerH. (2019). Causability and explainability of artificial intelligence in medicine. WIREs Data Mining and Knowledge Discovery, 9(4), Article e1312. https://doi.org/10.1002/widm.1312
26.
HudonA.DemazureT.KarranA.LégerP.-M.SénécalS. (2021). Explainable artificial intelligence (XAI): How the visualization of AI predictions affects user cognitive load and confidence. In DavisF. D.RiedlR.vom BrockeJ.LégerP.-M.RandolphA. B.Müller-PutzG. (Eds.), Information systems and neuroscience (pp. 237–246). Springer. https://doi.org/10.1007/978-3-030-88900-5_27
27.
KayserM. G.MenzatB.EmdeC.BerceanB. A.NovakA.MorgadoA. T. E.PapiezB.GaubeS.LukasiewiczT.CamburuO.-M. (2024). Fool me once? Contrasting textual and visual explanations in a clinical decision-support setting. In Al-OnaizanY.BansalM.ChenY.-N. (Eds.), Proceedings of the 2024 conference on empirical methods in natural language processing (pp. 18891–18919). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.1051
28.
KörberM. (2019). Theoretical considerations and development of a questionnaire to measure trust in automation. In BagnaraS.TartagliaR.AlbolinoS.AlexanderT.FujitaY. (Eds.), Proceedings of the 20th congress of the international ergonomics association (IEA 2018) (pp. 13–30). Springer. https://doi.org/10.1007/978-3-319-96074-6_2
29.
KundelH. L.NodineC. F.ConantE. F.WeinsteinS. P. (2007). Holistic component of image perception in mammogram interpretation: Gaze-tracking study. Radiology, 242(2), 396–402. https://doi.org/10.1148/radiol.2422051997
30.
LakkarajuH.KamarE.CaruanaR.LeskovecJ. (2019). Faithful and customizable explanations of Black box models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’19, Honolulu, HI, USA, 27–28 January, 2019, pp. 131–138. https://doi.org/10.1145/3306618.3314229
31.
LangerM.OsterD.SpeithT.HermannsH.KästnerL.SchmidtE.SesingA.BaumK. (2021). What do we want from Explainable Artificial Intelligence (XAI)? – A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research. Artificial Intelligence, 296, Article 103473. https://doi.org/10.1016/j.artint.2021.103473
32.
LeistR. A.ProfitlichH.-J.HunsickerT.SonntagD. (2025). Towards trustable intelligent clinical decision support systems: A user study with ophthalmologists. In Proceedings of the 30th International Conference on Intelligent User Interfaces, IUI ’25, Cagliari, Italy, 24–27 March, 2025, pp. 1470–1484. https://doi.org/10.1145/3708359.3712136
33.
LiptonZ. C. (2018). The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3), 31–57. https://doi.org/10.1145/3236386.3241340
34.
LohH. W.OoiC. P.SeoniS.BaruaP. D.MolinariF.AcharyaU. R. (2022). Application of explainable artificial intelligence for healthcare: A systematic review of the last decade (2011-2022). Computer Methods and Programs in Biomedicine, 226, Article 107161. https://doi.org/10.1016/j.cmpb.2022.107161
35.
MalterudK.SiersmaV. D.GuassoraA. D. (2016). Sample size in qualitative interview studies: Guided by information power. Qualitative Health Research, 26(13), 1753–1760. https://doi.org/10.1177/1049732315617444
36.
Pahud de MortangesA.LuoH.ShuS. Z.KamathA.SuterY.ShelanM.PöllingerA.ReyesM. (2024). Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging. Npj Digital Medicine, 7(1), Article 195. https://doi.org/10.1038/s41746-024-01190-w
37.
ParasuramanR.RileyV. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors: The Journal of the Human Factors and Ergonomics Society, 39(2), 230–253. https://doi.org/10.1518/001872097778543886
38.
ParasuramanR.WickensC. D. (2008). Humans: Still vital after all these years of automation. Human Factors, 50(3), 511–520. https://doi.org/10.1518/001872008X312198
39.
PorwalP.PachadeS.KambleR.KokareM.DeshmukhG.SahasrabuddheV.MeriaudeauF. (2018). Indian diabetic retinopathy image dataset (IDRiD): A database for diabetic retinopathy screening research. Data, 3(3), Article 25. https://doi.org/10.3390/data3030025
40.
RiethM.OnnaschL.HagemannV. (2024). Adaptable automation for a more human-centered work design? Effects on human perception and behavior. International Journal of Human-Computer Studies, 186, Article 103246. https://doi.org/10.1016/j.ijhcs.2024.103246
41.
SauerJ.KaoC.-S.WastellD. (2012). A comparison of adaptive and adaptable automation under different levels of environmental stress. Ergonomics, 55(8), 840–853. https://doi.org/10.1080/00140139.2012.676673
42.
SchemmerM.HemmerP.NitscheM.KühlN.VössingM. (2022). A meta-analysis of the utility of Explainable Artificial Intelligence in human-AI decision-making. In 2022 AAAI/ACM Conference on AI, Ethics, and Society, Oxford, UK, 19–21 May, 2021, pp. 617–626. https://doi.org/10.1145/3514094.3534128
43.
ShamszareH.ChoudhuryA. (2023). Clinicians’ perceptions of artificial intelligence: Focus on workload, risk, trust, clinical decision making, and clinical integration. Health Care (Basel), 11(16), Article 2308. https://doi.org/10.3390/healthcare11162308
44.
SheridanT. B. (2011). Adaptive automation, level of automation, allocation authority, supervisory control, and adaptive control: Distinctions and modes of adaptation. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 41(4), 662–667. https://doi.org/10.1109/TSMCA.2010.2093888
45.
ShinD. (2021). The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable AI. International Journal of Human-Computer Studies, 146, Article 102551. https://doi.org/10.1016/j.ijhcs.2020.102551
46.
StadeE. C.EichstaedtJ. C.KimJ. P.Wiltsey StirmanS. (2025). Readiness evaluation for artificial intelligence-mental health deployment and implementation (READI): A review and proposed framework. Technology, Mind, and Behavior, 6(2), 111–122. https://doi.org/10.1037/tmb0000163
47.
SterzS.BaumK.BiewerS.HermannsH.Lauber-RönsbergA.MeinelP.LangerM. (2024). On the quest for effectiveness in human oversight: Interdisciplinary perspectives. In ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT ’24), Rio de Janeiro, Brazil, 3–6 June, 2024, pp. 2495–2507. https://doi.org/10.1145/3630106.3659051
48.
SubramanianH. V.CanfieldC.ShankD. B. (2024). Designing explainable AI to improve human-AI team performance: A medical stakeholder-driven scoping review. Artificial Intelligence in Medicine, 149, Article 102780. https://doi.org/10.1016/j.artmed.2024.102780
TingD. S. W.CheungC. Y.-L.LimG.TanG. S. W.QuangN. D.GanA.HamzahH.Garcia-FrancoR.San YeoI. Y.LeeS. Y.WongE. Y. M.SabanayagamC.BaskaranM.IbrahimF.TanN. C.FinkelsteinE. A.LamoureuxE. L.WongI. Y.BresslerN. M.WongT. Y. (2017). Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA, 318(22), 2211–2223. https://doi.org/10.1001/jama.2017.18152
51.
TonekaboniS.JoshiS.McCraddenM. D.GoldenbergA. (2019). What clinicians want: Contextualizing explainable machine learning for clinical end use. In Proceedings of the 4th Machine Learning for Healthcare Conference, Ann Arbor, Michigan, 9–10 August 2019, pp. 359–380. https://proceedings.mlr.press/v106/tonekaboni19a.html
52.
TopolE. J. (2019). High-performance medicine: The convergence of human and artificial intelligence. Nature Medicine, 25(1), 44–56. https://doi.org/10.1038/s41591-018-0300-7
53.
TusfiqurH. M.NguyenD. M. H.TruongM. T. N.NguyenT. A.NguyenB. T.BarzM.ProfitlichH.-J.ThanN. T. T.LeN.XieP.SonntagD. (2022). DRG-net: Interactive joint learning of multi-lesion segmentation and classification for diabetic retinopathy grading. arXiv:2212.14615. arXiv. https://doi.org/10.48550/arXiv.2212.14615
54.
UsmanT. M.SaheedY. K.NsangA.AjibesinA.RakshitS. (2023). A systematic literature review of machine learning based risk prediction models for diabetic retinopathy progression. Artificial Intelligence in Medicine, 143, Article 102617. https://doi.org/10.1016/j.artmed.2023.102617
55.
van de MerweK.MallamS.NazirS. (2024). Agent transparency, situation awareness, mental workload, and operator performance: A systematic literature review. Human Factors, 66(1), 180–208. https://doi.org/10.1177/00187208221077804
56.
VenkateshV.MorrisM. G.DavisG. B.DavisF. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. https://doi.org/10.2307/30036540
57.
WabroA.HerrmannM.WinklerE. C. (2024). When time is of the essence: Ethical reconsideration of XAI in time-sensitive environments. Journal of Medical Ethics, 51(8), 516–520. https://doi.org/10.1136/jme-2024-110046
XiangY.ZhaoL.LiuZ.WuX.ChenJ.LongE.LinD.ZhuY.ChenC.LinZ.LinH. (2020). Implementation of artificial intelligence in medicine: Status analysis and development suggestions. Artificial Intelligence in Medicine, 102, Article 101780. https://doi.org/10.1016/j.artmed.2019.101780
