Sage Journals: Discover world-class research

Abstract

Objective

To investigate how AI-provided explanations impact efficiency, diagnostic accuracy, user perceptions, and workflow integration in ophthalmologists’ clinical diagnostic and treatment workflows, this study explores the challenges in human-AI interaction with transparency features in time-sensitive environments.

Background

While explainable AI (XAI) aims to foster trust and understanding, its introduction into complex work domains can unintentionally increase cognitive load and disrupt workflows, especially in high-stakes medical settings, potentially impairing system performance.

Method

The multi-phase, mixed-methods study included two parts. Study 1 (N = 32) was a between-subjects experiment in which ophthalmologists diagnosed diabetic retinopathy with AI support, with or without visual explanations (e.g., highlighting lesions). Measures included diagnostic accuracy, diagnostic time, trust, and usefulness. Study 2 (N = 11) employed qualitative methods, including think-aloud protocols and interviews, to explore clinicians’ experiences with AI in daily (treatment) workflows.

Results

In Study 1, explanations did not improve accuracy but increased decision time, reducing efficiency. Trends suggested lower perceived usefulness and trust in the explanation condition. Qualitative data from Study 2 supported these findings; clinicians found explanations time-consuming and disruptive, questioning their practical value, especially for routine cases.

Conclusion

A critical trade-off exists between pursuing AI transparency and the operational demand for efficiency. Explanations, while well-intentioned, can function as efficiency pitfalls in time-pressured clinical practice, highlighting the boundary conditions and challenges in designing effective human-AI systems.

Application

These insights inform future AI system design, favoring adaptable, on-demand explanations tailored to user needs. Such a user-centric approach supports complex cases without impeding routine task efficiency.

Keywords

trust cognitive load decision support systems workflow integration time-efficiency

Get full access to this article

View all access options for this article.

References

Abràmoff

M. D.

Garvin

M. K.

Sonka

(2010). Retinal imaging and image analysis. IEEE Reviews in Biomedical Engineering, 3, 169–208. https://doi.org/10.1109/RBME.2010.2084567

Amann

Vetter

Blomberg

S. N.

Christensen

H. C.

Coffee

Gerke

Gilbert

T. K.

Hagendorff

Holm

Livne

Spezzatti

Strümke

Zicari

R. V.

Madai

V. I.

, on behalf of the Z-Inspection Initiative. (2022). To explain or not to explain?—Artificial intelligence explainability in clinical decision support systems. PLOS Digital Health, 1(2), Article e0000016. https://doi.org/10.1371/journal.pdig.0000016

Antoniadi

A. M.

Guendouz

Wei

Mazo

Becker

B. A.

Mooney

(2021). Current challenges and future opportunities for XAI in machine learning-based clinical decision support systems: A systematic review. Applied Sciences, 11(11), Article 5088. https://doi.org/10.3390/app11115088

Bansal

Zhou

Fok

Nushi

Kamar

Ribeiro

M. T.

Weld

(2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, Yokohama Japan, 8–13 May, 2021. https://doi.org/10.1145/3411764.3445717

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Braun

Clarke

(2019). Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health, 11(4), 589–597. https://doi.org/10.1080/2159676X.2019.1628806

Brooke

(1996). SUS: A “quick and dirty” usability scale. In Jordan

P. W.

Thomas

McClelland

I. L.

Weerdmeester

(Eds.), Usability evaluation in industry (p. 6). CRC Press.

Buçinca

Malaya

M. B.

Gajos

K. Z.

(2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), 1–21. https://doi.org/10.1145/3449287

Bussone

Stumpf

O’Sullivan

(2015). The role of explanations on trust and reliance in clinical decision support systems. In 2015 International Conference on Healthcare Informatics, Dallas, TX, USA, 21–23 October, 2015, pp. 160–169. https://doi.org/10.1109/ICHI.2015.26

10.

Calhoun

(2022). Adaptable (not adaptive) automation: Forefront of human–automation teaming. Human Factors, 64(2), 269–277. https://doi.org/10.1177/00187208211037457

11.

Carrigan

A. J.

Charlton

Wiggins

M. W.

Georgiou

Palmeri

Curby

K. M.

(2022). Cue utilisation reduces the impact of response bias in histopathology. Applied Ergonomics, 98, Article 103590. https://doi.org/10.1016/j.apergo.2021.103590

12.

Carrigan

A. J.

Magnussen

Georgiou

Curby

K. M.

Palmeri

T. J.

Wiggins

M. W.

(2021). Differentiating experience from cue utilization in radiological assessments. Human Factors: The Journal of the Human Factors and Ergonomics Society, 63(4), 635–646. https://doi.org/10.1177/0018720820902576

13.

Chanda

Hauser

Hobelsberger

Bucher

T.-C.

Garcia

C. N.

Wies

Kittler

Tschandl

Navarrete-Dechent

Podlipnik

Chousakos

Crnaric

Majstorovic

Alhajwan

Foreman

Peternel

Sarap

Özdemir

İ.

Barnhill

R. L.

Brinker

T. J.

(2024). Dermatologist-like explainable AI enhances trust and confidence in diagnosing melanoma. Nature Communications, 15(1), Article 524. https://doi.org/10.1038/s41467-023-43095-4

14.

Combi

Amico

Bellazzi

Holzinger

Moore

J. H.

Zitnik

Holmes

J. H.

(2022). A manifesto on explainability for artificial intelligence in medicine. Artificial Intelligence in Medicine, 133, Article 102423. https://doi.org/10.1016/j.artmed.2022.102423

15.

Drew

Võ

M. L.-H.

Wolfe

J. M.

(2013). The invisible gorilla strikes again: Sustained inattentional blindness in expert observers. Psychological Science, 24(9), 1848–1853. https://doi.org/10.1177/0956797613479386

16.

Elhaddad

Hamam

(2024). AI-driven clinical decision support systems: An ongoing pursuit of potential. Cureus, 16(4), Article e57728. https://doi.org/10.7759/cureus.57728

17.

Fok

Weld

D. S.

(2024). In search of verifiability: Explanations rarely enable complementary performance in AI-advised decision making. AI Magazine, 45(3), 317–332. https://doi.org/10.1002/aaai.12182

18.

Franke

Attig

Wessel

(2019). A personal resource for technology interaction: Development and validation of the Affinity for Technology Interaction (ATI) scale. International Journal of Human–Computer Interaction, 35(6), 456–467. https://doi.org/10.1080/10447318.2018.1456150

19.

Gargeya

Leng

(2017). Automated identification of diabetic retinopathy using deep learning. Ophthalmology, 124(7), 962–969. https://doi.org/10.1016/j.ophtha.2017.02.008

20.

Gaube

Suresh

Raue

Lermer

Koch

T. K.

Hudecek

M. F. C.

Ackery

A. D.

Grover

S. C.

Coughlin

J. F.

Frey

Kitamura

F. C.

Ghassemi

Colak

(2023). Non-task expert physicians benefit from correct explainable AI advice when reviewing X-rays. Scientific Reports, 13(1), Article 1383. https://doi.org/10.1038/s41598-023-28633-w

21.

Gaube

Suresh

Raue

Merritt

Berkowitz

S. J.

Lermer

Coughlin

J. F.

Guttag

J. V.

Colak

Ghassemi

(2021). Do as AI say: Susceptibility in deployment of clinical decision-aids. Npj Digital Medicine, 4, 1–8. https://doi.org/10.1038/s41746-021-00385-9

22.

Guest

Bunce

Johnson

(2006). How many interviews are enough?: An experiment with data saturation and variability. Field Methods, 18(1), 59–82. https://doi.org/10.1177/1525822X05279903

23.

Baxter

S. L.

Zhou

Zhang

(2019). The practical implementation of artificial intelligence technologies in medicine. Nature Medicine, 25(1), 30–36. https://doi.org/10.1038/s41591-018-0307-0

24.

Herm

L.-V.

(2023). Impact of explainable AI on cognitive load: Insights from an empirical study. ECIS 2023 Research Papers. Article 269.

25.

Holzinger

Langs

Denk

Zatloukal

Müller

(2019). Causability and explainability of artificial intelligence in medicine. WIREs Data Mining and Knowledge Discovery, 9(4), Article e1312. https://doi.org/10.1002/widm.1312

26.

Hudon

Demazure

Karran

Léger

P.-M.

Sénécal

(2021). Explainable artificial intelligence (XAI): How the visualization of AI predictions affects user cognitive load and confidence. In Davis

F. D.

Riedl

vom Brocke

Léger

P.-M.

Randolph

A. B.

Müller-Putz

(Eds.), Information systems and neuroscience (pp. 237–246). Springer. https://doi.org/10.1007/978-3-030-88900-5_27

27.

Kayser

M. G.

Menzat

Emde

Bercean

B. A.

Novak

Morgado

A. T. E.

Papiez

Gaube

Lukasiewicz

Camburu

O.-M.

(2024). Fool me once? Contrasting textual and visual explanations in a clinical decision-support setting. In Al-Onaizan

Bansal

Chen

Y.-N.

(Eds.), Proceedings of the 2024 conference on empirical methods in natural language processing (pp. 18891–18919). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.1051

28.

Körber

(2019). Theoretical considerations and development of a questionnaire to measure trust in automation. In Bagnara

Tartaglia

Albolino

Alexander

Fujita

(Eds.), Proceedings of the 20th congress of the international ergonomics association (IEA 2018) (pp. 13–30). Springer. https://doi.org/10.1007/978-3-319-96074-6_2

29.

Kundel

H. L.

Nodine

C. F.

Conant

E. F.

Weinstein

S. P.

(2007). Holistic component of image perception in mammogram interpretation: Gaze-tracking study. Radiology, 242(2), 396–402. https://doi.org/10.1148/radiol.2422051997

30.

Lakkaraju

Kamar

Caruana

Leskovec

(2019). Faithful and customizable explanations of Black box models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’19, Honolulu, HI, USA, 27–28 January, 2019, pp. 131–138. https://doi.org/10.1145/3306618.3314229

31.

Langer

Oster

Speith

Hermanns

Kästner

Schmidt

Sesing

Baum

(2021). What do we want from Explainable Artificial Intelligence (XAI)? – A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research. Artificial Intelligence, 296, Article 103473. https://doi.org/10.1016/j.artint.2021.103473

32.

Leist

R. A.

Profitlich

H.-J.

Hunsicker

Sonntag

(2025). Towards trustable intelligent clinical decision support systems: A user study with ophthalmologists. In Proceedings of the 30th International Conference on Intelligent User Interfaces, IUI ’25, Cagliari, Italy, 24–27 March, 2025, pp. 1470–1484. https://doi.org/10.1145/3708359.3712136

33.

Lipton

Z. C.

(2018). The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3), 31–57. https://doi.org/10.1145/3236386.3241340

34.

Loh

H. W.

Ooi

C. P.

Seoni

Barua

P. D.

Molinari

Acharya

U. R.

(2022). Application of explainable artificial intelligence for healthcare: A systematic review of the last decade (2011-2022). Computer Methods and Programs in Biomedicine, 226, Article 107161. https://doi.org/10.1016/j.cmpb.2022.107161

35.

Malterud

Siersma

V. D.

Guassora

A. D.

(2016). Sample size in qualitative interview studies: Guided by information power. Qualitative Health Research, 26(13), 1753–1760. https://doi.org/10.1177/1049732315617444

36.

Pahud de Mortanges

Luo

Shu

S. Z.

Kamath

Suter

Shelan

Pöllinger

Reyes

(2024). Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging. Npj Digital Medicine, 7(1), Article 195. https://doi.org/10.1038/s41746-024-01190-w

37.

Parasuraman

Riley

(1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors: The Journal of the Human Factors and Ergonomics Society, 39(2), 230–253. https://doi.org/10.1518/001872097778543886

38.

Parasuraman

Wickens

C. D.

(2008). Humans: Still vital after all these years of automation. Human Factors, 50(3), 511–520. https://doi.org/10.1518/001872008X312198

39.

Porwal

Pachade

Kamble

Kokare

Deshmukh

Sahasrabuddhe

Meriaudeau

(2018). Indian diabetic retinopathy image dataset (IDRiD): A database for diabetic retinopathy screening research. Data, 3(3), Article 25. https://doi.org/10.3390/data3030025

40.

Rieth

Onnasch

Hagemann

(2024). Adaptable automation for a more human-centered work design? Effects on human perception and behavior. International Journal of Human-Computer Studies, 186, Article 103246. https://doi.org/10.1016/j.ijhcs.2024.103246

41.

Sauer

Kao

C.-S.

Wastell

(2012). A comparison of adaptive and adaptable automation under different levels of environmental stress. Ergonomics, 55(8), 840–853. https://doi.org/10.1080/00140139.2012.676673

42.

Schemmer

Hemmer

Nitsche

Kühl

Vössing

(2022). A meta-analysis of the utility of Explainable Artificial Intelligence in human-AI decision-making. In 2022 AAAI/ACM Conference on AI, Ethics, and Society, Oxford, UK, 19–21 May, 2021, pp. 617–626. https://doi.org/10.1145/3514094.3534128

43.

Shamszare

Choudhury

(2023). Clinicians’ perceptions of artificial intelligence: Focus on workload, risk, trust, clinical decision making, and clinical integration. Health Care (Basel), 11(16), Article 2308. https://doi.org/10.3390/healthcare11162308

44.

Sheridan

T. B.

(2011). Adaptive automation, level of automation, allocation authority, supervisory control, and adaptive control: Distinctions and modes of adaptation. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 41(4), 662–667. https://doi.org/10.1109/TSMCA.2010.2093888

45.

Shin

(2021). The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable AI. International Journal of Human-Computer Studies, 146, Article 102551. https://doi.org/10.1016/j.ijhcs.2020.102551

46.

Stade

E. C.

Eichstaedt

J. C.

Kim

J. P.

Wiltsey Stirman

(2025). Readiness evaluation for artificial intelligence-mental health deployment and implementation (READI): A review and proposed framework. Technology, Mind, and Behavior, 6(2), 111–122. https://doi.org/10.1037/tmb0000163

47.

Sterz

Baum

Biewer

Hermanns

Lauber-Rönsberg

Meinel

Langer

(2024). On the quest for effectiveness in human oversight: Interdisciplinary perspectives. In ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT ’24), Rio de Janeiro, Brazil, 3–6 June, 2024, pp. 2495–2507. https://doi.org/10.1145/3630106.3659051

48.

Subramanian

H. V.

Canfield

Shank

D. B.

(2024). Designing explainable AI to improve human-AI team performance: A medical stakeholder-driven scoping review. Artificial Intelligence in Medicine, 149, Article 102780. https://doi.org/10.1016/j.artmed.2024.102780

49.

Sweller

(2011). Cognitive load theory. In Mestre

J. P.

Ross

B. H.

(Eds.), Psychology of learning and motivation (Vol. 55, pp. 37–76). Academic Press. https://doi.org/10.1016/B978-0-12-387691-1.00002-8

50.

Ting

D. S. W.

Cheung

C. Y.-L.

Lim

Tan

G. S. W.

Quang

N. D.

Gan

Hamzah

Garcia-Franco

San Yeo

I. Y.

Lee

S. Y.

Wong

E. Y. M.

Sabanayagam

Baskaran

Ibrahim

Tan

N. C.

Finkelstein

E. A.

Lamoureux

E. L.

Wong

I. Y.

Bressler

N. M.

Wong

T. Y.

(2017). Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA, 318(22), 2211–2223. https://doi.org/10.1001/jama.2017.18152

51.

Tonekaboni

Joshi

McCradden

M. D.

Goldenberg

(2019). What clinicians want: Contextualizing explainable machine learning for clinical end use. In Proceedings of the 4th Machine Learning for Healthcare Conference, Ann Arbor, Michigan, 9–10 August 2019, pp. 359–380. https://proceedings.mlr.press/v106/tonekaboni19a.html

52.

Topol

E. J.

(2019). High-performance medicine: The convergence of human and artificial intelligence. Nature Medicine, 25(1), 44–56. https://doi.org/10.1038/s41591-018-0300-7

53.

Tusfiqur

H. M.

Nguyen

D. M. H.

Truong

M. T. N.

Nguyen

T. A.

Nguyen

B. T.

Barz

Profitlich

H.-J.

Than

N. T. T.

Xie

Sonntag

(2022). DRG-net: Interactive joint learning of multi-lesion segmentation and classification for diabetic retinopathy grading. arXiv:2212.14615. arXiv. https://doi.org/10.48550/arXiv.2212.14615

54.

Usman

T. M.

Saheed

Y. K.

Nsang

Ajibesin

Rakshit

(2023). A systematic literature review of machine learning based risk prediction models for diabetic retinopathy progression. Artificial Intelligence in Medicine, 143, Article 102617. https://doi.org/10.1016/j.artmed.2023.102617

55.

van de Merwe

Mallam

Nazir

(2024). Agent transparency, situation awareness, mental workload, and operator performance: A systematic literature review. Human Factors, 66(1), 180–208. https://doi.org/10.1177/00187208221077804

56.

Venkatesh

Morris

M. G.

Davis

G. B.

Davis

F. D.

(2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. https://doi.org/10.2307/30036540

57.

Wabro

Herrmann

Winkler

E. C.

(2024). When time is of the essence: Ethical reconsideration of XAI in time-sensitive environments. Journal of Medical Ethics, 51(8), 516–520. https://doi.org/10.1136/jme-2024-110046

58.

Wickens

C. D.

(2008). Multiple resources and mental workload. Human Factors, 50(3), 449–455. https://doi.org/10.1518/001872008X288394

59.

Wilkinson

C. P.

Ferris

F. L.

Klein

R. E.

Lee

P. P.

Agardh

C. D.

Davis

Dills

Kampik

Pararajasegaram

Verdaguer

J. T.

Global Diabetic Retinopathy Project Group . (2003). Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology, 110(9), 1677–1682. https://doi.org/10.1016/S0161-6420(03)00475-5

60.

Xiang

Zhao

Liu

Chen

Long

Lin

Zhu

Chen

Lin

(2020). Implementation of artificial intelligence in medicine: Status analysis and development suggestions. Artificial Intelligence in Medicine, 102, Article 101780. https://doi.org/10.1016/j.artmed.2019.101780

Efficiency Pitfalls of Explainable AI in Clinical Diagnostic and Treatment Human-AI Workflows

Abstract

Objective

Background

Method

Results

Conclusion

Application

Keywords

Get full access to this article

References