When Usability Undermines Explainability: A Case Study Designing AI-Infused Technologies in Healthcare

Abstract

Designing AI-infused technologies in healthcare involves multidisciplinary design considerations, like the usability of the display and explainability of the AI algorithm. We present three vignettes from a case study in which design decisions intended to improve usability inadvertently undermined explainability, and vice versa. Our results illustrate potential conflicts between disciplines and suggest the need for joint activity as a unifying design perspective. From these insights, we discuss revisions to our AI-infused technology.

Get full access to this article

View all access options for this article.

References

Amershi

Weld

Vorvoreanu

Fourney

Nushi

Collisson

Suh

Iqbal

Bennett

P. N.

Inkpen

Teevan

Kikin-Gil

Horvitz

(2019). Guidelines for Human-AI Interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–13. https://doi.org/10.1145/3290605.3300233

Arrieta

A. B.

Díaz-Rodríguez

Del Ser

Bennetot

Tabik

Barbado

Garcia

Gil-Lopez

Molina

Benjamins

Chatila

Herrera

(2020). Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58, 82–115. https://doi.org/10.1016/j.inffus.2019.12.012

Ericsson

K. A.

Simon

H. A.

(1980). Verbal reports as data. Psychological Review, 87(3), 215–251. https://doi.org/10.1037/0033-295X.87.3.215

Fram

(2013). The Constant Comparative Analysis Method Outside of Grounded Theory. The Qualitative Report, 18(1), 1–25.

Klein

Feltovich

P. J.

Bradshaw

J. M.

Woods

D. D.

(2005). Common Ground and Coordination in Joint Activity. In Rouse

W. B.

Boff

K. R.

(Eds.), Organizational Simulation (pp. 139–184). John Wiley & Sons, Inc. https://doi.org/10.1002/0471739448.ch6

Morey

D. A.

Gifford

R. C.

Rayo

M. F.

(2024). Early Investments for Teaming Dividends: A Human-Centered Approach to a Patient Decompensation Prediction Algorithm. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care, 13(1), 7–11. https://doi.org/10.1177/2327857924131046

Morey

D. A.

Rayo

M. F.

(2024). Situated Interpretation and Data: Explainability to Convey Machine Misalignment. IEEE Transactions on Human-Machine Systems, 54(1), 100–109. https://doi.org/10.1109/THMS.2023.3334988

Morey

D. A.

Walli

Cassidy

K. S.

Tewani

P. K.

Reynolds

M. E.

Malone

Jalaeian

Rayo

M. F.

McGeorge

N. M.

(2023). Towards Joint Activity Design Heuristics: Essentials for Human- Machine Teaming. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 21695067231193646. https://doi.org/10.1177/21695067231193646

Nielsen

(1994). Enhancing the explanatory power of usability heuristics. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 152–158.

10.

Popovich

Szecket

Nahill

(2019). Framing of clinical information affects physicians’ diagnostic accuracy. Emergency Medicine Journal, 36(10), 589–594. https://doi.org/10.1136/emermed-2019-208409

11.

Rayo

M. F.

(2017). Designing for collaborative autonomy: Updating user-centered design heuristics and evaluation methods. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 61, 1091–1095. https://doi.org/10.1177/1541931213601877

12.

Woods

D. D.

(1984). Visual momentum: A concept to improve the cognitive coupling of person and computer. International Journal of Man-Machine Studies, 21(3), 229–244. https://doi.org/10.1016/S0020-7373(84)80043-7