Deployments of artificial intelligence (AI) and machine learning (ML) in healthcare can both help and harm patient outcomes, amplifying calls for a human-centered approach to AI/ML development. This paper details one approach guided by three principles: (1) pursue human-machine team (HMT) performance, not algorithm performance, (2) build interpretability throughout, and (3) constrain development to deconstrain interactions. We describe how these principles influenced our development of two algorithms predicting patient decompensation events five minutes into the future. These algorithms showed comparable performance to other similar models with enhanced interpretability that greatly expanded HMT interaction possibilities. Our early investments in the potential for teaming appeared to pay dividends for the resultant HMT.
References
1.
AmershiS.CakmakM.KnoxW. B.KuleszaT. (2014). Power to the People: The Role of Humans in Interactive Machine Learning. AI Magazine, 35(4), Article 4. https://doi.org/10.1609/aimag.v35i4.2513
BeedeE.BaylorE.HerschF.IurchenkoA.WilcoxL.RuamviboonsukP.VardoulakisL. M. (2020). A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–12. https://doi.org/10.1145/3313831.3376718
4.
BorowskiM.SiebigS.WredeC.ImhoffM. (2011). Reducing False Alarms of Intensive Care Online-Monitoring Systems: An Evaluation of Two Signal Extraction Algorithms. Computational and Mathematical Methods in Medicine, 2011, e143480. https://doi.org/10.1155/2011/143480
5.
CaruanaR.LouY.GehrkeJ.KochP.SturmM.ElhadadN. (2015). Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1721–1730. https://doi.org/10.1145/2783258.2788613
6.
ClevelandW. S. (1981). LOWESS: A Program for Smoothing Scatterplots by Robust Locally Weighted Regression. The American Statistician, 35(1), 54–54. https://doi.org/10.2307/2683591
DonninoM. W.SalciccioliJ. D.HowellM. D.CocchiM. N.GibersonB.BergK.GautamS.CallawayC. (2014). Time to administration of epinephrine and outcome after in-hospital cardiac arrest with non- shockable rhythms: Retrospective analysis of large in-hospital data registry. BMJ, 348, g3028. https://doi.org/10.1136/bmj.g3028
9.
EscobarG. J.LiuV. X.SchulerA.LawsonB.GreeneJ. D.KipnisP. (2020). Automated Identification of Adults at Risk for In-Hospital Clinical Deterioration. New England Journal of Medicine, 383(20), 1951–1960. https://doi.org/10.1056/NEJMsa2001090
10.
EssayP.BalkanB.SubbianV. (2020). Decompensation in Critical Care: Early Prediction of Acute Heart Failure Onset. JMIR Medical Informatics, 8(8), e19892. https://doi.org/10.2196/19892
11.
HoffmanR. R.MillerT.ClanceyW. J. (2022). Psychology and AI at a Crossroads: How Might Complex Systems Explain Themselves?The American Journal of Psychology, 135(4), 365–378. https://doi.org/10.5406/19398298.135.4.01
12.
HorwoodC. R.Moffatt-BruceS. D.FitzgeraldM.RayoM. F. (2018). A qualitative analysis of clinical decompensation in the surgical patient: Perceptions of nurses and physicians. Surgery, 164(6), 1311–1315. https://doi.org/10.1016/j.surg.2018.06.006
13.
HorwoodC. R.RayoM. F.FitzgeraldM.BalkinE. A.Moffatt-BruceS. D. (2018). Gaps Between Alarm Capabilities and Decision-making Needs: An Observational Study of Detecting Patient Decompensation. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care, 7, 112–116. https://doi.org/10.1177/2327857918071028
KleinG.WoodsD. D.BradshawJ. M.HoffmanR. R.FeltovichP. J. (2004). Ten Challenges for Making Automation a “Team Player” in Joint Human-Agent Activity. IEEE Intelligent Systems, 19(6), 91–95. https://doi.org/10.1109/MIS.2004.74
16.
KreningS.FeighK. M. (2018). Interaction Algorithm Effect on Human Experience with Reinforcement Learning. ACM Transactions on Human-Robot Interaction, 7(2), 16:1-16:22. https://doi.org/10.1145/3277904
17.
KunarM. A.WatsonD. G.Taylor-PhillipsS.WolskaJ. (2017). Low prevalence search for cancers in mammograms: Evidence using laboratory experiments and computer aided detection. Journal of Experimental Psychology: Applied, 23(4), 369–385. https://doi.org/10.1037/xap0000132
18.
LeeJ. D.SeppeltB. D. (2023). Design for Human-Automation and Human-Autonomous Systems. In NofS. Y. (Ed.), Springer Handbook of Automation (pp. 433–455). Springer International Publishing. https://doi.org/10.1007/978-3-030-96729-1_19
19.
LiM.MoreyD. A.RayoM. F. (2021). Symbiotic Design Application in Healthcare: Preventing Hospital Acquired Infections. Proceedings of the International Symposium of Human Factors and Ergonomics in Healthcare, 10(1), 211–216. https://doi.org/10.1177/2327857921101138
20.
MathisM. R.EngorenM. C.WilliamsA. M.BiesterveldB. E.CroteauA. J.CaiL.KimR. B.LiuG.WardK. R.NajarianK.GryakJ., BCIL Collaborators Group. (2022). Prediction of Postoperative Deterioration in Cardiac Surgery Patients Using Electronic Health Record and Physiologic Waveform Data. Anesthesiology, 137(5), 586–601. https://doi.org/10.1097/ALN.0000000000004345
21.
McLellanM. C.GauvreauK.ConnorJ. A. (2014). Validation of the Cardiac Children’s Hospital Early Warning Score: An Early Warning Scoring Tool to Prevent Cardiopulmonary Arrests in Children with Heart Disease. Congenital Heart Disease, 9(3), 194–202. https://doi.org/10.1111/chd.12132
22.
MoreyD. A.RayoM. F. (2024). Situated Interpretation and Data: Explainability to Convey Machine Misalignment. IEEE Transactions on Human-Machine Systems, 54(1), 100–109. https://doi.org/10.1109/THMS.2023.3334988
23.
MostellerF.TukeyJ. W. (1968). Data analysis, including statistics. Handbook of Social Psychology, 2, 80–203.
24.
ObermeyerZ.PowersB.VogeliC.MullainathanS. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342
RayoM. F.FitzgeraldM. C.GiffordR. C.MoreyD. A.ReynoldsM. E.D’AnnolfoK.JefferiesC. M. (2020). The Need for Machine Fitness Assessment: Enabling Joint Human-Machine Performance in Consumer Health Technologies. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care, 9, 40–42. https://doi.org/10.1177/2327857920091041
27.
RayoM. F.Moffatt-BruceS. D. (2015). Alarm system management: Evidence-based guidance encouraging direct measurement of informativeness to improve alarm response: Table 1. BMJ Quality & Safety, 24(4), 282–286. https://doi.org/10.1136/bmjqs-2014-003373
28.
RudinC. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), Article 5. https://doi.org/10.1038/s42256-019-0048-x
29.
SamekW.WiegandT.MüllerK.-R. (2017). Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models. International Telecommunication Union Journal: ICT Discoveries, 1, 39–48.
SorkinR. D.WoodsD. D. (1985). Systems with Human Monitors: A Signal Detection Analysis. Human–Computer Interaction, 1(1), 49–75. https://doi.org/10.1207/s15327051hci0101_2
WickensC. D.DixonS. R. (2007). The benefits of imperfect diagnostic automation: A synthesis of the literature. Theoretical Issues in Ergonomics Science, 8(3), 201–212. https://doi.org/10.1080/14639220500370105
