In the coming years, artificial intelligence (AI) will pervade almost every aspect of the health care delivery system. AI has the potential to improve patient safety (e.g., diagnostic accuracy) as well as reduce the burden on clinicians (e.g., documentation-related workload); however, these benefits are yet to be realized. AI is only one element of a larger sociotechnical system that needs to be considered for effective AI application. In this paper, we describe the current challenges of integrating AI into clinical care and propose a sociotechnical systems (STS) approach for AI design and implementation. We demonstrate the importance of an STS approach through a case study on the design and implementation of a clinical decision support (CDS). In order for AI to reach its potential, the entire work system as well as clinical workflow must be systematically considered throughout the design of AI technology.
ArbabshiraniM. R.FornwaltB. K.MongelluzzoG. J.SueverJ. D.GeiseB. D.PatelA. A.MooreG. J. (2018). Advanced machine learning in action: identification of intracranial hemorrhage on computed tomography scans of the head with clinical workflow integration. NPJ Digital Medicine, 1(1), 9.
2.
BeedeE.BaylorE.HerschF.IurchenkoA.WilcoxL.RuamviboonsukP.VardoulakisL. M. (2020). A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. Paper presented at the: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems.
3.
BendaN. C.NovakL. L.RealeC.AnckerJ. S. (2022). Trust in AI: why we should be designing for APPROPRIATE reliance. Journal of the American Medical Informatics Association: JAMIA, 29(1), 207–212. https://doi.org/10.1093/jamia/ocab238
CarayonP. (2009). The Balance Theory and the Work System Model … Twenty Years Later Twenty years later. International Journal of Human-Computer Interaction, 25(5), 313–327. https://doi.org/10.1080/10447310902864928
6.
CarayonP (2019). Human Factors in Health(care) Informatics: Toward Continuous Sociotechnical System Design. Studies in Health Technology and Informatics, 265(1), 22–27. https://doi.org/10.3233/SHTI190131
7.
CarayonP.CartmillR.HoonakkerP.HundtA. S.KarshB.KruegerD.WetterneckT. B. (2012). Human factors analysis of workflow in health information technology implementation. In CarayonP. (Ed.), Handbook of Human Factors and Ergonomics in Health Care and Patient Safety (Second Edition ed., pp. 507–521). Taylor & Francis Group.
8.
CarayonP.HoonakkerP.HundtA. S.SalweiM. E.WangY.WiegmannD.PattersonB. (2019). Application of human factors to improve usability of clinical decision support for diagnostic decision-making: Results of a scenario-based simulation study. BMJ Quality & Safety.
9.
CarayonP.SalweiM. E. (2021). Moving toward a sociotechnical systems approach to continuous health information technology design: The path forward for improving electronic health record usability and reducing clinician burnout. Journal of the American Medical Informatics Association: JAMIA, 28, 1026–1028. https://doi.org/10.1093/jamia/ocab002
10.
CarayonP.Schoofs HundtA.KarshB. T.GursesA. P.AlvaradoC. J.SmithM.Flatley BrennanP. (2006). Work system design for patient safety: The SEIPS model. Quality & Safety in Health Care, 15(Supplement I), i50. https://doi.org/10.1136/qshc.2005.015842
11.
CarayonP.WetterneckT. B.HundtA. S.RoughS.SchroederM. (2008). Continuous Technology Implementation and Sustainability of Sociotechnical Change: A Case Study of Advanced Intravenous Infusion Pump Technology Implementation in a Hospital. In ZinkK. (Ed.), Corporate Sustainability as a Challenge for Comprehensive Management (pp. 139–151). Springer.
12.
CarayonP.WetterneckT. B.Rivera-RodriguezA. J.HundtA. S.HoonakkerP.HoldenR.GursesA. P. (2014). Human factors systems approach to healthcare quality and patient safety. Applied Ergonomics, 45(1), 14–25.
13.
CeliL. A.FineB.StoneD. J. (2019). An awakening in medicine: the partnership of humanity and intelligent machines. The Lancet Digital Health, 1(6), e255–e257. https://doi.org/10.1016/s2589-7500(19)30127-x
CoieraE. (2019). The last mile: where artificial intelligence meets reality. Journal of Medical Internet Research, 21(11), Article e16323. https://doi.org/10.2196/16323
16.
DétienneF. (2006). Collaborative design: Managing task interdependencies and multiple perspectives. Interacting with Computers, 18(1), 1–20. https://doi.org/10.1016/j.intcom.2005.05.001
17.
EndsleyM. R. (2017). From Here to Autonomy. Human Factors: The Journal of the Human Factors and Ergonomics Society, 59(1), 5–27. https://doi.org/10.1177/0018720816681350
18.
GarveyK. V.CraigK. J. T.RussellR. G.NovakL.MooreD.PreiningerA. M.JacksonG. P.MillerB. M. (2021). The potential and the imperative: the Gap in AI-related clinical competencies and the need to close it. Medical Science Educator, 31(6), 2055–2060. https://doi.org/10.1007/s40670-021-01377-w
19.
GrossmanC.PowersB.McGinnisJ. M. (2011). Digital infrastructure for the learning health system: The foundation for continuous improvement in health and health care: Workshop series summary. The National Academies Press.
20.
HoonakkerP. L. T.CarayonP.SalweiM. E.HundtA. S.WiegmannD.KleinschmidtP.PuliaM. S.WangY.NovakC.PattersonB. W. (2019). The Design of PE Dx, a CDS to Support Pulmonary Embolism Diagnosis in the ED. Studies in Health Technology and Informatics, 265(1), 134–140. https://doi.org/10.3233/SHTI190152
21.
International Organization for Standardization (2010). ISO 9241-210 ergonomics of human-system interaction - Part 210: Human-centred design for interactive systems.
22.
IsraniS. T.VergheseA. (2019). Humanizing artificial intelligence. Jsls: Journal of the Society of Laparoendoscopic Surgeons, 321(1), 29–30. https://doi.org/10.1001/jama.2018.19398
23.
KarshB.-T. (2004). Beyond usability: designing effective technology implementation systems to promote patient safety. Quality and Safety in Health Care, 13(5), 388–394. https://doi.org/10.1136/qhc.13.5.388
24.
KhanS.RichardsonS.LiuA.MecheryV.McCullaghL.SchachterA.PardoS.McGinnT. (2019). Improving provider adoption with adaptive clinical decision support surveillance: AAn observational study. JMIR Human Factors, 6(1), Article e10245. https://doi.org/10.2196/10245
25.
KlineJ. A.PetersonC. E.SteuerwaldM. T. (2010). Prospective Evaluation of Real-time Use of the Pulmonary Embolism Rule-out Criteria in an Academic Emergency Department. Academic Emergency Medicine, 17(9), 1016–1019. https://doi.org/10.1111/j.1553-2712.2010.00826.x
26.
LesselrothB. J.YangJ.McConnachieJ.BrenkT.WinterbottomL. (2011). Addressing the sociotechnical drivers of quality improvement: a case study of post-operative DVT prophylaxis computerised decision support. Quality and safety in health care, 20(5), 042689. http://dx.doi.org/10.1136/bmjqs.2010.042689qshc. 2010.
27.
LucassenW.GeersingG.-J.ErkensP. M. G.ReitsmaJ. B.MoonsK. G. M.BullerH.van WeertH. C. (2011). Clinical decision rules for excluding pulmonary embolism: A meta-analysis. Annals of Internal Medicine, 155(7), 448–460. https://doi.org/10.7326/0003-4819-155-7-201110040-00007
MathenyM.IsraniS. T.AhmedM.WhicherD. (2019). Artificial intelligence in health care: The hope, the hype, the promise, the Peril. NAM Special PublicationNational Academy of Medicine.
30.
MellesM.AlbayrakA.GoossensR. (2021). Innovating health care: key characteristics of human-centered design. International Journal for Quality in Health Care, 33(Supplement_1), 37–44. https://doi.org/10.1093/intqhc/mzaa127
31.
MeyerA. N. D.GiardinaT. D.SpitzmuellerC.ShahidU.ScottT. M. T.SinghH. (2020). Patient Perspectives on the Usefulness of an Artificial Intelligence-Assisted Symptom Checker: Cross-Sectional Survey Study. Journal of Medical Internet Research, 22(1), Article e14679. https://doi.org/10.2196/14679
OkoloC. T.KamathS.DellN.VashisthaA. (2021). It cannot do all of my work": Community health worker perceptions of AI-enabled mobile health applications in rural India. Paper presented at the CHI'21.
34.
PangtiR.MathurJ.ChouhanV.KumarS.RajputL.ShahS.GuptaA.DixitA.DholakiaD.GuptaS.GuptaS.GeorgeM.SharmaV. K.GuptaS. (2021). A machine learning‐based, decision support, mobile phone application for diagnosis of common dermatological diseases. Journal of the European Academy of Dermatology and Venereology, 35(2), 536–545. https://doi.org/10.1111/jdv.16967
35.
PatelB. N.RosenbergL.WillcoxG.BaltaxeD.LyonsM.IrvinJ.RajpurkarP.AmrheinT.GuptaR.HalabiS.LanglotzC.LoE.MammarappallilJ.MarianoA. J.RileyG.SeekinsJ.ShenL.ZuckerE.LungrenM. (2019). Human-machine partnership with artificial intelligence for chest radiograph diagnosis. NPJ Digital Medicine, 2(1), 111–210. https://doi.org/10.1038/s41746-019-0189-7
36.
PetersenC.SmithJ.FreimuthR. R.GoodmanK. W.JacksonG. P.KannryJ.LiuH.MadhavanS.SittigD. F.WrightA. (2021). Recommendations for the safe, effective use of adaptive CDS in the US healthcare system: an AMIA position paper. Journal of the American Medical Informatics Association: JAMIA, 28(4), 677–684. https://doi.org/10.1093/jamia/ocaa319
37.
RajaA. S.GreenbergJ. O.QaseemA.DenbergT. D.FittermanN.SchuurJ. D. (2015). Evaluation of patients with suspected acute pulmonary embolism: best practice advice from the Clinical Guidelines Committee of the American College of Physicians. Annals of Internal Medicine, 163(9), 701–711. https://doi.org/10.7326/M14-1772
38.
RuamviboonsukP.KrauseJ.ChotcomwongseP.SayresR.RamanR.WidnerK.CampanaB. J. L.PheneS.HemaratK.TadaratiM.Silpa-ArchaS.LimwattanayingyongJ.RaoC.KuruvillaO.JungJ.TanJ.OrprayoonS.KangwanwongpaisanC.SukumalpaiboonR.WebsterD. R. (2019). Deep learning versus human graders for classifying diabetic retinopathy severity in a nationwide screening program. npj Digital Medicine, 2(1), 25. https://doi.org/10.1038/s41746-019-0099-8
39.
SalweiM. E. (2020). Human-Centered Design of Health Information Technology for Workflow Integration. The University of Wisconsin-Madison.
40.
SalweiM. E.CarayonP.HoonakkerP.HundtA. S.WiegmannD.PattersonB. W.PuliaM. (2021). Workflow integration analysis of a human factors-based clinical decision support in the emergency department. Applied Ergonomics, 97(1), 1–16. https://doi.org/10.1177/00187208221078625
ScottI. A.CookD.CoieraE. W.RichardsB. (2019). Machine learning in clinical practice: prospects and pitfalls. Medical Journal of Australia, 211(5), 203–205. https://doi.org/10.5694/mja2.50294
43.
SendakM. P.GaoM.BrajerN.BaluS. (2020). Presenting machine learning model information to clinical end users with model facts labels. NPJ Digital Medicine, 3(1), 41–44.
44.
ShneidermanB. (2020). Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human-Computer Interaction, 36(6), 495–504. https://doi.org/10.1080/10447318.2020.1741118
45.
ShortliffeE. H.SepúlvedaM. J. (2018). Clinical decision support in the era of artificial intelligence. Jsls: Journal of the Society of Laparoendoscopic Surgeons, 320(21), 2199–2200. https://doi.org/10.1001/jama.2018.17163
46.
SloaneE. B.SilvaR. (2020). Artificial intelligence in medical devices and clinical decision support systems. In Clinical Engineering Handbook (pp. 556–568). Elsevier.
47.
SmithM. J.CarayonP. (1995). New technology, automation, and work organization: stress problems and improved technology implementation strategies. International Journal of Human Factors in Manufacturing, 5(1), 99–116. https://doi.org/10.1002/hfm.4530050107
48.
SmithM. J.Carayon-SainfortP. (1989). A balance theory of job design for stress reduction. International Journal of Industrial Ergonomics, 4(1), 67–79. https://doi.org/10.1016/0169-8141(89)90051-6
SujanMFurnissDGrundyKGrundyHNelsonDElliottMWhiteSHabliIReynoldsN (2019). Human factors challenges for the safe use of artificial intelligence in patient care. BMJ Health & Care Informatics, 26(1), 1. http://dx.doi.org/10.1136/bmjhci-2019-100081
51.
SujanM.FurnissD.HawkinsR. D.HabliI. (2020). Human factors of using artificial intelligence in healthcare: Challenges that stretch across industries. Paper presented at the Safety-Critical Systems Symposium.
52.
TristE. (1981). The evolution of socio-technical systems: A conceptual framework and an action research program. In Van de VenA.JoyceW. (Eds.), Perspectives on Organizational Design and Behavior (pp. 1–67). Wiley.
53.
TristE. L.BamforthK. W. (1951). Some social and psychological consequences of the longwall method of coal-getting. Human Relations, 4(1), 3–38. https://doi.org/10.1177/001872675100400101
54.
VaseyB.CliftonD. A.CollinsG. S.DennistonA. K.FaesL.GeertsB. F.TheD.-A. I. S. G. (2021). DECIDE-AI: new reporting guidelines to bridge the development-to-implementation gap in clinical artificial intelligence. Nature Medicine, 27(1), 186–187. https://doi.org/10.1038/s41591-021-01229-5
55.
WardenT.CarayonP.RothE. M.ChenJ.ClanceyW. J.HoffmanR.SteinbergM. L. (2019). The national academies board on human system integration (BOHSI) panel: Explainable AI, system transparency, and human machine teaming. and Ergonomics Society Annual Meeting. In the Proceedings of the Human Factors and Ergonomics Society Annual Meeting.
56.
WellsP. S.AndersonD. R.RodgerM.StiellI.DreyerJ. F.BarnesD.ForgieM.KovacsG.WardJ.KovacsM. J. (2001). Excluding pulmonary embolism at the bedside without diagnostic imaging: management of patients with suspected pulmonary embolism presenting to the emergency department by using a simple clinical model and d-dimer. Annals of Internal Medicine, 135(2), 98–107. https://doi.org/10.7326/0003-4819-135-2-200107170-00010
