Sage Journals: Discover world-class research

Abstract

Background. Artificial intelligence (AI) intends to support clinicians’ patient diagnosis decisions by processing and identifying insights from multimedia patient information. Objective. We explored clinicians’ current decision-making patterns using multimedia patient information (MPI) provided by AI algorithms and identified areas where AI can support clinicians in diagnostic decision-making. Design. We recruited 87 advanced practice nursing (APN) students who had experience making diagnostic decisions using AI algorithms under various care contexts, including telehealth and other healthcare modalities. The participants described their diagnostic decision-making experiences using videos, images, and audio-based MPI. Results. Clinicians processed multimedia patient information differentially such that their focus, selection, and utilization of MPI influence diagnosis and satisfaction levels. Conclusions and implications. To streamline collaboration between AI and clinicians across healthcare contexts, AI should understand clinicians’ patterns of MPI processing under various care environments and provide them with interpretable analytic results for them. Furthermore, clinicians must be trained with the interface and contents of AI technology and analytic assistance.

Keywords

artificial intelligence human-clinician teaming multimedia patient information patient diagnosis clinician satisfaction

Introduction

The growing popularity of machine learning algorithms or artificial intelligence (AI) in healthcare has revolutionized clinicians’ decision-making practices. Originally, clinical decision-making (CDM) was defined as a continuous and evolving process of collecting information and choosing between two or more options to formulate clinical diagnosis, interventions, interactions, and evaluations within a context.¹ With the recent advancement of AI in various industrial sectors, it has played a significant role in individuals’ decision-making by providing extracted insights from big data analytics.² In that sense, AI-assisted CDM in healthcare can be defined as the process of choosing among options by interacting with AI technology and its analytic suggestions. However, prior studies have documented mixed effects of AI assistance on clinicians’ decision-making. For example, while AI is praised for its high sensitivity and specificity in object-identifying tasks, instantaneity of reporting, and consistency in producing results,³ its hidden algorithmic biases have created ethical issues in end-of-life healthcare, where invasive treatments are not optimal for some individuals.⁴ Given that the expected performance of AI depends on the assumption that patient conditions are easily identifiable and that the data are unambiguous,⁵ complex health conditions are left to clinicians’ in-depth scrutiny and decision-making capability. As such, the role of AI in healthcare has been limited to informing or suggesting rather than direct decision-making.⁶ Under these circumstances, clinicians should be able to understand, interpret, and make decisions based on the patient information/summary provided by AI technology⁷ while AI needs to understand how clinicians process idiosyncratic patient data in various formats.

Such AI’s support in CDM may include multimedia-based patient information that often needs AI’s active processing capabilities. Referring to “the presentation of material using both words and pictures (p. 27),⁸” multimedia data is generated in the form of text, audio, videos, still images, and animation.⁹ This multimedia information can further be categorized by its delivery type (e.g., computer screen), presentation mode (e.g., images or videos), and sensory modalities (e.g., visual or auditory),¹⁰ which have been widely studied in contexts such as organization, education, and gaming.^8,11 Patient data are characterized by medical, social, genetic, and environmental information in various multimedia formats, all of which have helped clinicians formulate medical diagnoses and tailor their regimen of patient care.^12,13 Thus, AI in healthcare is anticipated to provide analytic assistance that includes summarized or multimedia patient information (MPI) such as image recognition interpretations, calculated medical risks for readmission and mortality, indications for automated prescribing medications, and patient triage index.⁵

More specifically, in a telehealth context, MPI can be the main communication conduit between clinicians and AI.¹⁴ Defined as healthcare consultations and patient diagnoses online or via telehealth platforms, telehealth consultations have recently surged and will remain a popular method for healthcare delivery.¹⁵ Given that telehealth services are characterized by the use of telecommunication modalities via audio (i.e., telephone), text, and video formats,¹⁴ virtual patient consultations under telehealth modalities require relevant stakeholders’ ability to process MPI on virtual care platforms. Under these circumstances, AI can assist synchronous and asynchronous modalities by managing high-volume demands and monitoring the availability of specialists on telehealth platforms¹⁶ and by assisting clinicians in information processing of MPI formats (e.g., images, text, audio, and video), respectively.^17,18 Figure 1 depicts two telehealth modalities and the AI’s expected assistance for clinicians’ decision-making. Although clinicians’ ability to perform AI-assisted decision-making with various MPIs is largely anticipated, little attention has been paid to whether and how AI can support clinicians’ MPI processing.

Figure 1.

The expected roles of artificial intelligence across telehealth modalities (adapted from Ref. 5).

To address this gap, this study explores how clinicians process MPI in traditional and telehealth contexts to identify the areas where AI can assist clinicians’ efficient CDM. The literature on CDM documents that clinicians’ decision-making has been influenced by MPI formats, stakeholders’ information literacy, and technology in use.^19-21 A recent study on AI-assisted decision-making focused on individual characteristics of clinicians (e.g., propensity to trust and personal innovativeness in IT) as key factors in their use of an AI-based decision support system.^22-25 Little is known about how AI and clinicians can collaborate and make shared decisions based on individual patients’ MPIs.⁵ Taken together, throughout this study, we explore three research questions to inform whether and how AI can facilitate shared decision-making with clinicians in healthcare under different care modalities.

RQ1. What type of support do clinicians need to manage MPI in traditional care contexts?

RQ2. How do clinicians process video MPI under a synchronous telehealth context?

RQ3: How do clinicians respond to AI images and audio MPI under an asynchronous telehealth context?

Method

Participants

Purposive sampling was used to recruit family nurse practitioner (FNP) students who were licensed registered nurses and were actively enrolled in a graduate nursing program at a large public university. We utilized purposive sampling due to limited access the users who are clinicians and AI users in healthcare contexts.²⁶ Our participants had experienced both AI-assisted decision-making and telehealth virtual consultations while currently performing healthcare tasks in family medical care in the United States. Since the school of nursing provided access to a simulation lab, clinicians who completed patient diagnosis tasks in the simulation lab were invited to participate in the online simulation surveys. Participants were between 18 and 65 years old, 56.3% were White, and 84.4% were female. In terms of education, 40.6% had a graduate degree and all had obtained their bachelor’s degrees before joining the graduate program. An estimated 87.5% of participants were working full-time or part-time as advanced practice nursing (APN) in hospitals and clinics as part of their preceptorship.

Procedures and materials

Our surveys captured frontline clinicians’ perceptions of patient diagnosis experience in traditional and telehealth contexts with or without AI assistance. Two cohorts (A and B) of FNP students featured different telehealth experiences. As a baseline, we captured cohort A’s behaviors, who experienced traditional, non-telehealth modality, for live patient diagnosis during 2019–2020 (n=30). For telehealth modalities with AI assistance, we surveyed cohort B, who diagnosed patients over a live telehealth context (via Zoom) and used AI technology with multimedia information (i.e., text, images, and audio) from 2020 to 2021 (n=57). To gauge clinicians’ MPI processing patterns under various diagnosis environments, each participant was given three diagnosis simulations and was asked to describe their most and least favorable characteristics of each simulation via survey questions on a 7-point Likert scale (ranging from “least satisfied” to “most satisfied”) as well as textual comments.

Case Study

Along with our multiyear data collection, we adopted a case-study method to explore clinicians’ MPI managing behavior for patient diagnosis in traditional and telehealth contexts without multimedia information provided by AI technology. By definition, a case study is “an intensive study of a single unit for the purpose of understanding a larger class of similar class (p. 342),”²⁷ whereas a unit is a spatially bounded phenomenon that can be observed at a single point in time or over a delimited period of time. A case study is known to be useful for forming descriptive inferences that ask “what” and “how” questions and utilize more comparative analysis than any causal assertion. Furthermore, it can be categorized by its spatial and temporal variation, such that a study unit can be physically fixed or variable, while the phenomenon of the unit can be changeable over time. According to Gerring’s research design typology, there are three types of case studies: case study I examines variation in a single unit over time, while case studies II and III break down a primary unit into subunits, which occur simultaneously and over time, respectively (III) (see Ref. 27 for an in-depth typology). Accordingly, this study conducted a case study II, such that while the primary unit of study (a simulation lab) is spatially fixed with temporal variation from 2020 to 2021, its subunits (FNP program cohorts) vary along with two cohorts of FNP students over time. As indicated in Figure 2, our study unit is a cross-sectional case study II that captures related but distinct questions on the two cohorts of FNP students from 2020 to 2021 (before and after telehealth simulations). In other words, we analyzed data from the FNP cohort of A (experiencing AI assistance with no live telehealth experience) for research question 1, whereas the cohort B data were used to answer our second research question (with Zoom-based telehealth experience).

Figure 2.

Conceptual flow of our case study.

Data Analysis

To conduct our qualitative case study, we analyzed our data using textual analysis and group mean comparison using a t-test and frequency analysis.

First, to answer RQ1, textual narratives of cohorts A and B were reviewed and categorized by an FNP faculty member. Each clinician’s textual narratives were then grouped into two categories—task-focused versus representation-focused or media-focused, following the suggestions from the seminal multimedia studies.^8,28 According to Lim and Benbasat (2000),⁸ task-focused people respond to potential problems by completing tasks, whereas media-focused people attend to the meaning of symbolic language from graphics, voice inflections, and body gestures (verbal versus non-verbal messages). Applying this to our research context, FNP students’ textual narratives can be categorized as task-focused when their comments include predetermined task descriptions to solve clinical tasks (e.g., taking blood), whereas media-focused comments indicate visual presentations of the virtual patients (e.g., body movements). A content analysis was conducted to analyze non-quantitative data and to derive insights by counting keywords, phrases, and other linguistic sets, or through the use of established coding algorithms.^29-31 To this end, we used Voyant, an open-source, web-based text reading and analysis tool, to identify prevalent keywords from simple sentences or phrases. This tool has been used across different research contexts to identify a text corpus or text in a document.³²

Next, an independent samples t-test was conducted in SPSS 27 (1000 bootstrapping samples) to examine the difference between the task-focused and media-focused groups. Factor variables included personal traits (i.e., individual innovativeness, social media habit, computer literacy, and multimedia technology use) and diagnostic capability. These survey instruments were adopted from well-established information systems literature and adapted to the context of patient simulation technology use using a 7-point Likert scale ranging from “strongly disagree” (1) to “strongly agree” (7). First, to measure diagnostic capability, we adapted and redefined an existing measure in a telehealth context as the perceived ability of processing MPI to understand and evaluate patients’ health conditions and problems. In this study, we contextualized and operationalized survey items of online diagnostic capability from Ref. 33. The four items of this construct are as follows: “it allows me to carefully evaluate the health condition of the patient,” “it allows me to thoroughly assess the health condition of the patient,” “it allows me to accurately evaluate the patient’s health condition,” and “it allows me to critically think during the simulation experience.” Next, individual characteristics such as individual innovativeness,³⁴ social media habits,³⁵ and computer literacy³⁶ were measured.

To answer RQ2, we compared cohorts A and B’s textual narratives based on live patient contexts (in-person versus Zoom-based telehealth contexts) and the type of experience (positive vs negative), all of which resulted in four blocks of keywords and phrases for separate content analyses (Table 3). Lastly, to explore RQ3, clinicians’ frequency of use and average satisfaction level with MPI formats were measured using a side-by-side bar chart in SPSS 27.

In summary, Figure 3 presents the method roadmap of this study.

Figure 3.

A method roadmap of this paper.

Results

RQ1: Understanding clinicians’ MPI processing behavior in a non-telehealth context

Within cohort A, the task-focused group (group 1, n=18) paid close attention to the given task description when processing MPI, while the media-focused group (group 2, n=12) concentrated more on media characteristics such as functionality and reality. Groups 1 and 2 demonstrated clear distinctive characteristics in their focus when interpreting multimedia information from three patient diagnosis simulations (Table 1). Namely, group 1 attended to symptoms to complete the given clinical task (e.g., “brachial artery was easy to palpate on this mannequin”) while group 2 paid more attention to the external conditions (e.g., “accurate skin color, skin changes, eye movements”).

Table 1.

Sample narratives of clinicians’ processing of MPIs under three diagnosis scenarios from cohort A.

Patient type		Task-focus group 1 (n=18)	Media-focus group 2 (n=12)
Standardized (live) patient	Positive review	“They simulate office or urgent care visits making it a common complaint seen regularly”	“Convenience, realistic”
Standardized (live) patient	Negative review	“Hard to simulate actual complaints”	“Someone watching us”
Simulated patient	Positive review	“Brachial artery was easy to palpate on this mannequin.” “More comfortable learning”	“Accurate skin color, skin changes, eye movements” “Very realistic”
Simulated patient	Negative review	“Unable to take blood pressure on these mannequins.” “Not as educational as live patient. Additional questions can’t be answered”	“Lack of face expression(s) and bodily movement” “No expressions, no immediate feedback, no gauging reaction”
AI-enabled diagnosis technology	Positive review	“History is concise and detailed” “Very thorough and prompts you to follow plan of care starting with interviewing”	“Convenience” “Not threatening”
AI-enabled diagnosis technology	Negative review	“Inability of asking additional questions” “Missing the hands-on portion”	“Fake and complex” “Unrealistic”

Next, we compared clinicians’ performance prospects and technology traits between the two groups using a t-test (Table 2). Those who focused more on task requirements when processing MPI were more familiar with video-based multimedia technology (p = 0.021) and positively perceived their performance of patient diagnosis in non-telehealth simulations (p = 0.009). These findings indicate that clinicians need to efficiently process patient information to fulfill their given care tasks, yet they may be distracted by the characteristics of the medium in which AI can support them by complementing their MPI processing.

Table 2.

Group mean difference of user characteristics and patient diagnosis performance between two groups.

			Focal sample		Bootstrapped samples (n=1000)
Variables	Mean (SD) of task-focus group (n=18)	Mean (SD) of media-focus group (n=12)	t	p	P	95% CI
Panel (a): Technological traits
Individual innovativeness	2.92 (0.75)	2.92 (0.95)	0.00	1.00	n/a	−0.6, 0.59
Social media habit	5.29 (1.37)	4.46 (1.54)	1.55	0.131	0.13	−0.23,1.84
Computer literacy	2.17 (0.79)	2.52 (0.997)	−1.09	0.29	s/a	−0.99,0.3
Multimedia technology use (1: video, 2: audio)	2.44 (0.51)	1.92 (0.67)	2.45	0.021**	0.027**	0.08,0.96
Panel (b): Perceived diagnosis performance
Diagnostic capability	5.94 (0.69)	5 (1.15)	2.82	0.009***	0.024**	0.13,1.67
p<0.05, p<0.01; *** p<0.001

RQ2: Clinicians’ video MPI use in a synchronous telehealth modality.

Next, we explored clinicians’ perceptions of various MPI formats using two telehealth modalities. According to the Centers for Disease Control and Prevention (CDC), telehealth modalities enable clinicians to synchronously and asynchronously connect patients. While a synchronous modality includes live or real-time interactions via telephone or audio-video technology, an asynchronous modality is characterized by using previously gathered messages, images, audio, or video data.³⁷ In this context, we captured synchronous telehealth encounters via Zoom conference sessions and compared clinicians’ live patient experiences via traditional clinical visits and Zoom visits. Two cohorts of APN students were asked to answer open-ended questions about their experience with a live patient encounter, either in-person (e.g., physical examinations, text-based patient charts) or via telehealth technology (e.g., video-based patient information, text-based patient chart), with the main difference being whether clinicians use a patient’s video MPI without AI assistance.

To answer our second question, we derived the main keywords based on the narratives given by cohort A (n=32) and B (n = 25). Table 3 highlighted frequent keywords reflecting clinicians’ processing of video MPIs in a synchronous telehealth modality, in comparison with traditional, in-person patient diagnoses. Cohort A, who had no telehealth experience, described their live patient simulations as “critical-thinking” and “interaction” with a “real” patient, which made it more “realistic,” while “being watched” and created anxiety about diagnosing patients under “complex, timed” situations. On the other hand, cohort B, whose telehealth experience, described their experiences with video-based patient diagnosis as “convenient” and “fast,” while the “inability to perform physical assessment” was listed as a major drawback in a synchronous telehealth modality.

1) For the use of live patient information among cohort A, an FNP faculty member stated:

“FNP students were efficient with history-taking, using their communication skills to build rapport with the patients. With their professionalism and empathy, they performed a focused physical exam and provided treatment and follow-up instructions. Of course, some students were nervous since they were being watched behind the one-way mirror. Still, as the live patient simulations progressed, they showed improvement from the first.”

2) Regarding video use under the Zoom-based telehealth context among cohort B, that same faculty member stated:

“FNP students did more history-taking when using video MPI because they had to ask more questions to get information compared to being able to do the physical exam in person. Thus, they also provided more instruction to the patients regarding follow-up care and signs and symptoms to assess in an emergency. They presented cultural sensitivity and practiced communication skills. However, due to contactless encounters with a patient under the virtual space, physical assessment is largely missing, so they needed to ask patients to help with the assessment. Majority of FNP students were influenced by video quality and internet connection when obtaining accurate data from the live patient, and they needed to be able to visualize patients’ facial and vocal expressions, which prolonged the length of the virtual visit in a Zoom breakout room.”

Table 3.

Keywords reflecting clinicians’ processing of MPIs in a synchronous modality.

Cohort A (n=32)Traditional patient diagnosis		Cohort B (n=25)Telehealth patient diagnosis via Zoom
Positive keywords	Negative keywords	Positive keywords	Negative keywords
Critical thinking	Anxiety	Fast convenience	Physical
Live patient	Watched	High	Patient
Feedback	Complex	Fast	Examination
Realistic	Time	Patient	Unable
Knowledge	Hard	Telehealth	Complex
Fast	Feedback	Informative	Assess
Interaction		Convenience
Patient		Real
Experience		Convenient
Live		Comfortable
Real		Feels
Convenience		Face
		Patients
		Time

Their processing of visual or video-based MPI was observed and evaluated by the FNP faculty who taught each cohort over a multiyear period.

Drawing from both the students’ keywords and the faculty assessments, an emerging discussion arising from a synchronous telehealth modality was that clinicians needed to collect more information from patients over a computer screen during a live Zoom videoconference, in addition to reading the given textual patient chart. Issues such as video quality and internet connection over Zoom might slow down the clinician’s ability to collect patient assessment data throughout the simulated virtual telehealth consultation. We propose that AI support for processing video MPI can decrease the length of prolonged telehealth visits.

RQ3: Clinicians’ image and audio MPI use in an asynchronous telehealth modality

Lastly, we examined clinicians’ use of image and audio MPI, which were provided by AI algorithms in an asynchronous telehealth modality. To this end, we distributed a follow-up survey to members of cohort B who experienced an AI-assisted patient diagnosis in 2021. Each participant was given two case scenarios that were replicas of cases they had encounter during the semester (e.g., cases of abdominal pain, absent menstrual periods) that required image, audio, or both MPI formats. Fourteen participants responded to our follow-up survey, in which they were asked to choose a specific patient MPI when using AI-assisted patient diagnosis technology (Table 4).

Table 4.

AI-provided MPI use case for cohort B.

	Diagnosis scenario 1 (abdominal pain)	Diagnosis scenario 2 (absent menstrual periods)
Scenario prompt	Please recall your recent case assignment on a patient who presents to your clinic with complaints of acute abdominal pain and abdominal tenderness	Please recall your recent case assignment on a young woman who presents to your clinic with complaints of no menses and she is not peri-menopausal or menopausal and is not pregnant
Necessary MPI	• Audio helps to hear bowel signs or lack of bowel signs	• Textual information is most relevant
	• Picture shows abdominal skin discoloration or structural abnormality	• Audio and image will not be ideal
	• Text can provide background information or other information	• Audio and image will not be ideal

As presented in Figure 4, when clinicians used AI recommended MPI formats in the two patient diagnosis scenarios, they tended to use images more frequently than audio information and their level of satisfaction with images was higher compared to using audio-based patient information. Interestingly, while case II does not require the use of patient images, clinicians still chose and were more satisfied with images. This implies that interpretation ability and accurate selection of various MPI formats at the time of decision making might be key for clinicians’ satisfaction with the asynchronous telehealth modality, in which AI can provide strong support.

Figure 4.

Average use frequency and satisfaction with audio and images.

Discussion

The objective of this study was to explore the potential and target areas of AI assistance in healthcare by observing and comparing clinicians’ current use of MPI data, which often involves clinicians’ information processing capabilities. We found that in a traditional, non-telehealth environment, clinicians display differential focus when processing live patient information, with those who are familiar with multimedia technology tending to focus more on patient care tasks than on complex media characteristics. Under the synchronous telehealth modality, clinicians positively evaluate the convenience of patient diagnosis over a telehealth platform, and yet their behavior is influenced by the characteristics of video MPI, such that video quality and Internet connection prolonged the length of telehealth patients’ virtual visits. Lastly, for the asynchronous telehealth modality, clinicians demonstrated differential frequency of use and satisfaction with image and audio MPI in various diagnosis cases.

More specifically, clinicians’ appropriate use of MPIs can be an enabler or a barrier to AI-assisted decision-making. First, clinicians need AI support when managing MPI under various care contexts. Depending upon their ex ante exposure to multimedia technology in everyday life, clinicians have demonstrated two distinctive focuses when processing various MPI—live patients, human-like mannequins, and AI-suggested multimedia information. Clinicians with media technology experience tend to identify visual cues relevant to task descriptions, whereas the rest are distracted by complex media characteristics. Second, clinicians differentially manage MPIs along with video, audio, and images in telehealth contexts. On the one hand, video-based patient consultation provides fast convenience that often lacks physical contact. Such a deficiency leads to prolonged visits to verbally gather questions on physical assessments. On the other hand, clinicians showed differential patterns in selecting MPI formats in asynchronous telehealth contexts by frequently using images compared to audio MPI, regardless of the recommended MPI formats under care scenarios. This is in line with prior studies that found that clinicians report higher satisfaction after using quality video and audio MPI³⁸ as well as image-based MPI under telehealth modalities.^39,40 The reason might be that clinicians can diagnose patients efficiently with quality MPI which can be easily interpretable and understandable by reducing multimedia-driven ambiguity. Taken together, the processing of complex MPI is a key area in which AI can support clinicians’ CDM, both in traditional and telehealth contexts.

This study highlights the importance of multimedia patient information as a critical component of AI-based diagnosis assistance and as a communication conduit between AI and clinicians. As telehealth services using virtual platforms continues to expand, clinicians must be able to formulate patient diagnoses by processing MPI on telehealth platforms. However, clinicians can become overwhelmed by the increased volume of MPI driven by the use of health mobile devices, which leads to information overload, burnout, or limited information processing time.^41-43 In this circumstance, we propose that more complex forms of MPI, such as audio- and video-based patient information can be processed and presented by AI algorithms such that clinicians can efficiently focus on patient-specific care tasks and rely on AI’s support (i.e., real-time alerts and nudges) when entering clinical data and adhering to clinical guidelines.⁴⁴ However, a paucity of research exists on AI assistance methods, which are critical for clinicians’ timely and efficient CDM.

Our findings provide policy implications for AI developers. Currently, clinicians do not have a standardized decision-making reasoning and preference toward displays of AI⁴⁵; therefore, AI developers require an understanding of clinicians’ MPI processing patterns to provide designs of AI assistance for MPI management under live and asynchronous telehealth modalities. To streamline the communication between clinicians and AI, the AI interface also needs to be clinician-centered and MPI presentations should be easy to understand to facilitate AI-assisted decision-making in healthcare. Second, for clinicians, our results emphasize the importance of incorporating multimedia information management into the clinical curriculum.⁴⁶ Multimedia literacy education can include literacy education consisting of information literacy (understanding the presentation mode), media tools (learning various types of multimedia in advanced nursing education), and communication skills.⁴⁷ This multimedia curriculum can help reduce duplicity of meaning in clinical tasks⁸ and reduce ambiguity in clinical communication.⁴⁸ Namely, such multimedia curriculum helps clinicians to maintain abilities to understand the optimal environments for performing telehealth patient care and develop technical skills to check internet infrastructure, video quality, and telehealth etiquette for patient satisfaction under telehealth contexts.⁴⁹

Lastly, as with all research, this study is not without limitations. First, we employed purposive sampling techniques to recruit FNP students who were working full-time (n = 17; 53.1%) or part-time (N = 11; 34.4%). In the graduate nursing program, each participant had previously experienced a virtual patient consultation in various formats. While our sample represents the target population of this study, national-level sampling would be beneficial for generalizing our results. Second, case studies are known to have limitations in representation, small sample size, scope and breadth of conditions, and descriptive inference in nature.²⁷ Future research can incorporate cross-sectional or panel data collection as well as predictive inference methods to examine AI-assisted decision-making. Lastly, as our results were derived from the simple use of text, images, and audio formats of MPI, they may not be generalizable to other types of multimedia formats such as video or 3D images. Further investigation is warranted to explore additional formats of MPI in telehealth contexts.

Conclusions

In keeping with recent interest in AI advancement and telehealth modalities for diagnosing patients, this study sheds light on the areas where AI can support clinicians’ clinical decision-making by exploring the current and potential uses of AI to assist clinicians in MPI processing and patient diagnosis behaviors. This research builds and expands on the existing literature regarding AI and healthcare by highlighting the importance of MPI in CDM as potential barriers for AI-clinician collaboration. Our findings further indicate that AI can assist clinicians’ decision-making by complementing various MPI processing and by making it easily interpretable to clinicians in telehealth contexts.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Hyeyoung Hah

References

Smith

Higgs

Ellis

. Factors influencing clinical decision making. Clin Reasoning Health Professions 2008; 3: 89–100.

De Vreede

Raghavan

De Vreede

G-J

(eds) Design Foundations for AI Assisted Decision Making: A Self Determination Theory Approach. In: Proceedings of the 54th Hawaii International Conference on System Sciences. 2021.

Shen

Zhang

CJP

Jiang

, et al. Artificial Intelligence Versus Clinicians in Disease Diagnosis: Systematic Review. JMIR Med Inform 2019; 7(3): e10010.

Lysaght

Lim

Xafis

, et al. AI-Assisted Decision-making in Healthcare: The Application of an Ethics Framework for Big Data in Health and Research. Asian Bioeth Rev 2019; 11(3): 299–314.

Krittanawong

. The rise of artificial intelligence and the uncertain future for physicians. Eur J Intern Med 2018; 48: e13–e4.

ANSI . Definitions/Characteristics of Artificial Intelligence in Health Care. Americal National Standards Institute (ANSI), 2020.

Banning

. A review of clinical decision making: models and current research. J Clin Nurs 2008; 17(2): 187–195.

Lim

Benbasat

. The effect of multimedia on perceived equivocality and perceived usefulness of information systems. MIS Q 2000; 24(3): 449–471.

Puentes

Bali

Wickramasinghe

, et al. Telemedicine trends and challenges: a technology management perspective. Ijbet 2007; 1(1): 59–72.

10.

Mayer

. Multimedia learning: psychology of learning and motivation. Elsevier, 2002, pp. 85–139.

11.

Park

Kim

D-g

Cho

, et al. Adoption of multimedia technology for learning and gender difference. Comput Hum Behav 2019; 92: 288–296.

12.

Topol

. Deep medicine: how artificial intelligence can make healthcare human again. Hachette UK, 2019.

13.

Khullar

. Welcoming medicine to the machine. Health Aff 2019; 38(9): 1593–1594.

14.

CMS.gov . 2020, https://www.cms.gov/newsroom/fact-sheets/medicare-telemedicine-health-care-provider-fact-sheet. MEDICARE TELEMEDICINE HEALTH CARE PROVIDER FACT SHEET 2020 [Available from:

15.

Telehealth McKinsey . A quarter-trillion-dollar post-COVID-19 reality?, 2020. https://www.mckinsey.com/industries/healthcare-systems-and-services/our-insights/telehealth-a-quarter-trillion-dollar-post-covid-19-reality#.McKinsey & Company2020 [Available from:

16.

Kuziemsky

Maeder

John

, et al. Role of Artificial Intelligence within the Telehealth Domain. Yearb Med Inform 2019; 28(1): 35–40.

17.

Long

Chen

, et al. Artificial intelligence manages congenital cataract with individualized prediction and telehealth computing. NPJ Digit Med 2020; 3(1): 112.

18.

Huang

Yang

Fong

, et al. Artificial intelligence in cancer diagnosis and prognosis: Opportunities and challenges. Cancer Lett 2020; 471: 61–71.

19.

Housten

Kamath

Bevers

, et al. Does animation improve comprehension of risk information in patients with low health literacy? a randomized trial. Med Decis Making 2019; 40(1): 17–28.

20.

Glaser

Nouri

Fernandez

, et al. Interventions to improve patient comprehension in informed consent for medical and surgical procedures: an updated systematic review. Med Decis Making 2020; 40(2): 119–143.

21.

Shortliffe

. Medical informatics and clinical decision making: the science and the pragmatics. Med Decis Making 1991; 11(4_suppl): S2–S14.

22.

Fan

Liu

Zhu

, et al.Investigating the impacting factors for the healthcare professionals to adopt artificial intelligence-based medical diagnosis support system (AIMDSS). Ann Operations Res 2020; 294(1): 567–592.

23.

Ozanne

Noseworthy

Cameron

, et al. Shared Decision Making in the Era of Telehealth: Implications for Practice and Research. MDM Pol Pract 2020; 5(2).

24.

Sasangohar

Bradshaw

Carlson

, et al. Adapting an outpatient psychiatric clinic to telehealth during the COVID-19 pandemic: a practice perspective. J Med Internet Res 2020; 22(10): e22523.

25.

Hsu

Liss

Frosch

, et al. Exploring provider reactions to decision aid distribution and shared decision making: lessons from two specialties. Med Decis Making 2016; 37(1): 113–126.

26.

Teddlie

. Mixed methods sampling. J Mixed Methods Res 2007; 1(1): 77–100.

27.

Gerring

. What is a case study and what is it good for? Am Polit Sci Rev 2004; 98(2): 341–354.

28.

Dennis

Fuller

Valacich

. Media, tasks, and communication processes: a theory of media synchronicity. MIS Quarterly 2008; 32(3): 575–600.

29.

Lacity

Janson

. Understanding Qualitative Data: A Framework of Text Analysis Methods. J Manag Inf Syst 1994; 11(2): 137–155.

30.

Kuckartz

. Qualitative text analysis: a guide to methods, practice and using software. Sage, 2014.

31.

Kripendorff

. Content analysis: an introduction to its methodology. Thousand Oaks, CA: Sage, 2004.

32.

Maramba

Davey

Elliott

, et al. Web-based textual analysis of free-text patient experience comments from a survey in primary care. JMIR Med Inform 2015; 3(2): e20–e.

33.

Serrano

Karahanna

. The Compensatory Interaction Between User Capabilities and Technology Capabilities in Influencing Task Performance: An Empirical Assessment in Telemedicine Consultations. Misq 2016; 40(3): 597–621.

34.

Agarwal

Prasad

. A conceptual and operational definition of personal innovativeness in the domain of information technology. Inf Systems Research 1998; 9(2): 204–215.

35.

Venkatesh

Thong

. Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology. MIS Quarterly 2012; 36(1): 157–178.

36.

Compeau

Higgins

. Computer self-efficacy: Development of a measure and initial test. MIS Quarterly 1995; 19: 189–211.

37.

CDC . Using Telehealth to Expand Access to Essential Health Services during the COVID-19 Pandemic. In: CfDCa Prevention. CDC; 2020.

38.

Saiyed

Nguyen

Singh

. Physician Perspective and Key Satisfaction Indicators with Rapid Telehealth Adoption During the Coronavirus Disease 2019 Pandemic. Telemed J E Health 2021; 27(11): 1225–1234.

39.

Gomez-Rice

Madrid

Izquierdo

, et al.

Does Clinical Photography Influence Satisfaction With Surgery in Adult Patients Operated on for Spinal Deformity?

Int J Spine Surg 2021; 14(6): 1037–1042.

40.

Whited

Hall

Foy

, et al. Patient and Clinician Satisfaction with a Store-and-Forward Teledermatology Consult System. Telemed J E Health 2004; 10(4): 422–431.

41.

Byström

Järvelin

. Task complexity affects information seeking and use. Inf Process Manag 1995; 31(2): 191–213.

42.

Sinsky

Privitera

. Creating a “Manageable Cockpit” for Clinicians: A Shared Responsibility. JAMA Intern Med 2018; 178(6): 741–742.

43.

Liu

H-C

Chuang

H-H

. An examination of cognitive processing of multimedia information based on viewers' eye movements. Interactive Learn Environments 2011; 19(5): 503–517.

44.

Siwicki

. How AI and analytics can automate CMS coding for physicians. Healthcare IT News, 2021.

45.

Barda

Horvat

Hochheiser

. A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare. BMC Med Inform Decis Mak 2020; 20(1): 257.

46.

Duff

Miller

Bruce

. Online virtual simulation and diagnostic reasoning: a scoping review. Clin Simulation Nurs 2016; 12(9): 377–384.

47.

Mashiyane

Bangani

Van Deventer

. The awareness and application of multimedia tools for information literacy instruction at an African university. El 2020; 38(4): 711–724.

48.

George