Sage Journals: Discover world-class research

Abstract

Background

Artificial intelligence (AI) and large language models (LLMs), are potential tools for enhancing healthcare delivery and clinical research. Presently, there is a scarcity of research regarding the viewpoints of medical students toward LLMs. This study aims to explore the perceptions of LLMs and their applications among this demographic.

Methods

A cross-sectional study was done using an online survey. It was designed using Google Forms and circulated from July 2023 to August 2023. The target population included medical students from Ecuador, Ethiopia, India, Mauritius, Pakistan, United Arab Emirates (UAE), and the United States (USA).

Results

A total of 1180 responses were collected from 10 medical colleges across 7 countries. The UAE had the largest number of responses (31.7%), followed by Ecuador (17.6%), and Mauritius (13.7%). And 77.4% respondents were already aware about LLMs before attempting the survey, most popularly, ChatGPT. Participants were most familiar with the specific use of LLMs in research (46%), and the lowest familiarity was seen in the use of LLMs in healthcare (37%). More than half of the participants (52%) support the use of LLMs healthcare, yet a significant number of them remained neutral (37%) or disagreed (31%) on LLMs being safe in this context.

Conclusion

While most of the medical students are aware of LLMs and their applications, the multinational survey demonstrated hesitancy among medical students in fully adopting LLMs into healthcare practice and clinical research. This study prompts further exploration of medical students’ attitudes and acceptance concerning the integration of LLMs for research methodologies, ethical considerations, and healthcare practice.

Keywords

medical students medical education large language models healthcare artificial intelligence

Introduction

The rapid advancement of artificial intelligence (AI) and natural language processing (NLP) technologies has opened up new possibilities in various fields, including the healthcare industry.^1,2 Large language models (LLMs), powered by deep learning algorithms and trained on massive amounts of data, have gained significant attention due to their potential applications in generating human-like text and facilitating language-based tasks.³, Specific implementations of LLMs, such as OpenAI's ChatGPT and Google's Gemini, have demonstrated remarkable capabilities in understanding and generating human-like text, making them potential tools for enhancing healthcare delivery, clinical research, and medical education.^4,5

In the context of medical education and research, LLMs have the potential to enhance learning experiences, facilitate information retrieval, and contribute to knowledge generation.^6,7 A 2022 study explored the knowledge and perceptions of medical students and interns regarding the increasing role of AI in medicine. The study found that over 65% believed that AI would impact their specialty choice, and 78% believed that every medical student should receive training on AI competencies.⁸

Recent studies, including those Alkhaaldi et al⁹ and Tangadulrat et al,¹⁰ have explored medical students’ perceptions of AI, including LLMs. However, focused investigations into the global perspectives of LLM use among medical students necessitate further exploration.

The primary aim of this cross-sectional study is to explore medical students’ perceptions of LLMs and assess their knowledge, opinions, and readiness for the integration of LLMs in healthcare, research, and medical education. By gathering data through a structured survey, we seek to investigate the awareness, understanding, and potential concerns related to the usage of LLMs among medical students.

Methods

Study Design and Timeline

The study is a cross-sectional survey using a self-designed questionnaire. Data was collected over 3 weeks from July 17 to August 7, 2023. The target population for this study included medical students from Ecuador, Ethiopia, India, Mauritius, Pakistan, United Arab Emirates (UAE), and the United States (USA). The reporting of this study conforms to the STROBE statement of cross-sectional studies (see Supplemental File 1).

Questionnaire and Validation

The survey covered 4 main areas. The first section assessed the student's awareness and utility of LLMs. The subsequent sections explored the specific application of LLMs in healthcare, research, and medical education, with regard to the areas of perception, knowledge, and willingness for LLMs in these fields. Finally, the survey concluded with questions related to the student's demographics. In total, the survey contained 35 questions, in which 28 were related to the study itself and 7 about demographic data.

The survey first underwent an internal validation stage by the core team. In this validation, the team members and mentors reviewed each question for content and construct validity. This was to ensure that the questions are seeking the information they intend to as well as they are not leading the respondents one way or another. Additionally, some questions were merged or deleted to avoid the duplication of content. It was then externally validated by four experts in the field of healthcare with adequate background in LLMs who suggested adding, removing, or modifying certain questions for further improvement of the quality and relevance of the survey tool.

Survey Dissemination Strategy and Data Extraction

The core team consisting of 7 researchers based in the UAE, USA, and Pakistan interacted via a dedicated LLM Research WhatsApp group in weekly meetings via video conferencing. An email and WhatsApp template was made to increase efficiency and reduce communication errors when distributing the survey link. Each core team member sent the survey to their networks using in-person messaging tools, mainly WhatsApp messages, Instagram direct messages, Snapchat messages, and emails. The core team then recruited the next group of collaborators who represented other countries (Ecuador, Ethiopia, India, Mauritius, Pakistan, and USA) and were in-charge of survey administration for their respective countries. The country leads were contacted via direct messaging, and each lead took up the responsibility of disseminating the survey in their local networks. Utilization of the “hub and spoke model” proved to be effective in gathering these responses, with the “hub” being the core team and the “spokes” being the collaborators.^11,12 Additionally, we employed a snowball sampling technique, where initial participants were asked to forward the survey to few other potential respondents, leading to a cascading recruitment process. A digital survey banner was designed for promotional messaging. A promotional YouTube video was recorded by the core team members and was shared to garner a greater number of responses and to increase the awareness of this study.¹³ The responses collected were extracted to a Microsoft Excel Sheet for further analysis.

Sample Size

The total number of medical students across 10 medical schools was estimated to be 7916. Assuming that 50% of the population was accessible to the study, with a 5% margin of error, and 95% confidence interval, sample size was determined to be of 367. This number was calculated using the sample size calculator by Raosoft, Inc. (http://www.raosoft.com/samplesize.html).

Ethical Considerations

The study was deemed IRB-exempt from Mayo Clinic Institutional Review Board.

The study adhered to rigorous ethical protocols to ensure participant privacy and data protection. Participant responses were collected using secure Google Forms, which anonymized all submissions to prevent identification.

Results

Demographics

The data extracted from the study included 1180 responses from 10 medical schools across 7 countries illustrated in Figure 1. The highest response was from UAE with 374 responses (31.7%) followed by Ecuador with 208 responses (17.6%) and Mauritius with 162 responses (13.7%). Pakistan had a total of 135 responses (11.4%) and Ethiopia had 116 responses (9.8%). The countries with the least responses were India with 113 responses (9.6%), and USA with 65 responses (5.5%).

Figure 1.

Distribution of Responses According to Country.

Majority of respondents were females (N = 719, 60.9%) followed by males (N = 459, 38.9%). There were also two respondents (0.2%) who selected “Other” as their gender.

The most common age group of respondents was 23 years (N = 192, 16.3%). It was also noted that 6 respondents selected “≥30 years” for their age. The mean age was 21.7. More than half the respondents were from the preclinical years (N = 651, 55.2%) and the remaining were in the clinical years (N = 529, 44.8%) of medical school, therefore the most common year group were third year medical students (N = 235, 19.9%) and second year medical students with 228 responses (19.3%). Analysis of the clinical specialities showed that medical students with an interest in pursuing surgery acquired the most responses at 162 responses (13.7%) followed by internal medicine at 154 responses (13.1%). There were minimal responses indicating interest in pulmonology and immunology, (N = 3 responses 0.25%) and only 2 individuals expressing interest in nuclear medicine (0.17%).

Most respondents (N = 953, 80.8%) had no research publications authored at the time of the survey, however 227 respondents (19.2%) have published at least one research paper at the time of this survey. From this pool of respondents, only 2 responses had more than 20 papers published as shown in Table 1.

Table 1.

Participant demographics.

Demographics	Frequency (N=1180)	Percentage
Gender
Male	459	38.9
Female	719	60.9
Other	2	0.2
Age
≤17	16	1.4
18-20	373	31.6
21-23	502	42.5
24-26	259	21.9
27-29	24	2.0
≥30	6	0.5
Mean age range	21.7
Country
Ecuador	205	17.4
Ethiopia	115	9.7
India	113	9.6
Mauritius	171	14.5
Pakistan	134	11.4
UAE	377	31.9
USA	65	5.5
Year of study
Year 1	203	17.2
Year 2	228	19.3
Year 3	235	19.9
Year 4	164	13.9
Year 5	222	18.8
Year 6	94	8.0
Year 7	34	2.9
Phase of study
Preclinical	651	55.2
Clinical	529	44.8
Articles published
0	953	80.8
1-3	178	15.1
4-6	35	3.0
7-9	9	0.8
10-20	3	0.3
>20	2	0.2
Mean number of publications	0.43
Medical school of respondent
Drexel University College of Medicine, Philadelphia, Pennsylvania, USA	65	5.5
Grant Medical College and Sir JJ Group of Hospitals, Mumbai, India	113	9.6
Universidad de Especialidades Espiritu santo, Samborondon, Ecuador	205	9.7
Shifa College of Medicine (SCM) Islamabad, Pakistan	134	15.6
Sir Seewoosagur Ramgoolam Medical College, Belle Rive, Mauritius	171	3.2
Hawassa University School of Medicine & Health Science, Hawassa, Sidama, Ethiopia	115	14.5
Mohammed Bin Rashid University of Medicine and Health Sciences (MBRU), Dubai, UAE	184	9.7
RAK Medical And Health Sciences University (RAKMHSU), RAK, UAE	38	17.4
University of Sharjah (UoS), Sharjah, UAE	114	11.4
Gulf Medical University (GMU), Ajman, UAE	41	14.5

Familiarity Toward LLMs

The study reports 898 respondents (76.1%) being already familiar with LLMs before taking the survey and a total of 1656 responses were obtained from this population. The most popular LLMs were ChatGPT (N = 866, 52.3%) followed by Microsoft Copilot (N = 292, 17.6%) and Gemini (N = 184, 11.1%). The least popular LLMs were Gorilla (N = 130, 7.9%) followed by Med-PaLM (N = 110, 6.6%) and Falcon LLM (N = 74, 4.5%).

Only 646 (72%) participants who are already familiar with LLMs have used it before. A total of 927 responses were obtained regarding the use of LLMs. The most used platform was ChatGPT (N = 612, 66%), followed by Microsoft Copilot (N = 134, 14.5%) and Gemini (N = 93, 10%). The least used platforms were Med-PaLM (N = 42, 4.5%), followed by Gorilla (N = 25, 2.7%) and Falcon LLM (N = 21, 2.3%)

Among the respondents, 692 of them were introduced to LLMs through social media (58.6%) followed by word-of-mouth (N = 547, 46.4%). Books/magazines and online courses seemed to have the least exposure to LLMs with 108 responses (9.2%) and 172 responses (14.6%), respectively.

On the 5-point Likert scale, the majority (51%) expressed either very or somewhat familiar with the various ways people can utilize LLMs. People were most familiar with the specific use of LLMs in research (N = 184, 46%), followed by 41% (N = 165) in medical education and the lowest familiarity was seen in the use of LLMs in healthcare (N = 107, 37%), tabulated in Table 2.

Table 2.

Level of awareness and experience with LLMs.

Awareness and use of LLMs	Frequency (N=1180)	Percentage
Have you heard about LLMs?
Yes	898	76.1
No	282	23.9
Which of the following LLMs have you heard about?
ChatGPT	866	52.3
Microsoft Copilot	292	17.6
Gemini	184	11.1
Gorilla	130	7.9
Med-PaLM	110	6.6
Falcon LLM	74	4.5
Have you ever used LLMs?
Yes	646	72
No	252	28
Which of the following LLMs have you used?
ChatGPT	612	66
Gemini	93	10
Microsoft Copilot	134	14.5
Falcon LLM	21	2.3
Gorilla	25	2.7
Med-PaLM	42	4.5

Abbreviation: LLM, large language model.

Ethical challenges of LLMs in research were acknowledged by 49% (N = 581). Lastly, regarding LLMs negatively affecting the reliability of research, 40% (N = 467) remained neutral and 31% (N = 130) agreed with this notion. Figure 2 and Table 3 expand further.

Figure 2.

Perceptions Regarding Barriers Towards the Use of Large Language Models in Research.

Table 3.

Perceptions of LLM applications.

Perceptions Regarding Application of LLMs in	Frequency (N=1180)	Percentage
Healthcare
Access new scientific information	537	45.5
Conversational agents (eg, providing customer service)	381	32.3
Drug discovery and development	269	22.8
Health behavior analysis and intervention	285	24.2
Language translation	501	42.5
Medical diagnosis and decision support	397	33.6
Optimize treatment plans	292	24.7
Patient monitoring and remote care	292	24.7
Standardized EMR	308	26.1
Telemedicine and virtual healthcare	278	23.6
Medical research
Bias detection	336	28.5
Data analysis and interpretation	751	63.6
Hypothesis generation and experimental design	398	33.7
Literature reviews	521	44.2
Natural language processing	332	28.1
Predictive modeling for patient outcomes	354	30
Quality control	332	28.1
Writing assistance and grammatical correction	516	43.7
Medical education
3D anatomy education	620	52.5
Help write personal statements and CV	437	37
Make flashcards and questions for revision	705	59.7
Remote learning	394	33.4
Simulations	464	39.3
Summarize complex topics	661	56
Language translation	259	21.9

Abbreviations: 3D, 3-dimensional; EMR, electronic medical report; LLM, large language model.

Perception Regarding Applications

With regard to medical education, a significant number of participants (N = 712, 60%) concurred that LLMs simplify understanding medical concepts. This was noted among the 654 respondents (55%) who agreed on the benefit of LLMs offering real-time feedback. Also, 617 participants (52%) acknowledged the use of LLMs in aiding exam revision, while 569 respondents (48%) saw themselves enhancing clinical reasoning and diagnostic skills using this modality.

The 3 most commonly perceived applications of LLMs in healthcare were reported to be “access new scientific information” (45.5%), “language translation” (32%), and “medical diagnosis and decision support” (22.8%) whereas the top 3 barriers are “lack of emotional intelligence” (51.2%), “lack of human interaction” (49%), and “ethical transparency” (40.7%) illustrated in Figure 3.

Figure 3.

Perceptions regarding Barriers Toward the use of Large Language Models in Healthcare.

The 3 most common applications of LLMs in research were reported to be “data analysis” (63.6%), “literature reviews” (44.2%), and “writing assistance and grammatical correction” (43.7%), whereas the top 3 barriers to LLMs in research are “lack of training in the uses of LLMs” (67.5%), “acceptance and adoption from public and governmen”’ (63.9%) and “lack of transparency and interpretability” (63%), illustrated in Figure 2.

Finally, the 3 most common applications of LLMs in medical education are “flashcards and revision questions” (59.7%), “summarize complex topics” (56%), and “3D anatomy education” (52.5%) summarized in Table 3.

Acceptance/Readiness Toward LLMs

The results further revealed that 608 participants (52%) support LLMs use in healthcare, yet a greater number remained neutral (N = 433, 37%) or disagreed (N = 372, 31%) on LLMs being safe in this context. Moreover, 656 respondents (56%) also rejected the possibility of LLMs replacing healthcare professionals, and 510 respondents (43%) believe that LLMs will gain acceptance in the healthcare field. Furthermore, in the context of LLM in medical research, more than half (N = 644, 55%) agreed that LLMs will have an important role in this field. This is complemented by 737 participants (62%) anticipating ease in conducting research due to LLMs, and 542 respondents (46%) also believe that LLMs will not replace researchers (Table 4).

Table 4.

Readiness for the use of LLMs.

Willingness Regarding the Use of LLMs in Healthcare		Frequency (N=1180)N	Percentage
Will LLMs have a role to play in healthcare:*
	Strongly disagree	37	3.1
	Somewhat disagree	112	9.5
	Neutral	366	31.0
	Somewhat agree	387	32.8
	Strongly agree	221	18.7
	Not applicable	57	4.8
Are LLMs currently safe to use in the healthcare field:
	Strongly disagree	94	8
	Somewhat disagree	278	23.6
	Neutral	433	36.7
	Somewhat agree	242	20.5
	Strongly agree	69	5.8
	Not applicable	64	5.4
Will LLMs replace healthcare professionals:
	Strongly disagree	410	34.8
	Somewhat disagree	246	20.9
	Neutral	260	22.0
	Somewhat agree	135	11.4
	Strongly agree	57	4.8
	Not applicable	72	6.1
Will LLMs be accepted in the healthcare field:
	Strongly disagree	66	5.6
	Somewhat disagree	143	12.1
	Neutral	399	33.8
	Somewhat agree	368	31.2
	Strongly agree	142	12
	Not applicable	62	5.3

*Rounding up to 2 decimal places results to a total of 100%.

Abbreviation: LLM, large language model.

Discussion

This pertinent study demonstrates the use of LLMs among medical students across medical schools in 7 countries, specifically focusing on its application in healthcare, medical education, and clinical research. Over 75% of participants were already aware of LLMs, particularly ChatGPT. Respondents were most familiar with the specific use of LLMs in research and the least awareness was seen with use of LLMs in healthcare. While 56% of participants rejected the notion that LLMs will replace healthcare workers, the pleurality (43%) nevertheless supported the acceptance of LLMs into the healthcare system. Additionally, 31% had reservations regarding the current safety of LLMs into practice. In the context of medical research, LLMs were seen as a convenient tool to increase the ease and efficacy of conducting research, however, 49% of responses highlighted the need for careful consideration of ethical issues in the incorporation of LLMs in clinical research. Lastly, concerning medical education, a significant proportion agreed LLMs aiding in simplifying medical concepts (56%) and exam revision (60%) especially by reducing cognitive load through dynamic summaries and interactive 3-dimensional (3D) models, allowing them to focus on deeper understanding and efficient exam preparation through personalized flashcards and practice questions.

Healthcare and Clinical Practice

Out of the number of respondents using LLMs, a vast majority (96.4%) were accustomed to ChatGPT, as compared to other language models. Given its positive perception on social media, it should come as no surprise that the majority of students learned about it through these platforms.¹⁴ In a 2023 study conducted by Alkhaaldi et al, a majority of recently graduated medical students in the UAE denied using ChatGPT in medical school, however around 63% students planned to use it during residency.⁹ The increasing presence of the chatbot and other language models among future physicians warrants close monitoring of its potential influence on their training and professional development.

The perceived applications of large language models in medicine encompass a vast and diverse landscape, impacting nearly every facet of clinical practice. Interestingly, respondents to our survey believed accessing new scientific information (46%) and language translation (42%) to be the most appropriate applications, followed by medical diagnosis and decision support (34%). In terms of accessing new scientific information, LLMs have shown promising potential in encoding clinical knowledge,¹⁵ however, the accuracy of such datasets is seemingly limited to models designed for medical usage as compared to general chatbots available for public use¹⁶ discovered that a synthetic generative LLM was able to outperform NLP models trained using real-world clinical text, and also found that physicians could not differentiate between clinical notes generated by LLMs compared with humans in terms of linguistic readability and clinical relevance. The retrieval of clinical knowledge and information using LLMs may lead to new scientific discoveries for diagnostic aid and therapeutic management, which presents a massive advantage to training and supporting healthcare professionals.

Slightly more than half of respondents (56%) disagreed with the notion that LLMs will replace healthcare workers, placing a point of contention on their authority and limitations in a clinical setting. While some studies suggest that LLMs, particularly multimodal ones, could augment healthcare professionals,¹⁷ others caution that the current limitations, such as inaccuracy and lack of an uncertainty indicator, preclude their autonomous deployment in clinical settings.¹⁸ Furthermore, certain studies have indicated that the performance of the models may not be consistent across different medical specialties. Wilhelm et al¹⁹ discovered error patterns in therapy recommendations generated by language models across the fields of ophthalmology, orthopedics, and dermatology, including general and nonspecific advice when prompted treatment recommendation requests on arbitrarily chosen diseases. Consequently, consideration for distinct needs and specifications of various specialties is necessary before LLMs can reach autonomy in a clinical setting.

However, LLMs have shown promise in assisting healthcare professionals in fields such as personalized medicine and routine documentation, including discharge summaries.^19,20 The specialized nature of tasks requiring understanding and interpreting clinical language presents unique challenges for LLMs, but task-specific learning strategies and prompting techniques can enhance their effectiveness in healthcare settings.²¹

A lack of human interaction (49%) and emotional intelligence (51%) were ranked as the most common potential barrier in the implementation of LLMs in clinical settings. Despite the promising potential that NLP-based AI systems hold in healthcare, concerns raised on their ability to mimic human interactions remain one of the most significant barriers to their widespread adoption in clinical settings involving patient interactions. This is due particularly to the perceived communication barriers, stemming from patients’ hesitancy in accepting the role of AI in traditional physician–patient interactions.²² Factors such as the loss in face-to-face interaction and personalized interactions in this context raise concerns about patient safety and the need for transparent and explainable AI.²³ Nadarzynski et al²⁴ found that participants felt that AI-based chatbot responses were more depersonalized and were worried about its inability to understand the emotionally sophisticated aspects of health, particularly mental health. To bridge the potential communication and emotional gap between users and LLMs in clinical settings, developers should prioritize the incorporation of interactive and engaging social cues. By implementing features that foster trust and connection, such as human-like communication and personalized assistance, it is possible to create an environment tailored to the patient's mental and emotional state, thereby simulating empathy.²² This focus on enhancing the patient experience could prove key in unlocking the full potential of LLMs as valuable tools within healthcare settings.

When training clinicians, emphasis must be provided on developing new communication skills to effectively interact with AI systems within the care delivery process, while patients need to overcome potential technological apprehension to fully engage with these tools.²⁵

Clinical Research

A significant number of students (55%) acknowledged the significant role of LLMs in the research field, aligning with findings in various papers. Ruksakulpiwat et al²⁶ highlight that ChatGPT, although not explicitly designed for medical research, has demonstrated impressive performance in this domain. The growing prominence of LLMs in medical research may be attributed to their ability to identify new research questions and enhance educational understanding.²⁷ Another study by Wang et al²⁸ investigated the ability of ChatGPT to produce effective Boolean queries for systematic review. The study involved ChatGPT being trained on a set of extensive prompts on over 100 systematic review topics. The results demonstrated that ChatGPT effectively generated queries leading to a high level of search precision.²⁸ This capability helps researchers avoid the inclusion of invalid and biased studies, potentially saving both time and healthcare resources.

Participants ranked the top research application of LLMs to be “data analysis and interpretation” (64%). Several studies highlight this feature in ChatGPT, indicating its growing role in conducting clinical research. In a systematic review conducted by Sallam et al,⁴ which assessed 60 records, it was found that 20 of them cited the efficient analysis of datasets in healthcare research as a notable benefit of using ChatGPT. Such ability to analyze large datasets can produce a foundation for clinical decision making. Furthermore, the mechanism through which language models like ChatGPT are able to analyse vast amounts of data is explained by Jeyaraman et al.²⁹ Natural language processing techniques, which are the core component of LLMs, can be utilized to extract crucial information from the texts, such as patient records, medical reports, and scientific journals, and present it in a structured way.²⁹

Nearly half (49%) of the participants held the view that LLMs will pose ethical challenges in research. Some of the significant ethical concerns are highlighted by Xames et al³⁰ such as unintentional plagiarism. ChatGPT's tendency to reproduce text without appropriate citations or attribution (also known as “hallucinations”) can pose a significant challenge for researchers. Thus, addressing this issue is crucial for researchers to guarantee that ChatGPT generates outputs that adhere to scholarly standards while being accurate and ethical. Another major ethical issue discussed by Xames et al is biases and inaccuracies. This is because conversational AI can amplify and replicate both human and algorithmic biases, creating a difficulty in distinguishing between accurate and misleading information.

When the participants were asked about LLMs potentially replacing researchers, 46% rejected this notion. Despite significant potential in assisting researchers in the processes of conducting a literature search, summarizing information, and language translation, Salvagno et al³¹ explores why ChatGPT is not expected to substitute for human researchers. The writing process of a scientific paper currently necessitates the judgment and oversight of human researchers who are experts in the field, to ensure the accuracy, coherence, and credibility of the content before being used or submitted for publication.³¹ Interestingly enough, Huespe et al³² conducted a study and found that a background section generated by GPT-3.5 for a critical care research question was nearly indistinguishable from the writing of a medical researcher. This highlights the potential of LLMs in enhancing the scientific tone of research writing, thereby augmenting the current role of researchers. The article however concluded that LLM generated content should be restricted to improving the grammar and coherence of article drafts, rather than drafting them entirely.³²

Medical Education

Around 59.7% of respondents believed that LLMs have a pivotal role in medical examination preparation and revision, particularly through the use of creating flashcards and practice questions.

Pendergrast et al specifically points toward the role of LLMs in the integration of one of the most popular third-party resources used by medical students known as Anki. They demonstrated how GPT-3.5 can effortlessly be used to generate, select, and tailor flashcards according to the medical school curricula and topic, thereby acting as an efficient tool to aid in academics.³³

A study conducted by Tangadulrat et al¹⁰ found that students were leaning toward ChatGPT as a tool for preparation for practical examination by generating a variety of scenarios and situations, testing the student's skills and knowledge, which would also provide real-time feedback. Precisely, 55.4% of respondents in our survey agreed that LLMs can provide students with access to real-time feedback. Moreover, a paper published by Chan et al³⁴ also agreed that AI provides real-time feedback, which would help identify any gaps in their learning, hence, enhancing their knowledge.

Moreover, Kung et al³⁵ revealed that ChatGPT was helpful in preparing medical students for the United States Medical Licensing Examination. Peculiarly, students will find ChatGPT helpful in many ways since it can provide multiple answers to one question, therefore, giving students a wide range of answers to choose from as well as expanding their knowledge.³⁶

However, it can be argued that there is a tendency of language models to generate false information, which means answers and clinical vignettes might not always be correct. This may be due to the way ChatGPT has been trained with information from a diverse range of datasets, which may not be specific to certain clinical scenarios and prompts. Additionally, a recent paper by Maryam Buholayka et al³⁷ explicitly stated that ChatGPT may produce invalid answers even though they may seem persuasive to the medical student. They tasked ChatGPT to write a case report based on report draft written by human authors under 5 different prompts. A comparison of the generated case reports showed that ChatGPT failed to uphold patient confidentiality, fabricate references and even conclude with an incorrect final diagnosis.

These concerns were further emphasized by attending physicians, residents and interns in another study, who were concerned about ChatGPT's answers being simple and failed to show a deeper understanding of the topic.¹⁰ This highlights cautious reviewing on the development of LLMs before they can be used as part of the medical curricula.

The implications of our study emphasize the need to incorporate AI literacy and practical training on LLMs in medical curricula. As many students are already familiar with these technologies, curricula could benefit from including modules that address the applications, ethical concerns, and critically evaluate the role of AI in healthcare.

Additionally, providing hands-on experience with LLMs in simulated environments, could enhance students’ skills and confidence in using these tools. This approach would better prepare future healthcare professionals to leverage AI responsibly and effectively, in both research and patient care.

Strengths and Weaknesses

When examining the strengths of the study, the generalisability among target population and diverse participant demographics of the study can be appreciated. This was achieved by gathering 1180 responses across 7 countries, providing a well-rounded representation of the medical student population and enhancing the statistical power of cross-sectional research. At the same time, the large sample size reduces the margin of error and increases the precision of estimates, making the findings more representative of the target population. Furthermore, using an online Google Form enabled the survey to be quickly distributed through various channels like email, social media, and online platforms. This is evident from the total responses generated within 3 weeks of the study initiation. The survey addresses a contemporary and increasingly relevant issue at the intersection of technology and healthcare which has not be studied in literature earlier. Lastly, the study provides actionable insights that can inform future direction to curriculum development, technology integration strategies, and regulatory frameworks in healthcare settings.

The study has several limitations, including reliance on voluntary participants, the survey being only in one language, and the use of self-reported data. To address the challenge of voluntary participation, multiple reminders were sent to nonrespondents, which significantly improved response rates and ensured higher survey completion. For future studies, incorporating incentives such as gift cards could further encourage participation from a more diverse group of respondents. The survey's limitation of being available only in English was mitigated by using simple and clear language to enhance comprehension; however, future surveys should consider translations into multiple languages to improve inclusivity. To minimize social desirability bias associated with self-reported data, participants were assured anonymity at the outset of the survey. Additionally, a pilot test was conducted with a small group before the survey's full rollout to identify and address potential issues, ensuring the reliability of the data collection process.

Another potential limitation of the study could be that some participants may be nonmedical students. However, questions about the medical university attended and year of study were included as indicators, the survey was primarily disseminated among medical students known to the authors and collaborators. This approach likely ensured that the participant pool predominantly consisted of medical students.

Future Research Recommendation

As AI continues to be rapidly adopted in different areas of healthcare, further research is strongly recommended to fully understand its potential, implications, and challenges from medical students’ perspectives.

One key area future studies should explore is how LLMs can be utilized differently in teaching and learning. Such research could contribute to the development of best practices that ensure AI tools are maximally effective for both educators and students. Additionally, investigating the differences in how LLMs are applied during preclinical and clinical years of medical school is important. Such studies could reveal unique needs and opportunities at each stage, enabling tailored approaches to AI integration that address the specific challenges and goals of these phases.

A potential direction for long-term research could involve exploring the impact of using LLMs during medical school and how it affects the clinical practice once they are fully qualified physicians. Does the use of AI tools during training positively or negatively influence clinical decision making and patient care?

It is also imperative to address the identified barriers in application of LLMs for healthcare and research and navigate solutions to increase medical students’ confidence in the usage of these tools.

Lastly, the ethical consideration surrounding the extent of AI integration is a priority. Research addressing ethics can be used for developing ethical frameworks and guidelines that can regulate AI use in medical schools.^38,39

Conclusion

The accelerating progression of AI has ushered in novel prospects in healthcare delivery, clinical research, and medical education. This creates a notable gap in understanding the perspectives and utility of LLMs among medical students and its ethical considerations. Although most respondents are already aware of LLMs and their applications, the survey demonstrated that there is still hesitancy among a significant portion of medical students in fully adopting it into healthcare practice, clinical research, and medical education.

This study prompts further exploration of medical students’ attitudes and acceptance concerning the integration of LLMs for future educational strategies, research methodologies, ethical considerations, and healthcare practice.

Supplemental Material

sj-pdf-1-mde-10.1177_23821205251331124 - Supplemental material for Medical Students’ Perceptions of Large Language Models in Healthcare: A Multinational Cross-Sectional Study

Supplemental material, sj-pdf-1-mde-10.1177_23821205251331124 for Medical Students’ Perceptions of Large Language Models in Healthcare: A Multinational Cross-Sectional Study by Faiza Ejas, Sameer Asim Khan, Amina Mujahid, Fatma AlJoker, Hans Mautong, Geovanny Alvarado-Villa, Abhishek Kashyap, Muhammad Umer Yasir, Kindie Woubshet Nigatu, Nethra Jain, Nandhini Iyer, Aman Sandhu, Shahab Sharafat, Sara Yahya, Mohamd Mahmoud Ghaly, Ibrahim Ibrar, Aakanksha Singh, Harpeet Grewal, Ivan Alfredo Huespe, Priyal Mehta, Zara Arshad, Rahul Kashyap and Faisal A Nawaz in Journal of Medical Education and Curricular Development

Footnotes

Acknowledgments

We are grateful to the Global Remote Research Scholars Program for providing the opportunity of global collaboration and keen mentorship for this study in collaboration with Mohammed Bin Rashid University of Medicine and Health Sciences.

ORCID iDs

Sameer Asim Khan

Sara Yahya

Mohamd Mahmoud Ghaly

Aakanksha Singh

Harpeet Grewal

Zara Arshad

Ethical Considerations

The study was deemed IRB-exempt from Mayo Clinic Institutional Review Board.

Consent to Participate

Your responses are entirely voluntary. This survey is designed to be anonymous, meaning that there should be no way to connect your responses with you. By completing and submitting the survey, you affirm that you give your consent for your answers to be used for the research and quality improvement purposes.

Author Contributions

FE, SK, AM, and FA were involved in study conception, survey design, data collection, analysis, and prepared the first draft of the manuscript. ZA, RK, and FAN reviewed, edited, and approved the final manuscript. HM, GA-V, AK, UY, KWN, NI, NJ, AS, SS, SY, MMG, and II contributed in data acquisition. AS, HG, IAH, and PM validated the survey. All authors contributed to the manuscript and approved the submitted version.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Supplemental Material

Supplemental material for this article is available online.

References

Sarker

. AI-based modeling: Techniques, applications and research issues towards automation, intelligent and smart systems. SN Comput Sci. 2022;3. Epub ahead of print 2022. DOI: https://doi.org/10.1007/s42979-022-01043-x

Bajwa

Munir

Nori

, et al. Artificial intelligence in healthcare: transforming the practice of medicine. Future Healthc J. 2021;8(2):e188-e194. DOI: https://doi.org/10.7861/fhj.2021-0095

Brants

Popat

Och

Dean

. Large language models in machine translation. In: Eisner

, ed. Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL). Association for Computational Linguistics; 2007:pp.858-867.

Sallam

. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare (Basel). 2023;11(6):887. DOI: https://doi.org/10.3390/healthcare11060887. PMID: 36981544; PMCID: PMC10048148.

Introducing ChatGPT. Openai.com, https://openai.com/blog/chatgpt (accessed March 3, 2024).

Safranek

Sidamon-Eristoff

Gilson

, et al. The role of large language models in medical education: applications and implications. JMIR Med Educ. 2023;9:e50945. DOI: https://doi.org/10.2196/50945. Available at: https://mededu.jmir.org/2023/1/e50945

Wen

Wang

. The future of ChatGPT in academic research and publishing: A commentary for clinical and translational medicine. Clin Transl Med. 2023;13(3):e1207. Epub ahead of print 2023. DOI:https://doi.org/10.1002/ctm2.1207. PMID: 36941774; PMCID: PMC10028016.

Jha

Shankar

Al-Betar

, et al. Undergraduate medical students’ and interns’ knowledge and perception of artificial intelligence in medicine. Adv Med Educ Pract. 2022;13:927-937. DOI:https://doi.org/10.2147/AMEP.S368519. PMID: 36039185; PMCID: PMC9419901

Alkhaaldi

SMI

Kassab

Dimassi

, et al. Medical student experiences and perceptions of ChatGPT and artificial intelligence: cross-sectional study. JMIR Med Educ. 2023;9:e51302. DOI: 10.2196/51302. PMID: 38133911; PMCID: PMC10770787.

10.

Tangadulrat

Sono

Tangtrakulwanich

. Using ChatGPT for clinical practice and medical education: cross-sectional survey of medical students’ and physicians’ perceptions. JMIR Med Educ. 2023;9:e50658. DOI: https://doi.org/10.2196/50658

11.

Elrod

Fortenberry

Jr . The hub-and-spoke organization design: an avenue for serving patients well. BMC Health Serv Res. 17. Epub ahead of print 2017. DOI: https://doi.org/10.1186/s12913-017-2341-x

12.

Amal

Banga

Bhatt

, et al. Global remote research scholars program. Guiding principles for the conduct of the Violence Study of Healthcare Workers and System (ViSHWaS): insights from a global survey. J Glob Health. 2024;14:04008. DOI: https://doi.org/10.7189/jogh.14.04008

13.

Kashyap

. 2023. Medical Students’ Perceptions of Large Language Models in Healthcare GRRSP_Survey Invite. YouTube. https://www.youtube.com/watch?v=IfmMMxWv1ig.

14.

Leiter

Zhang

Chen

, et al. ChatGPT: a meta-analysis after 2.5 months. ArXiv; abs/2302.13795. Epub ahead of print 2023. DOI: https://doi.org/10.48550/arXiv.2302.13795.

15.

Singhal

Azizi

, et al. Large language models encode clinical knowledge. Nature. 2023;620:172-180. DOI: 10.1038/s41586-023-06291-2

16.

Peng

Yang

Chen

, et al. A study of generative large language model for medical research and healthcare. NPJ Digit Med. 2023;6:210. Available at: https://doi.org/10.1038/s41746-023-00958-w

17.

Meskó

. The impact of multimodal large language models on health care’s future. J Med Internet Res. 2023;25:e52865. DOI: https://doi.org/10.2196/52865. Available at: https://www.jmir.org/2023/1/e52865

18.

Thirunavukarasu

Hassan

Mahmood

, et al. Trialling a large language model (ChatGPT) in general practice with the applied knowledge test: observational study demonstrating opportunities and limitations in primary care. JMIR Med Educ. 2023;9:e46599. DOI: https://doi.org/10.2196/46599. PMID: 37083633; PMCID: PMC10163403.

19.

Wilhelm

Roos

Kaczmarczyk

. Large language models for therapy recommendations across 3 clinical specialties: comparative study. J Med Internet Res. 2023;25:e49324. DOI: https://doi.org/10.2196/49324. Available at: https://www.jmir.org/2023/1/e49324

20.

Patel

Lam

. ChatGPT: the future of discharge summaries?. Lancet Digit Health. 2023;5(3):e107-e108. DOI: https://doi.org/10.1016/S2589-7500(23)00021-3

21.

Mannstadt

Mehta

. Large language models and the future of rheumatology: assessing impact and emerging opportunities. Curr Opin Rheumatol. 2024;36(1):46–51. DOI: https://doi.org/10.1097/BOR.0000000000000981.

22.

Wang

Zhao

Petzold

. Are large language models ready for healthcare? A comparative study on clinical language understanding. MLHC; abs/2304.05368. Epub ahead of print 2023. DOI: https://doi.org/10.48550/arXiv.2304.05368.

23.

Esmaeilzadeh

Mirzaei

Dharanikota

. Patients’ perceptions toward human–artificial intelligence interaction in health care: experimental study. J Med Internet Res. 2021;23:e25856. DOI: https://doi.org/10.2196/25856. PMID: 34842535; PMCID: PMC8663518.

24.

Nazar

Alam

Yafi

, et al. A systematic review of human–computer interaction and explainable artificial intelligence in healthcare with artificial intelligence techniques. IEEE Access. 2021;9:153316-153348.

25.

Nadarzynski

Miles

Cowie

, et al. Acceptability of artificial intelligence (AI)-led chatbot services in healthcare: a mixed-methods study. Digit Health. 2019;5:205520761987180. Available at: DOI:https://doi.org/10.1177/2055207619871808

26.

Yang

Xue

, et al. Translating cancer genomics into precision medicine with artificial intelligence: applications, challenges and future perspectives. Hum Genet. 2019;138(2):109-124. Available at: DOI:https://doi.org/10.1007/s00439-019-01970-5

27.

Ruksakulpiwat

Kumar

Ajibade

. Using ChatGPT in medical research: current status and future directions. J Multidiscip Healthc. 2023;16:1513-1520. DOI:https://doi.org/10.2147/JMDH.S413470

28.

Lahat

Shachar

Avidan

Shatz

Glicksberg

Klang

. Evaluating the use of large language model in identifying top research questions in gastroenterology. Sci Rep. 2023;13(1):4164.

29.

Wang

Scells

Koopman

, et al.

Can ChatGPT write a good Boolean query for systematic review literature search?

Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval; 2023; New York, NY, USA: ACM.

30.

Jeyaraman

Priya

Jeyaraman

, et al.

ChatGPT in medical education and research: a boon or a bane?

Cureus. 2023;15(8):e44316. DOI: https://doi.org/10.7759/cureus.44316

31.

Xames

Shefa

. ChatGPT for research and publication: opportunities and challenges. J Appl Learn Teach. 6(1):20. DOI:https://doi.org/10.37074/jalt.2023.6.1.20

32.

Salvagno

Taccone

Gerli

. Can artificial intelligence help for scientific writing? Crit Care. 2023;27. Epub ahead of print 2023. DOI: https://doi.org/10.1186/s13054-023-04380-2

33.

Huespe

Echeverri

Khalid

, et al. Clinical research with large language models generated writing—clinical research with AI-assisted writing (CRAW) study. Crit Care Explor. 2023;5(10):e0975. DOI:https://doi.org/10.1097/CCE.0000000000000975

34.

Pendergrast

Chalmers

. Anki Tagger: A Generative AI Tool for Aligning Third-Party Resources to Preclinical Curriculum. JMIR Med Educ. 2023;9:e48780. doi:10.2196/48780

35.

Chan

Zary

. Applications and challenges of implementing artificial intelligence in medical education: integrative review. JMIR Med Educ. 2019;5(1):e13930. doi:https://doi.org/10.2196/13930

36.

Kung

Cheatham

Medenilla

, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2(2):e0000198. doi: https://doi.org/10.1371/journal.pdig.0000198

37.

Klang

Portugez

Gross

, et al. Advantages and pitfalls in utilizing artificial intelligence for crafting medical examinations: a medical education pilot study with GPT-4. BMC Med Educ. 2023;23:772. DOI:https://doi.org/10.1186/s12909-023-04752-w

38.

Buholayka

Zouabi

Tadinada

. The readiness of ChatGPT to write scientific case reports independently: a comparative evaluation between human and artificial intelligence.Cureus. 2023;15(5):e39386. DOI:https://doi.org/10.7759/cureus.39386.PMID:37378091; PMCID:PMC10292135

39.

Nawaz

Opriessnig

Usman

, et al. From classroom to clinic: the impact of AI on medical education. In: Advances in medical technologies and clinical practice. IGI Global; 2024:63-90.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

6.34 MB