Abstract
OBJECTIVES
Implementation barriers and lack of standardized point-of-care ultrasound (POCUS) curricula make the development of effective POCUS curricula and methods of assessment challenging. The authors aim to develop a longitudinal POCUS curriculum through staged intervention. In the first stage, the authors hypothesized that the use of high-fidelity ultrasound simulation during the Internal Medicine clerkship would improve POCUS confidence and knowledge among medical students, minimizing the need for trained faculty.
METHODS
A quasi-experimental study of third-year students on the Internal Medicine clerkship at a large academic medical center in the United States was performed assessing the efficacy of ultrasound simulation use. The control group consisted of students who received baseline POCUS education during teaching rounds but did not have access to the ultrasound simulator. The experimental group consisted of students who, in addition to baseline POCUS education, had access to a high-fidelity ultrasound simulator throughout the clerkship for a minimum of 1 hour per week. Students in both the control and experimental groups completed a pre- and post-intervention confidence survey and knowledge-based examination.
RESULTS
Eighty-two percent (50/61) of students completed pre- and post-tests, with the control group demonstrating no significant difference in POCUS confidence or knowledge. After exposure to the ultrasound simulator, the experimental group demonstrated statistically significant improvement in POCUS confidence and overall POCUS knowledge (p < .01).
CONCLUSION
The use of high-fidelity ultrasound simulation can improve POCUS confidence and knowledge among medical students while addressing common barriers to the implementation of a POCUS curriculum. Despite showing statistically significant improvement in overall knowledge, the results did not appear to hold educational significance. Additional POCUS educational methods are necessary to overcome cognitive bias and potential overconfidence. The next stage of curriculum development will include resident-led POCUS workshops to supplement simulation.
Introduction
Point-of-care ultrasonography (POCUS) is a “goal-directed, bedside exam performed by a healthcare provider in real-time to answer a specific diagnostic question or guide an invasive procedure.” 1 Physicians across nearly all specialties who incorporate POCUS into their clinical practice can improve diagnostic expediency, procedural safety, and patient outcomes. 1
POCUS competency has been increasingly incorporated into Graduate Medical Education (GME) requirements. The Accreditation Council for Graduate Medical Education (ACGME) has named POCUS a major competency in multiple specialties, including Emergency Medicine and most recently, Family Medicine.2,3 Notably, Internal Medicine (IM) has not yet required POCUS training as an ACGME competency; however, numerous IM residency programs have still incorporated POCUS education into their standard curricula. 4
The addition of POCUS as a major competency at the GME level encourages Undergraduate Medical Education (UME) programs to incorporate ultrasonography training into their educational mission and structure to better prepare students for competitive residency programs.5–7 While the current instructional method and quantity of ultrasound training are highly variable, 62% of medical schools report ultrasonography training in their UME curriculum, and 79% report a belief that ultrasound training should be incorporated. 7 Common barriers to implementation of POCUS in UME include lack of trained faculty, lack of time in the curriculum, and lack of financial support for the purchase and maintenance of ultrasound equipment.7,8 Questions remain regarding the implementation of ultrasound training into UME, as well as the assessment of competency following the implementation of a curriculum.7,9
Our medical school, similar to many other institutions, is developing a longitudinal, multi-year POCUS curricula. Prior to July 2020, our institution lacked a formal longitudinal POCUS curriculum during the clinical undergraduate training years (years 3 and 4 of a standard United States medical school). A POCUS elective was available during year 4 through the Emergency Medicine department; however, space was limited to 12 out of 165 medical students per year. Lack of POCUS training in the clinical years presents a gap in UME, given its proposed benefits in diagnostic and physical exam skills, safer procedural guidance, and a better understanding of anatomic relationships and physiologic concepts. 7 Here, we present the results of the first stage of a stepwise intervention in the development of a longitudinal POCUS curriculum during the 6-week IM clerkship at a large academic medical center in the Southeastern United States. Our aim was to develop a curriculum that required minimal intervention by ultrasound-trained faculty, given the scarcity, and increased hands-on didactic time with the use of simulation. In the first stage of curriculum development, we hypothesized that use of high-fidelity ultrasound simulation alone would improve POCUS confidence, knowledge, and skill among third-year medical students.
Methods
A quasi-experimental study was performed assessing the efficacy of an ultrasound simulation curriculum implemented during the IM clerkship from January 2021 through May 2021 at the Medical University of South Carolina (MUSC). Students participated voluntarily in the study. The MUSC Institutional Review Board reviewed the study and deemed it exempt and waived the requirement for informed consent. All third-year IM clerkship students were invited to participate. Students were divided into two groups based on the location of their clerkship. Any students who did not fully complete the pre-intervention or post-intervention surveys were excluded from the study. The control group consisted of students assigned to MUSC Main Hospital who received informal, or “baseline,” POCUS education provided by faculty and residents during teaching rounds but did not have access to the ultrasound simulator. “Baseline” POCUS education included the following: all students received a brief introduction to ultrasound during the first two years of medical school that consisted of longitudinal 10-minute organ-specific POCUS trainings during the corresponding pre-clinical physical exam workshops. During the clerkship, the majority of hospitalist faculty were novices in the use of POCUS, while the residents at both clerkship locations were actively enrolled in a new longitudinal POCUS curriculum. The experimental group consisted of students assigned to the Ralph H. Johnson Veterans Affairs (VA) Medical Center who, in addition to the “baseline” POCUS education, had access to a high-fidelity ultrasound simulator for two 5-hour open-use sessions per week during the 6-week clerkship. Students were asked to spend at least one hour each week with the simulator, and simulator use was recorded using anonymous identification numbers assigned to students at the beginning of the rotation. Multiple students often used the simulator simultaneously; therefore, they were asked to sign in and out individually to capture the overall time used by each student.
The ultrasound simulator provided was the Simbionix Ultrasound MentorTM, a high-fidelity medical simulator for the training of ultrasound-related examinations and interventions. High-fidelity simulation consists of computer-generated cases of virtual sonographic anatomy, allowing multidisciplinary, realistic, hands-on POCUS training. The Simbionix Ultrasound MentorTM includes a computer with a large high-definition multi-touch screen, multiple standard ultrasound probes, and an ultrasound mannequin. 10 Further, it offers a probe guidance tool that can be turned on/off as desired. When on, the probe guidance tool shows a video representation of the simulation mannequin with an ultrasound probe that moves in real-time and corresponds with the learner moving the physical probe on the mannequin. There is a second partially transparent “shadow-probe” on the same screen that remains static and represents the goal placement and view. The cost of the simulator with basic and advanced cardiac, lung, and interventional educational modules totaled $148,500. Annual software licensing costs an additional $2,400. Simulation modules used by the students included Sonography Basic Skills, Bedside Echocardiography, COVID-19, and Lung Ultrasound. Each module included didactic videos and algorithms, followed by hands-on practice cases, allowing the acquisition of appropriate views and input of clinical findings.
Statistical analysis
Using G*Power Version 3.1, our sample size calculation and power analysis were calculated at N = 29; thus our sample of 61 clerkship students was deemed adequate for the study. A pre- and post-intervention survey was completed at the beginning and end of the clerkship. Students in both the control and experimental groups completed a pre-intervention survey that included demographic and needs assessment questions. Additionally, both groups completed a pre- and post-intervention confidence survey and knowledge-based examination that reviewed ultrasound physics, procedural use, image acquisition, recognition, and interpretation. The survey was previously validated 11 and is included in the supplemental material. Students’ confidence and perception of POCUS value was assessed using a 5-point Likert scale ranging from “1 = not at all” to “5 = extremely.” The pre- and post-intervention knowledge-based assessment consisted of 51 total questions that were further categorized as (1) ultrasound physics, (2) procedural use, (3) image acquisition, (4) image recognition, and (5) image interpretation. Paired t-tests were performed to compare pre- and post-intervention Likert scale responses. Pearson's chi-square and Fisher's exact tests were performed to determine differences in proportions between pre- and post-intervention knowledge-based categorical questions. The data were analyzed using SAS Version 9.4 (SAS Institute Inc., Cary, NC).
Results
Among clerkship students included in the study, a total of 82.0% (50/61) completed both the pre- and post-intervention surveys, with 42.0% (21/50) of participants being female. Eleven students were excluded due to incomplete surveys. Of the 50 participating students, 62.0% (31/50) reported no prior ultrasound training and 94.0% (47/50) reported rarely or never using ultrasound in their current practice (Table 1). There were a total of 45 unique sign-ins to the simulator among the 28 students in the experimental group. Fourteen of the 28 students in the experimental group did not have an entry in the simulator log. The range of simulator use was 0 to 247 total minutes per learner over the 6-week clerkship. The median was 28.5 minutes and the mean 86.2 minutes of simulator use per learner over 6 weeks. There did not appear to be any significant outliers.
Pre-intervention demographics among the control and experimental groups.
The control group showed no significant difference in POCUS confidence or knowledge of pre- and post-intervention analysis. The experimental group showed statistically significant improvement in POCUS confidence (Table 2). Additionally, the experimental group demonstrated no significant difference in pre- and post-intervention individual knowledge-based questions but statistically significant improvement in overall knowledge-based assessment scores (total of 51 questions) from 48.7% to 59.1% on pre- to post-intervention examination, respectively (p = .0024) (Figure 1).
Pre- and post-intervention examination results for overall ultrasound knowledge.
Pre- and post-intervention confidence survey results.
Note: Variables measured using a 5-point Likert scale ranging from “1 = not at all” to “5 = extremely”; p-values with bolded font designate those <.05; p*-value between pre- and post-intervention experimental groups.
Discussion
Implementation of the first stage of a multi-step intervention to develop a longitudinal POCUS curriculum during the Internal Medicine clerkship demonstrated use of high-fidelity ultrasound simulation alone increased POCUS confidence and overall knowledge among medical students. Advantages of simulation use include decreased need for paid or volunteer patient models, flexible curriculum scheduling, and guaranteed exposure to a variety of specific anatomical images using computer-generated cases.
While our study demonstrated a statistically significant increase in overall POCUS knowledge using ultrasound simulation, increase in knowledge scores was not as robust as expected. There were no statistically significant improvements in individual knowledge-based questions, suggesting the overall improvement may be statistically significant, but it is not educationally significant. Given 14 students in the experimental group did not have an entry in the simulator log, and overall students only spent an average of 86.2 minutes using simulation, it is unclear if additional time and effort would produce more educationally significant results. The improvement in POCUS confidence for our experimental group is also statistically significant, but it raises more concern than excitement. The improved confidence is incongruent with the lack of educationally significant improvement in knowledge, suggesting a Dunning–Kruger effect or cognitive bias and potential overconfidence. These results emphasize the importance of assessing confidence along with knowledge and skill following the implementation of a POCUS curriculum.
Furthermore, no POCUS-trained faculty were required for our intervention. This was initially hypothesized to be an advantage of our intervention and the use of simulation. Given our findings of increased confidence with minimal improvement in knowledge and skills, increased curricular involvement by POCUS-trained faculty may be beneficial. An investment of $150,000 in ultrasound simulation may advance and maintain POCUS skills. However, based on our results, additional investment of time and money to develop POCUS-trained faculty is necessary to avoid pure cognitive bias.
Limitations of our study include variable levels of expertise and amount of “baseline” POCUS education provided by faculty and residents during teaching rounds, which cannot be controlled for and may represent a confounding variable given our small sample sizes. Additionally, despite our instruction, there may have been a group of students who worked as a team, and thus we did not capture their unique sign ins at the time of simulator use, which would underestimate the number of hours students spent using the simulator. Students received an introductory tutorial on simulator use but were not proctored during the time they used the ultrasound simulator. Lack of proctoring may be advantageous for students less likely to participate in teaching rounds due to discomfort with the preceptor or fear of impact on clinical performance evaluation. However, the lack of consistent guidance in ultrasound use may have hindered the educational experience. Lastly, there is a lack of generalizability since many other UME programs may not have the funding for high-fidelity ultrasound simulation; however, more technology and options are becoming available.
Conclusion
Given the heterogeneity of current POCUS curricula in UME with mixed results, 7 and the numerous barriers to implementation, we believe a stepwise intervention to curriculum development and assessment is prudent. The use of high-fidelity ultrasound simulation can improve POCUS knowledge among third-year medical students while addressing several barriers to the implementation of a POCUS curriculum; however, additional POCUS learning methods are necessary to produce more robust educational results. The next stage of our stepwise approach to the development of a longitudinal POCUS curriculum will include the implementation of a resident-led POCUS workshop during the Internal Medicine clerkship to supplement simulation use. A resident-led workshop will allow real-time instruction and assessment of proper POCUS use with the goal of matching the observed increase in confidence in our experimental group with a more educationally significant increase in knowledge and skill.
Supplemental Material
sj-pdf-1-mde-10.1177_23821205231213754 - Supplemental material for Development and Assessment of Simulation-Based Point-of-Care Ultrasound Curriculum in Undergraduate Medical Education
Supplemental material, sj-pdf-1-mde-10.1177_23821205231213754 for Development and Assessment of Simulation-Based Point-of-Care Ultrasound Curriculum in Undergraduate Medical Education by Nancy L. Hagood, Monica Klaybor, Romik Srivastava, William McManigle, John Terrill Huggins, Pranav V. Shah and Marc E. Heincelman, Meghan K. Thomas in Journal of Medical Education and Curricular Development
Supplemental Material
sj-docx-3-mde-10.1177_23821205231213754 - Supplemental material for Development and Assessment of Simulation-Based Point-of-Care Ultrasound Curriculum in Undergraduate Medical Education
Supplemental material, sj-docx-3-mde-10.1177_23821205231213754 for Development and Assessment of Simulation-Based Point-of-Care Ultrasound Curriculum in Undergraduate Medical Education by Nancy L. Hagood, Monica Klaybor, Romik Srivastava, William McManigle, John Terrill Huggins, Pranav V. Shah and Marc E. Heincelman, Meghan K. Thomas in Journal of Medical Education and Curricular Development
Footnotes
Acknowledgments
The authors received no external funding or support for this research. The authors have no disclosures or other disclaimers to report.
Author contributions
All authors contributed to this article. Please see below:
Nancy Hagood: conceptualization, methodology, validation, writing—original draft/review/editing, and visualization; Monica Klaybor: conceptualization, methodology, validation, writing—original draft/review/editing, and visualization; Romik Srivastava: conceptualization, methodology, validation, writing—original, and visualization; William McManigle: conceptualization, methodology, validation, writing—review/editing, and visualization; John Terrill Huggins: conceptualization, methodology, and visualization; Pranav Shah: writing review/editing and methodology; Marc Heincelman: conceptualization, methodology, writing draft/review/editing, and visualization; Meghan Thomas: conceptualization, methodology, validation, investigation, formal analysis, data curation, writing—original draft/review/editing, and visualization.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
