Abstract
BACKGROUND
Point-of-care ultrasound (POCUS) has extensive clinical utility in internal medicine, but formal and uniform curricula in internal medicine are lacking.
OBJECTIVE
To determine the effectiveness of a longitudinal, flipped-classroom, academic half-day curriculum on internal medicine resident confidence, utilization, and changes in clinical management.
METHODS
We implemented an asynchronous, flipped-classroom, academic half-day curriculum from November 2020 to November 2021 and conducted an evaluation with a prospective, before-after cohort study. Curriculum included 4 rotating sessions comprised of 20 to 30 min of image interpretation followed by 1.5 to 2 h of image acquisition. Confidence was rated via Likert scale. Utilization was reported via indicating never, 1 to 2, 3 to 4, 5 to 6, or >6 times per month (recorded as 1-5, respectively). Image interpretation was assessed via a 6-question, multiple-choice video assessment.
RESULTS
Nineteen of 99 potential residents (19%) completed a pre- and post-curriculum evaluation. Residents attended a median of 4 sessions. Confidence improved from 2.47 to 3.53 (P = .002). Utilization did not improve overall (2.11-2.42, P = .22), but utilization of left ventricular function assessment (1.53-2.00, P = .046) and pulmonary assessment (1.53-2.00, P = .039) increased. The percentage of residents that had ever changed their clinical management by POCUS increased from 47% to 84% after implementation of the curriculum. Cardiac, pulmonary/pleural, volume assessment, and abdominal free fluid exams were reported as the most clinically useful.
CONCLUSION
Implementation of a longitudinal, academic half-day curriculum for POCUS resulted in improved confidence, increased POCUS utilization for the cardiac and pulmonary examination, and changes in clinical management based on POCUS.
Introduction
The clinical utility of point-of-care ultrasound (POCUS) includes procedural guidance,1,2 diagnostic capability, 3 therapeutic monitoring, 4 and increased patient satisfaction. 5 These benefits have led to guidelines and recommendations for its clinical use by internal medicine physicians. 6 However, early training in POCUS among internal medicine residency programs is lacking, with formal curricula in only 35% to 37.5% of programs.7,8
We incorporated formal POCUS training into our graduate medical education curriculum, including a precurriculum and postcurriculum evaluation to monitor effectiveness. Point-of-care ultrasound curricula in internal medicine programs follow various models; most are single-session workshops or longitudinal sessions which utilize a lecture-based methodology followed by hands-on delivery.9‐12 Our study was conducted to determine the effects of a longitudinal, asynchronous, academic half-day curriculum with flipped-classroom delivery on POCUS confidence, POCUS utilization, changes in clinical management, and image interpretation competency among internal medicine residents. While longitudinal curricula improve long-term retention, 13 the impact of the flipped-classroom format on confidence, clinical utilization, and changes in management among residents is unknown.
Flipped-classroom learning is a strategy that has learners complete pre-course materials to focus classroom time on problem-solving and experiential learning, rather than the traditional lecture format. 14 The format is advantageous by maximizing active learning, involving content through social interaction, and encouraging hands-on manipulation of materials; consistent with the educational theory of constructivism. 15 Combining this format with facilitated, small-group workshops amplifies peer collaboration and promotes self-efficacy; consistent with the social cognitive theory. 16
Methods
Setting and Participants
Our Midwestern internal medicine residency program is university-affiliated with 76 categorical positions. Prior to its introduction in November 2020, our residency program had no formal POCUS curriculum. All residents participated in the new POCUS curriculum, but involvement in the study was voluntary. Inclusion criteria in the study were an active resident status in our program and the ability to attend at least one session. There were no exclusion criteria other than declining to participate. During the 2020 to 2021 and 2021 to 2022 academic years, residents participated in our simulation lab with 6 to 8 ultrasound machines available for sessions.
Intervention
The Wright-Patterson Medical Center Institutional Review Board deemed this study exempt research (Protocol #: FWP20200013E) involving human subjects in accordance with 32 CFR 219.104(d)(1) and (2) after performing a limited review of this study required by 32 CFR 219.111(a)(7). Written consent was obtained from all participants. We conducted a prospective, before-after cohort study from November 2020 to November 2021 (academic years 2020-2021 and 2021-2022). The curriculum was implemented on a continuous basis starting in November 2020 after precurriculum assessments were completed. Postcurriculum assessments were completed between 6 and 12 months after the implementation of the curriculum. This time frame was initially chosen as a 6-month assessment of the curriculum, then extended to 12 months to create more opportunities for residents to complete the postcurriculum assessment. The assessment closest to 12 months was chosen, and duplicates were not included.
Kern's 6 steps for curriculum development were used. 17 The identified problem was the absence of a formal POCUS curriculum. Our precurriculum instrument was used as a targeted needs assessment. The goal of the curriculum was to educate residents about the level of entrustable professional activity, or adequate competency for unsupervised execution, regarding basic POCUS techniques clinically useful to internal medicine physicians. 18 Specific objectives were to identify clinical conditions and pathologies that would benefit most from the use of POCUS (Supplement 1).
The POCUS curriculum was included as a longitudinal, asynchronous, flipped-classroom activity with 2 to 3-h workshop sessions during an academic half-day that recurs monthly. It replaced traditional didactic lectures on other topics in our preexisting academic half-day curriculum. Residents on certain rotations are exempt from academic half-days and attendance is not 100%, so participants vary from month to month. Residents were given a course outline and objectives at the beginning of the curriculum (Supplement 1). For preparation prior to a session, residents were given objectives, PowerPoints, and textbook chapters by Soni et al. 19 Sessions were split into a large-group image review followed by small-group (4-5 residents) image acquisition teaching and practice. At the beginning of each session, residents were encouraged to obtain important views from previous sessions to promote retention. Image review was an approximately 20- to 30-min analysis of pathologic images. Image acquisition small groups were given a list of recommended views for deliberate practice, with the freedom to self-direct according to comfort and skill level. This hands-on portion was allotted significantly more time to align with Kolb's model of experiential learning. 20 Three to 6 experienced supervisors (ie, pulmonary and critical care attendings, cardiology attendings and fellows, chief residents, and the resident curriculum developer) led small groups and provided direct and immediate feedback. The 4 POCUS workshop sessions were knobology and cardiology, pulmonary and examination to differentiate shock, abdominal and renal, and vascular and volume assessment.
Outcomes
Prior to and at the conclusion of the POCUS curriculum, participants completed an instrument that assessed confidence and utilization frequency. Confidence was self-reported on a Likert scale (1 = strongly disagree through 5 = strongly agree). The scale regarded residents’ perceptions of global understanding of POCUS, then asked perceptions of global understanding specified by individual POCUS exams (ie, “I have a good understanding of POCUS in regards to assessing left ventricular function”). Utilization frequency (self-reported use during clinical time) was self-reported on a 5-point scale: 1 = never, 2 = 1 to 2 times per month, 3 = 3 to 4 times per month, 4 = 5 to 6 times per month, and 5 = greater than 6 times per month. Additionally, 6 multiple-choice questions assessed interpretation of ultrasound videos. The number of questions was chosen to optimize the Ottawa criteria component of feasibility. 21 Video images were diverse: identification of decreased left ventricular ejection fraction, normal lung imaging, subcutaneous emphysema, cardiac tamponade, normal liver tissue with normal inferior vena cava collapsibility, and mild hydronephrosis. Recognition of these images was chosen at the recommendation of faculty after they were asked for essential skills in image interpretation, a. The images were open-access and obtained via https://www.thepocusatlas.com/ and https://www.coreultrasound.com/. Supplement 2 contains the assessment, with the images as static displays of the videos. Answers were categorized as correct or incorrect without any partial points, and image interpretation scores were calculated as a percentage of correct out of the 6 multiple-choice questions. The planned use of the assessment was to provide scores to trainees for their educational benefit. For these reasons, we did not perform a reliability analysis on our ultrasound interpretation assessment tool.
REDCap, a web application hosted at Wright State University, was used to collect and manage data.22,23 Pre- and postcurriculum assessments were linked by an anonymous identifier. Assessments without an exact matching identifier were excluded.
Statistical Analysis
Mean or median is reported for continuous variables. Changes from precurriculum to postcurriculum were analyzed with the related samples Wilcoxon signed rank test. Pearson r and R2 are reported for the relationship between POCUS sessions attended and POCUS image assessment score. Inferences were made at the 0.05 level of significance. Analyses were conducted using IBM SPSS Statistics 25.0 (IBM).
Results
A total of 55 precurriculum and 59 postcurriculum assessments were completed. After removing duplicates, incomplete assessments, and assessments without linking unique identifiers, 19 records had complete pre- and postcurriculum assessments available for analysis (19% response rate). Residents were 42% PGY-1, 37% PGY-2, and 21% PGY-3 and attended a median of 4 sessions over a mean of 220 days. Only 3 (16%) residents reported never clinically using POCUS prior to curriculum implementation.
Table 1 shows that participants increased their confidence by more than one scale point on 3 of 4 items. Especially notable is the gain from 2.47 to 3.53 (P = .002) on “I have a good understanding of point-of-care ultrasound.” The 2 utilization items increased by a third of a scale point and were not significant. The 11% gain from 42% to 53% in image interpretation score was nonsignificant. Figure 1 shows change in confidence and utilization from precurriculum to postcurriculum by POCUS exam. Confidence increased significantly for all 9 POCUS exams while self-reported utilization increased significantly in 2 of 9 exams. Table 2 reports the proportion of residents that started using POCUS to change clinical management after the curriculum. Notable exams that were increasingly utilized to change clinical management were the cardiac exam (pre—15.8%, post—73.7%; P < .001) and pulmonary/pleural exam (pre—31.6%, post—68.4%; P = .02). Nonsignificant increases were seen with volume assessment (pre—26.3%, post—36.8%; P = .49) and abdominal free fluid (pre—26.3%, post—47.4%; P = .18) exams. The percentage of residents that reported ever having changed their clinical management by POCUS increased from 47% to 84% (P = .02) after the implementation of the curriculum. The types of exams reported to change management are described in Table 2. Figure 2 shows the relationship between the number of POCUS sessions attended and the postcurriculum POCUS assessment score. Pearson correlation (r = .549) was significant at P = .018, and the coefficient of determination was 30%.
POCUS outcomes for confidence, utilization, and image interpretation: precurriculum versus postcurriculum.
Related-samples Wilcoxon signed rank test.
1 = strongly disagree, 5 = strongly agree.
1 = never, 2 = 1 to 2 times per month, 3 = 3 to 4 times per month, 4 = 5 to 6 times per month, and 5 = greater than 6 times per month.

Confidence and utilization pre- and post-curriculum implementation by POCUS exam. N = 19. Related-samples Wilcoxon signed rank test. Confidence: 1 = strongly disagree, 5 = strongly agree. Utilization: 1 = never, 5 = more than 6 times per month.
Change in pocus use (clinical management) for exam type after implementation of pocus curriculum.
Note: Percentages calculated using a denominator of residents never using each POCUS exam prior to the curriculum and a numerator of those residents reporting use of each POCUS exam after the curriculum. Never-users prior to curriculum implementation (denominators): cardiac—16, pulmonary/pleural—13, volume assessment—14, differentiation of shock—17, skin and soft tissue infection—17, renal—19, abdominal free fluid—14, vascular/DVT—19.

Relationship between number of sessions attended and post-curriculum image interpretation score.
Discussion
Our program evaluation produced findings that can be used to redesign future iterations of internal medicine, asynchronous, flipped-classroom POCUS curriculum. The precurriculum instrument found that our residents did not believe their informal clinical experience with POCUS was adequate nor were they pleased with their understanding of POCUS. Our formal curriculum increased resident confidence and self-reported changes in clinical management by POCUS. Residents’ attendance was not extensive, a median of 4 education sessions, equating to approximately 3 h of image interpretation and 5.5 h of image acquisition. Thus, internal medicine resident confidence may improve rapidly with a modest amount of formal curriculum. Lastly, we found that time-on-task (number of sessions attended) was moderately correlated with knowledge/skill acquisition (postcurriculum image interpretation assessment score). The effect of correlation may have been blunted by prior training and experience, as evidenced by the 84% already clinically using POCUS prior to the curriculum. Still, the demonstrated improvement in confidence and competency with this longitudinal curriculum agree with previously demonstrated longitudinal curricula. 10 Our findings also indicate that informal clinical experience in POCUS does not provide adequate education (participants on average disagreed that “My current experiences using point-of-care ultrasound during clinical practice provide adequate education” precurriculum) and emphasizes the importance of formal POCUS curriculum implementation into internal medicine programs. Our POCUS curriculum notably impacted clinical management for cardiac and pulmonary/pleural exam and to a lesser extent volume assessment and abdominal free fluid, suggesting that these exams should be emphasized in internal medicine curricula since they proved to be the most clinically useful.
Other specialties have reported beneficial results when implementing flipped-classroom POCUS curricula.24,25 The flipped-classroom approach to POCUS education has several advantages. First, the flipped-classroom allows more time for image interpretation and image acquisition practice. Second, the knowledge and skills of experts are optimized by assigning them to provide small-group instruction and feedback. Third, learners can choose from a number of formats for self-education and preparation for POCUS sessions, including PowerPoints, text/graphic handouts, and web-based instruction. Our experience suggested residents were inconsistent in completing presession, preparatory materials for our POCUS curriculum. Unfortunately, we were unable to quantitatively assess presession preparation. Residents may need to be provided with protected time to assure adequate preparation for POCUS sessions.
Our study had limitations. First, the study was conducted at a single internal medicine residency. Thus, generalizability to other residency programs and specialties should be done with caution. Second, we could not conduct a randomized controlled trial since the formal POCUS curriculum was part of the residency's educational activities. Consequently, it would have been unethical to exclude some residents from the experience. However, the pre-post evaluation design was an alternative that limited confounding by making each resident his/her own control. Our initial design also did not include an a priori sample size determination. Third, we did not directly assess resident demographics or prior training in the assessment, factors which may contribute to perceptions. Though, we are unaware of any residents in our program with formal POCUS training outside of the residency curriculum. Fourth, we used a locally constructed three-part instrument to assess confidence, utilization frequency, and ultrasound interpretation. A search of the literature did not identify assessment measures with substantial validity evidence that corresponded to our curriculum and that matched our objectives for evaluating the curriculum. Additionally, the best evidence-based POCUS assessments are logistically challenging, require image acquisitions, and are limited to single clinical areas (cardiac or pulmonary only). Thus, our assessment was not evaluated for validity or pretested. However, it represents a low-stakes, formative assessment intended to drive future learning. Our assessment did not evaluate image acquisition due to concerns that expanding the assessment tool to meaningfully assess this component of competency would further limit participation in the study. The assessment also does not delineate confidence into subcomponents of POCUS competency (ie, image acquisition, interpretation, etc). Fifth, the image interpretation assessment was set at an advanced difficulty level, which may have limited residents’ ability to improve their scores. Sixth, since the frequency of utilization and changes in clinical management were self-reported, these outcomes were susceptible to bias. Seventh, while our study did not assess how increased use of POCUS impacted patient outcomes, previous literature suggests that patient outcomes (eg, hospital length of stay) may be affected. 26 Future research should explore the influence of formal POCUS curricula on patient outcomes.
Finally, with 76 residents in our residency program, we had hoped for enough voluntary participants for a medium effect size (ES) to be statistically significant. A medium ES is .50 as calculated with Cohen's formula: d = mean difference for 2 groups/pooled standard deviation. 27 With only 19 participants 2 of the 4 outcomes with a medium ES were not statistically significant (“POCUS is an important skill for practicing IM physicians” and image interpretation assessment score). If these 2 outcomes were to maintain the same mean and standard deviation with 31 participants, both would have been P < .05. Thus, for adequate statistical power, future educational studies of POCUS should enroll at least 35 participants for paired-comparison studies and 70 participants for studies of 2 independent groups (ie, 35 per group).
Conclusion
Implementation of a longitudinal, flipped-classroom, academic half-day curriculum for POCUS resulted in greater confidence, increased POCUS utilization for cardiac and pulmonary examinations, and changes in clinical management based on POCUS.
Supplemental Material
sj-docx-1-mde-10.1177_23821205231193284 - Supplemental material for Implementation and Evaluation of a Longitudinal Flipped-Classroom Point-of-Care Ultrasound Curriculum at an Internal Medicine Residency Program
Supplemental material, sj-docx-1-mde-10.1177_23821205231193284 for Implementation and Evaluation of a Longitudinal Flipped-Classroom Point-of-Care Ultrasound Curriculum at an Internal Medicine Residency Program by Brian P Elliott, Andrew J Berglund, Ronald J Markert and Kathryn M Burtson in Journal of Medical Education and Curricular Development
Supplemental Material
sj-pdf-2-mde-10.1177_23821205231193284 - Supplemental material for Implementation and Evaluation of a Longitudinal Flipped-Classroom Point-of-Care Ultrasound Curriculum at an Internal Medicine Residency Program
Supplemental material, sj-pdf-2-mde-10.1177_23821205231193284 for Implementation and Evaluation of a Longitudinal Flipped-Classroom Point-of-Care Ultrasound Curriculum at an Internal Medicine Residency Program by Brian P Elliott, Andrew J Berglund, Ronald J Markert and Kathryn M Burtson in Journal of Medical Education and Curricular Development
Footnotes
Acknowledgments
The authors appreciate the support of residents and faculty that participated in our curriculum.
DECLARATION OF CONFLICTING INTERESTS
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
FUNDING
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Disclaimer
The views expressed in this paper are those of the authors and do not necessarily represent the official position or policy of the Department of Defense or United States Air Force.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
