Abstract
INTRODUCTION
Physicians are looked upon to lead the healthcare team, a task that has grown increasingly complex and interdisciplinary, requiring a diverse extra-clinical skillset. Physician Executive Leadership (PEL) Plus is a student-run program that uniquely utilizes both didactic and real-world project-based approaches to deliver a business and management curriculum to medical students.
METHODS
We developed and implemented PEL Plus during the 2018-19 and 2019-20 academic years, geared at first- and second-year medical students. We provide an overview of this combined didactic and project-based curriculum, in addition to evaluating if the program was efficacious in teaching the desired skillset. We assessed short-term knowledge acquisition using multiple-choice questions, and investigated student perceptions of their learning and the program using Likert scales and narrative feedback. We also investigated the influence of student demographics on performance, in order to assess the appropriateness of our target audience.
RESULTS
28 students completed PEL Plus over the two years (14 students/year). Average performance on multiple-choice questions showed statistically significant improvement after the majority of sessions. There were no statistically significant effects of demographics on performance in the majority of sessions. Students self-rated stronger understandings of lecture topics after each session, and analysis of narrative feedback demonstrated thematic categories centred on teaching style, new knowledge, lecture content/material, projects, networking, program structure, and generic statements.
DISCUSSION
PEL Plus is an innovative and effective approach to teaching business, leadership, and management skills in undergraduate medical education. Development of similar programs at other institutions will positively impact the broader medical student community.
Keywords
Introduction
Physicians are often looked upon to lead the healthcare team, a task that has grown increasingly complex and interdisciplinary. However, due to the lack of consistent management education in the medical curriculum, many physicians are underequipped to take up these leadership roles, and to address issues in their medical practices.1,2 Issues may include managing conflicts, improving and providing care to the underserved, and implementing cost-effective and value-driven management of medical and surgical departments.3,4 Careful introduction of business, leadership, and management skills into the medical student repertoire will help to generate the next generation of physician leaders and innovators, which has the potential to ultimately improve patient care.
Hands-on application-based management curricula for medical students are scarce, 5 and include both student and faculty driven programs.1,6‐9 In 2020, the Association of American Medical Colleges reported only 1% of graduates completing an MD/MBA program. 10 The additional tuition and time spent in school likely contributes to this low adoption rate of combined formal management training programs. 11 Furthermore, patient care and the sciences must be prioritized and remain central to medical education, therefore careful consideration must be taken when developing a successful management curriculum. Founded in 2013, the Physician Executive Leadership (PEL) is a student-driven organization at Sidney Kimmel Medical College (SKMC) that has successfully developed programming to teach the extra-clinical foundations of medicine to future physician leaders and innovators. PEL has designed a curriculum for medical students based on six core pillars: Entrepreneurship & Innovation, Applied Leadership, Health Finance, Care Quality & Experience, Law & Ethics, and Health Policy. PEL's strong reputation among students and faculty has provided a robust platform on which to further develop a high-quality management curriculum in order to fill the perceived gap in medical education described above.
Accordingly, in 2018 we established PEL Plus, an integrated business and management program for first- and second-year medical students (MS1 and MS2, respectively) who have interests in healthcare innovation and leadership. Leaders from across the Jefferson enterprise were brought together to curate a combined didactic and project-based curriculum over the course of four months. Students utilized the session teachings to develop their summative projects and present a final pitch to the Jefferson faculty with their solution to a healthcare problem. Herein, we put forth two years of preliminary quantitative and qualitative user-based data on PEL Plus, highlighting the value of a combined didactic and project-based management curriculum for medical students’ extra-clinical skill development.
Methods
Establishing PEL Plus
In 2018, we established PEL Plus as a supplement to the existing PEL curriculum, for students interested in a deeper dive into business leadership. The course comprised of (1) educational sessions and (2) real-world healthcare-based projects. In contrast to the already-established PEL curriculum, which runs over the entirety of medical school, PEL Plus provides a condensed course to be completed by students over a four-month period (October – January) during their preclinical years. Since a guiding principle of PEL Plus was to be rigorous yet fit within the busy medical student schedule, we deemed it unfeasible to conduct the program during the demanding clinical years.
Recruiting speakers and project sponsors
We recruited speakers who were experts in their field and had experience teaching the planned educational topics. We pursued a networking strategy of reaching out to experts who were either connected to or employed by Thomas Jefferson University (TJU). There were no costs associated with inviting speakers due to their affiliation with TJU. We met with these experts to discuss the goals of PEL Plus, the material to be conveyed, and session dates. We highly prioritized student participation in the planning process, and as such chose session dates that did not conflict with exam weeks when students would be preoccupied. Topics planned for the first iteration of the course (2018-19 academic year) included Business Models I & II (BM1 & BM2), Design Thinking I & II (DT1 & DT2), Systems Thinking (ST), Leading Teams (LT), Managerial Decision Making (MDM), and Managing & Analyzing Data (MAD). The second year (2019-20 academic year) included BM1, BM2, DT1, DT2, ST, Building & Leading Teams (BLT), and Finance & Accounting Principles (FAP). These topics are listed in Appendix A.
We also employed a networking strategy through Jefferson Health's “Innovation Pillar” in order to recruit individuals who could provide and facilitate student projects. Similar departments/groups at other institutions may be a good place to start if a program similar to PEL Plus were to be replicated elsewhere. Once sponsors with suitable projects agreed to join our program, we articulated the expectations for them to be in touch with students and provide mentorship (eg, answering questions, providing relevant data, referring to relevant stakeholders).
It was critical to complete these processes prior to the start of the academic year to ensure a clear schedule of events and expectations for the expert speakers and project sponsors alike.
Recruiting students
We marketed PEL Plus using student social media groups and email during the start of the fall semester, in addition to holding an information session about the program. Subsequently, an online application via Google Forms was made available to MS1 and MS2 students on their respective class social media pages. Students applied by answering the following questions: (1) As the healthcare space continues to evolve, how do you see your role after completing your medical training? (1-2 paragraphs); and (2) Any other thoughts? (Optional). We accepted all individuals who expressed a genuine interest and had a vision for their future role as a physician leader. Otherwise, there were no prerequisite requirements for acceptance. A total of 14 MS1 and MS2 students were accepted to the program in each of 2018-19 and 2019-20, for a total of 28 students overall. Participants’ demographic information can be found in Table 1.
Student demographics.
Abbreviations: Medical Student Year 1 (MS1), Medical Student Year 2 (MS2)
This is the maximum number of respondents possible (28). It reflects the total number of participants who joined the program across both academic years, with 14 students participating each year.
One student with a science degree double-majored in economics. Their economics experience is captured in the business coursework section.
No student completed an MBA.
Participants were allowed multiple selections. Therefore, the number of responses across all options does not add up to the total number of respondents for the “Career Goals” question.
Execution of each session
To execute each session, it is necessary to have audio-visual (AV) capabilities and a suitable lecture space with a white/blackboard. We achieved this by coordinating with TJU registrar's office. We provided students with the list of topics to be discussed during the course, and each speaker was reminded approximately two weeks prior to their session to submit three multiple-choice questions regarding the content of their session. We compiled these questions into identical pre- and post-surveys to assess students’ short-term retention of the material. We also confirmed the general session plan with the speaker prior to the date. Of note, a formal syllabus with objectives, formal assessments, and reading requirements were made to be minimal (some resources were suggested throughout the course, but not required) because the goal of the program was not to impose an untoward workload onto students, which would diminish enthusiasm for this extracurricular program.
Each session was allotted three hours on one evening each week, and we encouraged speakers to fill the entire time. Speakers filled the time at their discretion with lecture, discussion, or other activities, though we did encourage speakers to be as interactive as possible with real-world examples and application-based exercises. Before beginning each session, the PEL Plus coordinator made announcements and administered the pre-survey link to the students. At the end of the session, the coordinator administered the post-survey, which contained identical questions as the pre-survey, in addition to three other questions. Appendix B highlights the questions asked in these surveys, where questions denoted by an asterisk were only present in the post-survey. We used these surveys to gauge student knowledge acquisition and obtain feedback, in addition to tracking attendance. We made clear that students could only miss one session in order to obtain recognition for completing PEL Plus. In this study, attendance at each session was ascertained for a given student when at least one of the pre- or post-surveys was filled out.
Introductory session and project selection
The introductory session was slightly different from the other sessions as it included brief presentations by the project sponsors. During this session, the project sponsors were invited to speak to the students for approximately 30 min about the various projects. After this, the session proceeded to follow the traditional session structure with the invited speaker. After this session, we sent an email to students, asking them to rank the projects in their order of preference. We then matched students to their first choice with a mixture of MS1s and MS2s in each group. Of note, particular focus was made to ensure that the first choice was attained to bolster student enthusiasm, and duplicate projects were permitted. Each group contained 3 to 4 students.
After the groups were established and matched with a project, we sent an introductory email to connect students with their project sponsor. Students were then directed to set up a meeting with their project sponsor to further scope out their deliverables for the final session of PEL Plus. Students were also advised to continually follow-up with their project sponsors to ensure they were staying on track.
Project objectives
The goal for each project was to provide students with an opportunity to utilize teamwork and problem-solving skills in order to identify a solution to a real-world healthcare problem. Students were advised to rely on session teachings to help frame their approach and final presentation. Students were additionally advised to identify and engage with relevant clinical, administrative, business, and academic stakeholders using their own and their sponsor's networks.
In their final presentations, each student group provided: (1) background information for their project (2) analysis done to solve the healthcare problem (3) conclusions and recommendations. Students sent their presentations to the PEL Plus coordinator several days in advance of the final presentation session, in order to ensure they would be uploaded appropriately to the AV equipment. The list of projects is presented in Appendix A.
Final presentation session
The final presentation session was conducted in late January/early February. Of note, this session occurred prior to MS2 students’ USMLE Step 1 dedicated study time, which typically begins in February-March at SKMC. This was done so that MS2 students could focus on PEL Plus without sacrificing other core aspects of their medical education. For this session, we invited individuals from the academic and business arms of the Jefferson enterprise, including, but not limited to, stakeholders within the project sponsors’ departments, SKMC academic deans, and Jefferson leadership. Each presentation was allotted 30 min, divided into 15 min for the presentation and 15 min for Q&A.
Summative feedback and demographic information
At the end of each academic year, we administered a summative feedback questionnaire to our students using Google Forms. We also asked participants for demographic information. These questions are in Appendix B.
Materials
The materials required for conducting PEL Plus are detailed in the above sections, with the core components consisting of facilities and human capital. To summarize, what is minimally required includes:
Projector for PowerPoint or other visual media Lecture space with AV capabilities and white/blackboard Subject-matter experts Project sponsors providing real-world projects Online survey software (eg, Google Forms, Qualtrics)
Evaluating the efficacy of PEL Plus
To assess the efficacy of PEL Plus in disseminating knowledge, and students’ short-term knowledge acquisition, we used quantitative multiple-choice questions. These questions were designed by each lecturer, focused on the lecture's content, and contained a single correct answer. To analyze these data, the fraction of questions answered correctly before versus after the session were compared for each session using a one-way analysis of variance (ANOVA) with planned comparisons and Bonferroni's post-hoc test. The same statistical approach was taken to assess for differences in performance when sessions were grouped according to the PEL pillars. The various PEL pillars (core competencies forming the basis of the PEL organization) are shown in Appendix C, along with the pillar under which each session fell. Four of the six overarching PEL pillars contained relevant sessions: Entrepreneurship & Innovation, Applied Leadership, Care Quality & Experience, and Health Finance pillars. We also averaged all sessions and used a parametric two-tailed t-test to assess for differences between pre- and post-session performance overall. Of note, where an individual failed to fill out both pre- and post-session questions, their answers were excluded from analysis. We also collected demographic data on participants, including prior educational and gap-year experiences, and investigated if these experiences influenced their performance on questions using an analysis of covariance (ANCOVA). This was done in order to ensure that PEL Plus was appropriately geared toward our audience.
We investigated students’ self-perceptions of their base knowledge level prior to each session, and how they felt their understanding of the topics changed after each session. We also asked students to evaluate the educational value of each session and the program as a whole. To perform these assessments, within the pre- and post-surveys we administered 5-point Likert scale questions where students could rate their level of agreement or disagreement with each statement on a scale of 1 to 5 (“1=strongly disagree” and “5=strongly agree”). Qualitative data were purely narrative free-text responses obtained after each session and at the culmination of each academic year. Two authors first independently and then collaboratively, coded the narrative comments into positive or negative/constructive feedback, and then developed thematic categories into which the feedback was organized. This was done to identify the main strengths and weaknesses of the program.
When an identical session ran in both academic years, the data from those sessions were combined across years (BM1, BM2, DT1, DT2). When sessions across the two years were similar in topic but contained unique questions, the data from these sessions were not combined. Statistical analyses were conducted using SPSS Statistics 1.0.0.1508 (IBM Corp, Armonk, NY) and Prism 6 (Graphpad Software, San Diego, CA).
Results
A total of 28 MS1 and MS2 students participated in PEL Plus over the 2018-19 and 2019-20 academic years, with 14 students in each year. In the first iteration of the course, attendance at each of the 8 sessions ranged from 12 to 14 of 14 students (mode and median = 13, 85.7-100.0%), and in the second iteration attendance at each of the 7 sessions ranged from 8 to 14 of 14 students (mode and median = 11, 57.1-100.0%).
To evaluate if students were indeed learning the content taught in lectures, we assessed their performance on knowledge-based multiple-choice questions, and compared their performance before and after the session. The use of pre- and post-testing to evaluate knowledge acquisition satisfied Level 2 (Learning) of Kirkpatrick's four-level model of evaluation. Average performance on questions showed statistically significant increases after all individual sessions (P < .05), except for DT1 (P = .06) and FAP (P = .07). When sessions were grouped by pillar and when all sessions were combined there were also statistically significant improvements in performance (P ≤ .01). The mean scores with standard deviations (SD), P-values, and absolute and % improvements in performance are presented in Table 2.
Performance on multiple-choice questions by session, pillar, and overall.
Abbreviations: Standard Deviation (SD)
Includes students across both academic years
While this session ran in both academic years, there was a malfunction of data collection at the start of this session in 2019-20 and so an analysis of performance on knowledge-based multiple-choice questions was unable to be performed for students in that year.
Using an ANCOVA, we assessed if differential experiences prior to medical school (time off vs. no time off between undergraduate education and medical school) or prior exposure to business coursework impacted our students’ abilities to learn from our sessions. The analysis compared post-session performance between groups while adjusting for pre-session scores. In all sessions there were no statistically significant differences in performance by students who took time off between their undergraduate degree and medical school compared to students who did not take time off. In the majority of sessions, exposure to prior business coursework had no statistically significant effect on performance, except in the FAP session, where students with prior coursework exposure performed statistically significantly better than those without (P = .02; Same as the Health Finance pillar). When all sessions were combined, there was no statistically significant difference in performance regardless of experiences prior to medical school (P = .72) or prior exposure to business coursework (P = .75). Results are presented in Figure 1, with all values tabulated in Appendix D.
Performance on questions by session, pillar, and overall, as a factor of prior experiences and business coursework exposure.
The surveys allowed students to provide self-reflections and program evaluations, which satisfied Level 1 (Reaction) of Kirkpatrick's four-level model of evaluation. Students rated stronger understandings of the session topics after all sessions compared to before. They also rated that the sessions increased their understandings of topics at ≥4 for all sessions except for MAD and FAP, which received average ratings of 3.42 and 3.88, respectively. All sessions were rated at ≥4 with respect to being valuable and interesting, except for MAD and FAP, which received average ratings of 3.42 and 3.75, respectively. On summative year-end feedback, students provided an average rating of ≥3.83 for all questions. All ratings are presented in Table 3.
Individual session and summative Likert-scale ratings.
Participants could select any whole number from 1 to 5 when answering each question presented in Table 3, where 1=strongly disagree with the statement, and 5=strongly agree with the statement.
These data contain responses from students across both academic years.
We asked participants for narrative free-text feedback on each session, with a response rate ranging from 21.4 to 36.0% across all sessions. We then generated seven thematic categories that encompassed the feedback. “Speaker & Teaching Style” focused on the level of engagement, use of activities and examples, the speaker's attitude and expertise, and time management. “New Knowledge” focused on learning new material, especially content outside the mainstream medical curriculum, and if this knowledge would be applicable in the participants’ futures. “Lecture Content/Material” focused on what was taught, whether or not content was appropriately targeted to the audience, the diversity of material, and if the content was interesting and informative. “Projects” focused on the scope of the projects, communication, guidance and expectations, and the interplay between lectures and projects. “Generic Statements” were simple (eg “great session”) and did not fit into any of the above themes. The summative narrative feedback also included “Networking”, in which students emphasized the opportunities they had to work with leaders and like-minded peers from diverse backgrounds, as well as “Program Structure”, which focused on the project-based nature of the program, lecture format, the program timeline, time commitment, communication, resources, and collaboration with other programs. A total of 23 students (82.1%) responded to the summative survey. Representative quotes in each thematic category are presented in Table 4.
Representative narrative free-text feedback according to thematic category.
The project sponsors included personnel from across the Jefferson network, including supply chain, clinical performance, and departmental leadership, as well as physicians. Projects took various forms, focusing on ways to optimize and implement leaner healthcare practices. As part of their solutions, students networked with relevant stakeholders, performed team-based research, and aggregated socioeconomic, financial, and clinical data. A list of the project titles can be found in Appendix A. Although there was no formal assessment to evaluate students’ performance on their projects, students presented a formal pitch to deans and stakeholders at the end of the year. This allowed students to demonstrate their application of the new knowledge learned in PEL Plus to real-world healthcare problems using teamwork and problem-solving skills.
Discussion
It is increasingly being acknowledged that medical professionals require a diverse skillset outside of clinical medicine in order to optimize workplace efficiencies and patient care.3,12,13 PEL Plus is one of few initiatives in North America delivering a combined didactic and hands-on business and management curriculum targeted specifically at medical students.1,5–9,14,15 Application-based learning is an effective way to enhance leadership skills,1,9,16,17 therefore this report highlights the utility for broader adoption of programs similar to PEL Plus.
In developing the program, a major hurdle was identifying project sponsors who would be able to provide sustained mentorship. Since establishing these connections, we now have a diverse network and strong relationships with these mentors such that we are able easily connect students with project sponsors and broaden the scope of project topics. Having run multiple iterations of the course, we have also identified the lecturers who have been well-received, making it easier to plan the course going forward. We also found that the program timeline, which avoided key testing dates within the mainstream medical curriculum, allowed students to focus their efforts on PEL Plus without becoming overwhelmed.
We collected quantitative data on student performance and observed statistically significant score improvements in the majority of domains, suggesting acquisition of knowledge through our program. In all but one instance, the ability to learn from our program was neither contingent upon participants’ experiences prior to medical school nor on prior exposure to business coursework, suggesting that our program is appropriately tailored to a diverse medical student audience. In the FAP session, students with prior business coursework performed better, suggesting they had an easier time learning the material. Though the sample size for this particular comparison was small, future students might nonetheless benefit from adjusting the complexity level of this lesson to fit a broader audience. Through 5-point Likert scale questions, students self-rated stronger understandings of topics, and they believed the sessions and overall program were valuable to their education. The MAD and FAP sessions each received average ratings in the 3-point range, suggesting that these sessions should be a focus for improvement. By thematically categorizing qualitative narrative feedback, we highlighted the key areas of our program that were important to students, along with ways to enhance educational value. Some examples for improvement included expanding the scope of project topics, ensuring appropriate communication and expectations for the projects, and ensuring that lectures emphasized interactive, healthcare-focused activities and application of skills.
Limitations of our study include the small sample size and the lack of a control group containing students not enrolled in the program. In evaluating knowledge acquisition, we focused on short-term knowledge retention which may not accurately represent long-term learning. Surveys only contained three knowledge-based multiple-choice questions, and future iterations of our assessments may benefit from additional questions to enhance validity. Program participation was voluntary and required an application, therefore participants were more likely to be self-motivated to learn the material, and selection bias was likely an influencing factor. In its current form, PEL Plus will remain feasible as long as there are interested students. However, now that the groundwork for the course has been laid and connections have been established with speakers and project sponsors, we believe it could be possible to integrate our course into the mainstream medical curriculum as a component of the Health Systems Science (HSS) competency and expand to all MS1 and MS2 students.
In the future it will be valuable to investigate knowledge retention beyond immediate post-didactic recall. For example, brief sets of questions could be administered prior to the start of each subsequent session to evaluate if long-term knowledge acquisition is occurring. Assessing knowledge attainment after conclusion of the course could be an additional route to investigating long-term knowledge retention, though this may prove challenging due to follow-up difficulties with students who are amidst their upper-year clinical training. One possible way to move beyond the first two levels of Kirkatrick's model and evaluate the third level (Behaviour), which involves implementation and transfer of skills, could be to facilitate students’ long-term engagement with their projects beyond conclusion of the course. This would be a practical way to understand the program's reach and impact on real-world healthcare outcomes, and to assess if the solutions developed by students are indeed being put into practice. As our alumni are beginning to graduate medical school, it will also be useful to follow-up with their career trajectories and evaluate if their roles are requiring the use of the core business and managements skills taught in PEL Plus. Finally, it will be interesting to assess the effect of online teaching modalities during the COVID-19 pandemic (2020-21 academic year) on knowledge and skill attainment.
In this report we presented two years of preliminary data, illustrating that PEL Plus is an innovative and feasible approach to expanding extra-clinical business and management skills early in medical training. With the role of the physician becoming ever more complex and requiring a diverse set of skills beyond those required solely for individual clinical practice, programs like PEL Plus have the opportunity to provide a skillset that will be important for future clinicians to utilize. These skills will allow clinicians to engage in effective team-based leadership with an understanding of systems-level healthcare issues when administering value-based and cost-effective quality care. PEL Plus exposes medical students to these aspects of team leadership, information management, entrepreneurship, innovation, systems and design thinking, and business and finance, which will prepare them for interdisciplinary careers in the changing healthcare field.
Footnotes
Acknowledgements
The authors wish to thank Dr. Rosie Frasso, Professor at the Jefferson College of Population Health, for her sound advice.
Declaration of Conflicting Interests
The authors declare that there are no conflicts of interest.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors
Ethical Approval
The study was exempt on December 3, 2020 by Thomas Jefferson's Office of Human Research Institutional Review Board (IRB) as it fell under the category of “research on the effectiveness of or the comparison among instructional techniques, curricula, or classroom management methods” (Reference #20E.1206).