Abstract
Background
Patient safety, quality improvement, and health equity training in graduate medical education are important. However, effective curriculum development in these complex domains remains challenging.
Objective
To evaluate the impact of EQuIPS (Equity, Quality Improvement, and Patient Safety), a mandatory, longitudinal curriculum for family medicine residents using Kirkpatrick Model of evaluation.
Method
This study employed a quasi-experimental design across three phases (pre-EQuIPS, during EQuIPS rollout, post-EQuIPS) with retrospective data collection. We analyzed data from residents graduating from academic years 2016 through 2024 where EQuIPS was implemented a multifaceted bundled intervention. The outcomes were measured using Kirkpatrick Levels 2, 3, and 4 defined as completion of certification, reported participation in projects, and leadership of projects demonstrated by presentation or publication. Demographics between the three groups were evaluated using the Chi-squared test for comparability. Statistical Process Control (SPC) charts were used for 3-phase analysis, while Kruskal–Wallis, Wilcoxon rank sum tests, and Poisson regression analysis were applied to draw inferences of association.
Results
A total of 51 residents graduated in the study period. The three groups were comparable with regards to gender or race while medical school background was significant (P < .001). SPC phase analysis revealed a process shift in resident engagement in interdisciplinary EQuIPS projects (Kirkpatrick Level 3) and completion of projects presented nationally or published (Kirkpatrick Level 4) post-implementation. The Kruskal–Wallis test indicated a significant difference in scholarly research involvement across the three phases (P < .001). Pairwise comparisons >10-fold increase in scholarly activity from pre- to post-implementation (P < .001) confirmed with Poisson regression after controlling for medical school.
Conclusion
The EQuIPS curriculum was associated with enhanced resident competency in equity, quality improvement, and patient safety. This model is a potentially valuable framework for other residency programs aiming to meet accreditation requirements and improvement and promote continuous improvement.
Keywords
Introduction
The Accreditation Council for Graduate Medical Education (ACGME) requires training in Equity, Quality Improvement, and Patient Safety (EQuIPS) in residency and fellowship programs. The actual requirements are interspersed in all the milestones of all six ACGME core competencies. Equity, safe and quality health care underpin the Quadruple aim, yet medical errors account for 250 000 to 400 000 U.S. deaths annually, making them third leading cause of death.1,2 Quality of care and equity are twins; one is not achieved without the other. Health equity has long been a central focus of public policy and public health, for example, Healthy People 2020 initiative. However, a study examining trends from 1993 to 2017 found minimal progress in advancing health equity in the United States over the past 25 years.3,4 Studies suggest that U.S. patients receive approximately only about half of the recommended care, despite the healthcare system being the most expensive globally. 5 The 1999 report To Err is Human 6 brought attention to medical errors prevalence and their impact on healthcare outcomes, which shifted the focus from individual blame to systemic issues. The COVID-19 pandemic further emphasized the importance of ethical decision-making and communication, yet formal training in these areas remains limited. 7 Despite recognizing leadership as a key competency, studies show that many faculty and students feel it's not adequately taught to future physicians. 8
Studies by Stevens et al 9 and Warm et al 10 demonstrated that residents exposed to QI education during their training delivered higher-quality care in their future practice. While studies like this highlight the value of integrating QI education into residency programs, there remains a notable gap in the literature that reports measurable outcomes related to equity, QI, and patient safety within graduate medical education (GME) curricula, especially using the Kirkpatrick model of evaluation.
Quality improvement in healthcare is driven by systematic, data-based strategies supported by organizations like ACGME, which mandates that residency programs teach QI through its Practice-Based Learning and Improvement (PBLI) competency. Additional support comes from groups like Agency for Healthcare Research & Quality (AHRQ) and Institute for Healthcare Improvement (IHI), which provide research, tools, and training to enhance patient safety and care quality. Despite these resources, there are many challenges in developing an effective training program that address the complexities of patient safety and quality improvement. The healthcare system is always evolving and its quality is affected by many different factors including social determinants of health (SDOH), care coordination and the application of QI tools across diverse settings. Residency programs, especially community programs with limited time and resources, may struggle to provide comprehensive QI education while balancing other aspects of residency training.
Additionally, fear of retribution, including legal consequences or criticism from colleagues, may discourage residents from reporting mistakes, ultimately limiting the opportunity for learning and improvement. 11
To overcome these barriers, medical educators must design improved curricula that empower residents with the skills to improve patient safety. Shorter, more focused curricula, with straightforward methods for assessing residents’ progress, can be an effective way to teach practical QI skills. 12
We introduced and implemented an innovative, longitudinal and integrated curriculum on EQuIPS. It is focused on teaching residents the core concepts and skills necessary to address healthcare disparities, enhance quality of care, and participate in patient safety work. By equipping residents with knowledge and tools to navigate the complexities of healthcare systems, programs like EQuIPS are aimed at improving both patient outcomes and the overall healthcare experience. This study reports quantitative analysis of the outcomes of EQuIPS using the Kirkpatrick model in a GME program.
Methods
Study Design and Study Setting
This is a quasi-experimental study with a pre and post design to evaluate the outcomes of a mandatory longitudinal curriculum—EQuIPS—for family medicine residents at WellSpan Good Samaritan Hospital. For this study, data was collected retrospectively from the existing residency database as part of a natural experiment that happened with the implementation of EQuIPS curriculum. Implementation of EQuIPS was part of a larger educational endeavor on increasing the residents’ scholarly activity while aligning with the institutional mission, vision, and priorities in a community hospital residency program. All residents graduating between June 2016 through June 2024 were included in the study as data were being collected retrospectively. The EQuIPS was implemented as a multifaceted bundled intervention with a roll-out period whereby the graduates of years 2019 and 2020 were partially exposed while the graduates of years 2021 through 2024 fully completed the curriculum. In this study, graduates of years 2016, 2017, and 2018 were considered pre-implementation. WellSpan Good Samaritan Hospital Family Medicine Residency is a medium size program for the discipline of family medicine in the United States. At the time of the study, it had accreditation for training 21 residents with 7 residents per year. The other curricular endeavors were reported in two other recent publications by Waheed et al. 13 This study followed the SQUIRE 2.0 Guidelines for quality assessment and improvement results. The WellSpan Health Institutional Review Board (IRB) determined it to be “Not Human Subjects Research” via study notification 1887432-2 since only de-identified aggregate/group data was collected from the existing database for educational and quality improvement purpose.
The Curriculum as Intervention
Grounded in ACGME milestones, the curriculum requires mandatory Quality Improvement reading modules, a Personal Wellness Quality Improvement (PWQI) project and IHI Patient certificate to teach residents essential skills for delivering equitable care, lead quality improvement initiatives, and improve patient safety. The changes in scholarly involvement after EQuIPS rollout indicated the impact of the curriculum on resident behavior and leadership. Table 1 shows the content area with learning objectives for the EQuIPS curriculum.
EQuIPS (Equity, Quality Improvement/Processes, and Patient Safety) Longitudinal Curriculum With Their Learning Objectives.
The quality improvement project training was packaged into a 13-step structured roadmap for scholarly activity with previously proven successful implementation,13–16 A summary of the alignment of the 13-step structured roadmap with Quality Improvement projects is shown in Figure 1. Kern's model for curriculum development was used to align new and existing elements of this curriculum in overarching training of family medicine. It is shown in Figure 2 as a schematic overview of the curriculum development process with the six steps of the Kern's cycle.
The Alignment of Quality Improvement and Patient Safety Projects With the 13-Step Structured Roadmap for Scholarly Activity.
Schematic overview of the curriculum development process with the six steps of Kern's Cycle.
This curriculum was rolled out as multifaceted bundled intervention. The implementation of this bundle was also part of the personnel requirements reported previously by Waheed et al. 13
Data Collection
This study evaluated three resident cohorts in chronological order based on rollout and implementation of the curriculum. Those who graduated: (a) before EQuIPS implementation (2016-2018 = “pre-EQuIPS”), (b) during its rollout (2019-2020 = “Rollout Phase”), and (c) after full EQuIPS implementation (2021-2024 = “post-EQuIPS”). Baseline demographics, including race (Asian, White, African American, Latino, Other), gender (male, female), and medical school background (LCME accredited, osteopathic, international), were compared across phases using the Chi-squared test. Meaningful resident engagement and accomplishment was assessed by evaluating the extent of scholarly output via Kirkpatrick levels. The Kirkpatrick model is a recognized framework for evaluating curriculum outcomes (11). Engagement and accomplishment was assessed using the Kirkpatrick evaluation model, focusing on Level 2 (demonstration of theoretical knowledge through completion of Institute for Healthcare (IHI) Patient Safety certificates, required QI modules, and personal wellness QI projects), Level 3 (behavioral change through participation in clinical/system-based QI projects, RCA Square activities, and M&M presentations), and Level 4 (project leadership and dissemination through regional/national presentations or peer-reviewed publications). Detailed definitions for each Kirkpatrick level are provided in Table 2. Kirkpatrick Level 1 was not included in our data collection given that it measures only reaction to the curriculum. The evaluation of the curriculum by the residents is optional for accreditation. By the design there would be no data for pre-EQuIPS and minimal for Rollout Phase for Kirkpatrick Level 1, there would be no meaningful comparison between the groups. It would not have added any more value to this study. Kirkpatrick Level 2 evaluates knowledge component. For this study, granular data on individuals was not collected. Rather the outcome was measured as dichotomous variable of completion or no-completion of IHI certificate, required QI modules, and personal wellness QI project. Our analysis focused on Kirkpatrick Levels 2, 3, and 4, while Level 1 was excluded since it reports reaction to the curriculum and done by all residents for all curricula. The data used for this study was already collected in the residency management software New Innovations. It was collected anonymously without any identifying information as count data. Although the data was presented to the Clinical Competency Committee for every resident per standard procedures, achievement of different Kirkpatrick levels did not need any additional judgement of an evaluator or score by an evaluator. Residents’ accomplishments were validated in the residency management suite, New Innovations, by the supervising attending or the program coordinator.
Outcomes of the Curriculum Evaluation on Kirkpatrick Model.
P-value calculated using Poisson regression analysis.
^ IHI certification requires post-test for the modules indicating the development of theoretical knowledge of the concepts. There was no certification before the implementation of the EQuIPS, only partial during the implementation, and 100% post-implementation.
* Statistical significance (P < .05).
Statistical Analysis
A combination of statistical methods was employed to evaluate the impact of the EQuIPS curriculum on resident engagement. Chi-squared tests were applied to compare baseline demographic variables (race, gender, and medical school background) collected as numerical data across three phases. Kirkpatrick Level 4 outcomes (scholarly dissemination) were analyzed using quantitative methods via Statistical Process Control (SPC) phase analysis. Since this study used a quasi-experimental study employing quality improvement approach, there was no formal sample size calculations done as a prior. A Shewhart U-control chart was constructed for the 3-phase analysis using JMP Pro 19.0 for SAS to assess the temporal trends of the curriculum as a multifaceted bundled intervention with a process improvement lens. The Kruskal–Wallis test was used to evaluate non-parametric outcome distributions to assess global differences in scholarly involvement in quality improvement and patient safety projects across the three curriculum phases (pre-, during, and post-implementation). Pairwise phase comparisons were performed using Wilcoxon rank sum tests with Bonferroni correction to control for multiple comparisons. Finally, Poisson regression was used to model the relationship between the implementation phase and scholarly output as count data. JMP Pro 19.0 for SAS was used to model Poisson regression analysis. All analyses were conducted using RStudio and Jamovi, with statistical significance set at P < .05. By using multiple methods in combination for this quasi-experimental design, we were able to comprehensively evaluate both temporal trends from process improvement lens and traditional statistical associations.
Results
Descriptive demographic analysis showed 14 of total residents in the pre-implementation, 12 residents in the Rollout phase, and 25 residents in the post-implementation phase. The Demographic analysis between the three groups was done by using the Chi-Squared test. There were no significant differences between the three groups with regards to Gender and Race among the three groups as shown in Table 3. There were significantly more graduates from International Medical Schools in the post-implementation group as compared to the other two while more Osteopathic graduates were observed in the pre-implementation group (P < .001) shown in Table 3.
Comparison of the Three Groups by Race, Gender, and Medical School Attended.
P-value calculated using Chi-squared test or Fisher's exact test where assumptions of Chi-squared test were not met because of the observed counts.
Statistical significance (P < .05).
Percentages are calculated based on the total sample size (N = 51).
The SPC 3-phase analysis, as shown in Figure 3, revealed a significant shift in the process indicating resident engagement and involvement in interdisciplinary EQuIPS projects (Kirkpatrick Level 3) and the completion of projects presented at regional/national conferences or published in peer-reviewed journals (Kirkpatrick Level 4) following the implementation of the EQuIPS curriculum. The number of EQuIPS project presentations at regional/national levels and peer-reviewed publications were 0.36 per resident per graduation year at the baseline. This increased to 1.3 per resident per graduation year in the roll-out period and to further increase at 4.9 per resident per graduation year in the post-implementation phase.
U-Control Chart of the Rate QI Projects per Resident per Graduation Year. UCL, Upper Control Limit; LCL, Lower Control Limit. The Blue Lines Indicate the Phases along the Time Scale on X-axis. The Average Number of QI Projects in the Three Phases Steadily Increased, but U Control Charts Adjusted for the Denominator as shown by the Variable UCL and LCL (Zigzag Top and Bottom Red Lines).
The Kruskal–Wallis test revealed a significant difference in scholarly research involvement (Levels 2 and 3 outcomes) among the three rollout groups for the EQuIPS curriculum (Pre-EQuIPS Implementation, During EQuIPS Rollout and Post-EQuIPS Implementation), with a P-value of <.001. Pairwise comparisons using the Wilcoxon rank sum test with Bonferroni correction further confirmed significant differences between all group pairs: Pre-EQuIPS Implementation versus During EQuIPS Rollout (P = .002), Pre-EQuIPS Implementation versus Post-EQuIPS Implementation (P < .001), and During EQuIPS Rollout versus Post-EQuIPS Implementation (P < .001). This can also be visualized in the 3-phase analysis using the U-control chart shown in Figure 3. It indicates a steady increase in the number of QI and PS projects per resident per graduation year. Poisson regression also mirrored these results indicating that the implementation of the EQuIPS program had a significant impact on increasing scholarly activity among residents. The Poisson regression plot is shown in Figure 4. It presents a Poisson regression model examining the number of scholarly activities involving EQuIPS projects over time. Poisson regression is appropriate for this analysis because the outcome variable represents count data, and the model estimates the expected rate of scholarly activities as a function of the predictor(s), assuming a log-linear relationship. The regression plot demonstrates a positive association between time (or implementation phase) and the number of scholarly activities, with the fitted regression line indicating an increasing expected count of EQuIPS-related scholarly outputs. This suggests that as the EQuIPS initiative matured, participation in scholarly activity increased in a sustained and systematic manner rather than occurring sporadically. The observed data points cluster reasonably close to the fitted line, indicating that the Poisson model provides an adequate fit to the observed counts. The absence of extreme deviations suggests no substantial overdispersion or model instability evident from the visual inspection alone, supporting the use of a Poisson framework for inference. From an interpretive standpoint, the upward trajectory of the regression line implies that the implementation of EQuIPS was associated with a progressive increase in scholarly productivity, consistent with a growing culture of scholarship and structured support for resident and faculty engagement. The log-scale nature of Poisson regression means that changes are multiplicative rather than additive, indicating that each incremental increase in the predictor corresponds to a proportional increase in expected scholarly output. Overall, this regression plot supports the conclusion that EQuIPS implementation was associated with a significant and sustained increase in scholarly activities, reinforcing the effectiveness of the program in promoting scholarship within the training environment (Figure 4).
Regression Plot of Number of Scholarly Activities Involving EQuIPS Projects on Poisson Distribution.
Table 4 summarizes the results of Poisson regression analysis which showed that as compared to the baseline there was 215% increase in the number of EQuIPS-related scholarly activities in the post-implementation phase.
Summary of the Results of Poisson Regression Analysis.
Phase 0 represents the pre-EQuIPS, phase 1 represents the rollout phase, and phase 2 represents post-EQuIPS.
Discussion
Research has consistently shown that early training in quality improvement (QI) methodologies improves patient outcomes and positively influences resident performance. Many studies, including the work by Warm et al, 10 have shown that residents exposed to QI education during their residency training deliver higher-quality care in clinical practice. However, most of these studies have been conducted in large academic medical centers.
Given the lack of studies evaluating QI curricula in community-based residency programs, our study is among the first to utilize measurable outcomes through the Kirkpatrick model to assess the impact of a structured QI curriculum on equity, patient safety, and overall quality improvement. The application of the Kirkpatrick model provided a structured and comprehensive framework for evaluating the effectiveness of our curriculum within the family medicine residency program. By focusing on Levels 2 (Learning), 3 (Behavior), and 4 (Results), we were able to move beyond surface-level satisfaction metrics and assess meaningful educational and clinical outcomes. The statistically significant findings across these levels suggest that the curriculum not only increased residents’ knowledge and skills (Level 2) but also translated into observable changes in clinical behavior (Level 3) and measurable improvements in patient care outcomes or system-level processes (Level 4). Excluding Level 1 (Reaction) was intentional, as all residents consistently met this threshold, and its inclusion would not have contributed value to our analysis. Using the Kirkpatrick model made our evaluation stronger by tying the results to clear, recognized levels of educational impact, which helped make our findings more understandable and meaningful. Unlike traditional QI curricula that focus primarily on process improvement projects or improvements of certain KPIs, the EQuIPS uniquely integrates principles of health equity, systems-based practice, and patient safety within a single longitudinal framework. This integration differentiates EQuIPS by embedding QI learning directly into residents’ clinical workflows while emphasizing equity-driven outcomes. This framework may be a useful tool for other residency programs seeking to systematically evaluate the effectiveness of multifaceted curricular interventions.
Our findings reveal statistically significant outcomes. It has transformed our residency program's training by creating a culture of continuous growth and improvement. Inspired by its success, this curriculum is now being adapted across other residency and fellowship programs across two sponsoring institutions. The results highlight the potential of targeted QI education to improve both resident competency and patient-centered outcomes. It can be adapted to other programs to achieve similar results.
Over the past two decades, quality improvement (QI) has become a core competency in physician training. A collective case study of four residency programs at one institution examined how QI is taught to residents. 17 It revealed that programs with hybrid QI curricula, combining both didactic and experiential learning, lead to higher resident engagement, skills, and attitudes toward QI. In contrast, programs with didactic-only curricula face challenges, including the perception that research-focused scholarly projects hinder QI involvement. Common barriers to QI involvement included limited time, mentorship, and pressure to prioritize research over QI. The study also found that learning climate and local safety culture significantly influences residents’ QI learning. Therefore, based on the previous findings, our study utilized both didactic and experiential learning in QI training, creating an environment conducive to QI activity.
Despite the U.S. healthcare system's substantial expenditure, there remains a need to ensure that investments lead to measurable improvements in healthcare delivery. However, the complexity of the healthcare environment demands a strong focus on system-based practice. This can only be done by using QI tools to respond to SDOH and improve care coordination within the system. By training our residents with QI skills, they will be able to recognize and work towards solving issues within the healthcare system.
Another aspect to note is that regardless of field of specialization, physicians fundamentally serve in leadership roles within healthcare teams and are held accountable for patient care outcomes. Yet, despite the essential role of leadership in clinical practice, there is still no universally accepted definition of “clinical leadership.” Research highlights this gap in clarity and training. One study found that while 79% of survey respondents view leadership as a core competency for physicians, 55% feel that leadership skills are not adequately taught during medical education. Additionally, 82% believe that stronger leadership abilities would enhance their effectiveness as physicians, and 61% do not agree that leadership training is consistently integrated across the medical curriculum. 8 EQuIPS addresses this gap by embedding leadership skill development directly within QI and patient safety initiatives, allowing residents to gain leadership experience through managing real improvement projects and team-based interventions. This integrated approach strengthens both leadership capacity and QI competency. It allows demonstration of leadership growth through hands-on participation in system-level improvement and dissemination within the system and outside world as presentations at local, regional, and national presentations as well as peer-reviewed publications.
This study has several limitations that should be acknowledged. First, quasi-experimental design without randomization introduces potential selection bias and limits causal inference. While demographic variables such as race and gender were comparable across cohorts, a significant difference in medical school background (eg, higher proportion of international medical graduates in the post-implementation group) may have influenced outcomes, particularly scholarly engagement. Second, the study was conducted at a single community-based residency program, which may limit generalizability to larger academic centers or other specialties. However, the Poisson regression model adjusted to control the confounders. Third, while the use of the Kirkpatrick model provided a structured framework for evaluation, Level 4 outcomes focused on scholarly output rather than direct clinical or patient safety outcomes. Additionally, outcome measures relied on data from residency management software rather than independent validation, which may be susceptible to reporting or confirmation bias. While the EQuIPS curriculum was rolled out and implemented, some other interventions related to scholarly activity were also being implemented. Those interventions may have acted as confounding and enhanced the effect and association of Level 4 outcomes. Lastly, the curriculum's long-term impact on clinical practice and patient outcomes after residency remains unknown and warrants future longitudinal follow-up studies.
Overall, our study highlights the value of implementing a structured, outcomes-based QI curriculum in a community hospital setting. A successful QI curriculum in such settings must be concise, focused, and practical. It should include tools for assessing resident attainment of learning objectives, offer hands-on opportunities for residents to lead their own QI initiatives, and provide simple yet effective evaluation methods to gauge the curriculum's impact. Creating a curriculum aimed at introducing QI concepts to residents early on in their training and using objective scales to assess the outcomes can better prepare future physicians to lead improvement efforts. Furthermore, by addressing barriers and tailoring the curriculum to meet local needs, it will benefit the entire healthcare system. Future work should examine the long-term impact of QI training on clinical practice, patient safety, and healthcare outcomes, helping to ensure that residency programs adequately prepare residents to lead efforts in improving care quality.
Conclusion
The implementation of EQuIPS curriculum in our community-based family medicine residency program was associated with a statistically significant and educationally meaningful shift in how residents’ engagement in scholarly activities related to equity, quality improvement and patient safety as assessed by using the Kirkpatrick model. Other GME programs could potentially use this approach when considering scholarly activity and curricula related to EQuIPS-related topics.
Supplemental Material
sj-docx-1-mde-10.1177_23821205261422899 - Supplemental material for Outcomes of an Equity, Quality Improvement, and Patient Safety (EQuIPS) Curriculum in Community Hospital-Based Family Medicine Residency Program
Supplemental material, sj-docx-1-mde-10.1177_23821205261422899 for Outcomes of an Equity, Quality Improvement, and Patient Safety (EQuIPS) Curriculum in Community Hospital-Based Family Medicine Residency Program by Erum Azhar, Syed Mustafa Ali Shah, Shehar Bano Awais, Fatima Hamid and Abdul Waheed in Journal of Medical Education and Curricular Development
Footnotes
Acknowledgements
Many thanks to Dr M. N. Khan and Dr Elizabeth Muhiire-Ntaki for initial input on curriculum writing in 2018 and serving on the guiding coalition for implementation of this curriculum in later years when it was rolled out.
Ethical Considerations
The WellSpan Health Institutional Review Board (IRB) determined it to be “Not Human Subjects Research” via notification 1887432-2.
Consent to Participate
Patient/Research Subject consent was not applicable since it is not a Human Subjects Research.
Author Contributions
The authors confirm contribution to the paper as follows: E.A., S.M.A.S, S.A., and A.W. contributed to study conception. S.A. and A.W. collected the data, designed the layout of the manuscript. E.A., S.M.A.S, and A.W. performed the data analysis. All authors were involved in interpretation of the results. The manuscript was written by E.A., S.M.A.S, S.A., and A.W. All authors contributed intellectually and edited the manuscript. All authors read and approved of the final manuscript. E.A. is the corresponding author. A.W. is the senior author and Principal Investigator responsible for the integrity of the work.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
