Advancing Equity in Graduate Medical Education Recruitment Through a Diversity Equity and Inclusion (DEI) Toolkit for Program Directors

Abstract

Objectives

To increase diversity and inclusion in graduate medical education (GME), the Accreditation Council for Graduate Medical Education (ACGME) issued new diversity standards requiring programs to engage in practices that focus on systematic recruitment and retention of a diverse workforce of trainees and faculty. The literature on how program directors (PDs) can incorporate and prepare for this standard is limited.

Methods

We developed a diversity, equity, and inclusion (DEI) toolkit for PDs as an example of an institutional GME-led effort to promote inclusive recruitment and DEI awareness among residency and fellowship programs at a large academic center.

Results

A survey was sent to 80 PDs before the launch of the toolkit and 6 months afterwards with response rates of 27% (22/80) and 97% (78/80), respectively. At baseline, 45% (10/22) anticipated that the DEI toolkit might provide better resources than those currently available to them and 41% (9/22) perceived that the toolkit might improve recruitment outcomes. At 6 months, 63% (49/78) found the toolkit helpful in the 2021-2022 recruitment season. By contrast, 2% (2/78) of PDs did not find the toolkit helpful, and 33% (26/78) said they did not access the toolkit. When asked if a PD changed their program's recruitment practices because of the toolkit, 31% (24/78) responded yes. Programs that changed recruitment practices started to require unconscious bias training for all faculty and residents involved in the residency interviews and ranking. Others worked on creating a standardized scoring rubric for interviews focused on four main domains: Experiences, Attributes, Competencies, and Academic Metrics.

Conclusion

There is a need to support PDs in their DEI journey and their work to recruit a diverse workforce in medicine. Utilizing a DEI toolkit is one option to increase DEI knowledge, skills, awareness, and self-efficacy among PDs and can be adopted by other institutions and leaders in academic medicine.

Keywords

diversity equity and inclusion graduate medical education program directors DEI toolkit holistic recruitment

Introduction

Black, Latinx, and other minorities are under-represented in medicine (URiM) and their representation continues to decline in most specialties.^1–3 Historically, medical students from URiM backgrounds are less likely to match into residency programs compared to their White counterparts^{3, 4} contributing to restricted representational diversity in graduate medical education (GME). To improve diversity in GME programs, the Accreditation Council for Graduate Medical Education (ACGME) issued new standards that require programs and their sponsoring institutions “to engage in practices that focus on systematic recruitment and retention of a diverse and inclusive workforce”.⁵

Similar to the Liaison Committee of Medical Education (LCME) diversity standards which resulted in gains in the percentage of female and URiM matriculants from medical schools,^{6, 7} the ACGME diversity standard has the potential to influence the diversity of the physician workforce. However, meeting the requirements of this standard can be challenging. In a study describing the perceptions of internal medicine program directors (PDs) and associate PDs, many reported limited knowledge of the new requirements.⁸ Some expressed concerns the standard reached beyond the PDs’ scope of influence. Institutional commitment to diversity was recognized as a prerequisite for successful implementation. Others have called on GME offices to lead and support efforts to diversify recruitment at an institutional level.⁹

In this perspective, we provide a structured approach to the development and implementation of a PD diversity, equity, and inclusion (DEI) toolkit as an example of an institutional GME-led effort to promote inclusive recruitment and DEI awareness among residency and fellowship programs at a large academic center. The toolkit is publicly available here: https://medicine.iu.edu/gme/why-iu/toolkit.

Setting and rationale

Indiana University School of Medicine (IUSM) is Indiana's only allopathic medical school and has nine campuses across the State. IUSM GME programs host more than 1300 residents and fellows in 112 ACGME accredited programs and 92 non-ACGME accredited programs; which are all held to the same standards. Several strategies and frameworks have been proposed to prepare programs for the ACGME DEI standard including educating PDs on creating diversity mission statements, implementing uniform approaches to inclusive recruitment, building recruitment pathways, and investing in trainees’ success through DEI training and professional development opportunities^9–11 . However, operationalizing these strategies in any large academic institution can be challenging. At large academic GME sites, recommendations and initiatives promoting DEI work can be difficult to navigate or easily lost among the overwhelming number of emails and updates. There is compelling evidence for the effectiveness of using innovative, high yield methods to engage busy faculty with competing responsibilities.¹² Such approaches also allow generalizability to all specialties, without being resource intensive, and enable prospective validation and evaluation. The toolkit was therefore envisioned as an optimal solution to provide a centralized, curated, comprehensive, high-yield, and convenient repository to support PDs in achieving DEI goals.

Methods

In August 2020, we launched a GME-diversity, equity, inclusion, and justice (DEIJ) committee with a call for volunteers.¹³ A subgroup of this committee was tasked with developing the DEI toolkit and followed the intentional step-wise approach outlined in Table 1. A detailed description of the technology used is provided in the supplemental section.

Table 1.

Eight-steps for developing an institutional DEI toolkit

DEIJ TOOLKIT DEVELOPMENT & IMPLEMENTATION STEPS
PHASE	DESCRIPTION
1. Define Stakeholders	Define target Audience for the toolkit (PD, trainees, faculty, and prospective residency applicants) Create an oversight committee for curating content, providing feedback, user acceptance testing Appoint a design and development team
2. Establish desired outcomes	Consider institutional DEIJ goals Identify DEIJ resource gaps/needs
3. Document technical requirements and universe of DEIJ resources	Select technology platform to house toolkit (consider accessibility, user friendliness, ability to visualize information, security/access controls) Document all existing and planned resources Compile all resources into a central repository
4. Design and develop the toolkit	Describe purpose and use of the toolkit – consider buy-in, support, and adoption Provide toolkit instructions and/or introductory video Highlight/spotlight important information, use intuitive navigation, visualize content Consider and align content and design with institution-specific factors
5. Conduct quality assurance and user testing	Review and test for quality and operability Oversight committee should test and provide feedback for improvement
6. Pilot the toolkit	Consider piloting the toolkit with a sample of programs Collect feedback for improvement
7. Rollout/implement toolkit comprehensively	Develop a communication strategy to facilitate buy-in and adoption Conduct demonstrations on how to use the toolkit
8. Develop an assessment tool	Measure and evaluate the adoption, use, and outcomes from implementing the toolkit

The DEI toolkit was modeled after the IUSM's existing diversity strategy which centers around three foundational pillars: Representational Diversity; Inclusive Environment; Cultural Humility and Antiracism. The toolkit provides strategies and resources to actualize each pillar (Table 2). The content consolidates school wide diversity initiatives, current literature recommendations on best practices for inclusive recruitment,^9–11 DEI resources within the organization, and upcoming DEI training events/conferences - allowing PDs access to resources, education and information on the work being done. After receiving support from key institutional stakeholders, the toolkit was presented at the Annual PD meeting during a 90-min plenary session on September 1, 2021.

Table 2.

A summary of DEI resources supporting each of the IUSM diversity pillars

DIVERSITY PILLARS	DEI TOOLKIT RESOURCES
Representational Diversity: https://medicine.iu.edu/gme/why-iu/toolkit/representational	Pursuing opportunities for building stronger connections with diverse groups within and outside IUSM through attending local/national meetings such as the Howard Residency Fair, the Latino Medical Student Association (LMSA), and Student National Medical Association (SNMA), and URiM second look and visiting electives programs are highlighted Developing guides on how to implement holistic recruitment strategies using a 10-Step Inclusive Recruitment Checklist developed by the IUSM GME Diversity Committee are presented. The checklist can be reviewed at the following link: IUSM GME Recruitment Checklist.pdf Creating a one-page summary of the AAMC guide on Holistic Principles in Resident Selection (11) to promote easy adoption. The one page summary can be reviewed at the following link: IUSM GME Summary of the AAMC Holistic Principles in Resident Selection.pdf Creating instructions on how to access IUSM retention and mentoring programs to grow and retain talent, leverage leaders, and create a culture of learning
Inclusive Environment: https://medicine.iu.edu/gme/why-iu/toolkit/inclusive-environment	Promoting DEIJ culture and leadership within IUSM to manage cultural change, create strategic accountability, and deliver transparent outcomes for DEI efforts Developing strategic plans, policies, and processes to socialize shared DEIJ vision and language among IUSM leaders, faculty, and trainees Establishing safe and welcoming learning environments that foster a sense of belonging within IUSM-affiliated hospital systems by educating faculty and trainees on the process and avenues for reporting mistreatment and sharing the annual mistreatment report with all stakeholders
Cultural Humility & Antiracism: https://medicine.iu.edu/gme/why-iu/toolkit/cultural-humility-antiracism	Supporting DEI training content and opportunities within IUSM to facilitate constructive intergroup and interprofessional interaction; reduce bias, discrimination, and prejudice; and instruct people from different backgrounds on how to collaborate effectively Encouraging and sponsoring education, training, and research in health inequities, health disparities, and social determinants of health to promote the wellbeing of IUSM patients and the communities served Employing qualitative and quantitative data to support evidence-based decision making, assess progress towards DEIJ goals and objectives, and ensure the accountability and transparency of DEIJ strategies, metrics, and outcomes Providing easy access to social media sites, DEIJ event calendars, DEI training modules including the ACGME Equity Matters DEI modules.

IUSM GME Recruitment Checklist.pdf: https://indiana.sharepoint.com/sites/O365-IUSMGME/Shared%20Documents/Forms/GME%20A%20to%20Z.aspx?id=%2Fsites%2FO365%2DIUSMGME%2FShared%20Documents%2FDiversity%2C%20Equity%2C%20and%20Inclusion%2FDEI%20Toolbox%2FDEI%20Toolbox%20Materials%2FGME%20Recruitment%20Checklist%2D2023%20revised%2Epdf&parent=%2Fsites%2FO365%2DIUSMGME%2FShared%20Documents%2FDiversity%2C%20Equity%2C%20and%20Inclusion%2FDEI%20Toolbox%2FDEI%20Toolbox%20Materials&p=true&ga=1

IUSM GME Summary of the AAMC Holistic Principles in Resident Selection.pdf: https://indiana.sharepoint.com/sites/O365-IUSMGME/Shared%20Documents/Forms/GME%20A%20to%20Z.aspx?id=%2Fsites%2FO365%2DIUSMGME%2FShared%20Documents%2FDiversity%2C%20Equity%2C%20and%20Inclusion%2FDEI%20Toolbox%2FDEI%20Toolbox%20Materials%2FEACM%20Model%202023%2Drevised%20%2Epdf&parent=%2Fsites%2FO365%2DIUSMGME%2FShared%20Documents%2FDiversity%2C%20Equity%2C%20and%20Inclusion%2FDEI%20Toolbox%2FDEI%20Toolbox%20Materials&p=true&ga=1

Evaluating the toolkit: survey development

We developed a survey to assess how the DEI toolkit may impact the awareness, knowledge, access, adoption, and use of DEIJ resources among IUSM PDs. The first section of the survey addressed questions about the current state of awareness and use of DEIJ resources by PDs. The second section was based on Diffusion of Innovations (DoI) theory^{14, 15} to gain insight into PDs’ perceptions of the toolkit and the potential barriers to adoption. Items in this section assessed perceptions of the five innovation attributes outlined in DoI theory (ie, relative advantage, compatibility, complexity, trialability, observability), and the PDs perceptions on how the toolkit may impact recruitment outcomes. The protocol was reviewed and approved as exempt by our local IRB with waiver of informed consent.

Survey administration and participants

The baseline survey was administered to PDs for core residency programs and fellowships who were getting ready to start a new recruitment season (2021-2022) before accessing the toolkit for the first time. Fellowship programs with fewer than two fellows per year, programs that were already in recruitment season, and those not participating in the 2021-2022 match cycle were excluded. Changes in recruitment practices for these programs attributable to the DEI toolkit would have been difficult to assess given the timing of the launch or the timing of the interview season. The baseline survey was open for 1 month and two reminders were sent encouraging the PDs to complete the survey. The survey was repeated 6 months after the PDs had the opportunity to use the toolkit through one recruitment cycle with reminders sent through the GME office encouraging completion. No incentives were offered for survey completion.

Results

The baseline survey was administered to 80 PDs of whom 22 responded (response rate 27%). Of the respondents, 45% (10/22) anticipated that the DEI toolbox might provide better resources than those currently available to them and 41% (9/22) perceived that the DEI toolkit might improve recruitment outcomes. Regarding adoption, 45% (10/22) reported they would be more inclined to use the DEI toolbox if it were adopted by others and 55% (12/22) reported they would have/make time to browse the resources available in the DEI toolbox.

The response rate on the six-month post intervention survey was 97% (78 responses/80). Of all respondents, 63% (49/78) reported finding the toolkit helpful in the 2021-2022 recruitment. By contrast, 2% (2/78) did not find the toolkit helpful and 33% (26/78) said they did not access the toolkit. Reasons for not accessing the toolkit included: “I was not the PD last year”, “I was not aware the toolkit existed”, “did not feel I needed it”, “I worked with the internal DEI committee”, and “too busy with other initiatives and do not have the bandwidth for anything extra”. When asked if “the recruitment practices provided in the toolkit align[ed] with the way the PDs currently conduct[ed] recruitment”, 46% (36/78) agreed and 54% (42/78) neither agreed or disagreed with this statement. When asked if a PD changed their program's recruitment practices for the 2021-2022 recruitment season because of the information provided in the toolkit, 31% (24/78) responded YES; 54% (42/78) responded NO, and 14% (12/78) responded that the toolkit did not contain practices not already adopted by their program.

Examples of strategies implemented by programs that changed recruitment practices included requiring unconscious bias training for all faculty and residents involved in the residency selection process, utilizing the AAMC guide on holistic reviews,¹¹ and the collation of criteria arrived at by a consensus of faculty and residents for the holistic review of applicants. These criteria resulted in the creation of a scoring rubric for the four main domains: Experiences, Attributes, Competencies, and Academic Metrics (EACM model). As a result of this approach, programs reported an increase in the number of URiM applicants invited to interview based on the EACM model.

Discussion

Implementation of new ACGME standards is often received with apprehension by program leadership and is fraught with barriers.^{16, 17} As such, preparation and dissemination of supportive guidelines for PDs is critical. By creating the PD DEI Toolkit, IUSM GME was able to curate a comprehensive list of resources and guidelines aimed at preparing residency and fellowship programs to engage in best practices for the recruitment and retention of URiM trainees. Training modules on unconscious bias and communication frameworks for upstander and allyship training for faculty and trainees were incorporated to create an inclusive learning environment that is welcoming and respectful to all. Institutional and program resources for reporting mistreatment were featured and publicized to promote a workplace that is free of harassment and discrimination.

Our work is innovative, as no published work on an institution wide effort to meet the ACGME diversity standard exists in the literature. The process used to develop the DEI toolkit can serve as a model for other GME institutions to adapt and adopt. While the preliminary data is encouraging with over a quarter of the PDs reporting changes to their recruitment practices because of the knowledge gained from the resources available in the toolkit, the PD survey also highlighted the need to further understand the details of how programs adopt change, the barriers, and facilitators to change, and other ways they can be supported.

Our report has several limitations. Our baseline survey response rate was low and as with all survey methodologies, response bias cannot be excluded and participants may have represented PD most or least engaged in DEI efforts. We are uncertain about how sustained the PDs’ perceptions regarding the utility of the toolkit will be nor do we have data on the long-term impact on DEI knowledge, recruitment, and retention outcomes. Another limitation is the lack of information on how users interacted with the toolkit. Plans to monitor traffic and user interface are already in development. Finally, although our survey was based on a well published method, the Dol, it was not validated, and pilot tested.

Conclusion

There is a need to support PDs as the GME community works towards training a diverse workforce in medicine. Our toolkit is one option to increase DEI knowledge, skills, awareness, and self-efficacy among PDs and can be adopted by other colleagues and leaders in academic medicine. While there is an enormous amount of work to be done before we achieve our diversity goals, establishing learning collaboratives and a sense of community around these efforts both within and between institutions may accelerate our progress. By making the IUSM toolkit public, it is our hope that others will be encouraged to share their resources with the GME community to advance the pace of our work and fulfill our collective mission.

Supplemental Material

sj-docx-1-mde-10.1177_23821205231203136 - Supplemental material for Advancing Equity in Graduate Medical Education Recruitment Through a Diversity Equity and Inclusion (DEI) Toolkit for Program Directors

Supplemental material, sj-docx-1-mde-10.1177_23821205231203136 for Advancing Equity in Graduate Medical Education Recruitment Through a Diversity Equity and Inclusion (DEI) Toolkit for Program Directors by Zeina M. Nabhan, Nicole Scott, Areeba Kara, Leilani Mullis, Travis Dams, Mark Giblin, Francesca Williamson and Curtis Wright in Journal of Medical Education and Curricular Development

Supplemental Material

sj-docx-3-mde-10.1177_23821205231203136 - Supplemental material for Advancing Equity in Graduate Medical Education Recruitment Through a Diversity Equity and Inclusion (DEI) Toolkit for Program Directors

Supplemental material, sj-docx-3-mde-10.1177_23821205231203136 for Advancing Equity in Graduate Medical Education Recruitment Through a Diversity Equity and Inclusion (DEI) Toolkit for Program Directors by Zeina M. Nabhan, Nicole Scott, Areeba Kara, Leilani Mullis, Travis Dams, Mark Giblin, Francesca Williamson and Curtis Wright in Journal of Medical Education and Curricular Development

Footnotes

Acknowledgements

The authors acknowledge the dedication of all the IUSM GME Diversity committee members, the support of the IUSM GME leadership, program directors, program coordinators, residents, and GME team. Special thanks to Miss Adrienne Phillips for her administrative support

Author contributions

All listed authors contributed intellectually to this manuscript, provided important input, and reviewed and approved the submitted work

Zeina M. Nabhan MD: wrote the first draft of the manuscript and contributed to the development of the survey and the DEI toolkit

Nicole Scott MD: worked on editing the methods section and contributed to the development of the DEI toolkit

Areeba Kara MD: worked on editing the introduction and conclusion section and contributed to the development of the survey

Leilani Mullis MD: worked on the results section and contributed to the development of the DEI toolkit

Travis Dams MD: worked on editing the discussion section contributed to the development of the DEI toolkit

Mark Giblin MA: developed the DEI toolkit platform and worked on the methods section

Francesca Williamson PhD: led the development of the survey and worked on editing the entire manuscript and the references

Curtis Wright MD: worked on editing the abstract section, edited the final draft of the manuscript, and contributed to the development of the DEI toolkit

Disclosures

The authors have nothing to disclose

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

This work was approved by the Institutional as an exempt study. Informed consent was not required. Ethical approval is not applicable to this type of work. IRB approval letter was added to the supplemental documents

Informed consent/funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Zeina M. Nabhan

Supplemental material

Supplemental material for this article is available online.

Abbreviations

References

Association of American Medical Colleges. Diversity in Medicine, Facts and Figures. U.S. Physician Workforce Data. Figure 18, Percentage of All Active Physicians by Race and Ethnicity. Accessed February 2, 2022. https://www.aamc.org/data-reports/workforce/interactive-data/figure-18-percentage-all-active-physicians-race/ethnicity-2018.

Xerali

Castillo-Page

Conrad

Nivet

. Analyzing Physician Workforce Racial and Ethnic Composition Associations: Geographic Distribution. Association of American Medical Colleges, 2018. Accessed July 27, 2022 https://www.aamc.org/media/7621/download

Lett

Orji

Sebro

. Declining racial and ethnic representation in clinical academic medicine: a longitudinal study of 16 US medical specialties. PLoS One. .2018;13(11):e0207274. doi:10.1371/journal.pone.0207274

Sondheimer

Xierali

Young

Nivet

. Placement of US medical school graduates into graduate medical education, 2005 through 2015. JAMA. 2015;314(22):2409-2410.

Accreditation Council for Graduate Medical Education. Common Program Requirements (Residency). Accessed July 27, 2022. https://www.acgme.org/globalassets/PFAssets/ProgramRequirements/CPRResidency2021.pdf .

University of South Florida. Liaison Committee on Medical Education (LCME) Standards on Diversity. Accessed July 27, 2022. https://health.usf.edu/∼/media/Files/Medicine/MD%20Program/Diversity/LCMEStandardsonDiversity1.ashx?la=en .

Boatright

Samuels

Cramer

, et al. Association between the liaison committee on medical education’s diversity standards and changes in percentage of medical students sex, race, and ethnicity. JAMA. 2018;320(21):2267-2269.

Martinez-Strengel

Balasuriya

Black

, et al. Perspectives of internal medicine residency program directors on the accreditation council for graduate medical education (ACGME) diversity standards. JGIM. 2021;36(9):2539-2546.

Geary

Wang

Cooper

Roberts

Yoo

. Analysis of electronic residency application service (ERAS) data can improve house staff diversity. JSR. 2021;257:246-251.

10.

Gonzaga

AMR

Appiah-Pippim

Onumah

Yialamas

. A framework for inclusive graduate medical education recruitment strategies: meeting the ACGME standard for a diverse and inclusive workforce. Acad Med. 2020;95(5):710-716.

11.

AAMC Guidelines for Holistic Principles in Resident Selection: An Introduction. Accessed July 27, 2022 https://www.aamc.org/system/files/2020-08/aa-member-capacity-building-holistic-review-transcript-activities-GME-081420.pdf

12.

Bar-on

Konopasek

. Snippets: an innovative method for efficient, effective faculty development. JGME. 2014;6(2):207-210., Accessed July 27, 2022 https://www.aamc.org/media/7621/download

13.

Kara

Wright

Funches

, et al. Serving on a graduate medical education diversity, equity, inclusion, and justice committee: lessons learned from a journey of growth and healing. Front Public Health. 2022;10:867035. doi:10.3389/fpubh.2022.867035. PMID: 35570954; PMCID: PMC9091374.

14.

Chew

Grant

Tote

. Doctors on-line: using diffusion of innovations theory to understand internet use. Fam Med. 2004;36(9):645-650.

15.

Rogers

Singhal

Quinlan

. Diffusion of innovations. In an integrated approach to communication theory and research. Routledge; 2014; 432-448.

16.

Antiel

Thompson

Reed

, et al. ACGME duty-hour recommendations–a national survey of residency program directors. N Engl J Med. 2010;363(8):e12.

17.

Drolet

Whittle

Khokhar

Fischer

Pallant

. Approval and perceived impact of duty hour regulations: survey of pediatric program directors. Pediatrics. 2013;132(5):819-824.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB

0.02 MB