Abstract
OBJECTIVE
A diverse physician workforce ensures equitable care. The holistic review of residency applications is one strategy to enhance physician diversity; however, little is known about current adoption and the factors that facilitate/impede the adoption of holistic recruitment practices (HRPs) by graduate medical education (GME) residency, and fellowship program directors (PDs). To describe the current state and explore, the barriers/facilitators to the adoption of HRPs at our institution.
METHODS
We disseminated information about HRP within our program between 2021 and 2022. In May 2022, a survey of 73 GME PDs assessed current recruitment practices and self-reported barriers to holistic recruitment. Holistic Recruitment Scores (HRSs) reflecting the adoption of best practices were tabulated for each program and compared to identify predictors of adoption.
RESULTS
73/80 (92%) of PDs completed the survey. Programs whose PDs had higher academic rank, total number of trainees, and female trainees in the past 3 years had higher HRSs. Program size was directly correlated with HRS. Most (93%) PDs felt their current efforts were aligned to increase diversity and 58% felt there were no barriers to the adoption of holistic review. The most reported barriers were lack of time and knowledge/expertise in diversity, equity, and inclusion (DEI), both reported by 16 out of 73 PDs (22%).
CONCLUSION
While most PDs implemented some HRP, institutional and departmental support of program directors through the commitment of resources (eg, staffing help and subject matter experts/coaches hiring) are crucial to overcome barriers.
Introduction
Increasing the racial diversity of the physician workforce is an important component of combating health inequities and improving health outcomes for all. 1 Physicians belonging to groups who have historically been underrepresented in medicine (URiM) (Black, Hispanic, and Native American) are more likely to work in underserved areas. 2 Yet even as Hispanic and Black populations have grown in the United States, and despite the standards introduced by the Liaison Committee on Medical Education (LCME), the Accreditation Council for Graduate Medical Education (ACGME), and the American Medical Association (AMA) for medical schools to attract and retain diverse student bodies, Hispanic and Black physicians are disproportionally more URiM today compared to 30 years ago.3–5 Contributing to this lack of diversity and difficulty in recruiting and retaining a diverse physician workforce are a higher risk of burnout in those working on these efforts, inadequate funding and/or legislative/social support for funding for physician training, inequities in advancement and attrition throughout the US education system, and a larger proportion of medical school matriculants coming from higher socio-economic backgrounds. To create a physician workforce that is representative of the demographic profile of the United States, we must recruit diverse individuals into medical schools and subsequently into graduate medical education (GME). 6
Strategies to diversify the physician workforce have been introduced by the AAMC and AMA and include the holistic review of medical school and residency applications.7–9 Early data show that holistic review practices (HRPs) increase the diversity of applications reviewed, 10 applicants interviewed, 11 and those matched into GME positions. 12 However, less is known about factors that may facilitate or impede the adoption of these practices by program directors (PDs). Such data are critical to bridge the gap between recommendations and real-world implementation.
Indiana University School of Medicine (IUSM) is Indiana's only allopathic medical school and has 9 campuses across the state. IUSM GME programs host more than 1300 residents and fellows in 112 ACGME-accredited programs and 92 non-ACGME accredited programs. To facilitate the adoption of HRP by our GME programs, the GME diversity committee at our institution developed several initiatives to support our PDs. These efforts included the development of a Toolkit to provide a centralized, curated, comprehensive, high-yield, and convenient repository of resources around holistic recruitment. The development of this toolkit is described elsewhere and is available here https://medicine.iu.edu/gme/why-iu/deij-toolkit. 13 In this work, we describe the state of adoption of holistic recruitment practices (HRPs) by residency and fellowship PDs at our institution, explore barriers to adoption, and identify variables that may predict adoption.
Methods
This is a cross-sectional quantitative survey analysis. The GME-diversity, equity, inclusion, and justice (DEIJ) committee, comprised of PDs, associate PDs, faculty members, residents, staff, and education experts, created and edited the survey instrument. Face and content validity was established in discussions during committee meetings, and questions were iteratively refined to improve clarity Multiple-choice, Likert-type, and open-ended questions addressed specific recruitment practices and barriers to holistic review. The survey instrument can be found in Supplemental File 1.
The survey was distributed electronically via Qualtrics to all residency and fellowship PDs at Indiana University School of Medicine in May 2022 and remained open for 4 weeks. Weekly reminders were sent, and no incentives were offered for survey completion. Inclusion criteria comprised all residency and fellowship PDs at our institution. PDs that oversee more than one program were only permitted to submit one survey.
PD demographics, including age, race/ethnicity, academic rank, gender, and years in current rank were collated. Program characteristics were obtained from GME records and included type of specialty (medical or surgical) and number of trainees in the past 3 years (total, URiM, and female).
The protocol was reviewed and considered exempt by the Indiana University Institutional Review Board according to US 45 CFR 46.104, and informed consent was waived.
The reporting of this study conforms to the STROBE Guideline statement. 14 The completed checklist can be found in Supplement File 2.
Data analysis
Demographics (Table 1) were calculated using descriptive statistics. We calculated a Holistic Recruitment Score (HRS) to reflect the number of HRPs adopted by each program. Higher scores reflected the adoption of more HRPs. Programs were awarded points for indicating the adoption of each current best practice (these are listed with an Asterix (*) in Table 2 with the value of each response indicated in the square brackets [ ] next to the response item in Table 2). All open-ended responses to questions with an “other” option were reviewed by 2 members of the research team. If the PD identified an HRP that was not on the list of response options for that question, we assigned their program one point for each item. The sum of these responses was calculated to provide the HRS for each program.
Demographics of program directors and programs.
Abbreviation: URiM, underrepresented in medicine.
Adoption of holistic recruitment practices.
Abbreviations: GME, graduate medical education; URiM, underrepresented in medicine; DEIJ, diversity, equity, inclusion, and justice; AOA, Alpha Omega Alpha.
Values are frequencies (percentages).
Indicates the question was included in the calculation of the diversity score.
[] Indicates value added to Holistic Recruitment Score if chosen by respondent.
Comparisons of scores across characteristics were performed using regression analyses when the independent variables were continuous and using analysis of variance models when the independent variables were categorical. Analysis of covariance models were then used for multivariable analyses, to determine if variables remained independently associated for those that were significant in the bivariate results. All analytic assumptions were verified, and all analyses were performed using SAS v9.4 (SAS Institute, Cary, NC).
Concordance between HRS and the PD's perception of holistic review practices was calculated using Spearman correlation between HRS and the survey item, “My program's recruitment practices in 2021 were aligned with principles that will increase the representation of trainees underrepresented in medicine (URiM) and in the GME workforce.” We will refer to this as perceived holistic review practices moving forward. HRS was categorized as low (< 1 standard deviation (SD) from the mean), medium (between 1 SD below and 1 SD above the mean), and high (> 1 SD above the mean). Perceived holistic review practices were categorized as low (strongly disagree or disagree with the survey item), medium (neither agree nor disagree with the survey item), or high (agree or strongly agree with the survey item).
Results
Precisely, 74 out of 80 (93%) PDs completed the survey. One survey did not have demographic information and was therefore removed from the analysis. Therefore, 73 surveys were used in this analysis. Participants predominantly identified as male (n = 50; 68.5%), White (n = 50, 68.5%), and were assistant professors (n = 32, 43.8%). Programs ranged in size from 3 to 114 with a range of 0% to 44% of trainees identifying as URiM. Program HRSs ranged from 8 to 45 (µ = 24.39; SD = 7.20). Complete demographics can be found in Table 1.
Adoption of HRPs
Overall, 68 (93.2%) program directors felt their program's recruitment practices were aligned with principles that will increase the representation of URIM trainees. Similarly, 67 (91.8%) felt their efforts to increase DEIJ in their program were supported by their department chair. The support that PDs most commonly reported receiving from their department chair included education/training (n = 66; 90.4%), involvement in initiatives (n = 39; 53.4%), a diversity forum (n = 30; 41.1%), and mentorship (n = 22; 30.1%). The least commonly reported support included staffing help (n = 7; 9.6%), additional protected time (n = 4; 5.5%), and hiring experts/coaches in DEIJ (n = 4; 5.5%).
Barriers to the adoption of HRPs
Reported barriers to utilizing programs and initiatives included having a small pool of URiM applicants (n = 11; 15.1%), followed by being a fellowship program (n = 6; 8.2%), and not being familiar with the programs offered (n = 5; 6.9%).
Most PDs (n = 42; 57.5%) reported no barriers to the adoption of holistic reviews. Among those who reported barriers, lack of time (n = 16; 21.9%) and/or lack of knowledge/expertise of holistic review (n = 16; 21.9%) were the most commonly selected. Additional barriers mentioned in the free response section included: a large applicant pool for a small number of training spots, nonuniformity of test scores (USMLE vs COMLEX), and a small applicant pool. A full list of barriers can be found in Table 3.
Barriers to DEI toolkit adoption.
Abbreviations: IUSM, Indiana University School of Medicine; URiM, underrepresented in medicine; DEI, diversity, equity, inclusion.
Values are frequencies (percentages).
Indicates independent answers (respondent could only answer one so frequency adds to the total sample size).
Predictors of DEI toolkit adoption
Bivariate analyses (Table 4) showed that a program's HRS was significantly correlated with the academic rank of the PD (P = .0147) with Endowed/Distinguished Professor having the strongest association (µ = 34.5; SE = 3.4). Additionally, program size (β = 0.03; SE = 0.01; P = .0303), total URiM trainees in the past 3 years (β = 0.22; SE = 0.10; P = .0374), and total female trainees in the past 3 years (β = 0.05; SE = 0.02; P = .0475) had a positive correlation with HRS, while percent female trainees in the past 3 years was negatively correlated with HRS (β = −6.15; SE = 3.00; P = .0441).
Bivariate analysis of holistic review score and program characteristics.
Abbreviations: PD, program director; URiM, underrepresented in medicine.
Values are regression slope parameters (standard error) for continuous variables and means (standard errors) of diversity score for categorical variables. P-values are from mixed linear models.
Multivariate analysis (Table 5) showed that only program size was significantly positively associated with HRS (β = 0.02; P = .0449). However, the variable total URiM trainees in the past 3 years were removed from this multivariate analysis because of the multicollinearity between total URiM trainees and program size.
Multivariate analysis of diversity score and program characteristics.
Abbreviations: PD, program director; URiM, underrepresented in medicine.
Values are regression slope parameters (standard error) for continuous variables and means (standard errors) of diversity score for categorical variables. P-values are from mixed linear models.
Concordance of HRS with perceived holistic review practices
Perceived holistic review practices were moderately positively correlated with HRS (ρ = 0.509, P < .001, 95% CI [0.310-0.666]). All programs with a high or medium HRS reported high perceived HRPs. There were 10 programs with a low HRS. Only one program with a low HRS felt their recruitment practices were not aligned with principles that would increase the representation of URiM trainees. The other 90% (n = 9) of programs with a low HRS reported either high (n = 5) or medium (n = 4) perceived holistic review practices (Figure 1).
The program director perceived holistic review practices by holistic review score.
Discussion
Holistic review improves diversity within the medical profession recruitment process in the United States 10 by balancing personal attributes with performance/aptitude. A standardized holistic review process aligned with a program's mission and values can replace the traditional metric-driven selection process for a more inclusive process. 15 We surveyed GME residency PDs to describe the adoption and barriers of HRPs. Within our institution, most PDs felt their current practices would increase representation. In addition, they reported feeling supported by their department chair. Predictors of the adoption of HRP, based on HRS, showed that the size of the program was significantly correlated with an increase in HRS, indicating smaller-sized programs may need different approaches to increase diversity, a finding that warrants further study. The finding of the current PD academic rank correlating with higher perceived HRP (Table 4) may be due to social desirability bias, a known confounder in qualitative and survey research. Stressing the importance of understanding this phenomenon, and that those further in academic rank may feel pressure to give what they perceive is the socially desired answer and skew their reported holistic efforts.
Programs with a higher total number of URiM trainees and female trainees in the past 3 years, had a higher HRS. Contrary to this, programs with a higher percentage of female trainees over the past 3 years saw a negative effect on HRS. This finding raises a concern about condensation whereas gender diversity increases, efforts toward racial diversity may decrease because programs may feel that they are reaching diversity goals through the increase in female trainees. While our study data does not break down gender identity by racial/ethnic identity, future studies should look at this concept more closely. Condensation is a process by which one changes the definition of diversity to shift attention away from the discussion of the distribution of power and privileges based on race and toward a more ambiguous and “safe” concept of a diversity of experiences and perspectives unrelated to power structures.16,17 Diversity efforts tend to be decentralized in medical schools and other institutions of higher education. As a result, there often is not a clear, centralized definition of diversity, leaving the door open for departments working toward a “safer” definition of diversity, excluding those with multiple marginalized identities, and ultimately, not progressing toward racial equality. 17 For DEI leaders to mitigate this phenomenon institutions should work toward a centralized definition of diversity, a collaboration of DEI efforts between departments, and keeping an intentional eye on all aspects of their defined diversity including racial/ethnic and socio-economic diversity.
Most of our PDs did not report barriers to the adoption of holistic review. However, those who did were likely to report having a small URiM application pool and/or lacking time or knowledge/expertise. A small URiM application pool should not prevent participation in the holistic review. To improve this misconception, messaging around HRP should be tailored to different programs, particularly programs of different sizes and traditionally with smaller numbers of URiM applicants. Many identified a lack of time and knowledge/expertise as a barrier. Along the same lines, the least common support from departmental chairs included staffing help, protected time, and hiring experts/coaches. This is not the first time these barriers have been reported in DEI literature.18–20 In a qualitative study assessing barriers to DEI efforts in medical schools, Ko et al 21 reported that performative attention without supporting policies or resources was a main challenge to increasing racial and ethnic diversity within medical schools, a practice further elaborated on in the book Diversity regimes: Why talk is not enough to fix racial inequality at universities. 17 If departments and institutions are truly committed to increasing diversity and inclusion, committing resources to reduce these barriers will be an important step in the process.
Program directors provided extra points in recommended holistic areas of personal/professional attributes, attributes that shape one's identity, and adversity faced. However, a significant, percentage (30.1%) provided extra points for outstanding metrics (ie, AOA status, USMLE scores, and medical school reputation), which have been shown to be biased toward nonminoritized/nonmarginalized applicants.15,22 While it seems many program directors are shifting toward a more holistic review, many are still utilizing a more biased, metrics-based review. As policies and legislation against diversity efforts are being passed all over the United States, including the recent SCOTUS decision striking down affirmative action, 23 it will become increasingly important that program directors understand the reasoning behind and the process of holistic review. 24 When done well it can increase racial and ethnic diversity within a program/institution. When done poorly, it may result in the same biases as metric review. 21 For example, counting how many extra-curricular activities an applicant has participated in can result in a bias toward those who had the time, money, and support to participate in such activities—which tend to be our non-URiM applicants. GME leadership should consider targeted holistic review education for PDs to ensure appropriate adoption of holistic review processes. This is the goal of our DEI toolkit. We will track adoption and results to measure the impact of this toolkit on diversity within our GME programs. However, it is important to remember that “holistic review is an important guiding principle but should not be conflated with specific justice-based criteria, continuous internal evaluation, and institutional accountability.” 21
Along the same lines, our results show that although there is an overall moderate concordance between HRS and perceived holistic review practices, most of the programs with low and medium HRS perceived their current practices aligned with best practices in holistic review. There could be many explanations for this discordance. For example, anticipating ACGME's keen focus on, and evaluation of, a program's initiatives for enhancing diversification of the physician workforce, PDs may agree with the statement that their recruitment practices aligned with principles that will increase the representation of URiM trainees and in the GME workforce. This phenomenon, social desirability bias, is well-known in survey research. For example, there is evidence of physicians highly rating the quality of their communication and collaboration with nursing, opposite of nursing's ratings in the same studies.25,26 However, it is just as likely that these PDs actually perceive their recruitment practices to be holistic and inclusive. Of note, the 9 programs with the widest discordance (low HRS but high perceived HRP) were all small programs with 1 to 3 trainees per year—further evidence that smaller-sized programs may need tailored approaches to increase diversity. While more research needs to be done to parse this out, it will be important for GME offices to ensure transparency when informing programs of their holistic recruitment performance, dedicate instruction and resources for PDs to better understand holistic recruitment, and provide guidance on ways to limit the social desirability effect in their self-assessments.
Limitations of this study include the turnover of program directors may have resulted in a change of PD demographics and accurate knowledge of the complete recruitment practices by the former PD. Additionally, the PDs taking the survey may not have been a part of the creation of the climate in their program. The mismatch between the objective measure, diversity of the program, and subjective rating by the program director may reflect unconscious bias. The same bias that led to lower diversity in their program may also have led to higher self-rating of diversity efforts. Paradoxically, lower self-rating may reflect a desire to improve the program's climate for URIM. Exploration of these findings in future studies would provide invaluable information on the discordant and subjective nature of programs' perception of their holistic efforts. Regarding survey design, some close-ended questions did not allow “not applicable” or a comment box for explanation leaving the reason for unanswered questions open to interpretation. Additionally, we asked PDs to report on DEI practices, but some could have perceived the responses as connected to whether their program's practices were morally good or bad. As a result, PD responses may have been influenced by potential moral dilemmas and social desirability bias. Selection and recall bias must also be considered. We tried to combat recall bias by administering the survey immediately after the closing of the recruitment season. Selection bias is less of an issue because of our 93% response rate. This survey has not been validated and we did not pilot test the survey. Finally, this study took place at a single institution which may limit generalizability.
Conclusions
Increasing diversity in the physician workforce is an important factor in combating health inequities, and one way to increase diversity is through holistic review. Our assessment showed that most program directors implemented at least some holistic review practices which differed by program size, URIM, and gender composition. Institutional and departmental support of program directors through the commitment of resources (eg, staffing help, protected time, and hiring of subject matter experts/coaches in DEIJ) are crucial for the adoption of HRPs.
Supplemental Material
sj-docx-1-mde-10.1177_23821205241260243 - Supplemental material for Facilitators and Barriers to the Adoption of Holistic Practices for Inclusive Recruitment in Graduate Medical Education
Supplemental material, sj-docx-1-mde-10.1177_23821205241260243 for Facilitators and Barriers to the Adoption of Holistic Practices for Inclusive Recruitment in Graduate Medical Education by Andreia B. Alexander, Levi Funches, Sheryl Katta-Charles, Francesca A. Williamson, Curtis Wright, Areeba Kara, James E. Slaven and Zeina Nabhan in Journal of Medical Education and Curricular Development
Supplemental Material
sj-docx-2-mde-10.1177_23821205241260243 - Supplemental material for Facilitators and Barriers to the Adoption of Holistic Practices for Inclusive Recruitment in Graduate Medical Education
Supplemental material, sj-docx-2-mde-10.1177_23821205241260243 for Facilitators and Barriers to the Adoption of Holistic Practices for Inclusive Recruitment in Graduate Medical Education by Andreia B. Alexander, Levi Funches, Sheryl Katta-Charles, Francesca A. Williamson, Curtis Wright, Areeba Kara, James E. Slaven and Zeina Nabhan in Journal of Medical Education and Curricular Development
Footnotes
The authors would like to acknowledge the IUSM Program directors for their participation in the survey and the GME staff Adrienne Phillips, Megan Overgaard, and Jess Esch Risk for their administrative support.
FUNDING
The author(s) received no financial support for the research, authorship, and/or publication of this article.
DECLARATION OF CONFLICTING INTERESTS
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Author Contributions
Andreia Alexander, MD: conceptual design, analysis, and interpretation of the data, wrote the first draft of the manuscript and critically revised the article. Levi Funches, MD: conceptual design, analysis, and interpretation of the data, wrote the first draft of the manuscript and critically revised the article. Sheryl Katta-Charles, MD: conceptual design, interpretation of the data, critically revised the article. Francessca Williamson, PhD: conceptual design, acquisition, and interpretation of the data, critically revised the article. Curtis Wright, MD: conceptual design, acquisition, and interpretation of the data, critically revised the article. Areeba Kara, MD: conceptual design, acquisition, and interpretation of the data, critically revised the article. James Slaven, MS: analysis and interpretation of the data, drafted and critically revised the article. Zeina Nabhan, MD: conceptual design, acquisition, analysis, and interpretation of the data, critically revised the article.
Ethical Approval
This work was approved by the Indiana University Institutional Review Board as an exempt study. Informed consent was not required. Ethical approval is not applicable to this type of work.
Informed Consent
This work was approved by the Institutional Review Board as an exempt study, and informed consent was not applicable.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
