Abstract
Objective
To report the selection criteria important to residency program directors (PDs) and whether they believe pass/fail scoring will impact underrepresented in medicine (URM), International Medical Graduate (IMG), or osteopathic (DO) residency applicants after the United States Medical Licensing Examination (USMLE) changed Step 1 score reporting to pass/fail in January 2022.
Methods
A Qualtrics survey was sent between August 2022 and January 2023 to 1141 US PDs from specialties with traditionally low residency selection rates: dermatology, ophthalmology, otolaryngology, orthopedic surgery, neurosurgery, interventional radiology, diagnostic radiology, radiation oncology, thoracic surgery, vascular surgery, plastic surgery, and urology. Contact information was obtained from AMA, FRIEDA, or program websites.
Results
We received 433 responses (38%). When asked to anticipate the difficulty student groups will face matching into their specialty, PDs reported: for URM, 24.0% increased, 46.0% unchanged, and 30.0% decreased; for DO, 49.19% increased, 44.58% unchanged, and 6.23% decreased and for IMG, 56.35% increased, 39.72% unchanged, and 3.93% decreased. When asked to rank the most important selection factors, the top two responses were Step 2 CK score and away rotation participation at their site.
Conclusion
PDs overwhelmingly believed residency selection difficulty would either increase or remain unchanged for DO (93.77%) and IMG (96.07%). In contrast, 76.0% reported difficulty for URM students would either decrease or remain unchanged. PDs ranked Step 2 CK score and away rotation participation as the most important selection factors. Despite PDs’ belief that the Step 1 pass/fail scoring system may mitigate one barrier for URM students, emphasis on Step 2 CK and away rotations place additional barriers.
Keywords
Introduction
The United States Medical Licensing Examination (USMLE) recently changed their score reporting techniques for the Step 1 examination from a 3-digit number to pass/fail.1–3 This change has resulted in uncertainty among future residency candidates, in particular those who are underrepresented in medicine (URM). The AMA definition of URM students includes “racial and ethnic populations that are underrepresented in the medical profession relative to their numbers in the general population”. Some believe this uncertainty, coupled with the recent US Supreme court ruling that higher education admissions are not to consider race, may place URM applicants at a considerable disadvantage. 4
URM applicants face unique barriers when applying to residencies, as they often experience a disparity in their Step 1 scores and an overall achievement gap when compared to their non-URM peers due to systemic inequities throughout their education prior to medical school matriculation. 5 It has been reported that the achievement gap has negative physical and psychological consequences on URM students, and after entering medical school they may internalize or externalize negative stereotypes which can impact standardized exam performance. 6 In the past, when compared to non-URM candidates, URM residency applicants had lower mean Step 1 and Step 2CK scores.7,8 Studies suggest that a reduction in the reliance on Step 1 scores may mitigate racial biases. 9 Due to the weight that residency program directors (PDs) historically ascribed to Step 1 scores in choosing applicants for residency interviews, URM applicants may have been screened out of interview opportunities, a point that was emphasized by the Invitational Conference on USMLE Scoring when they recommended minimiz(ing) “racial demographic differences that exist in USMLE performance”. 6
A few recent studies have also explored how the Step 1 scoring change may impact specific residencies’ application screening processes. In one study, about 50% of dermatology PDs believed that all medical students will have more difficulty matching into dermatology. Dermatology PDs anticipate focusing more on letters of recommendation (LOR), audition/away rotations, and Step 2CK scores. 10 Another study surveyed medical student deans; when asked to rank the importance of various application factors, the most frequently selected first-choice was Step 2CK score. Collectively, medical school deans reported feeling that students who apply to dermatology, neurosurgery, orthopedic surgery, ENT, and plastic surgery would be most impacted. 11 In a survey of vascular surgery PDs, 42% of respondents reported their intent to replace the Step 1 score with the Step 2CK score. 12 Neurosurgery PDs believed similarly that Step 2CK will be of heightened importance and may effectively “replace” the role of Step 1 within the residency selection process. 13 Similarly, a survey of plastic surgery PDs reported that LOR, Step 2 CK scores, and away rotations at their institution would be most important. Many stakeholders have expressed concerns about removing the standardized Step 1 score and posit additional issues that may arise from this scoring change.14–16 Some have highlighted the potential impact of structural biases on URM applicants, given the existing socioeconomic inequities that lead to reduced financial and community resources among the URM population. 17
A study by Makhoul et al 16 reported valuable data on the perspectives of PDs from 30 specialties on the pass/fail Step 1 scoring change; overall, 77.2% of PD respondents believed that all applicants will have a more difficult time matching into their specialty. Although they discussed how this new policy may impact the overall process and specifically addressed the potential impact on International Medical Graduate (IMG) and, to a lesser extent, DO applicants, they did not explore the impact on URM applicants. To date, no study has explicitly surveyed PDs regarding how they believe URM applicants will be impacted by the Step 1 pass/fail policy.
Twelve competitive US residency specialties were selected for this study due to their relatively low rates of admitting URM applicants, osteopathic graduates (DO), and IMG.18–20 Importantly, many who advocated for the Step 1 scoring change suggested that the change to pass/fail scoring will benefit URM candidates. We aimed to collect data to better understand the perspectives of PDs and included all three groups to ensure full inclusivity and to allow for a sort of control—as the difficulty of DO and IMG applicants matching into competitive specialties is well established. For the purposes of this study, URM included both allopathic and osteopathic grads, and IMG included both US and non-US grads.
We specifically focused on identifying what selection factors PDs from these twelve specialties consider to be most important now that Step 1 is pass/fail, and if they believe this change will impact the residency selection difficulty of URM, DO, and IMG applicants.
Methods
Study design
A cross-sectional study design was used in this online survey study. Coauthors (RK, JK, KHP) created the survey based on a literature review of relevant topics and exploration of other PD surveys. The questionnaire consisted of two mandatory sections utilizing ranking of importance of factors when selecting applicants and Likert scale multiple choice style questions to allow participants to express how strongly they agreed or disagreed with statements related to the factors they anticipate taking into consideration when reviewing residency applications (Online Supplemental Appendix). PDs, who did not complete the survey in its entirety, were excluded from results. One key reason a survey was utilized was due to the distance between participants across the United States. Another consideration is the unpredictability of a physician's schedule, and with the possibility to slowly complete an online survey over time, the likelihood of successful completion is increased. Past literature search was completed on PubMed and other PD surveys were reviewed prior to survey question creation.
Study participants
PDs in 12 competitive specialties were identified as study participants by stratified sampling: dermatology, ophthalmology, otolaryngology, orthopedic surgery, neurological surgery, interventional radiology, diagnostic radiology, radiation oncology, thoracic surgery, vascular surgery, plastic surgery, and urology. Inclusion criteria for the competitive specialties were twofold: accreditation by the Accreditation Council for Graduate Medical Education (ACGME) and a published mean Step 1 score average of ≥239 for those who matched into their specialty in the 2020 residency match. 21 Residency programs were broken down geographically, using definitions of regions set by The National Geographic Education – United States Regions. 22
Survey instruments and dissemination
A qualtrics survey was administered to all 1141 residency PDs from the twelve listed specialties. The survey link was sent to the publicly listed email addresses available on the AMA, FRIEDA, or the institution's website. The email that accompanied the survey was constructed with approval from NYMC Institutional Review Board. Potential participants were excluded if contact information could not be found on the sites listed or if they were a PD of a specialty that is not listed above. Informed consent was granted from all participants prior to administration of the survey, and participants were instructed to only continue with the survey if they granted consent; the NYMC Institutional Review Board approved this consent procedure (Online Supplemental Appendix). Duplicate surveys were sent to program coordinators to encourage collaboration and responses. Pretesting was completed with a limited nontarget audience who had awareness of the residency selection process (eg, staff members, but not PDs) to verify all online features of the survey were working properly.
Data collection
Data were collected from August 2022 to January 2023. Data were deidentified and aggregated prior to analysis. If any data were missing, the entire data set for that participant was excluded.
Statistical analysis
IBM SPSS version 28.0.0.0 (190) was employed for the analysis of both quantitative and qualitative data utilizing “crosstabs” and “descriptive statistics” functions; the sample size selected was predicated on survey responses. Continuous data were described using means and standard deviations, while categorical variables from the survey were presented as frequencies. Through the utilization of Microsoft Excel, the “Step 2CK score/COMLEX Level 2 score” and “away rotation at your site” variables were ranked, represented as percentages, and then converted into graphical illustrations.
Ethical considerations
This study (#15162) was deemed exempt by the NYMC Institutional Review Board by the General Medical and Behavioral panel on 7/28/2022.
Results
A total of 433 responses were received (38%) out of the 1141 accredited residency programs contacted. Within the 433 responses, each specialty was represented as follows: 11% (n = 49) dermatology, 18% (n = 76) diagnostic radiology, 4% (n = 19), interventional radiology, 10% (n = 43) neurosurgery, 8% (n = 36) ophthalmology, 13% (n = 58) orthopedic surgery, 10% (n = 42) otolaryngology, 5% (n = 23) plastic surgery, 5% (n = 20) radiation oncology, 2% (n = 7) thoracic surgery, 10% (n = 43) urology, and 4% (n = 17) vascular surgery. Response rates within each specialty ranged from 24% to 48% (Table 1). A geographical breakdown of responses appears in Table 2. Notably, the highest response rate was from PDs in the northeast (30.7%), followed by the southeast (24.7%).
Specialty response data.
Geographical response data.
Across all 12 specialties, 24% (n = 433) reported believing that the Step 1 pass/fail policy will make it more difficult and 30% reported it will make it less difficult for URM applicants to match into their specialty, with a 6% net effect of less difficult (Table 3). None of the twelve specialties reported greater than 40% of PDs anticipating increased residency selection difficulty for URM applicants. However, 46% of PDs reported the difficulty would stay the same. Regionally, two trends were observed with respect to likelihood of URM applicants to be selected for residency: PDs in the west had a higher number of those reporting “less difficult” and a lower number of those reporting “more difficult” compared to other regions.
Subgroup data.
PDs were asked to report whether they felt the Step 1 pass/fail policy would impact the residency selection difficulty experienced by DO applicants (Table 3). Overall, 49.2% (n = 433) of all PDs reported it would be more difficult, while 6.2% reported it would be less difficult (43% net effect more difficult) and 44.6% reported it would be the same as in past years. Specialties in which greater than 50% of PDs reported increased difficulty for DO applicants included: plastic surgery (69.6%), neurosurgery (60.5%), otolaryngology (57.1%), and vascular surgery (53%). Notably, none of the neurosurgery, plastic surgery, and thoracic surgery PDs believed the pass/fail policy would decrease match difficulty for DO applicants. Regionally, two trends in the frequency of responses were observed: Fewer PDs in the west selected “more difficult,” while more PDs in the midwest responded “less difficult.”
Across all 12 specialties, 56.4% (n = 433) of PDs reported believing that the Step 1 pass/fail policy will make it more difficult for IMG applicants to be selected for residency, whereas 3.9% believed it would be less difficult (52% net effect more difficult), and 39.7% of PDs believed it would be the same as in past years (Table 3). Specialties in which greater than 50% of PDs believed it would increase difficulty for IMG included: Plastic surgery (78.3%), radiation oncology (75%), ophthalmology (69.4%), diagnostic radiology (61.8%), neurosurgery (58.1%), thoracic surgery (57.1%), vascular surgery (53%), and urology (51.6%). Notably, none of the orthopedic or plastic surgery PDs believed the pass/fail policy would decrease match difficulty for IMG applicants.
When PDs were asked to rank specific academic criteria (1-7) in the order of importance for interview selection (Options were: Step 2 CK score, Alpha Omega Alpha membership, class rank, tier of medical school, published research, research year/fellowship, and clinical grades), 61% of PDs ranked Step 2CK score as either #1 or #2 (Figure 1).
Percent specialty ranking Step 2 CK.
When PDs were asked to rank personal attribute criteria (1-6) in the order of importance (Options were: Away rotation at your site, strong clinical letter of recommendation, previous knowledge of the applicant, leadership in clubs, volunteer hours, and personal statement), 75% of PDs ranked an away rotation at their site as #1 or #2 (Figure 2).
Percent specialty ranking away rotation at their site.
Discussion
In the absence of a numerical Step 1 score, the majority of PDs from 12 competitive specialties anticipate the two most important residency interview selection factors will be Step 2CK score and participation in an away rotation at their site. While this is not surprising and supports the results of previous work, this study is unique in that we focused on twelve of the most competitive specialties with some of the lowest match rates for our three selected subgroups and explicitly asked PDs about the potential impact on these applicants. Notably, our study built on a previous study 16 by specifically asking how the pass-fail scoring change would impact URM applicants. Overall, there was a stark contrast in PDs’ beliefs regarding the difficulty of matching between IMG and DO applicants (collectively) and URM applicants. The intention of changing the USMLE scoring to pass/fail was to mitigate barriers. However, data reported in this and other studies illuminate potential unintended consequences for all three student subgroups.
Historically, students would apply for residency with only a Step 1 score and would take Step 2CK after their residency application was already submitted. This often reduced the pressure felt by students when applying. However, given the results of this and other studies that PDs believe Step 2CK is one of the new most important factors when considering an applicant's residency selection, there is no pressure for students to opt to take this exam prior to their application submission date. To address this issue, many medical schools have shifted their recommendations and curricular timelines, requiring their students to complete Step 2CK prior to the beginning of their fourth year. 23
In this study, 46.0% of PDs believed that the match difficulty will remain the same for URM applicants and 30.0% of PDs reported it would be less difficult. Step 1 was previously reported by PDs as the most important factor in ranking applicants, 13 and non-White examinees historically score lower on both Step 1 and Step −2CK.7,8 Hence, to date, URM applicants have represented a disproportionately smaller percentage of those selected for interviews and are some of the least represented among these 12 competitive specialties.24,25 The previous Step 1 scoring system placed URM applicants at a disadvantage, and some proponents of the scoring change feel the pass/fail system will mitigate this divide. 6
While 76% of PDs from competitive specialties believed the score change will make it less difficult or stay the same for URM students, it should be noted that performance on Step 1 has been shown to be an indicator for Step 2CK performance, an exam that now has enhanced emphasis.14,26 Indeed, previously published data demonstrate that URM residency applicants had a lower mean Step 1 and Step 2CK score when compared to non-URM candidates. This raises the concern that one standardized exam may simply replace another.7,8 In addition, with Step 2CK being administered so late in training, it increases the amount of pressure on students to enhance their score, 19 while often requiring them to secure away rotations prior to knowing their score that may impact where they can match.
In addition, we noted an interesting trend: a higher percentage of PDs from western programs believed it would be less difficult for URM students to match than PDs from other regions. Importantly, the western region had the lowest number of responses, so these results should be viewed with caution. Nevertheless, this trend aligns with results reported in an earlier study that surveyed vascular surgery PDs about their use of Step 1 scores to screen applicants. Those located in the western region were less likely to report using Step 1 as a screening tool than other regions, so it is possible that western regions may have less of a need to prioritize Step 2CK in the absence of Step 1 scores. 12 Studies have demonstrated that individuals who are URM are more likely to practice in underserved areas, 22 and it is possible that the larger metropolitan areas in the western states are implementing more holistic admissions tools to diversify resident cohorts.
Our response rate by region made it clear that more PDs from the Eastern region participated. Among the Eastern PDs, we observed more PDs who reported using Step 1 as a screening tool and believe it will be more difficult for URM students to match. This might reflect the preferences of Eastern PD's as well as PDs from across the country for objective measures as an indicator of resident success. As mentioned before, such findings may reflect variations in how programs approach the admission process more objectively or holistically; however, it is important to consider differences in the number of applicants applying to western residency programs versus eastern residency programs. Given possible differences in the volume of applicants to programs each year, the Step 1 screening tool may be utilized more frequently in larger programs to limit applications to a number that can realistically be reviewed.
Participation in away rotations has always been an essential part of the medical student experience, especially for those interested in pursuing a competitive residency. Students obtain clinical experience among different patient populations and health systems while having the opportunity to network with residency PDs. However, away rotations can be expensive and impose financial constraints on students, which may particularly impact those from lower-income backgrounds. 27 In addition, URM students have faced barriers to receiving mentorship and making professional connections, which can limit opportunities to participate in away rotations. 28 Fortunately for URM students, numerous medical schools and professional organizations have recognized these disparities and developed programs to work toward providing more equitable access to some away rotations. 29 While it is encouraging that some funded away rotations exist for URM applicants, they are not universally available and have limited enrollment. 30
The current study has many limitations, including the variation in response rates across the regions and the twelve specialties, which prevented subspecialty analysis and thus limited specialty generalizations. There may have been a response bias toward PDs who feel strongly in favor of, or opposed to, the scoring change. While lower response rates were noted, it was encouraging to see that several of our results supported those of previous studies. It is also important to note a limitation with the questionnaire: while pilot testing was conducted, it was with a very small sample size (1%) and not of the target-audience. Limitations to this study include the inability to complete power analysis because there was not enough data (eg, no pre- and post-survey for comparison). An additional limitation is our speculation that URM and all students may feel pressured to complete away rotations thereby undertaking a greater financial burden. By including Step 2 scores as an option in our survey, we also implicitly made the assumption that given the Step 1 change in score reporting, students will choose to take Step 2CK prior to submitting their residency application. Finally, PDs from the 12 different specialties have varying experiences and might prioritize different factors in selecting residents beyond USMLE scores, making it challenging to generalize findings.
Conclusion
Our results suggest that the majority of PDs believe the USMLE Step 1 score change policy will not impact, or could even lessen, barriers for URM students to match into competitive specialties. Paradoxically, PDs believed that Step 2CK score and away rotation participation would become the two most important selection factors in the absence of Step 1 numerical scores. Hence, despite the beliefs of PDs that the Step 1 pass/fail scoring will mitigate barriers for URM students, potential unintended barriers need to be adequately considered and addressed.
While URM applicants have historically scored lower on Step 1 and Step 2CK than non-URM applicants,7,8 residency programs are left with few if any, other external objective criteria with which to predict those applicants who will be successful. We encourage residency PDs to rectify this by determining a minimum score for which they can expect a future resident to be successful, rather than focusing on one resident outscoring another.
Concerns about the residency selection process are compounded by the recent US Supreme Court ruling that disallows higher education admissions to consider race and several state initiatives that are dismantling Diversity, Equity, and Inclusion initiatives. It is unclear what downstream effects may result from these recent events; therefore, medical education stakeholders need to strongly promote a holistic and inclusive residency application process.
Supplemental Material
sj-docx-1-mde-10.1177_23821205241254161 - Supplemental material for Underrepresented Applicants Post-USMLE Pass/Fail: A National Survey of Competitive Residency Directors
Supplemental material, sj-docx-1-mde-10.1177_23821205241254161 for Underrepresented Applicants Post-USMLE Pass/Fail: A National Survey of Competitive Residency Directors by Rebecca Kindler, Julia Kahn, Anaz Uddin and Kristina H Petersen in Journal of Medical Education and Curricular Development
Footnotes
Author's Contribution
Rebecca Kindler contributed to the conceptualization, methodology, validation, resources, data curation, writing-original draft, visualization, supervision, project administration. Julia Kahn contributed to the conceptualization, methodology, validation, resources, writing-original draft, visualization. Anaz Uddin contributed to the software and data analysis. Kristina H. Petersen contributed to the conceptualization, methodology, validation, resources, data curation, writing-review & editing, supervision, project administration, funding acquisition.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the NYMC Medical Student Summer Research Fellowship and the Department of Biochemistry & Molecular Biology.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
