Can Life Experiences Predict Readiness for Residency? A Family Medicine Residency’s Analysis

Abstract

BACKGROUND

Program directors for Family Medicine residencies must navigate an increasingly complex recruitment landscape. With increasing United States allopathic and osteopathic graduates and continued high volumes of international graduates, the ability to identify application characteristics that predict quality residents both for filtering applications for interview offers and ranking is vital. Our study concentrates on the predictive value of reported life experiences including volunteerism, work experiences, prior career, research experience, and participation in medical student organizations including student leadership.

METHODS

Through a retrospective cohort study, we extracted the described life experiences from resident application materials. We then obtained initial clinical performance data on the Family Medicine inpatient service during the first six months of residency to determine readiness for residency. This analysis occurred in 2020 and included all matriculants in the graduating classes of 2013 through 2020 for a single residency. Of 110 matriculating residents, data were available for 97(88%).

RESULTS

Applicants with a history of a prior career demonstrated improved overall readiness for residency with competency domain-specific advantages in Interpersonal and Communication Skills and Systems-Based Practice. In contrast, applicants reporting participation in research performed below peers in all competency domains. Applicant reports on volunteerism, work experience, academic productivity and student involvement did not correlate with initial clinical performance.

CONCLUSIONS

Residency directors should recognize applicants with prior careers as likely having strong communications and systems-based practice skills. All other examined experiences should be evaluated within the context of broader applicant assessments including research experience which overall has a potential negative correlation to clinical readiness.

Introduction

Family Medicine residency programs distill a significant volume of applicant data during the recruitment process. What criteria is useful during initial sorting and the predictive value of corresponding applicant data remains unclear.^1,2 Based on the premise that past accomplishments predict successful residents, ³ we investigated whether volunteerism, work experience, prior career, research experience, or student organizational engagement were predictive of resident performance.

The importance of the biopsychosocial model of care in the specialty of Family Medicine seems to naturally support the humanism associated with volunteerism as a predictive attribute in applicants. Literature supports self-reported growth in multiple areas of the Accreditation Council for Graduate Medical Education (ACGME) competency domains after meaningful participation in service activities. One study in a developed Asian country demonstrated personal self-reported growth in empathy, communication skills, organization, decision-making, and interpersonal and leadership skills after participation in a community service experience in a cohort of medical students. ⁴ Unfortunately, literature is lacking in determining a correlation between applicant self-reported volunteerism and clinical performance as a resident.

Applicant participation in research and in associated publication activity has been reported for several medical specialties. One study of 1020 matriculants at the Uniformed Services University of the Health Sciences determined no relationship between self-reported research and success in medical school or graduate medical education. ⁵ An Internal Medicine evaluation of applicant reported research also showed no correlation to clinical performance, ⁶ while an investigation into the selection process for surgical residents found that prior research experience and research publications had a negative correlation with clinical performance ratings during residency. ⁷ Research remains a central focus in residency applications, interviews and ranking conversations for many specialties despite specialty literature questioning its value. It is worth noting the distinction between self-reported applicant research and scholarship that occurs within the context of residency training; there is a positive correlation between participation in research during residency and clinical performance. ⁸ Literature in these areas remains lacking in Family Medicine.

We further examined two areas in which little literature is available for comparison: work experience and engagement in medical school organizations. An applicant history of work experience both during education and as a separate, prior career is a self-reported component in an Electronic Residency Application System (ERAS) application. Work experience likely provides opportunities for growth in communication skills, professionalism, teamwork, organizational skills and work ethic. These attributes correlate well with multiple ACGME competencies providing a theoretical framework for applicant advantage. However, a multicenter study on the predictors of top performers in Emergency Medicine found no correlation between full-time work experience and resident performance. ⁹ Similarly, investment of personal time during medical school in organizational activities and leadership such as student government would create opportunities for anticipated growth across multiple ACGME competency domains. Although extracurricular activities such as these are looked upon favorably by program directors, ¹⁰ literature evaluating whether these characteristics were predictive of clinical performance in residency for Family Medicine or other specialties is limited.

Methods

This research occurred in an urban, medical school-based residency. Research included all matriculating residents for the graduating cohorts from 2013 through 2020. Research occurred in 2020.

Data from self-reported information in ERAS applications were extracted utilizing a standard form. This included volunteer and work experiences subdivided by type, duration and cumulative hours. A prior career defined as two or more years of established full-time employment while not engaged in education was separated from work experience for separate analysis. Research experience was divided into basic science research, non-basic science research and academic productivity through peer-reviewed publications, posters, abstracts and presentations. Engagement in medical school student organizations including leadership was gathered. Evidence of residency readiness was determined utilizing a composite of resident clinical assessments during Family Medicine inpatient rotations occurring during the first six months of training. Assessment reports were generated for each resident in composite form with utilization of quantitative data with a range of three to eight assigned evaluators. Assessments included representative milestones for the inpatient setting in each of the ACGME competency domains utilizing developmental behaviors and level of expectations as defined by the Family Medicine Residency Review Committee. The assessment tool length and value of data gathered were evaluated through a three-month pilot prior to implementation. The assessment scale mirrored the ACGME Family Medicine milestones with a one representing a novice, a two an advanced beginner, a three competent, a four proficient (graduation target) and a five an expert. The tool was 25 questions in length. Assessments were completed for all residents on the inpatient service at the conclusion of each week by the assigned rounding faculty. Delivery of assessments to faculty and results to residents occurred electronically through the residency management system.

All matriculating residents through the National Resident Matching Program (NRMP) were included in research analysis. Five residents separated from the residency due to transfer, academic or personal reasons prior to graduation. Three residents matriculated outside of the NRMP process. Data was included as available.

We selected Welch's two-sample t-test as the statistical procedure since dealing with disparate sample sizes between comparison groups.

This study was determined to be exempt by the Indiana University School of Medicine Institutional Review Board. The need for informed consent is waived by the Indiana University School of Medicine Institutional Review Board as it is secondary research. All methods were carried out in accordance with relevant guidelines and regulations.

Results

Of 110 residents who entered the program during the study period, complete assessment information was available for 97 (88%). Applicant data on the 97 applicants varied. Table 1 describes resident characteristics.

Table 1.

Demographic and life experience characteristics among matriculating residents.

	N
Female	43
Participation in scholarly activity	20
US allopathic medical school graduate	24
International medical school graduate	73
Work experience prior to medical school	60
Engagement in medical school organizations	38
Engaged in a career prior to residency Medical professional (N = 20) Researcher (N = 2) All other (N = 1 each)	27
Formal research experience Non-basic science only (N = 13) Basic science only (N = 19) Both (N = 12)	32
Publications and/or posters Named author on a publication (N = 13) Named author on a poster (N = 20) Named author on both (N = 10)	33
Volunteer experiences Healthcare setting (N = 60) Non-healthcare setting (N = 67)	82

Using the presence or absence of six life experiences, we performed comparisons between applicant groups to determine if their academic performance or clinical ratings differed. Results of the statistical procedure were: (1) applicants with less than 30 months of volunteer experience scored significantly higher in the domain of professionalism (Table 2); (2) applicants with no work experience scored significantly higher in the domain of system-based practice (Table 3); (3) applicants with a history of a prior career (distinct from work experience) scored significantly higher in the domains of Interpersonal and Communication Skills Systems-Based Practice, and on their composite score (Table 4); (4) applicants with no research experience scored significantly higher in every competency domain (Table 5); and (5) there were no significant differences for any of the domains between groups in the areas of academic productivity (publications and posters) and organizational engagement in medical school.

Table 2.

Welch's t-test for matriculating residents volunteer experience.

AssessmentDomain	Mean scores for residents having volunteer experience prior to residency (SD)		P value
	<30 months (n  =  13)	≥30 months (n  =  31)
Patient Care	3.07 (.90)	2.52 (.87)	.195
Medical Knowledge	2.91 (.89)	2.43 (.83)	.118
Interpersonal and Communication Skills	2.93 (.92)	2.42 (.93)	.108
Professionalism	3.08 (1.0)	2.35 (1.02)	.040*
Systems-Based Practice	2.51 (1.0)	2.02 (1.07)	.161
Practice-Based Learning and Improvement	2.96 (.82)	2.59 (.86)	.195
Composite	2.91 (.85)	2.39 (.91)	.085

*Difference is statistically significant @ P < .05.

Table 3.

Welch's t-test for matriculating residents work experience.

Assessment domain	Mean scores for residents having work experience prior to residency (SD)		P value
	Yes (n  =  56)	No (n  =  41)
Patient Care	2.68 (0.83)	2.85 (0.91)	.337
Medical Knowledge	2.61 (0.80)	2.81 (0.90)	.250
Interpersonal and Communication Skills	2.63 (0.85)	2.86 (1.05)	.258
Professionalism	2.59 (.94)	2.74 (1.05)	.438
Systems-Based Practice	2.24 (1.03)	2.68 (1.04)	.045*
Practice-Based Learning and Improvement	2.61 (0.80)	2.85 (0.83)	.456
Composite	2.58 (0.84)	2.78 (0.95)	.305

*Difference is statistically significant @ P < .05.

Table 4.

Welch's t-test for matriculating residents prior career.

AssessmentDomain	Mean scores for residents having a career prior to residency (SD)		P value
	Yes (n  =  24)	No (n  =  46)
Patient Care	2.91 (0.73)	2.60 (0.88)	.123
Medical Knowledge	2.89 (0.69)	2.54 (0.86)	.072
Interpersonal and Communication Skills	3.04 (0.82)	2.51 (0.96)	.019*
Professionalism	2.89 (0.80)	2.47 (1.06)	.066
Systems-Based Practice	2.76 (0.88)	2.14 (1.11)	.015*
Practice-Based Learning and Improvement	2.92 (0.67)	2.65 (0.81)	.143
Composite	2.91 (0.74)	2.49 (0.91)	.042*

*Difference is statistically significant @ P < .05.

Table 5.

Welch's t-test for matriculating residents research experience.

Assessment domain	Mean scores for residents having research experience prior to residency (SD)		P value
	Yes (n  =  20)	No (n  =  77)
Patient Care	2.19 (0.709)	2.90 (0.842)	.001*
Medical Knowledge	2.23 (0.700)	2.82 (0.841)	.003*
Interpersonal and Communication Skills	2.14 (0.766)	2.88 (0.928)	.001*
Professionalism	2.05 (0.851)	2.81 (0.960)	.001*
Systems-Based Practice	1.71 (0.786)	2.62 (1.03)	.001*
Practice-Based Learning and Improvement	2.35 (0.758)	2.88 (0.769)	.008*
Composite	2.11 (0.734)	2.81 (0.870)	.001*

*Difference is statistically significant @ P < .01.

Higher professionalism scores in residents without a history of volunteerism reached statistical significance. However, at the calculated effect size of d  =  .73, statistical power was limited to .57 with our available sample size of 44. Power analysis revealed that in order for an effect of this size to be detected reliably as significant at α  =  .05, a sample of 112 participants would be required. Similarly, the higher scores in systems-based practice for residents without work experience (prior careers excluded) reached statistical significance but a power analysis yielded a coefficient of only .24. Generating an effect size of this magnitude with a more suitable power coefficient of .80 would require a sample size of 318.

A power analysis for results pertaining the characteristics of having a prior career and research experience demonstrated higher predictive values. PGY1 residents who began graduate training after having a previous career scored significantly higher in the domain of interpersonal communication skills compared to PGY1 residents who had not had a prior career (x̅  =  3.04 vs 2.51, P  =  .019; d  =  .62, achieved power  =  .68). Previous career interns also scored significantly higher in system-based practice when contrasted with their non-career peers (x̅  =  2.76 vs 2.14, P  =  .015; d  =  .62, achieved power  =  .67). Power analysis revealed that in order for an effect of this size to be detected reliably as significant at α  =  .05, a sample of 152 participants would be required. In examining the influence of research experience, PGY1 residents with no research experience prior to residency scored higher in every domain when compared to PGY1 residents who participated in one or more such experiences. All of those differences were statistically significant at a level of P  =  .008 or lower. Effect sizes among the six domains ranged from d  =  .69 to d  =  1.01. In all comparisons for this independent variable achieved power was >.77.

Discussion

Our study revealed two unique findings. The first is that residents with a prior career performed better than peers in the competency domains of Interpersonal and Communications Skills and Systems-Based Practice along with an overall higher clinical performance composite. This finding links identifiable applicant data to areas considered valuable during recruitment.^2,9 The second was applicants reporting any participation in research were rated lower across all ACGME competency domains compared to peers. Prior studies in various specialties have demonstrated a lack of correlation between research and clinical performance with at least one study demonstrating a negative correlation in General Surgery. ¹⁰

Residents in our study who had a prior career scored significantly higher in the ACGME competency domains for Interpersonal and Communication Skills and the Professionalism. Although variable in nature, a prior career likely provides a very direct opportunity to develop skills across these domains through a structured environment that is not experienced by other applicants prior to residency. Previous studies demonstrate that program directors place Interpersonal and Communication skills only behind Clinical Skills and Medical Knowledge when ranking the importance of resident characteristics, ² and the importance of team-based functioning abilities is critical. ¹¹ Although a larger sample size with an improved power analysis is desirable, the significance of these findings in the study cohort warrant heightened consideration of prior careers during the application process for program directors seeking candidates with more advanced readiness skills in the areas of communication, interpersonal skills and systems-based practice.

In contrast, residents in our study with research experience prior to residency scored significantly lower in all six domains. The literature contains many studies demonstrating research as non-predictive for resident performance, and at least one study demonstrating a negative correlation between research experience and resident performance in General Surgery. ⁷ The cause or causes for the observed differences may include: applicants for both medical school and residency who are unsuccessful often obtain research experience to increase competitiveness; personal characteristics or personality types attracted to research differ from those that foster clinical excellence in Family Medicine; applicants with research history begin residency with less clinical exposure; or a number of other possibilities. Although reflecting on potential relationships between personal characteristics and subsequent readiness for residency, it is important to acknowledge we were unable to demonstrate causation through this study. It is also important to remember these differences between groups are within the context of applications for residency. The benefit of research participation and scholarship inquiries during residency are well documented to improve resident development and clinical performance.

We also explored several other life experience categories such as volunteering, work experience, academic productivity and student organizational engagement during medical school, but these failed to yield any significance in predicting resident performance. Although another study showed that Emergency Medicine residents with five or more presentations and publications outperformed their peers, ⁹ our study on Family Medicine residents did not yield the same result. Our results were more consistent with the conclusion that publication history is not a strong predictor of performance among residents. ⁶

Our study is predominantly limited by the nature of self-reported data in a highly competitive application process. ERAS data has been documented in the literature as being potentially inflated or even deceptive. ¹² This likely contributes to a lack of significance in areas you might expect more predictive value. Our study is further limited by utilizing data from a single academic institution that may not be as predictive for community-based residencies. However, the mix of international and allopathic residents is representative of the national norms. This study focused on applicant readiness for starting a residency; further studies including data from additional institutions with a focus on residency outcomes and specialty satisfaction after graduation from residency would be valuable extensions to this analysis.

Although this study provides valuable insights into evaluating students with established prior careers and with a history of research experience, it validates the continued need to explore better ways of assessing for characteristics of a successful resident through the application and interview process. The application materials alone are unable to provide enough predictive value. As the trends toward virtual recruitment continue to evolve, more studies are needed to determine predictive values of various applicant assessment tools during the process.

Conclusion

This research study from a single academic-based United States Family Medicine residency demonstrates the history of a prior career among residency applicants is a predictor of stronger communications skills, systems-based practice skills and overall clinical competency assessments compared to peers. This study also concludes a history of research, while often looked upon favorably during application process, does not correlate with higher clinical assessments and as a group actually correlates with lower overall clinical assessment scores.