Abstract
Summary
Objective
A recent independent review on diversity and inclusivity highlighted concerns that barriers to surgical career progression exist for some groups of individuals and not others. Group-level differences in performance at the Intercollegiate Membership of the Royal Colleges of Surgeons (MRCS) examinations have been identified but are yet to be investigated. We aimed to characterise the relationship between sociodemographic differences and performance at MRCS.
Design
Retrospective cohort study.
Setting
Secondary care.
Participants
All UK MRCS candidates attempting Part A (n = 5780) and Part B (n = 2600) between 2013 and 2019 with linked sociodemographic data in the UK Medical Education Database (https://www.ukmed.ac.uk).
Main outcome measures
Chi-square tests established univariate associations with MRCS performance. Multiple logistic regression identified independent predictors of success, adjusted for medical school performance.
Results
Statistically significant differences in MRCS pass rates were found according to gender, ethnicity, age, graduate status, educational background and socioeconomic status (all p < 0.05). After adjusting for prior academic attainment, being male (odds ratio [OR] 2.34, 95% confidence interval [CI] 1.87–2.92) or a non-graduate (OR 1.98, 95% CI 1.44–2.74) were independent predictors of MRCS Part A success and being a non-graduate (OR 1.77, 95% CI 1.15–2.71) and having attended a fee-paying school (OR 1.51, 95% CI 1.08–2.10) were independent predictors of Part B success. Black and minority ethnic groups were significantly less likely to pass MRCS Part B at their first attempt (OR 0.41, 95% CI 0.18–0.92 for Black candidates and OR 0.49, 95% CI 0.35–0.69 for Asian candidates) compared to White candidates.
Conclusions
There is significant group-level differential attainment at MRCS, likely to represent the accumulation of privilege and disadvantage experienced by individuals throughout their education and training. Those leading surgical education now have a responsibility to identify and address the causes of these attainment differences.
Introduction
Equity and fairness are fundamental values that must be prioritised within medical training and assessment to create a diverse and inclusive medical workforce. 1 This is not the case in UK surgery as evidenced by limited diversity within the consultant surgeon population and senior leadership roles, presented in a recent independent review on diversity and inclusivity led by Baroness Helena Kennedy. 1 This report highlighted widespread concern that barriers to progression in surgical careers exist for some groups of individuals and not others. The first step in addressing the issue – and thus enabling equity and fairness within surgery – is identifying barriers to progression.
Recent research identifying group-level differences, or differential attainment, at the Intercollegiate Membership of the Royal Colleges of Surgeons (MRCS) examinations poses the question of whether this examination is a barrier to career progression for some groups. 2 Scrimgeour et al. 2 found that men, White candidates and younger candidates were significantly more likely to pass MRCS at the first attempt, suggesting that individual differences in personal and social circumstances may impact examination outcomes. However, this study included data on a limited number of sociodemographic differences. It was, therefore, unable to adjust for numerous other confounding variables including prior academic performance, which is known to be the best predictor of later success in medical assessments.3 –6 Therefore, it remains to be seen whether sociodemographic factors such as gender and ethnicity are independently associated with differential attainment at MRCS or whether their association with performance is largely related to other individual differences or ability, as indicated by prior academic attainment.
Successful completion of MRCS Part A (written component) and Part B (objective structured clinical examination) is a prerequisite for entry into UK higher surgical specialty training. The use of MRCS as a ‘gatekeeper’ for entry to higher surgical specialty training means that examination performance is likely to have a significant and lasting impact on the career progression of trainees. These concerns have prompted investigation into differential attainment in postgraduate examinations, including MRCS, to highlight and address potential barriers to career progression for some groups of trainees.1,7 In the current study we aimed to characterise and understand the relationship between sociodemographic differences and performance at MRCS. Given that prior academic attainment is known to be the best predictor of later success in medical assessments,3 –6 we also adjusted analyses for candidates’ performance at medical school to isolate the true relationship between sociodemographic factors and MRCS success.
Methods
This was a retrospective cohort study using data from the UK Medical Education Database (UKMED) and the four Royal Colleges of Surgeons of the UK and Ireland (Edinburgh, Glasgow, England and Ireland). UKMED links educational outcomes for all trainees within the UK by regularly crosslinking data from a number of sources, including the Higher Education Statistics Authority (HESA) Limited and the General Medical Council. For more information on UKMED please refer to: https://www.ukmed.ac.uk/. Anonymised data were extracted for all UK graduates who attempted either MRCS Part A or Part B between April 2013 and May 2019. The study period was established between these dates to include the maximum number of candidates with educational performance measure (EPM) scores (described below) before the COVID-19 pandemic, thereby eliminating the effect this may have had on training and assessment.
We focused on examining differences between groups on the basis of gender, ethnicity, age and indicators of socioeconomic status.4,8 –16 The following standardised and anonymised data were extracted from UKMED prior to analysis: gender, ethnicity, age; graduate status at the time of entry to medical school; parental education; parental occupation; participation of local areas (POLAR) quintile (which classifies areas of the UK into categories according to the level of participation of young people in higher education and ranges from quintile 1 (lowest participation in higher education) to 5 (highest participation)); school type; entitlement to income support and free school meals; index of multiple deprivation (IMD) quintile (IMD identifies small zones of deprivation throughout the UK mapped to socioeconomic domains and range from quintile 1 (most deprived) to quintile 5 (least deprived)); and EPM score (see later). First attempt examination scores were used throughout, as these have been shown to be the best predictor of future performance in postgraduate examinations.2,4,5
Measures of socioeconomic status
Variables were linked to MRCS performance on an individual level by UKMED. Measures of educational background included: parental education (university-educated or not); parental occupation (mapped to national statistics socioeconomic codes on a scale of 1 to 5 and dichotomised into managerial and professional occupations (1) versus other occupations (codes 2–5) as used in previous studies).15,16 POLAR scores were dichotomised for analysis with POLAR quintiles 1 and 2 representing students from the lowest participation areas versus students from quintiles 3, 4 and 5.15,16 High-school education was dichotomised into state (non-fee paying) or fee-paying school.
Measures of socioeconomic status were similar to those used in previous studies15,16 and included IMD quintile and entitlement to income support and free school meals. IMD quintiles were dichotomised into 1 and 2 (commonly used in higher education to identify most disadvantaged, or ‘widening participation students’) versus quintiles 3, 4 and 5. The dichotomisation of these variables also results in larger subgroup sizes, maximising statistical power for analyses and enabling more precise estimates of effect sizes. Both POLAR and IMD use UK postcodes for the classification of applicants. Therefore, POLAR and IMD scores were included in analyses only for non-graduate entry medical students, as these were most likely to represent the parental/childhood home (as opposed to a university dwelling for those who had undertaken a prior degree).
Although many non-UK educated doctors sit the MRCS, the focus of our study was on UK graduates because sociodemographic indicators are context-specific. For more information on the measures of socioeconomic status used in this study, please refer to the UKMED data dictionary (https://www.ukmed.ac.uk/).
Adjustment for prior academic attainment
The performance of medical school graduates in the UK is quantified by an EPM score on completion of medical school. The EPM is the sum of three scores; points awarded for additional degrees (maximum of 5 points), points awarded for publications (maximum of 2 points) and a decile score based on their performance throughout medical school, ranging from 34 points for the 10th (lowest) decile to 43 points for students in the 1st (highest) decile. Previous studies have found that the EPM decile score demonstrates the most predictive value6,17 of the three component parts of the EPM. Additionally, points for additional degrees and publications are more accessible to students from more affluent backgrounds, who are able to ‘pay for points’, creating a financial barrier to success. 6 Therefore, and as per previous studies, EPM decile scores (not EPM total score) were used as a measure of prior academic attainment. 18
Statistical analysis
All analyses were conducted using SPSS® v22.0 (IBM, Armonk, NY, USA). Univariate analysis using chi-square testing was initially employed to determine any associations with first attempt MRCS pass/fail outcomes. To avoid a high level of multi-collinearity within regression models, Spearman’s Rho correlation coefficients were first calculated for each measure of socioeconomic status (Supplementary Table 1). Where a high correlation coefficient was found between two variables, only one was entered into logistic regression models as per previous studies (e.g. free school meals was carried forward instead of income support and parental education was carried forward instead of parental occupation). 16 There was a large statistically significant correlation between age at graduation and being a graduate on entry to medicine (r = 0.50, p < 0.001). Of all candidates aged >29 years at graduation from medical school, 80% (390/485) were graduate students, and 98% (4640/4735) of all candidates that did not have a degree prior to starting medicine were aged <29 years at the time of graduating from medical school. Therefore, as these two variables represent the same cohort (i.e. mature candidates vs. their younger peers), graduate status being a binary variable was carried forward in regression models. Missing data are stated where present and analyses were performed on a complete case basis to allow easier interpretation and validation by others.
Initial univariate logistic regression models were developed to identify predictors of success at MRCS at the first attempt. Further regression models were developed to identify predictors of MRCS success that were independent of other sociodemographic factors. The first multivariate model adjusts for sociodemographic factors with little or no missing data. The second also adjusts for a measure of prior academic attainment (EPM decile scores). The third multivariate model adjusts for all sociodemographic factors and the final logistic regression model identifies independent predictors of MRCS success after adjusting for all sociodemographic factors and prior academic attainment. Effect sizes are given as odds ratio (OR) (95% confidence interval [CI]). Potential interactions between significant predictors were also examined and stated where found.
When handling, storing and analysing data, the highest standards of security, governance and confidentiality were maintained. In line with the HESA standards (www.hesa.ac.uk), all counts presented have been rounded to the nearest 5 to ensure person-level anonymity.
Results
Sociodemographics of study cohort.
Univariate analysis of Membership of the Royal College of Surgeons (MRCS) examinations first attempt pass rates by sociodemographic variables.
SD: standard deviation; n: number of candidates.
Logistic regression model showing predictors of pass at the first attempt at MRCS Part A for UK medical graduates before and after accounting for prior academic performance.
Note: First category displayed within each variable was used as the reference.
CI: confidence interval.
Black candidates and candidates from other minority ethnic groups were less likely to pass MRCS Part A at the first attempt when adjusting for gender, ethnicity, graduate status and prior attainment (OR 0.66, 95% CI 0.45–0.98 and OR 0.65, 95% CI 0.47–0.88, respectively). However, after adjusting for all sociodemographic factors including measures of socioeconomic and educational background, these were no longer independently predictive of MRCS Part A outcomes.
Logistic regression model showing predictors of pass at the first attempt at MRCS Part B for UK medical graduates before and after accounting for prior academic performance.
Note: First category displayed within each variable was used as the reference.
CI: confidence interval.
Of note, there was no significant change to the final logistic regression model results for MRCS Part B when entitlement to income support and parental occupation were carried forward instead of free school meals and parental education. None of these measures of socioeconomic status were statistically significant independent predictors of MRCS Part B success (p > 0.05).
Discussion
Our findings highlight significant associations between individual differences (sociodemographic factors) and performance at MRCS, despite few of these being independent predictors of MRCS success.
Gender
Differential attainment describes variation in performance between different groups taking the same assessment. In this study, we found that men were significantly more likely to pass MRCS Part A at the first attempt than women, although gender was not independently associated with MRCS Part B success when all sociodemographic factors were accounted for (see also literature2,4,6). Although it is reassuring that no differential attainment was identified in the face-to-face components of the MRCS examination, making examiner discrimination less likely, further work is required in order to rule out gender bias at a question level. Men have also been found to perform significantly better in surgical training appraisals 19 and Fellowship of the Royal College of Surgeons,4,20 indicating that this pattern is widespread. Interestingly, while gender differences in other high-stakes medical assessments are well documented, it is usually women that perform better than men.8,10,21,22
Further research is required to ascertain whether men outperform women in surgical assessments or whether structural or systemic factors exist in training and assessment which privilege men. We suspect the latter, given the culture of surgery has long been criticised for its inflexibility, discrimination and lack of female role models.1,23 The growing number of female consultant surgeons in many specialties and the increasing election of female surgeons to senior leadership roles may help change culture and patterns of progression in surgery. 1
Ethnicity
As per previous studies, MRCS success was associated with ethnicity.2,6,18 Our data also reflect patterns in differential attainment seen throughout the wider literature. 24 White candidates have been found to perform better at school, 3 medical school 14 and in almost all postgraduate medical examinations 8 including the Membership of the Royal College of Physicians,10,12 Membership of the Royal College of General Practitioners,12,13 Membership of the Royal College of Psychiatrists 11 and United States Medical Licencing Examination. 9
The reasons for this attainment gap remain poorly understood. Differential attainment exists despite adjusting for prior academic attainment. This suggests that differential attainment exists at a structural level, with some groups accumulating disadvantage throughout their education,24,25 including at the postgraduate level, 26 which potentially results in the systemic attainment gap seen in postgraduate assessments.
Maturity
Younger candidates (defined as aged <29 years in previous studies2,5,27) and non-graduates performed significantly better at MRCS than their peers. Graduates have been found in other studies to perform at least as well as undergraduate students in medical school examinations 28 but appear to perform less well throughout postgraduate training4,8,19,27 and on other postgraduate examinations.4,9 Whether this differential attainment exists as a result of competing time, family and financial demands 27 or other barriers to success in postgraduate training for older doctors-in-training, or a combination thereof, is currently unknown.
Educational background
Despite similar prior attainment (EPM decile score), candidates from lower higher-education participation neighbourhoods or who were first-in-family to attend university performed significantly worse at MRCS Part A, while candidates with parents in non-managerial or professional occupations also performed worse at MRCS Part B. This pattern reflects that seen in the wider literature.15,16,24 Additionally, candidates who attended fee-paying schools performed significantly better at MRCS Part A and were 51% more likely to pass MRCS Part B on the first attempt. Taking these findings together and reflecting the wider literature,15,24 our findings indicate that candidates from more privileged backgrounds do better at MRCS. The reasons for these patterns of performance remain unclear but may be related to differences in opportunities, role modelling, mentoring and educational support.
Socioeconomic status
Candidates from lower socioeconomic backgrounds also performed less well. Those entitled to free school meals and from the most deprived areas of the UK performed significantly worse at MRCS Part A and Part B, while those eligible for income support also performed significantly worse at MRCS Part B. Candidates from less affluent backgrounds are known to enter university with lower high-school grades and have been found to perform worse at medical school, indicating an accumulation of educational disadvantage over time.15,16,29 While it is unlikely (although this remains to be excluded) that MRCS questions are biased against candidates from less affluent backgrounds, surgical training has long been criticised for perpetuating financial barriers to success. Mandatory training courses, conference fees and even the MRCS itself pose significant financial hurdles for trainees. Those from more affluent backgrounds may be able to afford more courses, conferences and other learning opportunities, accruing advantages that may contribute to stronger performance at MRCS.
Implications
These data reveal differential attainment in MRCS at a group level that cannot be attributed to learner deficit, suggesting that differential attainment may be the result of the assessment itself or variation in learning and training experiences. Further work is required to examine and understand the potentially complex and multifaceted reasons for these differences. Future research should include quantitative work, scrutinising MRCS Parts A and B for signs of unfairness (e.g. question and/or examiner bias) which has been investigated and largely ruled out for other postgraduate examinations in which differential attainment has been found). 12 In addition, qualitative work is needed to examine differences in the nature of the learning and assessment environment between groups of trainees.
The wider literature suggests that differential attainment at postgraduate assessment is the lens through which we see the accumulation of educational privilege and disadvantage. This privilege and disadvantage may take place at a macro (policy/systemic), meso (institutional/local) and micro (individual) level at each stage of a candidates’ education and training. 30 This must be borne in mind by those leading surgical education and training who now have a responsibility to act on these data, to identify and address the causes of differential attainment in the MRCS. 1 It is clear that systemic change is required in surgery if differential attainment and inequity in education and training is to be addressed. There needs to be a move away from the ‘deficit model’ of thinking and acknowledgement that differential attainment seen in the postgraduate setting is likely the result of a combination of inequity in social and educational opportunities and accumulated historical bias and discrimination experienced by some groups of individuals and not others.1,26 Key differences in experiences of the medical learning environment by some groups of trainees have been highlighted within the literature and include a lack of belonging, a lack of mentors, coaching and role models in senior leadership positions, reduced social capital and its subsequent limitation on the development of supportive social networks.1,24,26,30 This, combined with ongoing reports of bullying, harassment, microaggressions and discrimination, creates a hostile and unsupportive learning environment experienced by some groups of trainees and not others.1,26,30 A greater understanding of these issues and how they affect surgical trainees will enable the development of supportive, inclusive and equitable surgical training programmes.
Additionally, these data highlight groups of individuals at increased risk of failing MRCS. Training providers can use these results to provide additional support and resources to those most in need, mitigating some degree of accumulated disadvantage and enabling equitable training and career progression. Such support and resources may reasonably include mentorship programmes, greater flexibility in training programmes and the provision of grants and bursaries to make access to training opportunities and revision resources equitable. While it may be argued that targeted support programmes stigmatise the recipients, the current model of equal support offered at a policy level and remediation training offered only to those who have already failed MRCS at multiple attempts has so far disproportionately benefited some groups and perpetuated differential attainment at the postgraduate level. There is a paucity of studies looking at the effectiveness of actions designed to address postgraduate differential attainment; therefore, it is imperative that any future interventions are sustainable, have clearly defined outcome measures that aim to reduce differential attainment and are audited regularly for their effectiveness.
Strengths and limitations
To our knowledge, this large cohort study is the first to assess the relationship between specific sociodemographic factors and MRCS success after adjusting for measures of prior academic attainment. Our findings provide a starting point for understanding how the accumulation of social and educational disadvantage or privilege can impact MRCS performance.
Cohort studies using large datasets are inevitably limited by the data that are available and the limits of statistical analysis. For example, ethnicity was categorised within the dataset into one of five groups to maximise the sample size and resultant statistical power for comparisons. While pragmatic, this approach has been criticised for failing to recognise the diversity and intersectionality of identities and experiences within such broad groupings. 24 Similarly, the term ‘Black and Minority Ethnic groups’ is not granular but mirrored terminology used in previous research and has allowed us to contextualise and compare the study findings. Despite the UKMED database containing data from a number of sources, a degree of missing data for sociodemographic variables is inevitable and it can be argued that this may limit the generalisability of the results. However, the UKMED database is one of the most comprehensive sources for longitudinal medical education data, providing a unique opportunity to conduct analyses on a complete case basis with no multiple imputation performed, allowing comparability with future studies. Sensitivity analyses were performed including multivariate regression analyses adjusting for sociodemographic factors with little or no missing data. Comparison of these with full multivariate regression analyses that included all factors revealed similar findings between these groups ruling out bias caused by missing data in the subgroup analyses.
As in previous studies, the outcome measure of pass/fail was used since this is what is meaningful to those sitting MRCS.2,6,18 Individual MRCS question and station data were not available, potentially hiding group-level variation in performance. Future studies aiming to rule out bias in questioning will require this data to enable a more forensic analysis using differential item functioning.
A-level (high-school exit examination) scores were considered an alternative measure of prior academic attainment. However, continued grade inflation limits the spread and predictive value of A-levels (resulting in the increased reliance on admissions testing for medical school selection purposes) and A-level performance is known to be influenced by external factors such as social class and educational background. Given this, we believe EPM decile scores were the most suitable measure of prior academic attainment in this study.
Finally, our focus was UK graduates because of the availability of socioeconomic measures for this group. International medical graduates are known to exhibit significant differences in performance in many postgraduate examinations compared to UK graduates. Further work is required to investigate whether differential attainment exists for international graduates at MRCS. This will likely require a significantly larger study population to enable meaningful statistical analyses given the likelihood of higher levels of missing sociodemographic and prior academic attainment data, and the difficulty of comparing across contexts.
Conclusions
This study identified significant differences in MRCS performance between sociodemographic groups, likely to represent the accumulation of privilege and disadvantage experienced by individuals throughout their education and training. Further work is required to identify the causes of this differential attainment, rule out bias at MRCS and examine differences in the nature of the learning environment between groups of trainees.
Supplemental Material
sj-pdf-1-jrs-10.1177_01410768221079018 - Supplemental material for Differential attainment at MRCS according to gender, ethnicity, age and socioeconomic factors: a retrospective cohort study
Supplemental material, sj-pdf-1-jrs-10.1177_01410768221079018 for Differential attainment at MRCS according to gender, ethnicity, age and socioeconomic factors: a retrospective cohort study by Ricky Ellis, Peter A Brennan, Amanda J Lee, Duncan SG Scrimgeour and Jennifer Cleland in Journal of the Royal Society of Medicine
Footnotes
Declarations
Acknowledgements
The authors would like to acknowledge Iain Targett at the Royal College of Surgeons of England, for his help with data collection and John Hines and Gregory Ayre from the Intercollegiate Committee for Basic Surgical Examinations for their support during this project. Our thanks to members of the UKMED Research Group who provided useful feedback on an earlier version of this manuscript, and whose comments helped refine the paper. The authors would also like to acknowledge Daniel Smith for his help with the UKMED database. Data Source: UK Medical Education Database (“UKMED”). UKMEDP043 extract generated on 25 July 2018. We are grateful to UKMED for the use of these data. However, UKMED bears no responsibility for their analysis or interpretation. The data include information derived from that collected by the Higher Education Statistics Agency (“HESA”) Limited and provided to the GMC (“HESA Data”). Source: HESA Student Records 2007/2008 to 2015/2016. Copyright Higher Education Statistics Agency Limited. The Higher Education Statistics Agency Limited makes no warranty as to the accuracy of the HESA Data, cannot accept responsibility for any inferences or conclusions derived by third parties from data or other Information supplied by it. The dataset used in this study was acquired from the UK Medical Education Database and is held in Safe Haven. All counts have been rounded to the nearest 5 according to HESA data standards to ensure person-level anonymity, as per UKMED policy. Data access requests must be made to UKMED. Full information for applications can be found at 
.
Provenance
Not commissioned; peer reviewed by Carla Hope, Mala Rao and Julie Morris.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
