Abstract
Introduction
The oral part of European Diploma in Intensive Care diploma examinations changed in 2013 into an objective structured clinical examination-type exam. This step was undertaken to provide a fair and reproducible clinical exam. All candidates face identical questions with predefined correct answers simultaneously in seven high throughput exam centres on the same day. We describe the factors that are associated with success in part 2 European Diploma in Intensive Care exam.
Methods
We prospectively collected self-reported data from all candidates sitting European Diploma in Intensive Care part 2 in 2015, namely demographics, professional background and attendance to a European Diploma in Intensive Care part 2 or generic objective structured clinical examination preparatory courses. After testing association with success (with cutoff at p < 0.10) and co-linearity of these factors as independent variables, we performed a multivariate logistical analysis, with binary exam outcome (pass/fail) as the dependent variable. Structural equation modelling was used to gain further insight into relations among determinants of success in the oral part of the European Diploma in Intensive Care.
Results
Out of 427 candidates sitting the exam, completed data from 341 (80%) were available for analysis. The following candidates' factors were associated with increased chance of success: English as native language (odds ratio 4.3 (95% CI 1.7–10.7)), use of Patient-centred Acute Care Training e-learning programme module (odds ratios 2.0 (1.2–3.3)), working in an EU country (odds ratios 2.5 (1.5–4.3)), and better results in the written part of the European Diploma in Intensive Care (for each additional SD of 6.1 points odds ratios 1.9 (1.4–2.4)). Chance of success in the European Diploma in Intensive Care 2 decreased with increased candidates ‘age (for each additional SD of 5.5 years odds ratios 0.67 (0.51–0.87)). Exam centres (7 in total) could be clustered into 3 groups with similar success rates. There were significant differences in exam outcomes among these 3 groups of exam centres even after adjustment to known candidates' factors (G1 vs G2 odds ratios 2.4 (1.4–4.1); G1 vs G3 odds ratios 9.7 (4.0–23.1) and G2 vs G3 odds ratios 3.9 (1.7–9.2)). A short data collection period (only one year) and 20% of missing candidates' data are the main limitations of this study.
Conclusions
Younger age, English as native language, better results in written part of the exam, working at a European country and the use of PACT for preparation, were factors associated with success in the oral part of the European Diploma in Intensive Care exam. Despite the limitations of this study, the differences in outcome among the exam centres will need further investigation.
Keywords
Introduction
The European Diploma in Intensive Care (EDIC) is awarded to candidates who pass a two-stage exam process at the end of their intensive care medicine (ICM) training (which should be certified and performed over a minimum of two years). Part one of the EDIC is a multiple-choice test (MCQ), whilst part 2 is oral. Since 2013, the oral part of the exam (called also part 2) has been held outside hospitals, in only seven preselected high-volume European centres (Amsterdam, Copenhagen, Dublin, London, Porto, Prague, Vienna and Zurich). These centres coordinated by an expert director can offer seats to 36 candidates twice a year. Therefore, 504 are the maximum number of candidates that can attempt EDIC part 2 exam every year.
The exam is a modified Objective Structured Clinical Examination (OSCE 1 ) where all candidates are asked the same questions with pre-specified correct answers. The ESICM Examinations Committee meticulously prepares the exam content with the assistance from a consultant educationalist. The same committee has the duty of supervising the process of selection and training of EDIC examiners in each centre. All examiners have a half-day training before every single exam and full explanation of the content of the exam. A candidate has to obtain a pre-specified minimum number of points in three clinical case scenarios (CCSs) and three computer-based stations (CBSs) in order to pass the exam. During each CCS (25 min each) two examiners test one candidate’s ability to make routine decisions during the course of a patients ICU stay. These patients' histories are derived from ‘real life’ cases intended to reflect real ICU situations and problems as closely as possible. During the CBSs (12 min each) one examiner tests the candidate’s capability to recognize common patterns of clinical imaging, different curves relevant in ICU care (monitor traces, ventilator screens etc.) and biochemical constellations (for more details about EDIC exam see www.esicm.org). 2
Authors are aware of the variability of results obtained by candidates sitting the EDIC exam in Europe. These differences in outcome for EDIC part 2 have not been investigated: they could be attributable to candidates’ factors as well as exam-related variables.
Methods
Study design
We performed a retrospective analysis of prospectively collected candidates’ self-reported data held by the Department of Professional Development of the ESICM. We gathered data from 427 candidates who performed the oral part of EDIC exam (EDIC 2) on 26 March 2015 and 5 November 2015 in 7 exam centres. As part of the candidates’ related characteristics we collected their EDIC part I result (number of points obtained in MCQ test for each EDIC part 2 candidate). By using forename, surname and date of birth to identify each candidate, we used a relation database software (MS Access 2016, MicroSoft©, USA) to merge data from all data files available to us. This included the results of 2015 edition of EDIC 2 and all results of EDIC 1 from Spring 2013 onwards, and the lists of participants of all editions of EDIC 2 Preparatory Courses. From this data set we collected self-reported candidates’ data, which included demographic data (age, sex, country of medical school, working country, English as primary language, English as study language). We obtained and entered into the multivariate analysis a complete data set for 336 candidates (79% of 427 candidates sitting EDIC 2 in 2015), whilst 88 (21%) candidates had at least one missing value; these candidates were included in the univariate analyses, but excluded from multivariate and structural equation modelling analyses. The most common missing value was the EDIC I result (n = 41; 10%), mostly due to old records from EDIC part 1 being unavailable.
Statistics
In categorical variables we firstly reduced the number of categories by grouping the variables into two or three groups within which there was no significant differences in outcome. Detailed step-by-step description is in the Supplementary Appendix. Afterwards we performed univariate regression analysis and selected factors which were associated with outcome at p < 0.10 and tested whether they were mutually correlated. If there was a significant correlation between these factors, suggesting that they may not be independent of each other, we selected the factor with stronger association with exam outcome. These variables were subsequently tested by multivariate logistical regression.
Structural equation modelling is a novel computer-based technique, allowing for exploring the relationships among multiple variables in complex systems, mostly used in behavioural and social sciences. 3 The computing method is described in detail elsewhere. 4 We used Stata (version 14.1, StataCorp©, USA) for all analyses and p < 0.05 is considered significant.
Results
Outcome of EDIC 2 exam in 2015
Recruitment into the study is showed in the flowchart (Figure 1). There were 427 participants who sat the EDIC part 2 in 2015. The success rate in 7 examination centres were as follows: in centre A 13/57 (23%), B 31/70 (44%), C 36/69 (52%), D 36/69 (52%), E 36/63 (57%), F 37/58 (64%), G 21/28 (75%) and in centre H 13/17 (76%). In total, 221 (52%) participants passed and 206 (48%) failed the exam. There were 107 (25%) candidates failing in CCSs, 170 failing in CBSs (40%) and 71 (17%) failed both. The profile of successful and unsuccessful candidates in EDIC 2 is shown in Table 1.
Flowchart of the analysis. Note: No show = candidate signing up for the exam but not attending it. Characteristics of EDIC part 2 candidates in 2015. PACT: patient-centred acute care training; CoBaTRICE: competency-based training in intensive care.
Factors associated with success in EDIC 2
Univariate logistical analysis of factors associated with success in EDIC 2 as a binary outcome (pass/fail).
EDIC: European Diploma in Intensive Care.
Structural equation modelling confirmed the results of the multivariate analysis and, in addition, it revealed that some factors only influenced success in one domain of the exam, e.g. the use of PACT influences the outcome of exam via better success in CCS, whilst candidates who used CoBATRICe did better in CBS (see Figure 2)
Global model of factors associated with success in oral part of EDIC exam as per structural equation modelling analysis. CBS: computer-based scenario; CCS: clinical case scenario; E1: EDIC Part 1 (written); E2 session: morning, afternoon or evening session of the oral exam.
Discussion
The EDIC exam had been developed in 1989 in order to standardize the assessment of education and training in ICM in Europe and beyond. Even though the exam is recognized as a surrogate for national accreditation only in few European countries, 5 it has gained a wide acceptance as proof of advanced knowledge in the field of ICM. The oral part (known as part 2) was reformed in 2013 into an OSCE-type exam with the aim of increasing objectivity and reproducibility of the exam. In order to test whether these goals have been achieved, we looked into factors associated with success in the EDIC part 2. Ideally, exam outcome should only be determined by candidates’ performance. Exam-related factors (i.e. exam content, which was different between Spring and Autumn edition, exam centre, examiners’ training, others) should not play a role.
Not surprisingly we found that candidates who passed the MCQ test (EDIC part 1) with a higher margin of safety performed better in part 2 (see Table 2). In line with results of the EDIC 1 exam (unpublished data) candidates practicing in Europe were more likely to succeed compared to those who are practicing elsewhere. Arguably, the reason for that could be that members of the Examination Committee preparing the exam are exclusively from European countries and they are practicing in Europe, in line with the mission of the exam. These cultural differences are notoriously difficult to overcome and are tested in several field of education. The exam is oral, held in English and despite the exam content being written mostly by authors from non-English-speaking countries, candidates whose native tongue is English are more likely to succeed in EDIC part 2. The use of PACT (Patient-centred Acute Care Training e learning program promoted by the ESICM, www.PACT.org) was associated with success, suggesting that thorough systematic preparation may be more important than a priori familiarity with the format of the exam. The EDIC part 2 preparatory course started in January 2015 and it is receiving excellent feedback from its attendants; however, it seems to be helping only good candidates (Figure S1). Our data provide no explanation why younger candidates are more successful as compared to those who sit the exam later in their career. The literature suggests that learning methods change and efficacy deteriorates with ageing. 6 The exam is very practically oriented and accumulated experience of older candidates may help, particularly in CBSs, which test pattern recognition. Nonetheless, both CBSs and CCSs are influenced negatively by increasing candidates’ age (see Tables S1 and S2). This is particularly surprising in view of the fact that we have not seen any difference between candidates currently actively practicing ICM and those who were not.
Even after adjustment to all above-mentioned candidates’ factors there were significant differences in exam outcome among exam centres. The Examinations committee meticulously prepares the exam content, which is the same in every centre. In line, for the total pool of candidates, there was no difference overall in outcome between spring and autumn session of the exam, which used different exam content. Pre-specified candidate’s correct answers are recorded on a tablet and transferred online for blinded central processing at the University of Heidelberg. Wilkinson et al.
7
clearly demonstrated that in OSCE-type tests examiners’ factors are mostly responsible for the exam-related variability in outcome and it is the impression of the authors that the exam centre is a surrogate variable for another factor, most likely variability in examiners’ behaviour, e.g. prompting. Other factors are unlikely to play a role as in each centre all candidates are prohibited from the use of any electronic devices, internet access and opportunities for unfair behaviour are minimized the same way in all the centres. There is evidence in the literature showing that examiners’ training helps to standardize the exam.8–10 In EDIC part 2, each candidate faces 9 different examiners who are carefully selected (ESICM-SOP) and have passed a standardized training,
11
which includes a mandatory train-the-examiner course and pre-exam workshop. In addition, before becoming an active examiner, it is mandatory to observe at least one session of the exam. In light of this, differences among centres are difficult to explain. However, it should be stressed that our data bring no proof that there is a real influence of the centres on exam outcomes. We only retrospectively collected a limited amount of candidates’ self-reported data (with 21% missing values) over 1 year of the exam. It should be stressed that missing values bring a risk of bias. However, the number of points achieved in EDIC part 2 was not different between candidates with and without EDIC 1 results available (546 ± 86 vs 545 ± 73, p = 0.83) and that in candidates with known EDIC 1 results, the gap between EDIC 1 and 2 did not affect the EDIC 2 outcome (data not shown). Even within the group of candidates with a complete dataset available, there might be important differences in candidate factors among exam centres which we have not accounted for and
Examinations Committees with the endorsement of the Executive Committee of the ESICM decided to perform a series of measures in order to further improve the quality and standardization of the EDIC exam. This includes: (1) unification of the electronic database system to allow prospective candidates’ data collection and regular exam outcome evaluation; (2) introducing the individual examiner’s performance monitoring 12 ; (3) introducing a scheme of regular examiners’ rotations among centres; (4) gaining accreditation of The Council for European Specialists Medical Assessment (CESMA) for the EDIC exam.
In conclusion, in an analysis of one year outcomes of the EDIC part 2 exam, we identified that younger age, working in EU countries, English as first language, EDIC 1 score and the use of PACT are candidates’ factors associated with success in the exam. Our results also generate a hypothesis that there still may be differences among exam centres not explained by candidates’ factors, which are currently being explored and will be addressed.
Addendum
The measures described in the article have been introduced. In Spring 2016 exam session, there were no significant differences among exam centre groups in the OR of success in the exam: G1 vs G2 = 2.2 (95% CI 0.9–5.1); G1 vs G3 = 1.2 (95% CI 0.6–2.7); G2 vs G3 = 0.6 (95% CI 0.2–1.5).
The Examinations Committee will continually monitor exam quality indices and since Autumn 2015 this includes collecting data on individual examiners’ performance. The CESMA has agreed to appraise the EDIC using best-designed exams in the European Union of Medical Specialists (UEMS) as a benchmark.
Footnotes
Acknowledgements
The authors thank Mrs Dominique DeBoom and ESICM office staff for help with data collection and secretarial assistance. We have not sought any funding for this study.
Declaration of conflicts of interest
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: PW, RK, RS and FD are authors and lecturers on EDIC part 2 preparatory courses. AG, FD, CS and BM are EDIC part 2 exam centre directors.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.