Abstract
Summary
Objective
To discover whether applicants regard structures interviews as a fair method of selection for jobs.
Design
Audit study of short-listed candidates for postgraduate specialty training programmes in the London Deanery.
Setting
Postgraduate applications for the London Deanery.
Main outcome measures
Satisfaction or otherwise with the application and selection process for postgraduate specialty training programmes amongst short-listed candidates in the London Deanery. Questions were asked under five categories: the applicant, the advertisement, the application form, the short-listing process, and the interview.
Results
89 of 118 forms were completed and analysed. Candidates thought the advertisement was clear on who to contact (97%), when short-listed candidates would be notified of their interview (66%) and when interviews would occur (93%). The design of the application form and the short-listing process both scored a median of 1 or 2 (strongly agree or agree) on all points. The interview process itself also scored well, with most candidates scoring broadly positively.
Conclusions
As in the previous study, the overall response was broadly a positive one from the candidates' perspective, with the majority of candidates finding the system fair and objective.
Introduction
The selection of medical trainees into postgraduate specialty training programmes requires a process which in a fair and transparent manner selects the best candidates. The process of selection into cardiology in the London Deanery has changed over the past eight years, 1 , 2 resulting in a structured application form and interview. An initial audit demonstrated that overall, candidates thought that this structured interview was more objective than previous unstructured interviews. 3 Since this initial audit there has been no further formal assessment of this selection process from the candidates' perspective; this study was therefore designed to obtain the views of the short-listed candidates on the current process.
History
Interviews have always formed an important part of the process of selecting candidates at all grades within medicine, including selection into specialist training. The traditional approach has involved using a CV supplied by each candidate to short-list those for interview. Those short-listed would have an open interview, in which different questions were usually asked of different candidates. Potentially this leads to bias, with some candidates performing better than others based on the different questions they had been asked. There were also concerns that it may lead to canvassing, discrimination and patronage. 4
The reforms of specialist training in medicine in the UK 3 and the development of the specialist registrar grade (SpR) aimed to link the number of training posts to the predicted number of consultant opportunities. This gave increased importance to the selection process, since those selected could expect to become consultants after this period of training. The selection process for entry into this specialist training grade would need to be more rigorous and structured than previously. 5 This new selection process would also need to be viewed as fair, open and accountable by both selectors and candidates. As a ‘high stakes’ point in the candidates’ careers, it needs to be fit for purpose and to accordingly select the candidates with the ‘best’ profile for the specialty.
There were several stages in the development of this new recruitment process. The first was to define the attributes required to be an NHS consultant, and then use this to develop a competency framework through discussion in a series of focus groups. This was then used to derive the person specification which clearly defined the various attributes to be assessed during the selection process. 5 The point where each of the specific attributes of the person specification was to be assessed were clearly defined on the application form, at the structured interview, from the references, or a combination of all three. Although SpR interviews no longer exist, a similar process was used for Modernising Medical Careers (MMC), with explicit person specifications from which attributes to be assessed during the selection process are clearly defined.
The London Deanery procedure
As the structured application form was specifically designed to assess these various attributes, CVs were not accepted. By using a structured marking scheme, the application form was used to derive a short-list of candidates for interview.
The interview was structured with three panels of interviewers. At each panel, each candidate was asked the same set of initial stem questions. These stem questions were asked to each candidate by the same member of the interview panel, and preferably with the same wording. The stem questions were then followed up with more detailed ‘probe’ questions based on the initial responses. 7
When the initial structured application process was devised and evaluated by the Institute for Employment Studies (IES), initial feedback suggested that both assessors and applicants found this new approach was fairer and more open as compared to the previous system. 6 One of the recommendations of the IES, however, was that a periodic evaluation exercise would allow the system to evolve further. After the initial assessment from the candidates' perspective, 6 there has been little further evaluation of this method of selection. We therefore devised a questionnaire to further evaluate the short-listed applicants' view of this process.
Methods
The number of candidates attending the interview was obtained from London Deanery records. All candidates who attended an interview for SpR/NTN appointments for Cardiology/G(I)M in the Pan-Thames London Deanery in May 2005, November 2005 and May 2006 were asked to complete a short questionnaire after the interview itself but prior to announcement of the results of the selection procedure. The questionnaire contained five sections: About the advertisement About the application form About the short-listing process itself About the interview, and About you the candidate.
Several types of questions were used in the form. In the section ‘About you the candidate’, date of graduation and date of passing the MRCP examination were requested. The number of previous attempts at obtaining a SpR NTN and the number of previous interviews each candidate had attended were also requested. Candidates were also asked about other degrees they had obtained apart from the MBBS qualification (or equivalent). Candidates were asked about their current employment. Out of the four possible regions (North East, North West, South East and South West) each candidate was asked whether they would accept appointment to any of the four London areas. They were also asked if expressing this preference of region within the London rotation was the most acceptable and fair way to determine this. If they disagreed they were asked to give a preferred method. They were also asked specifically if returning to the previous system of interviews by the four separate London geographic areas would be preferable.
In the section entitled ‘About the advertisement’, candidates were asked if they were aware that an advertisement was due to be placed in advance, whether a clear point of contact was identified, if it was clear when a decision on short-listed candidates would be announced, and if it was clear when the interviews would occur.
For the above sections either the means and standard deviations or the percentage of candidates agreeing with the various statements was calculated, as appropriate.
For the other three sections – ‘About the application form itself’, ‘About the short-listing process’, and ‘About the interview’ – there were several statements which each candidate was asked to grade according to the following scale: 1, Strongly Agree; 2, Agree; 3, Neutral; 4, Disagree; 5, Strongly Disagree. As the responses to these questions appeared highly skewed, the median and interquartile range for each of these statements was calculated. At the end of each of these sections there was an open-ended question asking if there was any way in which the application form, short-listing process or interview could be improved. The individual responses have been described in the results section.
Results
There were 94 completed forms out of a total of 118 candidates who attended an interview, representing an 80% response rate. Five of the forms were incomplete. The results are presented in the same five sections as the questionnaire.
About you the candidate
Higher degrees achieved by candidates at the time of interview
Current employment at the time of interview
Results of questions about the application form for the SpR interview
1, Strongly Agree; 2, Agree; 3, Neutral; 4, Disagree; 5, Strongly Disagree
Less than half of short-listed candidates (39%) had not previously applied for a cardiology NTN. 49% had previously been interviewed for a cardiology NTN. The mean number of applications was 1.6±2.0 and the mean number of previous interviews was 1.0±1.4. All of these previous attempts had been made between 2002 and 2006, with the exception of one candidate who claimed that he had made an application in 1997.
Most short-listed candidates (77%) thought that expressing a preference using a form at the time of interview for one of the four London regions was acceptable. Three candidates suggested that a better method would be to indicate a preference well before the interview (such as on the structured application form) and three candidates thought that the preference should be indicated after the interview. Only two candidates suggested returning to the previous system of separate interviews for the four separate London Regions. Despite these preferences, the majority of short-listed candidates (81%) would have accepted an NTN from any of the four London geographical areas.
About the advertisement
Results of questions about the short-listing process for the SpR interview
1, Strongly Agree; 2, Agree; 3, Neutral; 4, Disagree; 5, Strongly Disagree
About the application form itself
There was a favourable response to the design of the application form (Table 3). There was, however, some concern over the space available for completion of the application form, with 16% of short-listed candidates commenting that there was insufficient space to answer all the questions fully. Seven of these responses specifically commented about insufficient space for research publications. Two candidates thought that the application form should be a PDF folder.
About the short-listing process
The responses were in general positive with regard to the short-listing process (Table 4). When asked if there was any way in which the short-listing process could be improved, six candidates commented either that it took too long to notify successful short-listed candidates or that there was no stated date by which the results of the short-listing would be available. One candidate thought that there should be a more explicit and objective scoring system and one candidate thought that there should be increased weighting given to the reports from referees. One candidate felt that the form was not explicit about the candidates' personal involvement in obtaining research funding.
About the interview
Results of questions about the SpR interview itself
1, Strongly Agree; 2, Agree; 3, Neutral; 4, Disagree; 5, Strongly Disagree
Six candidates suggested that either more time per panel was needed or that more panels were needed. Other forms of assessment were also suggested, including psychometric testing, assessment of clinical skills and assessment of organizational skills. One candidate felt that there should be clearer instructions prior to the interview of the format and content.
Discussion
The structured application form and interview were both designed to make the selection process for trainees fair and objective. 6 When the structured process was initially devised there was an extensive evaluation of trainees' opinions but since this initial evaluation there has been little subsequent formal assessment. 6 In our study we had a high overall response rate, with 94 out of 118 short-listed candidates (80%) completing the questionnaire and within those responses only the occasional question not being answered.
We have confirmed that short-listed candidates consider this structured process (both the structured application form and the structured questioning in the interview) a fairer and more objective assessment than the traditional CV and open interview. A similar result was obtained when this method of selection was introduced, with 76% of respondents thinking that the structured application form was fairer than short-listing by CV and 59% that the process better enabled candidates to demonstrate their strengths. 6 All written documentation – both the initial advertisement and the structured application form – were easily understood.
There are a large proportion of short-listed candidates entering research prior to attempting to enter cardiology specialist training. There is clearly a need to try to further elucidate why candidates consider that it is desirable to undertake a period of research training in cardiology at such an early stage in their career, when many candidates will not have had experience of cardiology beyond general professional training. With the current application form there are several areas where points are scored either directly or indirectly for research. These include points for a higher research degree but also points for peer-reviewed papers and presentations at meetings. Several short-listed candidates obviously felt that they were more likely to be short-listed if they had done a period of postgraduate research; some candidates may have felt that postgraduate research is essential to gain the required number of points to ensure short-listing. The comments around insufficient space to list research publications would reflect this concern.
Postgraduate research experience is one of several desirable attributes of the person specification but is not an essential attribute. Credit should be given for previous high quality research. To maintain fairness to all candidates it must be possible to be short-listed without undertaking postgraduate research. When the structured application form was devised the space on the form for research publications was deliberately limited so that the emphasis would not only be on the number of publications per se but also the quality of those publications to maintain fairness and prevent undue weight being given to postgraduate research. Every comment made about insufficient space on the application form for publications was made by a short-listed candidate either in a postgraduate research post or who had completed a higher research degree. No short-listed candidate suggested a reason for increasing the space available for research publications or why this may be fairer to all candidates. One candidate suggested that the form should be more explicit in establishing the personal involvement of the candidate in obtaining their research funding. This may be an important point, as credit should be given to those whose research proposal has undergone peer-review and has attracted competitive funding only where they were actively involved in this process. This issue of research funding cannot have been a major concern, as overall candidates overall felt the application from to be fair and objective.
The majority of short-listed candidates had previously applied for cardiology specialist training and a significant proportion had previously been interviewed. The selection process was changed in 2004 with the introduction of Pan-Thames Interviews twice yearly replacing separate interviews in each of the four areas of London at various times of the year. Only two candidates suggested a return to this system of separate interviews for the four separate London areas.
The short listing process was felt to be fair and objective. A few comments were made about the time it took to notify successfully short-listed candidates or that there was no explicit date given when the results of short-listing would be available. Although it is preferable to notify candidates about the results of short-listing as soon as possible, it is important that all candidates are notified at the same time for the sake of fairness. Only one candidate suggested that the system was not as objective as it should be, but without any suggestions as to how the system might be made more objective. One candidate suggested that increased weighting should be given to the references. In order to maintain objectivity and fairness, references were excluded from the marking scheme when the structured application form was devised, as they are subjective opinion.
The interview process itself was also felt to be generally fair and objective. It is of interest that inclusion of presentations of any kind was not considered to be of benefit. We are unable to provide the reasons for this from our study. Several short-listed candidates suggested increasing the overall time of the interview either by increasing the number of interview panels or having a longer time with each interview panel, but overall the structured interview process was felt to be fair. With a longer time per panel there is the danger that more follow up ‘probing’ questions after the initial stem question will be asked. This could lead to substantially different questions being asked of each candidate and the structured interview has been devised in such a way to try to limit this to be fair to all candidates. The option of more interview panels to assess other qualities such as an aptitude test or some form of psychometric testing (as suggested by one candidate) could be explored in the future as long as they can be shown to be a valid, reproducible and fair method of assessment. Simulated clinical scenarios involving trained actors would be a further possible way of extending the scope of the structured interview process.
Limitations and suggested further work
Some caution must be used in assessing the objectivity of the short-listing process from the responses, since all respondents had succeeded in being short-listed. To gain a better overall view of this, the views of all those filling in an application form would be required, and could be addressed in the future. One further area that was not explored was the feedback that all candidates (successful or otherwise) receive after the results are announced to each candidate. We chose not to include this as it would mean that the questionnaire would be completed after the results of the interview were known, and this could introduce a large potential bias. This study was carried out before the introduction of MMC. With the significant changes to medical training that have occurred with MMC, further work is required to ensure that from the candidates' perspective the process continues to be fair and objective.
Summary points
In summary, this questionnaire has shown that from a candidates' perspective the current structured procedure does represent a more objective and fair assessment of candidates. It shows similar results to those found when the structured application form and interview were first introduced. Continued monitoring of this will be valuable in the future, especially with the proposed changes to specialty training through the MMC programme. This includes a major change to the way in which doctors will be recruited into specialty training, using an electronic application portal, national person specifications and application documentation for each specialty, as well as a more standard approach to selection interviews.
Footnotes
DECLARATIONS
Footnotes
Acknowledgements
None