OCI Examination,RANZCP

To the Editor

This letter was prepared in response to a letter received by the ANZJP expressing some concerns about processes connected with the Observed Clinical Interview (OCI) examination of the RANZCP. The author subsequently withdrew their letter, but it was considered that it may be of value to current trainees, overseas trained psychiatrist candidates required to pass the Modified Observed Clinical Interview (MOCI examination) and Fellows to provide information about this component of the examination, which has undergone a number of changes over recent years. The OCI, or Observed Clinical Interview, is the RANZCP version of what is referred to in the literature as the long case. The long case has been an essential tool in the assessment of clinical competence as it evaluates trainees’ skills with real patients under realistic conditions (Wass and van der Vleuten, 2004). Overseas-trained psychiatrists, if required to do so, sit the Modified Observed Clinical Interview (MOCI).

The M/OCI process since 2003 has been highly standardised. Each candidate has 50 minutes to interview a patient, observed by two examiners. They are then given 20 minutes’ thinking time (in private), followed by a viva exam in which they must present the salient features of the case, mental state examination, synthesis, identify any important gaps in the history and give a diagnosis and differential diagnosis in 7 minutes (10 minutes in the MOCI) or less. They are then asked questions by their examiners for approximately 13 minutes, before being asked to present their action (management) plan. The examiners will ask questions on the action plan if there is time available. The maximum viva times are 30 minutes for the OCI and 40 minutes for the MOCI. Examiners are restricted to asking questions about material the candidate has presented in their viva, using prescribed stems (e.g. ‘Please elaborate on your description of the problematic relationship between the patient and his mother’ or ‘Please justify your diagnosis of schizoaffective disorder’). Examiners are limited to two questions on any topic. Following the viva, examiners are required to mark each of five domains (data gathering process, data gathering content, mental state examination, synthesis, action plan). Following the education review commissioned by the RANZCP in 2008/2009, a change was instituted in 2012 whereby candidates were required to pass two out of three M/OCI exams to be awarded an overall pass in this examination. Readers will be aware that prior to 2003 candidates were required to pass two out of two long case exams. (although a high score on one case might ‘compensate’ for a near miss on the other, thus allowing a pass).

The letter originally received by the ANZJP expressed expresses the opinion that candidates are disadvantaged because they find it difficult to understand what is expected of them to pass the OCI. The RANZCP Committee for Examinations (CFE) has published information on the College website which is freely available to all trainees and includes detailed guidelines on how to formulate a case as part of the OCI, details about the timing and structure of the examination and how each candidate’s performance is assessed. Trainees also have access to the instructions for OCI examiners, marking guidelines and proformas.

In addition to these resources, the development of the Accredited Examiner Panel (AEP) means that there are now close to 200 Fellows in Australia and New Zealand who have been trained to the requirements of the OCI and MOCI exams, and who have been calibrated to the required standard. These accredited examiners represent an important resource for M/OCI examination candidates as they are able to give precise, contemporaneous and personalised feedback to assist the candidate in developing his or her skills, as well as in understanding what is required to pass.

The College applies internationally validated training processes to ensure that all examiners operate at the standard expected for examiners in the field of postgraduate medical education. Accredited examiners are psychiatrists with at least 3 years’ experience since attaining Fellowship, whose nomination to the panel was endorsed by their local RANZCP branch, who have attended compulsory training sessions run by members of the CFE, and who have been observed to examine by senior examiners. They are not given full accreditation until the CFE is satisfied that they have met the required standard. In addition, accredited examiners are regularly required to undertake exercises to maintain their calibration to the identified standard.

The exam process has been designed to minimise examiner subjectivity, by requiring each examiner to mark independently initially, but then come to consensus through discussion. This process is also standardised, with examiners instructed that the higher marking examiner should begin the discussion with the aim of presenting a convincing argument as to why the higher mark should prevail. This consensus procedure is a safeguard against potential bias from examiners who may be ‘super-specialists’ in an area, or who may have placed too much emphasis on some limited aspect of performance by the candidate. A process is available to assist with achieving consensus where this fails at the time of the exam. In 2012, with a total of 716 examinations, there was no instance of examiners failing to reach consensus. Given the rigour of the examiner training and calibration process, and the fact that each candidate may sit up to three examinations involving six examiners, it is highly unlikely that factors related to one examiner would influence the overall outcome.

Each candidate receives a breakdown of his or her scores by domain after the examination. Examiners are also expected to provide feedback to individual candidates on their performance and also generic feedback to the CFE post examination. The individual feedback is transcribed and provided to each candidate and the generic feedback informs an examination report which is published on the RANZCP website. Also on the website are general observations collated from examiners over a period of time outlining the characteristics of candidates who perform well in the OCI versus those who perform poorly.

The College thus provides ample opportunity for candidates to become aware of what is required of them in this examination and the ‘expected norms’. Examiners are not expected to reiterate this information. It is intended that candidates will discuss their feedback with their supervisor or other Fellows, such as their Director of Training or a local accredited examiner, who will be able to assist the candidate in using the feedback to inform their further examination preparation. The pass rates for the OCIs have followed the same trend for several years: the pass rates in 2010 and 2011 varied between 47% and 63%; in 2012 the pass rate has varied between 50% and 58%. These pass rates are below those of the Observed Structured Clinical Examination (OSCE), which range between 72% and 83%.

All examinations must necessarily reflect a balance between reliability and validity, and indeed other constructs (van der Vleuten, 1996). The OCI is viewed as a valid assessment of a candidate as it is based on a highly authentic task and reflects candidates’ work practices in clinical settings. It is recognised that the OSCE is a more reliable examination. However, it cannot and does not test certain skills that the RANZCP deems to be critical for a Fellow: namely, the ability to engage with a real patient, elicit a history in an effective but empathic manner, perform and accurately interpret a mental state examination, synthesise biological, psychological, social and cultural information in a way that demonstrates an appropriate understanding of the individual, and propose a tailored plan of management. The OCI allows assessment of these skills as a holistic process within a time frame that closely approximates a real clinical setting.

Review of the literature shows that there is some variation on the exact number of OCIs required and the exact length of testing to guarantee a reliable assessment. Based on the advice of external educational experts, the RANZCP increased the number of OCIs required in order to increase the reliability of the tool. Though it was expected that the reliability would still likely fall short of 0.8, this was accepted given the high level of validity of this exam.

The assessment of the skills currently assessed by the OCI is being refined and revised within the framework of the new Competency Based Fellowship Program (CBFP). The CBFP will see further changes to the OCI/long case that are designed to maximise the reliability and validity of the examination and the development of skills by trainees.