The validity and reliability of the observed clinical interview (OCI) exams

The recent article about the validity and reliability of the observed clinical interview (OCI) exams (Lampe and Jenkins, 2013) did not provide any evidence to substantiate the claim that the OCI exam is an effective assessment tool.

The OCI is the not-so-improved descendant of the long case clinical examination technique that originated in the mid-19th century in Cambridge. The long case was one of the least reliable and most often biased forms of assessment to survive into the 21st century. The Royal Australian and New Zealand College of Psychiatrists (RANZCP) has regarded this style of exam as a cornerstone of its clinical examinations process since the 1970s, with a sustained and curious disregard for its lack of reliability, validity, feasibility and fairness.

For the last 30 years, clinical education has been established as a seriously evidence-based discipline. It has become increasingly apparent that the long case does not yield results that achieve reasonable levels of reproducibility and the evidence begins to form a challenge to the validity of this exam. (Norcini, 2002; Tyrer, 2007) In 2003, the long case examination in the MRCPsych (Royal College of Psychiatrists) was replaced by an observed structured clinical exam (OSCE).

The Royal College of Physicians and Surgeons of Canada (RCPSC) replaced the long case psychiatric clinical examination with the OSCE in 2004. The American Board of Psychiatry and Neurology replaced the long case with an approximately 1-hour long clinical vignette section with four vignettes in 2006. The vignette cases may be presented in a written format or as a short video clip. Candidates are examined by one or more examiners with particular reference to the content of the vignettes.

The RANZCP remained the last college of psychiatrists to retain the long case as part of its clinical assessment process. It appears that the College cannot see an acceptable way to extricate itself from the millstone it calls the OCI.

The RANZCP underwent an external review of the exam process after the Australian Medical Council (AMC) commented on its low pass rates, difficulties progressing through fellowship completion, and high assessment load. The OCI exam came under critical scrutiny, particularly as regards the fact that it was a ‘hit and miss’ exam, with little chance for trainees to show their full range of skills, as they only had one chance to display their skills, on one day, with one patient and only two examiners.

The difficulties of assessing a candidate on the basis of a single case, which cannot be standardised for diagnostic complexity, cooperation of the patient, style of the examiner and a wider range of other factors, were recognised. The exam report stated that this was not a reliable way to test a trainee’s clinical skills and the OCI should be dropped, or moved into workplace assessment.

A review of the 2008 examinations by Dr Gareth Holsgrove, dated May 2009, recommended, ‘a fair and transparent set of steps towards licensure’ and that ‘each component should be passed in its own right’ (Holsgrove, 2009). The College appears to have interpreted this to mean that a pass in the OSCEs can stand alone, but a pass in the OCI cannot. The reason given is that a single OCI is not deemed as reliable as the OSCEs.

The multiple OCI exams were introduced in 2012 and at present the College requires that a pass in two out of three OCI attempts be recorded in immediately adjacent examination cycles.

The authors reported that the recent OCI exam pass rate in an examination set to test basic training skills varied between 50% and 58% and this appeared to be a lower pass rate than in the previous years. These pass rates included the candidates who had attended this exam more than once; therefore, a detailed statistical analysis of these results would be valuable. It is inconceivable that between 42% and 50% of candidates who are in training for several years are deemed unfit to progress to Advanced Training. The low pass rate of the OCI exam in the setting of a theoretical improvement in its reliability raises serious questions about the validity of the examination and the integrity of the training program.

The RANZCP’s former chair of the Board of Education pointed out that the OCI exam is not sustainable. He proposed that trainee assessment activities would significantly improve in their reliability, validity and utility by having the clinical examination system that used the increased number of OSCE stations (Mellsop, 2012).

We should recognise the need for the College to maintain the integrity of psychiatry training through sound quality control. Indeed, achieving and maintaining the highest standards should be the main goal of every specialist training program. It is interesting to see that other specialist medical colleges are able to maintain high standards as regards exam pass rates similar to those given for the RANZCP OSCE exam (72–83%). (Australian Government Department of Health and Ageing, 2013).

It is encouraging to see that the previous and current RANZCP presidents Maria Tomasic and Murray Patton have unequivocally recognised the need to revise the examination and training programs and bring them in line with other colleges both domestically and abroad. The new Competency Based Fellowship Program (CBFP) for psychiatry training, which the RANZCP is implementing, will introduce a whole raft of new assessments. It seems very likely that the workplace-based structured observed clinical activity (SOCA) will replace the current multiple long case exams.

In my opinion, there is an urgent need for the College to apply the Holsgrove recommendations honestly and replace the OCI immediately with workplace-based competency assessments. This will guarantee fairness and ensure that all trainees currently enrolled into two separate fellowship programs will suffer no disadvantage.

See Letter by Lampe and Jenkins, 47(11): 1089–1090.