Abstract
In most jurisdictions, criminal courts rely on evidence from mental health experts to determine whether a person is fit to stand trial (FST) or not guilty due to mental illness (NGMI) [1]. An incorrect decision about fitness (also referred to as competence or capacity) to stand trial can deny a defendant the opportunity to put their case, or conversely can result in an unfit defendant receiving an unfair trail. An incorrect finding about NGMI can result in a criminal conviction instead of appropriate hospital treatment, or allow a malingerer to escape an appropriate verdict.
FST and NGMI are legal issues to be decided by the courts, but in practice courts rely on the opinion of mental health experts [2–4]. Hence, in many jurisdictions expert witnesses are allowed to express opinions about the ‘ultimate issue’ before the court, of whether the accused is FST or might be NGMI.
There are only a few studies that have examined the reliability of the opinions of mental health professionals about FST or NGMI [5–7], and two of those used opinions about hypothetical cases [6, 7]. The lack of empirical studies is surprising because it has often been alleged that expert evidence is biased [8–12], and in many jurisdictions the opinions of experts are presented in the form of written reports that can be readily compared [13, 14].
The accusation that opinions about FST are biased may have stemmed from the finding of an association between demographic variables and FST [15], although subsequent studies found that clinical factors, rather than sex or race, were the main predictors of opinion about FST [2, 16]. None of these studies went on to assess the level of agreement between the experts. Skeem et al. found that agreement between experts about FST in Utah was satisfactory (κ = 0.64) but did not report if there was evidence of bias arising from whether the expert was engaged by the defence or by the prosecution [5].
Although the level of capacity required to be considered fit to participate in a trial is unlikely to vary much between English-language jurisdictions, there are differences in the way FST is determined. In some North American jurisdictions and in the UK semi-structured interviews have been introduced in an attempt to overcome a perceived lack of reliability in opinions about FST [17–19]. These instruments, however, may not be able to capture all the relevant clinical information and do not address variations in capacity required to participate in proceedings with varying levels of complexity, and hence are unlikely to replace an expert's opinion [20]. Semi-structured interviews for the assessment of NGMI have not been developed, probably because the effect of delusional beliefs on moral reasoning is difficult to quantify. Hence in every jurisdiction the opinion of experts is considered when making decisions about NGMI.
In Australia FST is assessed according to the common law standard in the Victorian case of R v Presser, which describes what an accused needs to understand in order to be tried [21], and R v Kesavarajah, in which the High Court of Australia determined that a person must remain fit for the duration of the trial [22]. Assessments for fitness using the Presser criteria examine the defendant's knowledge of the procedure followed in adversarial trials, intellectual function and communication skills. FST is often assessed some months prior to a trial, which requires the expert to make a prediction about the defendant's mental performance in the future.
By contrast, an opinion about whether an accused is NGMI is based on a retrospective assessment of a person's state of mind at the time of an offence and is usually assisted by documents such as medical records and contemporaneous descriptions of the person's behaviour. In most English-language jurisdictions the way that mental illness defence is defined is derived from the 19th century English case of R v McNaghten, which turns on the extent to which a psychotic illness affected the person's ability to understand that their actions were wrong [23]. In Australia the McNaghton rules were extended by the decision of R v Porter, which considered the effect of disorganized thinking on moral reasoning [24]. Hence in common law, or in the criminal codes of some States of Australia, a defence of mental illness is based on (i) the diagnosis of a severe form of mental illness that (ii) resulted in a false belief or grossly disorganized thinking, which (iii) prevented the person from understanding that their behaviour was morally wrong.
In the present study we examined the level of agreement in expert opinions about FST and NGMI using a consecutive series of criminal matters in which there were written reports by two or more mental health experts. We developed three a priori hypotheses.
The first hypothesis was that agreement about FST and NGMI would be greater in pairs of experts engaged by the same side when compared to experts from opposite sides. A higher level of agreement about FST or NGMI between experts engaged by the same sides would suggest that the role of experts in adversarial proceedings was a source of bias.
The second hypothesis was that there would be an association between agreement about FST and agreement about the diagnosis of disorders that often affect FST. If confirmed, this would suggest that opinions about FST were associated with clinical factors.
The third hypothesis was that there would be an association between agreement about NGMI and pairs of reports in which there was also agreement about the diagnosis of schizophrenia. If confirmed, this would suggest that agreement about NGMI was associated with the diagnosis of serious mental illness.
Methods
Sample of reports
Copies of reports from a consecutive series of 110 criminal cases concluded between 2005 and 2007, in which there were two or more reports written by psychiatrists or psychologists, were made available by the Office of the Director of Public Prosecutions (ODPP) in New South Wales (NSW), Australia. There were a total of 270 reports, 226 by 30 psychiatrists and 44 by 15 psychologists. Defence experts wrote 148 reports and prosecution experts wrote 122 of the reports.
The 110 cases all involved serious offences that were dealt with in the higher courts. They included 30 charges of murder or attempted murder (referred to as homicide offences), 35 charges of wounding or serious assault, 14 charges of sexual assault, 12 serious property offences, 10 drug cases and the remaining eight cases included fraud, kidnapping, arson and firearms offences. Permission to perform the study was obtained from the Justice Health Research and Ethics Committee and the NSW Director of Public Prosecutions.
Data collection
The data points collected were: (i) whether the expert was engaged by the defence or prosecution; (ii) whether the expert was a psychiatrist or a psychologist; (iii) the defendants’ age, gender, marital status, occupation and criminal convictions; (iv) whether the charge was of a homicide offence; (v) whether the main diagnosis was intellectual disability, acquired brain injury or schizophrenia spectrum psychosis (defined as schizophrenia, schizophreniform disorder, schizoaffective disorder, delusional disorder or psychotic disorder not otherwise specified); and (vi) the expert's opinion about FST and NGMI.
All three authors independently rated reports from 28 cases, with one disagreement (about NGMI in a case in which automatism was raised). Each of the remaining 82 cases were rated by two authors, with no disagreements.
Statistical analysis
Analysis of agreement between three or more raters presents a methodological dilemma arising from the number of degrees of statistical freedom [25]. For example, when a defendant is assessed by three experts (A, B and C) there are three pairs (AB, AC and BC). If two pairs of reports agree then the third pair must also agree.
There are statistical techniques to allow for multiple raters, but in the present study there were varying numbers of assessors per case. Solutions that were considered included examining only some of the reports in each case or excluding cases with three or more reports [26, 27]. The arbitrary omission of cases or report pairs, however, could alter the level of agreement. Uncertainty about the effect of degrees of statistical freedom prevented an assessment of the overall level of agreement about FST and NGMI in the present study, but did not effect the examination of report pairs from opposing sides. For example, if report writers A and B are both engaged by the defendant and the prosecution engages C, the pairs AC and BC are independent because AB is omitted. A lack of statistical independence does occur in a few cases with three reports by experts from the same side, causing an increase in overall agreement [27], and a second analysis of agreement was performed after the omission of the third report pair (based on the date of the interview) in those cases.
K statistics [28] were used as an initial measure the level of agreement about FST and NGMI. The level of agreement defined by kappa has been classified as poor, <0.2; fair, 0.2–0.4; moderate, 0.4–0.6; good, 0.6–0.8; and very good, 0.8–1.0 [29].
The proportions of the experts providing an opinion about FST or NGMI were examined using χ2 test. Definitive conclusions about the tendency of experts to make opinions about NGMI or FST, however, cannot be drawn using this method, or from the kappa analysis because some offenders were seen by multiple experts from the same side.
Univariate and multivariate generalized estimating equations (GEEs) were used to allow the inclusion of clustered data (in this case clustered by case) with independent variables that lack statistical independence. The defendant was used as the subject variable and the report pair was the within-subject variable. GEEs were used to assess the association between report pairs with the same expert role and the dependant variables of agreement about FST or NGMI. Demographic variables, expert's profession, defendant's criminal record and presence of homicide charge were controlled for in a multivariate GEE main effects model if they had a significance of below p < 0.1 determined by univariate GEE. Computational options included a logit link function, binomial probability distribution and an unstructured correlation matrix.
A power analysis was performed prospectively to estimate the sample size needed for the GEEs [30]. We assumed that (i) one-third of report pairs would disagree about FST and one-third would disagree about NGMI; (ii) two-thirds of report pairs that agreed about FST and two-thirds of report pairs that agreed about NGMI would contain report pairs written by experts from the same side; (iii) one-third of report pairs that disagreed about FST and one-third of report pairs that disagreed about NGMI would contain report pairs written by experts from the opposite sides. This determined that the study would require 26 report pairs with disagreement and 79 pairs with agreement to have an 80% chance of finding an association between disagreement or agreement and the roles of the experts at a probability of 0.05.
The statistics were performed using SPSS for Windows version 15.0 (SPSS, Chicago, IL, USA).
Results
Opinions regarding FST
An opinion about FST was found in 198 reports about 82 defendants. In 54 cases there were two reports, in 22 cases there were three, and in six cases there were four reports. Hence there were 156 report pairs (54+[22×3]+[6×6]).
Defence experts were less likely to find that the accused was FST (58%, 59/101 reports) than prosecution experts (77%, 75/97 reports, χ2=8.08, p = 0.004). There were four pairs of reports in which prosecution experts found a defendant to be unfit for trial after the defence expert had found the defendant to be FST, and 10 pairs of reports in which the defence expert found that the defendant was not FST but the prosecution expert determined that they were FST. Although these results appear to suggest that defence experts were more likely to find the defendant FST, they might be explained by the opinions about defendants who were assessed by more than one expert from the same side.
Agreement about FST in the reports pairs from opposite sides was fair and agreement in reports written by experts from the same side was moderate. Kappas for experts from the same side were modestly increased by four cases with three expert opinions from the same side (Table 1). GEE indicated that experts from the same side were not more likely to agree about FST than experts from opposite sides. There was a higher level of agreement about FST, in homicide matters and in report pairs written by experts from the same profession (Table 2), and a multivariate GEE found that the only factor independently associated with agreement about FST was presence of homicide charge.
Agreement between experts from the same and opposite adversarial sides about FST and availability of an NGMI defence
CI, confidence interval; FTS, fitness to stand trial; NGMI, not guilty by reason of mental illness. †Kappa 0.352 (95%CI = 0.056–0.647) after the removal of a report pair from four cases with three pair; ‡kappa 0.577 (95%CI = 0.225–0.929) after the removal of a report pairs from four cases with three pairs
GEE analysis of factors associated with agreement about FST
CI, confidence interval; FST, fitness to stand trial; GEE, generalized estimating equation.
Opinions regarding NGMI
An opinion about NGMI was found in 146 reports about 61 defendants. There were 43 cases with two reports, 12 cases with three reports, and six cases in which there were four reports, a total of 115 report pairs, (43+[12×3]+[6×6]).
Defence experts found that the accused was NGMI in 39 of 79 reports (49%) and prosecution experts concluded that the defendant was NGMI in 38 of 67 reports (57%, χ2=0.789, p = 0.375).
Agreement about NGMI between experts from opposite sides was moderate, and there was good agreement between experts from the same side. Kappas for experts from the same side were increased by four cases with three expert opinions from the same side (Table 1). A GEE showed that experts from the same side were not more likely to agree about NGMI, and agreement about the diagnosis of schizophrenia-related psychosis was the only factor that predicted agreement between experts about NGMI (Table 3).
GEE analysis of factors associated with agreement about NGMI
CI, confidence interval; FST, fitness to stand trial; NGMI, not guilty by reason of mental illness.
Discussion
The main findings were as follows: (i) agreement about FST was fair or moderate and agreement about NGMI was moderate or good; (ii) there was little evidence of bias arising from the expert's adversarial role; (iii) agreement about FST was higher in homicide matters; and (iv) agreement about NGMI was higher when there was agreement about the diagnosis of schizophrenia-spectrum psychosis
Although the kappa tests suggested that experts engaged by the same side were more likely to agree about FST and NGMI than experts from opposite sides, the kappas fell well within the 95% confidence intervals. A further analysis using a GEE found that the adversarial role of the experts was not a significant factor determining agreement about FST or NGMI, and being engaged by the same side was a weak predictor of agreement about FST and NGMI, with B coefficients well below 0.1. Hence, the negative finding was unlikely to be a type II error and a larger study would be unlikely to find evidence of significant bias in opinion about FST or NGMI.
The non-significantly increased agreement in experts from the same side found in the kappa and GEE analysis might be explained by some experts tending to agree with earlier experts from the same side in pairs from cases with three or more reports This raises a limitation of the present study, which was that the experts were aware of the opinions of previous experts and the prosecution usually ordered a report only after being served with a report by the defence. The lack of blinding to the opinion of other raters is an unavoidable consequence of the use of real, rather than hypothetical cases. It did not seem to affect the opinions of experts in a study of reports prepared for civil matters, however, in which there was a low level of agreement between experts [27]. A second limitation of the present observational study is that some defence reports may not have been served on the prosecution. Reports withheld by the defence, however, would be more likely to have agreed with prosecution reports and hence the inclusion of those reports would probably have lowered agreement between experts from the same side and increased agreement between experts from opposite sides.
The second hypothesis was partially confirmed because report pairs in which there was agreement about the presence of intellectual disability were more likely to agree about FST. Agreement about the diagnosis of schizophrenia-spectrum psychosis and acquired brain injury, however, was not associated with agreement about FST. In addition, the association between agreement about the diagnosis of intellectual disability and FST was not independent of the association between homicide and FST.
The third hypothesis was confirmed, because agreement about NGMI was associated with agreement about the diagnosis of schizophrenia-spectrum psychosis. The higher level of agreement about FST in homicide matters could have been due to the use of more experienced experts in those matters. The 14 experts who wrote reports in the homicide matters wrote an average of 13 of reports in the present study, whereas the remaining 31 experts wrote an average of fewer than three reports each. Furthermore, the experts who provided fewer than three reports were less likely to refer to the Presser criteria, suggesting that they were less familiar with the legal test for FST.
The lower level of agreement about FST compared with opinions about NGMI may be due to the unsatisfactory nature of the Presser criteria to determine FST, which emphasize the defendant's legal knowledge and do not attempt to set a uniform threshold for FST. Moreover, assessments of FST are often performed many months apart, and disagreement about FST could be due to fluctuation in the patient's condition or the difficulty experienced by the expert in predicting the person's mental state at some future date.
Conclusions
After taking into consideration the limitations arising from an observational study of expert opinion, we found little evidence of bias in expert opinions about FST or NGMI. The high level of agreement about NGMI suggests that although the legal test seems to have little relationship to modern concepts of mental illness, the standard is well understood by mental health experts and is applied in a reliable way. The level of agreement about FST, however, was only modest and suggested the need for reform to the procedure for assessing FST, both in the criteria to be applied and the way in which the assessments are performed.
The present findings demonstrate the value of using quantitative methods to establish the extent to which courts in similar jurisdictions can rely on expert opinion.
Footnotes
Acknowledgements
We would like to thank the New South Wales Director of Public Prosecutions, Mr Nicholas Cowdrey QC, for permission to conduct the study, Mr Craig Hyland at the ODPP, for his assistance in collating the reports, and Dr Peter Arnold for his editorial assistance in the preparation of this paper. Dr Nielssen has provided expert opinion for the prosecution and defence in criminal cases in NSW.