Abstract
Introduction
Over the last 15 years, criteria have been introduced for identifying individuals at ‘ultra-high risk’ (UHR) of psychotic disorder – that is, in the putatively prodromal phase [1–3]. These criteria are based on a combination of known trait and state risk factors for psychosis onset, including age range of highest risk of psychosis onset, family history of psychotic disorder, and attenuated psychotic symptoms. Help-seeking young people who meet these criteria have a high risk of becoming psychotic over 1–2 years [2, 4, 5]. Together with studies that indicated that antipsychotic medication could decrease or delay onset of psychosis [6] and reduce symptoms [7] in high risk patients, these findings have led in recent years, to the practice of prescribing antipsychotic medication to UHR patients [5, 8–11]. This practice is based on clinical impression of an incipient psychosis (i.e., a clinical judgment that a particular UHR patient may soon progress to full-threshold psychotic disorder and may therefore benefit from antipsychotic medication). However, the evidence to date provides only partial support for the effectiveness of antipsychotic medication in delaying or preventing onset of psychosis in the UHR population and is not strong enough to warrant prescription of antipsychotics outside the context of controlled clinical trials (see [12] for a review).
This situation raises a number of issues, including the validity of clinical impression as a predictor of transition to full-threshold psychosis in the UHR population. The issue of clinical versus actuarial (i.e., standardised or statistical) judgments has been debated in the social and medical sciences [13–15], in areas such as psychiatric diagnosis, detection of brain damage and characterisation of personality (see [13, 14] for reviews). Actuarial methods have generally been found to exceed clinical methods in judging a particular condition or outcome [14].
Although the clinical impression method has been researched in patients with established psychotic disorders (e.g., see Grube's [12] study of the ‘praecox feeling’, a classic exemplar of the clinical method, in patients with schizophrenia), it has not yet been investigated as a means of predicting future onset of psychosis in the UHR population. In the current study, we investigated whether clinical impression reported by clinicians upon patients’ entry to a UHR clinic identified high risk patients who progressed to full-threshold psychotic disorder. If clinical impression predicts psychosis outcome in UHR samples, it may be a useful supplementary tool for ‘closing in’ on UHR patients at highest risk for onset of psychotic disorder and therefore informing treatment choice for particular UHR patients [16]. Conversely, if clinical impression does not predict psychosis outcome in UHR samples, then this would indicate that the prescription of antipsychotic medication in this group based on clinical impression of incipient psychosis is a questionable practice.
Method
Participants
Participants consisted of UHR patients referred to the PACE clinic, ORYGEN Youth Health between August 2000 and February 2007. The PACE clinic is a specialised clinical and research service for young people at ultra high risk of psychotic disorder [17]. PACE accepted 684 UHR referrals over this period. 168 of these cases (25%) were rated by outpatient case managers (OCMs) for their clinical impression of whether the patient would develop psychosis. Inclusion criteria were meeting the UHR criteria (see Table 1), being aged between 15 and 30 years (in 2005 the upper limit of the age range was reduced to 25 years) and residing in OYH's catchment area (western metropolitan Melbourne). Exclusion criteria included: presence of psychotic disorder, known organic cause for presentation, known intellectual disability (IQ < 70) and an inability to speak English. The missing ratings were due to OCMs not completing all documentation when seeing new patients.
Ultra-high risk (UHR) and full threshold psychosis criteria
Note: See Yung et al. [3] for the full operationalized criteria.
Materials
Clinical impression was rated at baseline by asking OCMs working at the PACE clinic a written question ‘Do you think the person will develop psychosis in the next 12 months?’ (‘Yes’ or ‘No’ (dichotomous) response possible). No instructions were provided for how the OCM should judge this, nor were the reasons for the OCM's judgment recorded. All OCMs (seven in total) were experienced clinical psychologists. They ranged from having 3–11 years experience working with early psychosis patients.
UHR criteria at baseline were assessed with the Comprehensive Assessment of At Risk Mental States (CAARMS [18]) and Global Assessment of Functioning (GAF [19]). The outcome of psychosis was assessed at 12-month follow-up by the CAARMS using previously defined criteria [3]. If no CAARMS data were available, State medical records were checked to ascertain if any psychiatric services were accessed by the individual and if so, records were reviewed to determine psychosis status. This was required on 31 participants (18.5% of the sample). Subtype of psychosis was assessed using the SCID-IV [20] or, for participants checked in State medical records, the diagnosis of the treating clinical team.
Procedure
OCMs rated their clinical impression of risk for psychosis immediately following their first appointment with the new PACE patient. The clinical impression rating was completed along with other documentation (clinical notes, writing letters regarding attendance, updating registration details, etc.) as part of standard clinical practice. The first appointment was generally approximately one hour in length and consisted of assessment of presenting symptoms and history of symptoms, psychoeducation and information regarding the clinic, and assessment of any risk issues (see [21] for description of the treatment approach at the PACE clinic).
A total of 92 patients (55% of the sample) were followed up regularly (weekly–fortnightly) by their OCM and doctor for a 12-month period. 76 (45% of the sample) were discharged from the service prior to 12 months.
Results
Of the 168 cases, 102 (60.7%) were female. The mean age was 18.34 years (SD = 2.75 years, range = 14–26 years). Intake group information is provided in Figure 1. Follow up data on psychosis status were available for all 168 participants, 92 (55%) of whom were rated via CAARMS interview and 76 (45%) using State medical records.
Intake group and number (%) in each group.
Transition to psychosis
Fifteen of the 168 subjects in this sample developed psychosis by 12 months (8.9%).
Predictive validity of clinical impression of risk for psychosis
OCMs thought that 37 participants (22% of the sample) would develop psychosis by 12 months. Of these 37, 12 developed psychosis and 25 did not. An additional three subjects who were thought would not develop psychosis did in fact develop psychosis within 12 months (see Table 2). Thus there was a sensitivity of 0.80, specificity of 0.84, positive predictive value (PPV) of 0.32 and negative predictive value (NPV) of 0.98 for the prediction of psychosis using clinical judgment.
Cases who transitioned to psychotic disorder by clinical prediction of whether the patient would develop a psychotic disorder
Psychotic diagnoses
Of the 15 cases who developed a psychotic disorder, four developed a schizophrenia spectrum psychosis (schizophrenia, schizophreniform disorder, schizoaffective disorder), two an affective psychosis (mood disorder with psychotic features), and nine developed other psychoses (psychotic disorder NOS, delusional disorder).
Predictors of clinical impression of risk for psychosis
Table 3 summarises demographic and intake variables for participants for whom clinical impression of risk for psychosis was rated positive versus those for whom it was rated negative. Participants who were rated positive were older, higher on the CAARMS Perceptual Abnormalities scale, and met than more than one of the UHR intake groups.
Comparison of demographic and intake variables by whether the clinical impression ratings were rated as positive or negative
Notes: UTC, CAARMS Unusual Thought Content Scale; PA, CAARMS Perceptual Abnormalities Scale; DS, CAARMS Disorganised Speech Scale; GAF, Global Assessment of Functioning Scale. CAARMS scales are rated on a 0–6 continuous scale. ∗P < 0.05.
Predictors of completing the clinical impression of risk for psychosis rating
Table 4 presents demographic and intake data comparing PACE patients for whom the clinical impression of risk for psychosis was rated at all versus patients who were referred over the same time period for whom this was not rated. The two groups did not differ with regards to age, gender, severity of attenuated psychotic symptoms, or intake group. However, the group for whom clinical impression was rated was lower on general functioning (GAF score).
Comparison of demographic and intake variables between UHR cases for whom the clinical impression rating was completed and cases for whom it was not completed
Notes: UTC, CAARMS Unusual Thought Content Scale; PA, CAARMS Perceptual Abnormalities Scale; DS, CAARMS Disorganised Speech Scale; GAF, Global Assessment of Functioning Scale; APS, Attenuated Psychotic Symptoms Group; CAARMS scales are rated on a 0–6 continuous scale; ∗∗P < 0.01.
Discussion
In this study, we investigated the predictive validity of clinical judgment of whether a UHR patient would develop full-threshold psychosis. The results indicated that clinical judgment at first clinical appointment was a poor predictor of whether UHR patients developed a psychotic disorder within 12 months (PPV = 0.27). However, a negative clinical judgment (i.e., clinical prediction that the patient would not develop psychosis) was a strong predictor that the UHR patient would not develop a psychotic disorder within 12 months (NPV = .98). This latter finding was due to the low rate of transition to psychotic disorder in the sample overall (8.9%), combined with a reasonably high rate of negative clinical judgment ratings (78%). OCMs were more likely to have the impression that the patient would develop psychosis if the patient was older, experienced more perceptual abnormalities, and met more than one UHR intake group. This may partly have been due to OCMs’ familiarity with previous research indicating that longer duration of symptoms [2, 22], correlated with being older, and belonging to more than one UHR intake group [3] are associated with increased risk of transition to psychosis. It also indicates that clinicians may have greater clinical concern regarding incipient psychosis if patients present with more marked perceptual abnormalities, such as hallucinatory experiences, rather than other attenuated psychotic symptoms, such as subthreshold paranoid symptoms.
The poor predictive power of clinical judgment of whether UHR patients will develop psychosis has both clinical and research implications. As noted above, over recent years there have been increased rates of prescription of antipsychotic medication to patients who have been identified as being at high risk of psychotic disorder, but do not yet meet criteria for full-threshold psychotic disorder. This increased prescription of antipsychotics has been based on clinical impression of incipient psychosis and has occurred outside the context of controlled trials [9, 10]. However, the current results indicate that clinicians, even experienced clinicians such as those involved in this study, cannot reliably predict which patients will go on to develop psychotic disorder, at least over the short term. Thus it seems that the use of antipsychotic medication outside the context of a controlled trial is not justified, particularly given the significant side-effects associated with antipsychotic medication for UHR patients [7, 23].
In terms of research, the poor predictive value of clinical judgment suggests that the assessment of variables that may predict onset of psychotic disorder in UHR samples needs to continue to be rigorously assessed. Assessing a range of clinical, neurocognitive, and neurobiological variables may assist with identifying individuals in the truly prodromal phase (i.e., true positives), rather than relying on clinical impression.
This study suffered from a number of limitations. First, all participants received active treatment. The treatment varied across the cohort depending on whether they were involved in a randomised controlled trial being conducted at the PACE clinic. Some of the participants in this trial received low-dose antipsychotic medication [24]. Other treatments received by this cohort consisted of cognitive-behaviour therapy, supportive therapy, and antidepressant medication [25]. This active treatment may have delayed or even prevented onset of psychotic disorder in cases whom clinicians predicted would develop psychosis and who would indeed have gone on to develop psychosis if they had not received treatment (UHR ‘false false positives’ [26]). In other words, treatment may have confounded the predictive validity of clinical judgment of risk for psychosis.
Second, inter-rater reliability amongst OCMs was not established for clinical impression of risk for psychosis. Therefore, it is possible that there was diversity in clinical judgments between OCMs. Third, the fact that a clinical judgment of risk for psychosis was not made for every new patient seen at the PACE clinic is a potential source of bias. Our analysis indicated that the only difference on a selection of demographic and symptom data between patients for whom this rating was made and patients for whom it was not made was on the functioning measure, i.e., OCMs were more likely to rate their clinical impression of risk for psychosis (either positive or negative) for patients with poorer overall functioning. However, poorer functioning was not associated with a positive clinical judgment (i.e., prediction that the patient would develop psychosis). This may indicate that OCMs are more likely to complete all relevant documentation for patients who have lower functioning, possibly due to an increased general concern about their clinical presentation, rather than a specific concern that such patients are at increased risk of psychosis.
Future work should address the predictive validity of clinical judgment of whether UHR patients will develop full-threshold psychosis with larger samples and over a longer follow-up period, as well as investigate inter-rater reliability of clinical judgment ratings. Recording the reasons for clinical judgment of whether a patient will develop psychosis or not would be useful in increasing understanding of the clinical heuristics at play. It would also be of interest to investigate clinical impression of whether a patient will develop psychosis, and the reasons for this, in different clinical settings, e.g., between general practitioners and clinicians working in a specialised early intervention service. Finally, future research should investigate whether clinical impression (and certain types of clinical impression) is a more valid predictor of onset of certain types of psychotic disorders than others. For example, the clinical impression of the ‘praecox’ feeling (an intuitive sense of not being able to develop empathic ‘contact’ with the person [27]) in UHR patients may predict future onset of schizophrenia but not other psychotic disorders. This possibility is suggested by evidence that the ‘praecox’ feeling is a good indicator of a schizophrenia diagnosis in patients with established psychotic disorders, when compared with standardised classification systems such as the ICD-10 and DSM-IV [28].
Conclusion
This study demonstrates that even clinicians experienced with managing UHR patients are poor predictors of which patients will develop frank psychosis. It is therefore likely that other health care professionals, such as general practitioners and private psychiatrists, would also have difficulty predicting psychosis outcome in patients. We therefore recommend that they do not prescribe antipsychotic medication on the basis of their belief that the patient will develop psychosis. Rather, in line with the treatment guidelines for early psychosis [29], UHR patients should be monitored and treated for current symptoms. If rapid deterioration occurs in the context of worsening attenuated psychotic symptoms, then that may be an indication for antipsychotics even before onset of frank psychosis. However, otherwise it is recommended to adopt a ‘watchful waiting’ approach. Our findings are of particular relevance given that the UHR criteria are currently being considered for inclusion in the next iteration of the Diagnostic and Statistical Manual of Mental Disorders (DSM-V) [30–32]. This may lead to application of the criteria widely across different health services with the possible consequence of increased prescription of antipsychotics if clinical impression is adapted to guide their use.
Footnotes
Acknowledgements
This research was supported by the Colonial Foundation and a National Health and Medical Research Council (NHMRC) Program Grant (# 566529). BN was supported by a Ronald Philip Griffith Fellowship and a NARSAD Young Investigator Award. AY was supported by a NHMRC Senior Research Fellowship.