Abstract
The possibility of treating psychotic disorders during the prodromal phase is an alluring prospect for a number of reasons. The prodromal phase is characterized by a considerable array of psychiatric symptoms and disability, including self-harming and other health-damaging behaviours [1], [2]. A substantial amount of the disability that develops in psychotic disorders accumulates prior to the appearance of the full positive psychotic syndrome and may even create a ceiling for eventual recovery [2], [3]. In addition, recent studies have indicated that at some point in the transition from prodromal phase to full-blown psychotic disorder alterations in brain structure (and presumably function) occur [4]. If the prodrome can be recognized prospectively and treatment provided at this stage, then existing disability could be minimized, recovery may be possible before symptoms and poor functioning become entrenched and the possibility of preventing, delaying or ameliorating the onset of diagnosable psychotic disorder arises. Neurobiological changes that occur around the time of onset of full-blown psychotic disorder could also be prevented, minimized or reversed. Thus the prodromal phase presents two possible targets for intervention: (i) current symptoms, behaviour or disability; and (ii) prevention of decline into frank psychotic disorder.
Aside from these two treatment aims, there are a number of other benefits of prodromal treatment [2], [5]. Individuals experiencing this early phase of the disorder may engage more quickly with treatment than those who present late when psychotic symptoms are entrenched, social networks are more disrupted and functioning further deteriorated. Additionally, the individual may be more likely to accept treatment if full-blown psychosis does emerge compared to the individual who has had a longer duration of symptoms before assistance is sought. This may especially be the case given that the person is likely to have already developed a therapeutic relationship with a treating team. Effective treatment can be provided rapidly if the person does develop psychosis, possibly avoiding the need for hospitalization and minimizing the deleterious effect of extended untreated psychosis [6]. Finally, pre-psychotic intervention offers the chance to research the onset phase of psychotic illness, which may provide insight into the core features of the psychopathology and psychobiology of psychosis [2].
A first step in providing treatment to individuals who are possibly in the prodromal phase of psychotic disorder it to accurately identify this population [7]. The Personal Assessment and Crisis Evaluation (PACE) Clinic adopted a ‘close-in’ strategy to identifying the population, which uses combinations of established trait and state risk factors for psychosis with common phenomenology from the prodromal phase of psychotic disorders, as well as narrowing identification to the age range of highest risk [2], [5], [7]. The term ‘at risk mental state’ (ARMS) was used to refer to the phase prospectively identified as the possible precursor to full-blown psychosis [2], [5], [7]. The criteria introduced to identify an ARMS were termed the ‘ultra-high risk’ (UHR) criteria. UHR criteria consist of the following groups: (i) attenuated psychotic symptoms group, who have experienced subthreshold, attenuated forms of positive psychotic symptoms during the past year; (ii) brief limited intermittent psychotic symptoms group (BLIPS), who have experienced episodes of frank psychotic symptoms that have not lasted longer than 1 week and have spontaneously abated; and (iii) trait and state risk factor group, who have a first-degree relative with a psychotic disorder or who have a schizotypal personality disorder in addition to a significant decrease in functioning during the previous year or chronic low functioning (see Yung et al.[8] for the full operationalized criteria). The person must be aged between 14 and 30 years and cannot have experienced a psychotic episode for longer than 1 week or received neuroleptic medication prior to referral to the PACE Clinic.
The introduction of the UHR criteria provided a valid and reliable means of identifying individuals at high risk of imminent onset of psychotic disorder [8], [9], with rates of transition to frank psychotic disorder within 12 months ranging from 35% to 54% [8], [10–12]. The two main instruments used to assess UHR status are the Comprehensive Assessment of At Risk Mental States (CAARMS) [13], developed by the PACE Clinic, and the Structured Interview for Prodromal Syndromes (SIPS) [14]. Other instruments used to detect a possible prodromal state include the Bonn Scale for the Assessment of Basic Symptoms [15] and the Schizophrenia Prediction Instrument–Adult version [16], both of which derive from the basic symptom approach, the Early Recognition Inventory [17] and the Basel Early Detection Checklist [18] (for review see [19]).
The importance of identifying UHR status as accurately as possible is twofold: first, to maximize identification of cases at risk of onset of psychosis (and not turn them away from UHR clinics) and, second, to minimize treatment of people who were never truly at risk of psychosis (i.e. misclassified cases), with treatments specifically designed to reduce risk of progression to psychosis. Research indicates that ‘misclassification’ of UHR status is not uncommon. For example, Nelson and Yung report that 11.4% of cases accepted to the PACE Clinic were in fact experiencing first-episode psychosis (FEP) at time of referral [20]. Broome et al. and Loewy et al. reported that 18% and 33% of cases were psychotic upon referral to UHR clinics, respectively [21], [22]. There may also be UHR-positive subjects who are not identified (i.e. are judged to be sub-threshold for UHR status), even by established early intervention services, as suggested by the Yung et al.[23] results.
In order to assist mental health professionals to identify UHR cases, the PACE Clinic and Early Psychosis Prevention and Intevention Centre (EPPIC) Statewide Services recently developed a training DVD and manual in the use of the CAARMS. The current paper presents the outcome of a series of training workshops with mental health professionals based around this DVD. The research aim was to investigate whether the training workshops assisted mental health professionals in their confidence and ability to accurately identify UHR cases and distinguish these from non-UHR and FEP cases.
Method
Participants
Recruitment of participants took place during two separate training programmes. The first, in September–October 2006, was held at six sites throughout the UK. The second training programme was held at two sites in Sydney, Australia in May 2007. Group sizes ranged from nine to 24 participants. In total, 137 mental health professionals (89 female, 40 male, eight unidentified) attended the training sessions. The mean age was 36.44 years (SD = 8.1 years).
Materials
Intervention: CAARMS training DVD
The CAARMS training DVD (www.orygen.org.au) consists of four filmed interview vignettes. The interviews were acted by staff of the PACE Clinic. Two of the vignettes are designed to represent UHR patients, one vignette is designed to represent a patient with FEP, and one vignette is designed to represent a patient who reports some psychotic-like experiences but who does not meet UHR criteria (i.e. subthreshold for UHR status). Because the DVD is for training purposes, the interview questions are directed towards acquiring the clinical information required to rate the first four scales of the CAARMS (the scales required to determine UHR status), rather than conducting a full clinical interview. The first four scales of the CAARMS consist of: Unusual Thought Content (UTC), Non-Bizarre Ideas (NBI), Perceptual Abnormalities (PA), and Disorganized Speech (DS). The DVD also consists of an introduction to the area of UHR research and explanation of how each of the vignettes would be rated on the CAARMS. The DVD is approximately 75 min in duration.
Evaluation: pre- and post-training written vignettes
Four written vignettes were presented to participants before the training and four different vignettes were presented to participants after the training. Two vignettes in each set (i.e. pre-test set and post-test set) were designed to represent UHR-positive subjects (one meeting the attenuated psychotic symptoms criteria and the other the trait plus state risk factor criteria), one was designed to represent a FEP subject, and one represented a subject who was sub-threshold for UHR status. The vignettes presented before and after the workshop were matched for UHR inclusion group and difficulty of rating by the authors using a consensus approach.
Participants were asked to rate the vignettes on the first three scales of the CAARMS (UTC, NBI and PA). Disorganized Speech (DS) could not be rated because the vignettes were written instead of visual presentations. Scoring details for the CAARMS have been described in greater detail elsewhere [13]. Briefly, CAARMS scoring consists of a global rating (level of severity of symptom), a frequency rating and a dichotomous inclusion/exclusion rating. Participants were asked which inclusion/exclusion category they believed the written vignette represented. The vignettes were less than one page in length, presenting the necessary clinical information required to make a judgement of UHR status. The vignettes were written by the authors based on patients seen at the PACE Clinic (anonymity preserved). An example of one of the vignettes is presented in Table 1. The authors established gold standard ratings of the vignettes on the CAARMS scales using a consensus approach. Ratings on the pre-training vignettes were not discussed during the training session.
Example of vignette used in the CAARMS training evaluation†
CAARMS, Comprehensive Assessment of At Risk Mental States; UHR, ultra-high risk.
†The patient is UHR-positive due to meeting attenuated psychotic symptoms criteria.
Evaluation: confidence in identifying ARMS and usefulness of workshop/DVD
Before the workshop, participants were asked to record their demographic details, professional discipline, years working in mental health, previous experience with the UHR population, familiarity with the concept of the ARMS, and their confidence in identifying ARMS and in using the CAARMS. After the workshop, participants were asked to re-rate their confidence in identifying ARMS and in using the CAARMS. They were also asked to rate whether the workshop and DVD were useful. All ratings were made on a 0 (‘not at all’) to 10 (‘extremely’) Likert scale.
Procedure
Training workshop
The training workshop consisted of four modules.
(1) Theoretical background: education regarding the theoretical background of early psychosis research and intervention, focusing on the prodromal period, was provided in the form of a powerpoint presentation. The following areas were addressed: advantages of pre-psychotic intervention; research into the pre-psychotic period; the development of the UHR criteria; and ethics of pre-psychotic intervention. Time was allowed for general discussion at the end of the presentation.
(2) Rating written vignettes: four written vignettes were rated after the presentation and before viewing the DVD (pre-training vignettes).
(3) Viewing and discussion of the CAARMS Training DVD: the DVD was screened to participants. Each of the vignettes in the DVD was discussed among participants and workshop facilitators, focusing on how each vignette would be rated on the CAARMS.
(4) Rating written vignettes: four written vignettes were rated at the end of the workshop (post-training vignettes).
Results
Participant characteristics
Participants came from a range of disciplines, including nurses (n = 49, 35.8%), psychologists (n = 23, 16.8%), psychiatrists (n = 19, 13.9%), social workers (n = 13, 9.5%), research staff (n = 8, 5.8%), occupational therapists (n = 7, 5.1%), students (n = 4, 2.9%), medical doctors (n = 2, 1.5%) and other (e.g. support worker, care coordinator, n = 12, 8.8%). The majority of participants (73.1%) had not previously attended a training session specifically addressing ARMS/UHR. Further sample characteristics are presented in Table 2.
Sample characteristics (mean ± SD)
ARMS, at-risk mental state.
Confidence and usefulness of workshop/DVD
Table 3 presents data relating to participants’ familiarity with the concept of ARMS and confidence in identifying ARMS and using the CAARMS before and after the workshop. Participants reported moderate levels of familiarity and confidence with the concept of ARMS prior to the workshop and were not confident about using the CAARMS. After the workshop, participants reported a significant improvement in their self-reported confidence with ARMS/CAARMS (p < 0.001). Participants strongly agreed that the workshop was useful (mean score = 8.2/10) and that the DVD was a useful training resource (mean score = 8.1/10).
Pre- and post-workshop familiarity and confidence ratings (mean±SD)
ARMS, at-risk mental state; CAARMS, Comprehensive Assessment of At-Risk Mental States.
Inclusion/exclusion criteria ratings
At the pre-workshop assessment the proportion of participants who correctly identified the inclusion/exclusion criteria varied between vignettes (Table 4). The majority of participants were able to correctly rate the UHR-positive vignettes (scenario one = 92.9%, scenario four = 88.8%). The majority of participants who correctly rated the UHR-positive vignettes also correctly identified the appropriate inclusion/exclusion group (Table 5). Participants were less competent in accurately rating the UHR-negative vignettes (scenario two = 42.4%, scenario three = 48.8%). In addition, only approximately half of the correct UHR-negative ratings were rated as belonging to the correct UHR-negative group (sub-threshold UHR or FEP; Table 5).
Percentage of participants who correctly rated vignettes as meeting UHR inclusion/exclusion criteria
These data are based on participants who completed the rating form at both pre- and post-training assessment.
APS, attenuated psychotic symptoms; UHR, ultra-high risk.
Percentage of participants with correct vignette ratings
These data are based on all available data, irrespective of whether the participant only completed pre- or post-training ratings.
At the post-workshop assessment the proportion of participants who correctly rated the UHR-positive vignettes decreased for one vignette (92.9–76.5%) and remained the same for the other vignette (88.8–87.7%). χ2 tests indicated that these changes were not significant (Table 4). The proportion of participants who correctly rated the UHR-negative vignettes increased (42.2–78.8%, 48.8–58.8%) but χ2 tests indicated that these changes were not significant (Table 4).
As indicated in Table 5, there were a substantial number of missing cases for the pre- and post-workshop vignette ratings. Of the 137 participants, only 74 (54%) completed all four pre-workshop vignette ratings and 79 (57.7%) completed all four post-workshop vignette ratings. We assessed whether there were differences between participants who completed and who did not complete vignette ratings on demographics and previous training, familiarity and confidence in assessing ARMS and using the CAARMS (Table 6). The only difference between the groups was that participants who did not complete the post-workshop vignette ratings (‘non-completers’) had undergone a greater number of hours training in the concept and identification of the ARMS prior to this workshop than participants who did complete the post-workshop vignette ratings (‘completers’).
Vignette completers vs non-completers
ARMS, at-risk mental state; CAARMS, Comprehensive Assessment of At-Risk Mental States.
Completers, participants who completed all four vignettes; non-completers, participants who did not complete all four vignettes. †χr2; ‡t. ∗p < 0.05.
∗p < 0.05.
Comments on the session
Participants were asked three open-ended questions on the post-workshop evaluation form. The first question, ‘Was the workshop useful?’ elicited 11 positive comments (e.g. ‘very useful and clearly explained workshop’), one negative comment (‘too research orientated’) and seven neutral comments (e.g. ‘I feel that I require a lot of practice’). The second question, ‘Was the DVD useful?’ elicited 11 positive comments (e.g. ‘variety of approaches to interviewing very useful’), five negative responses (e.g. ‘didn't agree with some of the ratings’), and three neutral comments (e.g. ‘practical experience more useful’). Finally, participants were invited to comment on any aspect of the training day. The response was overwhelmingly positive, with 21 positive comments (e.g. ‘very valuable training, useful, efficient’), nine neutral comments (e.g. ‘it was interesting… but could have been summarized more’) and three negative comments (e.g. ‘quite a lot to absorb in one session, perhaps better to break up into more than one session –- though perhaps not very practical’). The majority of the neutral and negative comments referred to the training being enjoyable but that it was too much information to absorb in a single day's training.
Discussion
In order to achieve the goal of preventative intervention for schizophrenia and other psychotic disorders, it is necessary to accurately identify UHR cases. The current paper reports data from a series of training workshops with mental health professionals designed to assist with the use of the CAARMS questionnaire to identify UHR cases. The findings indicate that participants’ confidence in identifying ARMS/UHR cases and in using the CAARMS increased as a result of the workshop. Participants’ ability to correctly identify UHR-positive cases did not improve as a result of the workshop, but there was a trend for participants’ ability to correctly identify UHR-negative cases (either above or below UHR threshold) to improve as a result of the workshop. In other words, the workshop did not improve sensitivity of UHR identification (accurately identifying UHR-positive cases), where ability was already high, but did enhance specificity of UHR identification (accurately identifying UHR-negative cases).
Examination of the data indicates that participants’ baseline (i.e. pre-workshop) ability to identify UHR-positive cases was high. This introduced a ceiling effect for the training workshop's potential to improve correct identification of UHR-positive cases. That is, there was little room for improvement. This may have been due to participants’ relatively high level of experience working in mental health (approximately 10 years) and previous experience with ARMS (approximately 25% of the total sample). Participants’ baseline ability to identify UHR-negative cases was poor (fewer than half were correctly identified). This indicates that although experienced mental health professionals may be fairly competent in identifying UHR cases, they are less able to identify non-UHR cases. The results indicate that there is equal difficulty with identifying below-UHR-threshold and above-UHR-threshold (i.e. FEP) cases.
This pattern of results indicates that clinical services may have a relatively low threshold for identifying UHR cases and a high threshold for identifying FEP cases. That is, individuals who are UHR negative due to being below the UHR threshold are identified as UHR positive, and FEP-positive cases are identified as FEP negative but UHR positive [20]. In terms of service delivery, this low threshold for UHR status is preferable to a high threshold because clinical services will be provided to a wide group of patients, which will include UHR cases, rather than turning UHR patients away from a clinical service due to being falsely identified as not meeting UHR criteria.
We mentioned in this paper that an important reason for the accurate identification of UHR status is minimizing treatment of people who were never truly at risk of psychosis (i.e. UHR-negative cases) with treatments specifically designed to reduce risk of progression to psychosis. A low threshold for identifying UHR cases will of course include such cases but a treatment approach that is generalized and non-specific in the first instance and then becomes more specific as there is failure to achieve recovery or the target syndrome becomes more clearly defined, in accordance with the staging model of psychiatric disorders [24], [25], addresses this concern. All UHR subjects present with a clinical need and therefore require clinical services, even if this is targeted towards non-psychotic complaints such as mood or anxiety disorders rather than risk for psychosis [5]. A related point is that it is important to maintain a low-stigma environment at UHR services, both due to the general need to reduce stigma associated with mental illness and due to the fact that UHR-positive subjects may be false positives for psychotic disorder [2].
Similarly, the misidentification of FEP subjects as UHR subjects is not overly concerning from a clinical perspective if a clinical service is available for UHR patients. In this scenario, the FEP patient will receive clinical care at a UHR service and be identified as experiencing frank psychosis as treatment progresses and will then be commenced on appropriate treatment [20]. This highlights the importance of UHR services: in the absence of such services the misidentified FEP subject will be turned away from clinical services, resulting in an extended duration of untreated psychosis. But from the perspective of research studies, accurate UHR-negative, UHR-positive and FEP identification is important in order to ensure sample validity. Given that the workshops reported in the present paper were found to improve identification of UHR-negative cases, CAARMS workshops or similar forms of training should be provided to researchers prior to commencing research studies with the UHR population.
In terms of methodological limitations, there were a substantial number of missing cases for the post-workshop ratings. The data indicate a selective completion of post-workshop ratings. Specifically, participants with more training in the concept and identification of the ARMS (as operationalized by a greater number of hours previous training in ARMS) were less likely to rate the post-workshop vignettes than participants with less prior training in ARMS. This suggests a bias towards participants who felt that they needed further practice with using the CAARMS and identifying UHR status completing the post-workshop ratings, while participants with more previous training decided not to complete these ratings. It is possible that more training prior to the workshop may have enhanced the effect of the workshop in improving ability to use the CAARMS and identify UHR status. Therefore, the post-workshop data may have been skewed towards participants who did not improve substantially in their ability to use the CAARMS and identify UHR status (due to less previous training), and data were lacking from participants who did improve in these areas (due to more previous training). The qualitative results lend some support to this interpretation because they indicate that some participants felt overwhelmed by the large amount of information presented in the training session. This may have particularly been the case for participants with less previous training in ARMS, thus diminishing the positive impact of the workshop on UHR identification for these participants. Finally, although the pre- and post-workshop vignettes were matched for UHR inclusion group and difficulty of rating using a consensus approach, it is possible that accurate matching was not achieved. This may have confounded the post-workshop outcome data. In future workshops we will reverse the order of presentation in the pre- and post-workshop vignettes in order to assess this possibility.
In conclusion, the results suggest that a 1 day workshop consisting of education about UHR identification and intervention and written and video-based vignette training may assist with using the CAARMS to correctly identify UHR cases, particularly UHR-negative cases. But, given the limitations of the study noted here, further evaluation of UHR training workshops is required. In the current sample the moderately high ability to identify UHR-positive cases prior to the workshop may have introduced a ceiling effect for the training workshop's potential to improve correct identification of UHR-positive cases. The results suggest that UHR training programmes with experienced mental health professionals should pay particular attention to the correct identification of UHR-negative cases.
Footnotes
Acknowledgements
The authors would like to thank the participants in the training workshops and acknowledge the support of the Colonial Foundation.