Abstract
Most psychotic disorders, such as schizophrenia, begin with a prodromal period of altered functioning or symptomatology before the onset of frank or threshold psychosis [1]. Clinicians and researchers alike have viewed this prodromal period as potentially important from a preventive perspective: if it could be recognized prospectively and treatment given at the early stage, then the course of illness may be altered. However, prodromal symptoms and signs tend to be non-specific [1] and, therefore, prospective detection is complicated by a likely high false-positive rate, that is, not all people who seem to be experiencing prodromal symptoms will make the transition to threshold psychotic disorder. Thus, prospectively we are dealing with degrees of risk, and the mental state thought to be a prodrome is best termed as At-Risk Mental State or ARMS, a state that confers high, but not inevitable risk of development of psychotic disorder in the near future [2], [3].
By literature review [4] and retrospective study of prodromes in first-episode cases [5], we found that a consistent feature of late prodromes, that is incipient psychosis, was the presence of ‘attenuated’ or ‘subthreshold’ psychotic features. These differ from frank psychotic symptoms in their intensity, frequency and/or duration. For example, a persecutory idea that is held with less than delusional conviction has less intensity than a fully formed persecutory delusion. If an individual fleetingly holds a persecutory belief with delusional conviction for only 1 hour, but this then recedes and the patient then realizes that the belief was false, then this may be seen as a subthreshold form of a psychotic disorder and is distinguished from a threshold psychotic disorder on the basis of duration: the abnormal experience has not been present for long enough. Similarly, if someone had this belief for 1 hour twice a month, it may be considered below the threshold for the diagnosis of a psychotic disorder on the basis of frequency: it is not happening often enough. If the frequency and duration were to increase, for instance, to a point where the belief was occurring continuously for a week, then the patient may be considered to have crossed the threshold in terms of intensity, frequency and duration, and frank psychotic disorder would be said to be present.
It is likely that not all people with sub threshold forms of psychotic symptoms and syndromes will develop a full-blown psychotic disorder. However, a combination of these subthreshold syndromes with other risk factors for psychotic disorder may increase the likelihood of imminent onset of psychotic disorder in an individual. We recently used such an approach for identifying a group said to be at ultra high risk (UHR) for psychotic disorder (i.e. putatively prodromal) [6–8]. For example, the combination of subthreshold psychotic symptoms and age range in the peak period of risk for first-onset psychotic disorder (15–29 years) may identify people with incipient psychotic disorder. Initially, the Brief Psychiatric Rating Scale (BPRS) [9], psychotic subscales (unusual thought content, suspiciousness, hallucinations and conceptual disorganization) and the Comprehensive Assessment of Symptoms and History (CASH) [10] delusional conviction scale were used to assess the intensity of psychotic-like experiences. However, additional criteria were needed for frequency and duration of the experiences, as this degree of detail in relation to subthreshold symptoms is missing from these instruments. The recency of these symptoms, that is when they last occurred, was also added as degree of risk may fluctuate depending on current or recent symptomatology.
In our previous study, operationalized cut-off points for symptoms considered to be subthreshold and threshold for frank psychotic disorder were developed [7], [8]. The definition of frank psychotic disorder is based on the presence of clear-cut threshold level psychotic symptoms (delusions, hallucinations and formal thought disorder) occurring several times per week for at least 1 week. This threshold is essentially that at which neuroleptic medication would probably be commenced in common clinical practice. This definition of onset of threshold psychotic disorder is, of course, somewhat arbitrary, but does at least have clear treatment implications and applies equally well to substance-related symptoms, symptoms that have a mood component and schizophrenia spectrum disorders. It is not entirely concordant with the DSM-IV [11] psychotic disorder categories, which include, for example, brief psychotic disorder. This category allows a psychotic diagnosis to be made on the basis of the person experiencing as little as 1 day of one psychotic symptom with no frequency of the experiences specified. Thus, an individual may be conceptualized using our criteria as UHR, but meet the DSM-IV criteria for brief psychotic disorder. An example would be someone who experiences 1 week of infrequent auditory hallucinations which then resolve.
As hypothesized, meeting the UHR criteria was associated with a high rate of onset of psychotic disorder (Kaplan–Meier estimate of 0.41 with 95% confidence interval (CI) (0.25, 0.53) within 12 months), a rate several hundred-fold above that expected in the general population [7].
Having defined these UHR criteria, the next step was to develop an instrument which could include all dimensions of psychopathology needed for the definition. That is, one that could measure intensity, frequency, duration and recency of subthreshold psychotic symptoms. The Comprehensive Assessment of At-Risk Mental States (CAARMS) was designed for this purpose. Additionally, the CAARMS measures other symptoms thought to be indicative of imminent psychotic disorder such as negative, dissociative and ‘basic’ [12], [13] symptoms. Basic symptoms are subjective deficit features, the experiential equivalents of the behaviourally defined negative symptoms, and are thought to occur as precursor symptoms in the prodromal phase of schizophrenia, as well as in the deficit state. Some basic symptoms, particularly thought, language and perceptual disturbances, have recently been found to predict subsequent psychosis in a German study of individuals referred to a psychiatric service and followed up over 8–9 years [13].
The CAARMS cut-offs for UHR status and frank psychotic disorder were developed based on the previous BPRS/CASH criteria. The CAARMS has two functions: (i) to provide a comprehensive assessment of psychopathology thought to indicate imminent development of a first-episode psychotic disorder; and (ii) to determine if an individual meets UHR status based on criteria derived from the CAARMS assessment (Table 1). The CAARMS was first designed in 1996 and has undergone stepwise revision over the last 9 years. This paper describes the pilot evaluation of the CAARMS.
CAARMS-defined ultra high risk and psychotic disorder threshold criteria
Description of the CAARMS
The CAARMS is a semistructured interview schedule designed for use by mental health professionals who are already able to assess and evaluate patients' information. It is designed for repeated use over time, for example, monthly to 6 monthly.
The CAARMS includes the following subscales: disorders of thought content (e.g. delusional mood, overvalued ideas and delusions), perceptual abnormalities (e.g. distortions, illusions and hallucinations), conceptual disorganization (e.g. subjectively experienced difficulties with forming thoughts and objective assessment of formal thought disorder), motor changes (e.g. subjectively experienced difficulties with movement and objective signs of catatonia), concentration and attention (assessing both the subjective experience and objective rating), emotion and affect (assessing subjective sense of change in emotions and objective rating of blunting of affect), subjectively impaired energy (a basic symptom) and impaired tolerance to normal stress (a basic symptom). Scores for each subscale range from 0 to 6.
Testing of the CAARMS
Aims
This study aimed to evaluate the CAARMS in terms of its ability to perform the two functions described earlier and to examine the inter-rater reliability of the CAARMS.
Method
Function 1: Does the CAARMS include measures of psychopathology that indicate risk for psychotic disorder?
Two approaches were taken to test this function of the CAARMS. First, we aimed to show that UHR patients with high scores on the CAARMS had a higher risk of onset of psychotic disorder within a brief time period than patients with low scores. This is a type of predictive validity. Second, we needed to show that symptoms in the UHR range were not common in the general population, that is, showing discriminant validity.
Predictive validity of high CAARMS score in the ultra high risk group
A sample of 49 young people (aged 15–29 years), defined as being at UHR using BPRS/CASH criteria (7]. The UHR criteria are described in detail elsewhere [7]. To summarize, they consist of three groups: (i) presence of attenuated psychotic symptoms; (ii) history of brief self-limited psychotic symptoms; and (iii) positive family history of psychotic disorder plus persistent low functioning. Individuals in group 1 have subthreshold psychotic symptoms in terms of the intensity or frequency of their experiences. Group 2 patients have a history of frank psychotic symptoms that spontaneously resolve within 7 days (i.e. subthreshold in terms of duration). The final group includes young people with a presumed genetic vulnerability to psychotic disorder who have a recent and marked deterioration in functioning. All participants gave informed consent.
BPRS/CASH-defined ultra high risk criteria
The sample was followed up monthly for 12 months using the CAARMS and BPRS as well as other instruments to assess mental state and functioning [7]. This enabled examination of the predictive power of the CAARMS subscale scores and overall score for onset within 12 months to be assessed by comparing the group which became psychotic during follow-up with the group which did not. The CAARMS scores for each subscale and overall CAARMS score (mean of the subscale scores) were analysed as ‘potential predictors of psychosis’ using survival analysis and Cox regression. Analysiswas similar to that done in the earlier paper on the same sample [7].
Discriminant validity
The discriminant validity of the CAARMS was assessed by determining its ability to distinguish UHR patients from non-patient adolescents and young adults. A control group was recruited from a near by centre for the unemployed. Assessment of these young people included demographic data, Structured Clinical Interview for DSM-IV [14] to ensure that none had a threshold mental disorder and the CAARMS. We hypothesized that UHR patients would score significantly more highly on the CAARMS than the control group.
Function 2: To determine if an individual has reached the UHR status: can the UHR criteria be validly defined using the CAARMS?
Two approaches were taken to test this function of the CAARMS. First, a comparison was made between the BPRS/CASH method of determining UHR status and the CAARMS-based method. That is, a test of the concurrent validity of the CAARMS. Second, the CAARMS defined UHR criteria were applied to a new sample of not apparently psychotic help-seeking young people which was followed up 6 months later to determine rate of onset of psychotic disorder.
Concurrent validity
The CAARMS-derived cut-off points for UHR status were compared to the BPRS/CASH-defined criteria. The CAARMS-based criteria were applied by two independent researchers, blind to patients' eventual outcome, to the intake data of each of the 49 patients in the above cohort in order to assess the concordance of the two methods of determining UHR status. Unfortunately, because of the design of the main study, people who did not meet BPRS/CASH-based UHR criteria were not followed up and, therefore, the CAARMS-based criteria could not be applied to them. Hence, there is the possibility that we may have missed some patients who would have met the CAARMS-intake criteria. However, we expect such a number to be small and we have borne this point in mind when interpreting the results. This study provides a measure of concurrent validity.
Predictive validity of CAARMS-defined UHR criteria
A new sample was recruited to test the predictive validity of the UHR criteria as measured by the CAARMS. A revised version of the CAARMS was used in this study, but the UHR criteria were essentially unchanged. The sample was recruited from ORYGEN Youth Health, a public specialist mental health service for young people aged 15–24 years living in western metropolitan Melbourne, Australia. It services a catchment of approximately 1 million people. The service has a component for psychotic and UHR individuals (the PACE Clinic) and a separate service for non-psychotic young people. Consecutive referrals to the service for non-psychotic young people were invited to participate in a longitudinal follow-up study.
The CAARMS criteria for UHR status were applied to the sample and the rate of transition to psychosis in the those meeting these criteria (CAARMS-positive status) was compared to the rate in those not meeting the criteria (CAARMS-negative status). This is a measure of the predictive validity of the CAARMS-defined UHR criteria.
Inter-rater reliability of the CAARMS
Inter-rater reliability of the CAARMS was assessed by joint interviews of 34 UHR patients attending the PACE Clinic at baseline. There were four pairs of raters with the role of interviewer rotating between them. Raters were either psychiatrists or trained research psychologists.
Results
Function 1: (i) Predictive validity in a UHR group: do high CAARMS scores predict onset of psychotic disorder?
In the BPRS/CASH-defined UHR group, 20 of 49 patients (40.8%) developed psychotic disorder within the 12-month follow-up period [7]. High CAARMS overall score was significantly associated with development of psychotic disorder. The CAARMS measures of disorders of concentration and attention (CA), emotion and affect (EA), impaired energy (E) and impaired tolerance to normal stress (S) were highly predictive of psychotic disorder. Disorders of thought content (TC), perceptual abnormalities (PA), conceptual disorganization (CD) and motor changes (M) were not predictive. Dividing the subscales into positive (TC, PA, CD and M) and negative (CA, EA, E and S) symptoms showed the negative symptoms to be highly and consistently predictive of psychotic disorder, but that positive symptoms were not predictive, see Table 3.
Function 1: (ii) Discriminant validity: do UHR patients have higher CAARMS scores than controls?
Cox regression results (p-values) examining associations between CAARMS measures and onset of psychotic disorder
A control group of 48 young people (mean age 21.1 years, 37.5% males) was recruited. No member of the control group had a history of psychotic disorder or a family history of psychotic disorder. The means of the baseline CAARMS individual subscale scores of the 49 BPRS/CASH-defined UHR patients, and 48 controls were compared (Table 4). As hypothesized, UHR patient scoreswere significantly higher than control scores on all subscales of the CAARMS(p<0.001).
Function 2: (i) Concurrent validity: do the CAARMS-defined UHR criteria identify the same patients as the BPRS/CASHdefined UHR criteria?
Patient (baseline) and control CAARMS scores
Of the 49 patients meeting the BPRS/CASH-based UHR criteria (mean age 19.1 years, 51.0% males), 45 also met the CAARMS based criteria (91.8%). Those meeting both BPRS/CASH criteria and CAARMS criteria had a mean age of 19.2 years and 55.6% were male. Of the four who met BPRS criteria but not CAARMS criteria, two were adjudged to have had a psychotic disorder in the past (before intake into the study), using the CAARMS-based definition; one was found to be psychotic at intake and one had psychiatric symptoms not severe enough to meet criteria. In the original study [7], the one who was deemed psychotic at intake using the CAARMS criteria reached the threshold for BPRS-defined psychotic disorder shortly after study baseline. The other three remained non-psychotic by 1 year after entry.
Of the 45 patients meeting CAARMS-intake criteria, 19 became psychotic within 12 months from entry and two are known to have become psychotic after 1 year. The Kaplan–Meier estimate of the 12-month transition rate is 0.42 with 95% CI (0.26, 0.55). For the original BPRS intake criteria, the Kaplan–Meier estimate of the 1-year transition rate was 0.41 with 95% CI (0.25, 0.53) [7]. Thus, these two sets of criteria appear to be similar.
Function 2: (ii) Predictive validity: do individuals meeting CAARMS-defined UHR status have a higher rate of transition to psychotic disorder than those not meeting these criteria?
In the sample of 150 non-psychotic help-seekers, 43 met CAARMS defined UHR criteria at baseline and 107 did not. At 6-month follow-up, the whole sample had a rate of transition to psychotic disorder of 4% (six of 150). Of the 43 who met CAARMS-based UHR criteria at baseline, five developed psychotic disorder and 38 did not (Table 5). Only one person who did not meet UHR criteria at baseline developed psychotic disorder at 6-month follow-up. Thus, in this sample, those who were CAARMS-positive were at significantly increased risk of onset of psychotic disorder during the follow-up period compared those who were CAARMS-negative, with a relative risk of 12.44 (95% CI=1.5–103.41, p=0.0025). The sensitivity, specificity, PPV (positive predictive value) and NPV (Negative predictive value) of the CAARMS criteria for predicting psychosis onset were 0.83, 0.74, 0.12 and 0.99, respectively.
Number meeting ultra high risk (UHR) criteria and number developing psychotic disorder within 6 months
Inter-rater reliability
Seven raters were used to form pairs to provide data for examining inter-rater reliability using 34 UHR patients. Table 6 shows the intraclass correlation coefficients (ICC) of each of the eight main subscales. As can be seen, good to excellent agreement was found with all scales, with only the impaired energy subscale displaying an ICC lower than 0.7. The overall agreement (total CAARMS score) was 0.85.
Intraclass correlation coefficients (ICC) of the eight CAARMS subscales
Discussion
The CAARMS was evaluated in a number of ways and was found to perform well. The CAARMS had good to excellent inter-rater reliability. The CAARMS-defined UHR criteria appeared to have good agreement with the original BPRS/CASH-defined criteria. Meeting these criteria conferred a significant risk for psychotic disorder on an individual in the sample of non-psychotic helpseekers, that is, good predictive validity of the CAARMS defined criteria was supported. In the original UHR sample, high CAARMS scores predicted transition to psychotic disorder, that is, good predictive validity of the symptom measures was found.
High CAARMS scores were not found in a control group, therefore, the CAARMS showed good discriminant validity. These controls were relatively marginalized (unemployed young people), yet did not score highly on this instrument of risk for psychotic disorder. An area of potential criticism of this study is that the control group was not followed up to determine rate of onset of psychotic disorder. However, we believe that the expected incidence of psychotic disorder in this sample would be so low as to make follow-up of this group not worthwhile. The incidence of schizophrenia in young adults in the general population is in the order of 3.0 per 10 000 per year. In a population with high unemployment levels (as well as a range of other risk factors for schizophrenia), the Afro-Caribbean population of South London and Ealing, the incidence rate was 5.9 per 10 000 per year [15]. Applying this to the control group (n=48), 0.028 individuals would be expected to become psychotic within 12 months. Even multiplying this risk 10-fold, as all the controls were unemployed, an annual incidence of 0.2 cases would be expected. Thus, it is unrealistic to expect that any members of the control group would have become psychotic in a 12-month follow-up period.
One interesting finding was that high levels of ‘negative’ type symptoms (cognitive and emotional disturbances, low energy and impaired stress tolerance) assessed by the CAARMS were more predictive of psychosis than subthreshold ‘positive’ symptoms in the UHR group. The reasons for this are unclear; however, it may be that the UHR criteria are already selecting people with high levels of attenuated psychotic symptoms and what then distinguishes those who go on to psychotic disorder from those who do not are persistent negative and basic symptoms. That is, negative and cognitive symptoms in the presence of subthreshold positive symptoms are predictive of psychotic disorder in the UHR group. It may be that difficulty coping with stress may precipitate the transition from non-distressing positive symptoms to actual psychotic disorder, as postulated by Van Os and colleagues [16], [17].
In both the UHR PACE sample and the sample of non-psychotic help-seekers, the majority of those meeting UHR criteria did not develop a psychotic disorder. As expected, the predictive power of the UHR criteria was dependent on the base rate of psychotic disorder in the sample. In a sample with high incidence of psychotic disorder (the UHR group, 41% over 12 months) individuals with CAARMS-positive status had a high likelihood of onset within 12 months (42%). However, in the other clinical sample (the non-psychotic helpseekers), which had a much lower incidence of psychotic disorder (4% over 6 months), only 11.6% of UHR-positive individuals had onset of psychotic disorder within the follow-up period. (However, this was only a 6-month follow-up period rather than 12 months.) In this latter sample, individuals who were CAARMS-positive were more likely to develop a psychotic disorder than those who were CAARMS-negative. However, the fact that most people who were CAARMS-positive did not develop a psychotic disorder within the follow-up period has important implications for the use of the CAARMS and other similar instruments. The degree of risk an individual has for psychotic disorder depends not only just on their CAARMS status but also on the population to which they belong. It may be justifiable tomonitor a CAARMS positive person who is seeking help and check for worsening symptoms. The extent to which he/she should be further treated depends on how they presenting problems. As we have previously noted [18], caution is needed in using the CAARMS or similar instruments as screening instruments in the general population as the incidence of psychotic disorder in unselected populations is so low that the false-positive rate would be extremely high and make screening inappropriate.
The CAARMS is the first instrument developed to specifically assess the UHR group, although some preliminary data have already been reported on a similar instrument, the Structured Instrument for Prodromal Syndromes (SIPS) [19], which originally drew on the CAARMS as a resource. The CAARMS has now been translated into French and Swedish, and a Japanese version is in development. Ongoing testing and development of the CAARMS is continuing.