Abstract
Mental health services are increasingly required to measure the effectiveness of outcomes in clinical delivery [1]. The Second National Mental Health Plan [2] has set forth priorities for the development of mental health services in Australia. One of these priorities is the integration of consumer outcome measures into the daily practices of clinical services. Through the late 1980s [3] and 1990s [4], the Commonwealth Government sought a nationwide approach to measuring the effectiveness and outcomes of mental health services. As part of this initiative, early-psychosis programmes around Australia are required to demonstrate their effectiveness in providing sound clinical outcomes.
Early intervention with psychosis patients has been found to be effective [5, 6], however, ongoing research and evaluation of services for this population are central to the success of new initiatives [7]. McGorry et al. [7] note that evaluation allows clinicians to judge the value of an intervention and that outcome data can influence policy makers. Good quality evaluation can help protect innovative services in an environment of scarce resources. The current research works to this end.
The Early Psychosis Outcome Evaluation System (EPOES) was developed by a working party of clinicians working in early-episode psychosis services, and a research psychologist (NP). An essential requirement of the system was to provide outcome evaluation that is relevant to and feeds directly back to the clinician and patient, the service and the system. The EPOES was developed upon the matrix model [8] where outcome data is fed back at all levels including the individual case, clinic and systems levels. It was designed to measure serially over time the effectiveness of early psychosis programmes, where clinicians can produce instant reports on the clinical status of patients in their management. EPOES is a Microsoft Access Database platform, which produces individual and service-wide data. It captures demographic and clinical outcome data, produces individual graphical case reports of patient progress on clinical outcome, and exports data for analysis. Codified forms are coupled with the electronic system making the evaluation of the system automatically established and part of the clinical work practices of staff. EPOES aims are to serve each level of outcome measurement, by allowing the clinicians to be directly involved in outcome research. Often outcome measurement can be slow, esoteric and irrelevant to clinical practices. EPOES tries to overcome these deficiencies by inverting the process, where individual clinical data is considered first, then pooled data at the clinic level, then finally systems level data for research. It is argued that such a system can effectively contribute to the literature on early psychosis programme development by first meeting the needs of clinicians, then turning to the broader needs of the research community in the area of early psychosis.
Method and sample
The study is a naturalistic prospective study of a cohort of earlyepisode psychosis patients. It employs a repeated measures design that tracks outcomes over time. All patients referred to any one of the four participating specialist early-episode psychosis programmes in Perth, Western Australia, were invited to participate in the study. Ethics approval was granted from each participating programme with written informed consent obtained from subjects. The four current participating programmes are: Early Intervention Psychosis Program (EIP), Alma Street Centre, Fremantle; Early Episode Psychosis–A System of Care (EPP), Rockingham Kwinana Mental Health Service; First Psychosis Liaison Unit (FPLU), Mill Street Centre, Bentley; and First Episode Psychosis Program (FEPP), Joondalup Community Mental Health Services, Joondalup.
Entry criteria included: residing within the catchment areas of the programmes and the person presenting with a first psychotic illness (or within 12 months of onset). Variation in age ranges of the programmes exist: Fremantle, aged between 17 and 40; Rockingham-Kwinana, aged 18–40; Bentley, aged 16–35; and Joondalup, aged 16–30 years.
The average participation rate across the four services was 35.6%. Since the ethics approval did not allow the research to perform case note reviews of non-participants, the study is not able to determine diagnostic and demographic differences between the participating and non-participating populations.
Procedure
Data collection is coordinated and entered into the EPOES database by the case manager of the patient. All case managers received 1 day intensive training on how to conduct a Brief Psychiatric Rating Scale (BPRS) interview and use, interpret and administer the other instruments. To be eligible to use EPOES, each case manager had to receive adequate interrater agreement across two case scenarios during the training. Measures were collated by the case manager from three sources: case manager, patient and family member of the patient. The initial assessment is conducted within 4–6 weeks of acceptance into an early psychosis programme. Subjects were approached to participate in the study after sufficient rapport was gained by the case manager. The assessment procedure was repeated every 6 months and also at discharge from the service.
Measures
The measures used in EPOES are gathered from three perspectives. The first perspective is from case manager assessment of patients service use, psychiatric functioning and substance use; the second perspective is from the patients self-reported experience of psychopathology and social functioning; and the third perspective asks family members to report their perceived level of family burden and general health. Each of the perspectives are linked in EPOES through a unique identifier. This allows for cross-referencing of data between each of the outcomes measured by the different informants.
Case manager assessment
Demographics: gender, age, onset of psychosis are collected during the case manager assessments. Age of onset is determined when positive symptoms become evident (not including prodromal signs and symptoms). This is estimated using information from the patient, family or significant others and any other collateral history available. ICD-10 Diagnosis was made by the consultant psychiatrist on the treating community team via clinical interview, observation and collateral information. Current medication. Brief Psychiatric Rating Scale [9]. Global Assessment Scale (GAS) [10]. Recent substance use history [Kavanagh D: personal communication] asks patients about their use of substances (e.g. coffee, cigarettes, amphetamines, cannabis); frequency, type and quantity over the previous 3 months.
Patient self assessment
Demographics: gender, age, service provider. The Social Functioning Scale (SFS) [11] assesses social functioning relevant to the needs and impairment of individuals with schizophrenia and related disorders. It is a 62-item scale consisting of six subscales: Withdrawal, Interpersonal Communication, Recreation, Pro-social Activities, Independence Competence, and Independence Performance. The Brief Symptom Inventory (BSI) [12], a 53-item scale measuring psychological functioning; scores from the Positive Symptom Total (PST), Global Severity Index (GSI) and four of the nine symptom dimensions: Depression, Anxiety, Paranoia and Psychoticisim are used in the study.
Family/carer assessment
Demographics: gender, age, relation to patient, living status. The Burden Assessment Scale (BAS) [13], a 19-item scale measuring the level of burden experienced from caring for someone with a mental illness. It produces five subscales: Disruptive Activities, Personal Distress, Time Perspective, Guilt and Social Functioning. The General Health Questionnaire-12 (GHQ-12) [14] assesses the overall general wellbeing of care-givers; it produces a total score and a positive response score, where four or more positively scored items reflect caseness of psychological distress [15].
Analysis
Demographic descriptors of the sample such as age, gender, diagnosis, medication and substance use were collated. χ2 tests of significance were used to assess differences in categorical measures such as gender and diagnosis. Analysis of variance (ANOVA) and paired-wise t-tests were used to analyse continuous data. Bonferoni adjustments for experiment-wise error were made for subscale analysis for each outcome measure. Pearson's product moment correlation coefficients were used to examine the relationships between continuous variables. To examine diagnostic differences, patient diagnoses were grouped into three categories: (1) Schizophrenia and schzophreniform diagnosis; (2) Substance Induced Psychosis (SIP only); and (3) Other (all other diagnoses). Schizophreniform subjects were placed with schizophrenia subjects from recent evidence suggesting their longitudinal diagnosis to be schizophrenia [16].
Duration of untreated psychosis represents positive psychotic features displayed before seeking formalized treatment. This period is estimated in months by the patients and significant others.
A poly drug-use score was calculated by the addition of a positive response to use of cannabis, amphetamines, sedatives, hallucinogenics, or opiates. Use of tea/coffee, alcohol or cigarettes was excluded from the analysis to focus on illicit substances. To examine the relationship between poly drug use and psychopathology, the poly drug use score was correlated with the total and subscale scores of the BPRS and BSI.
Results
Patient characteristics
The baseline sample consisted of 66 males with a mean age of 25.6 years (SD = 5.47) and 18 females with a mean age of 26.22 years (SD = 6.82). There was no statistically significant age difference between males and females (F = 0.157, df = 1,82, p = 0.69).
Diagnosis at baseline
Table 1 presents the distribution of diagnoses of patients at baseline. The largest category (one-third), are substance induced psychotic (SIP) disorder followed by schizophreniform disorder.
Diagnosis of patient at baseline
Using diagnostic groups of schizophrenia (n = 26), substance induced psychosis (n = 26) and all other diagnoses (n = 31), χ2 analysis revealed gender did not contribute to differences in diagnosis (χ2 = 2.30, df = 2, p = 0.316). The frequencies in each diagnostic group were proportionally equivalent in males and females. Age did not differ between diagnostic categories of schizophrenia (mean = 26.1, SD = 7.0, n = 26), substance induced psychosis (mean = 24.4, SD = 4.7, n = 26) and all other diagnoses (mean = 26.7, SD = 5.6, n = 31) (F = 1.24, df = 2, 80, p = 0.29).
Duration of untreated psychosis (DUP)
The average DUP was 7.1 months. There was no statistically significant differences in DUP between males (DUP = 7.30 months, SD = 11.63, n = 62) and females (DUP = 6.52 months, SD = 11.50, n = 17) (F = 0.060, df = 1, 77, p = 0.80); (one male case was excluded as an outlier). Diagnoses when grouped into schizophrenia (DUP mean = 11.6 months, SD = 17.4, n = 25), substance induced psychosis (DUP mean = 4.3 months, SD = 5.1, n = 25) and all other diagnoses (DUP mean = 5.7 months, SD = 7.8, n = 29) were found not to differ on DUP (F = 2.96, df = 2,76, p = 0.058). Observation of the means however, indicates that schizophrenia subjects had twice the length of DUP.
Medication
All patients were prescribed medication at baseline. The most common medications prescribed were Olanzapine (46, 31.5%), Risperidone (34, 23.3%), and Sodium Valproate (11, 7.5%). Olanzapine and Risperidone accounted for 54.8% of medications. All typical antipsychotic medication accounted for 4.1% of medication prescribed. Average daily Chlorpromazine equivalent dose at intake was 362.17 mg with a range from 100 to 1200 mg. Of the two most frequently prescribed atypical antipsychotics, Olanzapine recorded higher levels (mean = 444.4 mg, SD = 193.80) of Chlorpromazine equivalent dose than Respiridone (mean = 278.78 mg, SD = 208.80) (t = 3.60, df = 1,76, p = 0.001).
Differences in psychopathology between gender and diagnosis
There were no statistically significant gender differences in psychopathology as measured by the BPRS total score and subscales: Thinking Disorder, Withdrawal, Anxiety Depression and Activation. However, males recorded significantly higher levels of Hostility and Suspicion than females (Male mean = 2.4, SD = 1.1, n = 66; Female mean = 1.6, SD = 0.8, n = 18) (F = 9.76, df = 1,82, p = 0.002). There were no statistically significant gender differences on BSI subscale scores.
Patients with a diagnosis of SIP recorded significantly higher levels of BPRS Activation (mean = 1.83, SD = 0.9, n = 26) than those with schizophrenia (mean = 1.09, SD = 0.2, n = 11) or other diagnoses (mean = 1.58, SD = 0.7, n = 46) (F = 4.17, df = 2,80, p = 0.02). Post hoc analysis revealed a significant difference between SIP and schizophrenia (Tukey HSD mean difference = 0.74, p = 0.02) but not with other diagnoses. Substance induced psychosis also recorded higher levels of Thought Disorder (mean = 2.76, SD = 1.1, n = 26) compared with schizophrenia (mean = 1.98, SD = 1.0, n = 11) or other diagnoses (mean = 2.20, SD = 1.1, n = 46) though it narrowly missed statistical significance (F = 3.01, df = 2, 80, p = 0.055). The subscales of the BSI revealed no statistically significant differences between the diagnostic groups.
Social functioning
Examination of gender differences in patient social functioning revealed that males recorded more withdrawal than females (increase denotes improvement) (Male mean = 9.54, SD = 2.4, n = 59; Female mean = 11.2, SD = 2.9, n = 17) (F = 5.54, df = 1,74, p = 0.02), and lower levels of independently performed activities (Male mean = 26.5, SD = 8.5, n = 59, Female mean = 31.9, SD = 4.8, n = 17) (F = 6.43, df = 1, 74, p = 0.013). Gender did not significantly differ on any other social functioning scale subscales or the total score.
Diagnosis did not contribute to differences in social functioning, except with schizophrenia patients showing poorer scores on social withdrawal than the other psychosis group (F = 3.17, df = 2,75, p = 0.047).
Substance use
Sixty per cent of patients (n = 47) reported using cannabis within 3 months of their baseline assessment. Thirty per cent (n = 23) reported using amphetamines/ecstasy/cocaine within 3 months of their baseline assessment. Almost 26% of patients (n = 20) reported sedative use 3 months prior to baseline assessment.
Table 2 indicates cannabis use was associated with higher levels of psychopathology on BPRS subscales, notably thought disorder (Thinking Disorder), paranoid ideation (Hostility and Suspicion), anxiety/ depression and agitation (Activation).
One way analysis of variance on Brief Psychiatric Rating Scale (BPRS) total score and subscales with marijuana use and non-use at baseline
There was no significant difference between gender and cannabis use (χ2 = 0.001, df = 1, p = 0.982), but cannabis users tended to be younger (mean = 24.6, SD = 5.2) than non-users (mean = 27.3, SD = 6.2) (F = 3.92, df = 1,76, p = 0.051).
Amphetamine/ecstasy/cocaine use was significantly associated with increased levels of Anxiety/Depression (User mean = 3.2, SD = 0.93, n = 23; Non-user mean = 2.5, SD = 1.1, n = 54) (F = 6.92, df = 1,76, p = 0.01) and Hostility/Suspicion (User mean = 2.8, SD = 1.1, n = 23; Non-user mean = 2.1, SD = 1.1, n = 54) (F = 7.89, df = 1,76, p = 0.006). The BPRS total score also significantly differed between amphetamine users and non-users (User mean = 45.2, SD = 9.2, n = 23; Non-user mean = 37.7, SD = 12.5, n = 54) (F = 6.642, df = 1,76, p = 0.012). Other subscales did not differ significantly between users and-non-users. There was no significant age difference between amphetamine users (mean = 24.6, SD = 4.6, n = 23) and nonusers (mean = 25.9, SD = 6.2, n = 54) (F = 0.49, df = 1,75, p = 0.49).
The poly drug-use score showed mild positive correlations with the total and subscale scores of the BPRS and BSI (Table 3).
Correlations of poly drug use with total and subscale scores on the Brief Psychiatric Rating Scale (BPRS) and Brief Symptom Inventory (BSI) at baseline (n = 77)
Family member characteristics
Data from 66 family members had been collected at baseline. The majority of family members were mothers (n = 33, 50%). Fathers were the next largest group (n = 17, 25.8%) followed by spouse (n = 9, 13.6%) and sister (n = 3, 4.5%). At 6 months, data were available from 21 family members; 16 (76%) were mothers. The majority of respondents lived with their relative at the baseline assessment (n = 55, 83.3%). At 6 months, 18 of the 21 lived with their relative (86%).
General Health Questionnaire-12
Psychological distress at baseline was reported by 59% (n = 39) of family members. There were no statistically significant differences on GHQ-12 positive scores between mothers (positive score = 4.9, SD = 3.4) and fathers (positive score = 6.6, SD = 4.3) at baseline (GHQ-12 positive score F = 2.48, df = 1,48, p = 0.12). The sample was also grouped into parents and other family members. The GHQ-12 positive score differentiated parents (mean = 5.4, SD = 3.8, n = 50) from other family members (mean = 3.3, SD = 3.4, n = 16) with parents reporting significantly more psychological distress than other family members (F = 4.05, df = 1,64, p = 0.048).
Burden Assessment Scale
The baseline family member sample was grouped according to mothers and fathers, then parents and other carers. Comparison of the BAS total score and 5 subscales (Disruption, Personal Distress, Time Perspective, Guilt and Social Functioning) revealed no statistically significant differences in reported family burden between these groups. Equivalent levels of family burden are reported across all carers.
Outcome at 6 months of treatment
Twenty-three patients had been successfully followed-up at 6 months. The following analyses are based on these cases using paired-wise t-tests to assess within subject effects over time. The BPRS total score and subscale scores statistically significantly reduced (excluding withdrawal after Bonferoni adjustment) at 6 months compared to baseline (Table 4). Patients also reported improvement in psychiatric functioning with all subscales of the BSI showing a reduction from baseline to 6 months (excluding depression after Bonferoni adjustment) (Table 4).
Paired t-tests on the BPRS, GAS, BSI, SFS, BAS and GHQ-12 total scores and subscales at baseline and six months
The GAS significantly increased from baseline to 6 months (Table 4), indicating an overall functional improvement in patient outcome. Analysis of subscale scores of social functioning from baseline to 6 months revealed no changes after Bonferoni adjustment. The results indicate no improvement in social functioning within 6 months.
Overall family burden did not significantly reduce from baseline to 6 months (Table 4), with the exception of the subscale Time Perspective that measures concern about the future wellbeing of their relative. Family members reported significantly less concern for their relative's future.
The positive response score of the GHQ-12 significantly reduced from baseline to 6 months (Table 4), indicating family members returned within normal population ranges of psychological wellbeing 6 months after treatment for their family member. At 6 months only 15% of family members reported psychological distress compared to 55% at intake.
Discussion
Early Psychosis Outcome Evaluation System is an effective system for the collation of descriptive clinical outcome data. The system is innovative in that clinicians participating in EPOES now conduct research and evaluation as part of their everyday work duties. Acceptability of EPOES centres around the functional capacity of the system to provide immediate feedback to clinicians on the status of patient's clinical outcome. The success of EPOES lies in the fact that the system was generated by clinicians specializing in early psychosis who were interested in research and outcome evaluation rather than a research group interested in clinician outcome in service delivery. Ownership of EPOES occurs due to the immediate and direct involvement of clinicians to generate the design, implementation and analysis of data that is important in the day-to-day delivery of their clinical services.
The system works on the matrix model [8], where information is fed back at the individual clinical, service and statewide systems levels. In the case of EPOES the system is inverted where the priority lies in providing clinical outcome information to clinicians first, then to the service then finally to the overall system of care. Unlike other clinical outcome evaluation systems implemented in psychiatric services, EPOES was initiated, designed and implemented by clinicians. High participation rates among the service sites over the past 4 years is mainly attributed to the system focusing primarily on providing clinical information relevant to the treating clinician. Outcome reports were first constructed at the individual case level. It was anticipated that if the data collected were relevant at this level then systems analysis (i.e. evaluating a service model) would be easier and more realistic. With the focus of the 1998 Second National Mental Health Plan [2] on outcome assessment, EPOES has proven a precursor to how successful outcome evaluation systems can survive beyond their initial implementation. This was done by first considering informatics central to clinical service delivery as opposed to meeting the needs of researchers, or service managers and administrators. This strategy has proven successful with clinicians actively using EPOES as part of their clinical service delivery and not some abstract data collection system used for someone else's benefit or research interests.
Preliminary results identify a significant reduction in psychopathology, both observed and self-reported at 6 months from baseline. Global Assessment Scale scores also significantly improved over 6 months of treatment. Gains were not observed in the depression subscale of the BSI and the withdrawal subscale in the BPRS. This is indicative of the difficulty in treating the negative spectrum of psychotic disorders. Family member general health also significantly improved, although family burden remained unchanged in the strictest statistical sense (see Table 4). In terms of patient social functioning, prolonged role functioning difficulties are identified in that there was no social functioning improvement at 6 months. These results suggest social functioning deficits associated with psychotic disorders [17]. It is important however, to qualify these results. These results are on a small sample (n = 23) and hence are only preliminary. For example, family burden and family member general health were slightly under the statistical threshold, indicating the results to be at the lower bounds of power. In addition to these concerns, the research has been unable to determine interrater reliability within clinicians and between service sites over the course of the study although this was determined during training. It is the intention with further research that these issues of internal consistency of assessment between service sites are addressed.
The frequent use of substances by early psychosis patients confirms previous research [18–22]. Substance misuse has been associated with a more severe course of schizophrenia [18], earlier and more psychotic relapses in young cannabis-using patients diagnosed with a first schizophrenic episode [20] and increased rates of hospitalization [21]. The need for services to address effectively both substance use and psychosis is evident.
Although EPOES is not a classical randomised control trial it can allow for such a design to occur by employing EPOES into regional areas where the minimum standards of operating an early psychosis intervention service are not present. We aim to address this limitation in a controlled experimental design by inviting non-specialist service providers of early psychosis patients to participate in using EPOES. Since EPOES uses a naturalist design in ‘real life’ clinical settings, it is difficult to get control sites to participate where one may expect that their service is inferior or inadequate. By providing control sites with a system that helps them immediately inform their clinical practice (not after the experiment is completed 2 years after implementation) then the evaluation system is likely to find willing candidates. By employing research designs that compare real life clinical practices as opposed to highly funded research projects, then generalizability of the results tends to be higher and more accurate. In addition to this, it is the intention of the researchers to examine treatment fidelity and its contribution to service utilization such as bed days and admission rates.
Footnotes
Acknowledgements
Thanks to Craig Headford for developing the EPOES programme. The Early Psychosis Outcome Evaluation System was funded by Quality Improvement Project: Mental Health Division Department of Health Western Australia.
Written on behalf of the Early Psychosis Group Western Australia.