Abstract
Three decades ago the debate about where best to treat the seriously mentally ill focused on whether treatment in the community offered a viable alternative to hospital care [1, 2]. With community treatment now established as an essential component of mental health care, the debate has shifted to a number of different concerns. These include questions about how best to provide community treatment in rural and regional settings [3]; whether the improved outcomes associated with assertive community treatment (ACT) are comparable to those associated with standard community care [4, 5]; and what is the right balance between community and hospital care since it is now recognized that community services form part of an integrated system of service delivery which includes inpatient care [6, 7].
For these reasons there has been a shift away from studies of the ‘efficacy’ of experimental services offering ACT to those investigating the ‘effectiveness’ of existing clinical services [8, 9] in which some form of community treatment is offered as an addition to existing forms of care.
An opportunity to undertake a study of this kind occurred in 1993 when, as part of Australia's National Mental Health Policy [10], the mental health service in the Northern Region of Tasmania developed plans to add an extended-hours community mental health team (CMHT) to its hospital-based service situated in the regional city of Launceston. The research team undertook to conduct a naturalistic study of the effectiveness of this development by evaluating the service in the before and after stages of this addition.
Service description
The Northern Region of Tasmania is generally described as rural although most people live near a population centre. In 1993 it had a population of 127 000, half of whom lived in the city of Launceston where the service was based. At that time mental health services in the region were recognized as suffering from ‘an overwhelming lack of resources’ [11]. They consisted of an acute psychiatric ward at the Launceston General Hospital with 24 beds and use of two seclusion rooms. In addition to the ward nursing staff there were three-and a- half psychiatrists, two registrars, four social workers and two psychologists, working in two teams across inpatient and outpatient settings, as well as one outpatient clinic nurse, three clerical staff and two community psychiatric nurses.
A small day centre nearby had two nurses, a social worker and one-and-a-half trained auxiliary nurses. Its activities included a small work programme run by staff. A recently opened long stay unit with 15 beds was located 20 km away. The secure ward of the state psychiatric hospital, located 200 km away was used for a small number of patients. There was a child and adolescent mental health clinic with one medical officer and three allied health professional staff. In 1993/1994 the services had 531 admissions and 9219 outpatient and community contacts, excluding the Day Centre. Apart from 4.6 EFT private psychiatrists, with the exception of the Richmond Fellowship, non-government services specifically devoted to mental health were absent.
The CMHT began operating in November 1994 and was fully operational by June 1995. The staffing included a nurse team leader who carried a clinical caseload, 6 shift workers (4 nurses, 2 allied health professionals), a clerical officer and 0.3 visiting psychiatrist. This represented an overall increase to services in the region. In addition, the acute ward was remodelled to create a separate high dependency unit and a reduction in acute beds from 24 to 20. Transfers to the state psychiatric hospital declined to minimal levels.
Community mental health team model
The Team was designated to provide services within Launceston and its suburbs. Its functions included a substantial component of emergency assessments/crisis management and ongoing case management. Its aims reflected those of the National Mental Health Strategy and included the reduction of inpatient utilization, rehabilitation, reduced symptomatology, improved social functioning and community integration.
While service personnel regarded ACT as the most desirable model of community treatment, budgetary restrictions meant that the service fell short of this. Operating hours were 0800 hours to 2300 hours rather than the desired 24 hours; caseloads averaged about 20 rather than 10; both emergency work and formal rehabilitation activities were limited. Responsibility for managing substance abuse was held by another agency. However, the service did have the potential to see patients daily or more frequently; it actively followed up patients who dropped out of treatment; it had direct involvement in admissions and discharge planning; caseloads were shared; there were highly established communication mechanisms within the team and case management took place in the community rather than the hospital campus. These differences meant that the team should be seen as offering standard community treatment rather than ACT [12, 13].
In making this decision clinicians and managers rejected the option of reducing hospital beds in order to develop community services, partly because the population dispersal meant that even if ACT was available it would not reach all the service's regions. The service's relatively conservative staff culture also led to concerns that cutting hospital beds further would reduce inpatient care to below an acceptable level.
Methods
The study population comprised individuals who were:
– resident in the North region of Tasmania; – aged between 15 and 65 years old; – admitted to the acute psychiatric inpatient unit of the Launceston General Hospital between April and September 1993 (pre-CMHT group) or were either admitted or given intensive treatment by the CMHT between October 1996 and September 1998 (post-CMHT group), and – had a provisional or confirmed diagnosis of either schizophrenia or schizophrenia-like conditions (delusional disorder, schizophreniform disorder, schizoaffective disorder) or a major affective disorder (major depressive episode, biopolar disorder) on DSM-III-R criteria. Acquired brain injury was the only exclusion. Comorbidities were not taken into account in the selection. Involuntary patients were included in the study.
Two additional criteria applied to the post-CMHT group:
– they were not prescribed Clozapine at the time of recruitment;
– they matched the pre-CMHT group on age, gender, diagnosis and severity of illness as measured by the BPRS.
The research was conducted in two Stages. Stage 1 of the study collected data on 60 respondents (pre-CMHT group) prior to the establishment of the CMHT. Stage 2 of the study collected data on 39 respondents after the CMHT had been introduced and was fully operational. At each stage there were four phases of data collection: within 36 hours of admission or intensive treatment by the CMHT, 1 month, 6 months and 12 months later.
The analysis was conducted on 37 matched patients in each group. All assessments were completed by members of the research team who were independent of the service and had no prior knowledge of the patients.
A file analysis of service use was conducted on all consenting respondents from the hospital records of the North region and the State psychiatric hospital.
The following assessment schedules were used:
– the Brief Psychiatric Rating Scale (BPRS) [14]; – the Global Assessment Scale (GAS) [15]; – the Life Skills Profile (LSP) [16]; – Stein and Test's 18-item assessment of activity and social relationships; – the Rosenberg Self-Esteem scale [17]; – Stein and Test's Satisfaction with Life Scale [1]; – a semistructured questionnaire based on Hoult & Reynolds [2], which included questions on patient demographics, social functioning, social problems, medication, compliance with medication, side-effects, community adjustment and treatment satisfaction as well as collection of qualitative data.
Interview training included formal trials of interrater reliability for the BPRS and the GAS against scores by research team psychiatrist (RS). The reliabilities achieved were satisfactory for comparison of group data.
Statistical methods
For variables measured on interval or ratio scales, means and standard deviations were calculated for each study phase. For categorical variables, frequencies and percentages of responses were calculated. A significance level of.05 was adopted for statistical tests. To control for Type 1 error rate, a Bonferroni adjustment was made within families of tests as presented in the tables. Because attrition reduced the number of available pairs over 12 months, the groups were analysed as independent groups rather than matched pairs. Scale variables were initially analysed using a mixed design 2-way ANOVA, with the two study groups being the between-subjects factor, and the four phases being the within-subject factor. To compensate for attrition the ANOVAs were followed up with independent groups t-tests to compare the two groups at each phase. With ordinal or nominal data the comparisons across the four phases within each study group were conducted using the following statistical tests:
– The Friedman test for J matched groups was used for all variables which had values in an ordered series (e.g. very satisfied to very dissatisfied). – The Cohran's Q-test was used for dichotomous variables (Yes/No).
For comparisons between the two study groups, the Mann–Whitney U-test was used when the data were in ordered categories. Separate comparisons were performed for each stage. Contingency table chisquared tests were used for data in non-ordered categories or for dichotomous variables [18].
Percentages were calculated excluding data missing through attrition and non-response.
Results
For the pre-CMHT group there were 74 patients who met the research criteria. Refusals (14), incorrect diagnoses (3) and lack of a match (20) left a final sample size of 37.
For the post-CMHT group there were 129 patients who met all research criteria for Stage 2 apart from matched severity of illness which was unknown. Follow-up difficulties (18), refusals (30), lack of a match on severity of illness or duplicate matches (39), and incorrect diagnoses (5) left a final sample size of 37.
Attrition rates for the pre-CMHT group were 6 (16.2%) at 1 month and 7 (18.9%) at 6 months and 12 months. For the post-CMHT group they were 2 (5.5%) at 1 month, 3 (8.1%) at 6 months and 6 (16.2%) at 12 months.
Analysis of the service utilization data for the first full year of the study (1993/1994) showed that the distribution of gender and diagnosis was representative for the two main diagnostic groups. The matching strategy ensured that at baseline the two groups were comparable on age, gender, diagnosis and severity of illness as measured by the BPRS. They were also comparable on other demographic variables with no significant differences between them (see Table 1).
Initial patient characteristics
Patient outcomes
Table 2 presents the results for the main measures of clinical outcome, activity levels and social relationships. A Multivariate analysis of variance to assess changes from admission to 12 months for the first five variables listed was conducted. This revealed a highly significant improvement over the 12-month period for both groups (Wilks Lambda = 0.464, F5,25 = 5.78, P = 0.001). There was no significant group main effect, nor phase–group interaction. Follow-up univariate analyses with Bonferroni adjustments showed significant improvements over time for each variable except satisfaction with life.
Comparison of clinical outcomes, activity level and social relationships for the pre-CMHT and post-CMHT groups
Separate independent group t-tests comparing the two groups at each phase for each of the five variables did, however, show a significant difference for the BPRS at 6 months, where the mean score for the post-CMHT group (36.74) was significantly worse than the mean for the pre-CMHT group (30.8), t(62) = 3.23, P < 0.001.
Mean activity level and social relationship scores showed a broadly similar pattern of improvement over the study period although the ANOVA for activity level showed a significant interaction between group and phase (F3,133 = 3.81; P < 0.05). The mean activity level for the pre-CMHT group was significantly higher (P < 0.05) than for the post-CMHT group at 6 months although at 12 months this situation was reversed.
Both groups reported similar outcomes for social functioning and social problems (see Table 3) with improvements showing in subjective measures such as ‘accommodation problems’ and ‘feelings of loneliness’ but few in objective ones such as ‘living situation’ and ‘source of income’. With the Bonferroni adjustment, no significant differences were obtained between groups, although some trends were apparent. For ‘satisfaction with things done’ the baseline figure in the pre-CMHT group was much lower than in the post CMHT group (42.8% at least moderately satisfied compared with 70.2%) (U = 391, P = 0.040).
Comparison of Pre-CMHT and Post-CMHT 12-month outcomes for social functioning, social problems and treatment satisfaction
Service utilization
A file analysis of service utilization revealed that the post-CMHT group had a lower number of mean readmissions (1.9 for the post-CMHT group compared with 2.5 for the pre-CMHT group). The range in the pre-CMHT group was also wider (1–11 compared with 1–8 for the post-CMHT group). T-tests showed none of these differences reached significance. The post-CMHT group had a higher range of total contacts (23.2 contacts compared with 19.7 in the pre-CMHT group) which was almost entirely accounted for by CMHT visits. Use of rehabilitation services was similar in both groups. At 12 months no more than seven respondents in either group had attended any nonmental health rehabilitation service in the study year.
At 12 months, medication adherence was high in both groups [89.3% in the pre-CMHT group (n = 25); 84% in the post-CMHT group (n = 21)] and although more pre-CMHT group respondents had stopped or reduced their medication in the last year [40% (n = 12) compared with 28.6% (n = 8) in the post-CMHT group] the difference was not significant.
At 12 months there were no significant between-group differences in patient or relative reported suicide attempts or threats although there was one completed suicide in the pre-CMHT group during the study period.
Treatment satisfaction
Satisfaction with treatment was generally high in both groups (see Table 3). There was a trend favouring higher levels of satisfaction in the post-CMHT group with ‘satisfaction with caring’ being significantly higher than in the pre-CMHT group at 12 months (U = 246.0; P = 0.024). At 12 months the pre-CMHT group reported higher medication side-effects (Chi-squared = 5.85; P = 0.035). At 12 months twice as many post-CMHT respondents would ‘definitely/probably not’ prefer treatment at their own address. At 12 months the pre-CMHT respondents were more likely to feel they needed help outside available hours (U = 190; P = 0.016).
Clozapine
The availability of atypical drugs for the treatment of psychoses from September 1993, three months after data collection for the pre-CMHT group had begun, was a potentially confounding variable. Clozapine has the strongest reputation for additional clinical effectiveness in treatment resistant schizophrenia and its use involves extensive support and education of the patient. While no matched pre-CMHT group respondents were prescribed Clozapine at the time of interview a small number of the post-CMHT group respondents were (3 respondents at 6 months; 4 respondents at 12 months). However, oneway ANOVAs, using the 6 scale variables GAS, BPRS, satisfaction with life, life skills, activity level, social activity score and Rosenberg score, comparing the pre-CMHT group with the post-CMHT group for all patients and for patients excluding those on Clozapine did not affect the outcome of these F tests. This suggests that Clozapine did not act as a confounding variable in this study.
Attrition
While attrition rates could have affected the comparability of the two groups, an analysis using the same 6 scale variables as the clozapine analysis showed no significant difference between the means of the pre-CMHT group for those who remained in the study and those who didn't. However, for the post-CMHT group there was a significant difference for satisfaction with life. The mean score (3.45) for those subjects who remained in the study was significantly higher than the mean (2.61) for those who dropped out (F(1,36 = 10.6; P < 0.01). In addition, those who remained in the study had higher mean activity levels (8.33) and social activity scores (3.97) than those who dropped out (6.14 and 2.14, respectively), with both of these differences approaching significance. Thus, there is the possibility of a differential loss to the study of those individuals who were functioning less adequately for the post-CMHT group.
Conclusions
The results suggest that, overall, the addition of the extended-hours CMHT had limited effects on patient outcomes. While there was a tendency for improvement to occur over the 12 month study period for both groups, there were very few significant between group differences. Clinical, social functioning and social problems outcomes were broadly similar and although levels of treatment satisfaction were generally higher in the post-CMHT group, in only one of these variables did the difference reach statistical significance. There was no difference in suicidal incidents.
The quasi-experimental design of the study allowed the potential introduction of confounding variables which reduce the certainty that any changes in outcome measures can be attributed to changes in service delivery arrangements. One possibility is that the results were affected by changes other than those identified in the study design. However, the service environment was stable over the study period. The exception to this was the refurbishment of the acute inpatient ward and it is possible that this did influence the higher levels of the post-CMHT group's satisfaction with acute care.
The sample size together with the effects of attrition has resulted in a lack of power that may have given rise to Type 2 errors. It also means that the study can make no claim that the results can be generalized to the study population as a whole.
It is also possible that 12 months is not a sufficiently long enough period for improved outcomes to occur and that a longer study period might have been produced different results.
Despite these limitations the results for the two groups demonstrate a high degree of consistency. This suggests that they do reflect the situation of the respondents and that it is reasonable to draw conclusions from them.
A key issue raised by these results is that while model services such as ACT have been associated with improved outcomes the service environment plays a critical role. This service had to choose between providing adequate hospital care and poorly developed community services or offering potentially inadequate hospital care and well-developed community services. It is this that explains why most patients were admitted rather than treated in their homes at times of crisis. Thus, hospital care remained at the heart of the service – a pattern that observers have noted elsewhere [19].
This may also explain the surprising result that at 12 months post-CMHT patients were less in favour of home treatment than their pre-CMHT counterparts with just over half indicating a preference for hospital-based treatment. The qualitative data suggests they believed that the CMHT was not really offering an alternative to hospital care. Also, the number of re-admissions throughout the study may indicate that the posthospitalization support available to patients remained underdeveloped.
The absence of other service have contributed to these outcomes. It may have been too much to expect that the addition of the CMHT in itself could make a measurable difference. While the aims of the CMHT included supporting patients in the community and accessing other services, the absence of specialist mental health programmes targetting housing and rehabilitiation restricted what could be achieved.
The extended hours service did improve access insofar as fewer post-CMHT patients said they would need help outside operational hours. Medication compliance was higher in the post-CMHT group and these patients also experienced significantly fewer side-effects at 12 months. The reduction in admission bed days and total days spent in hospital suggests the team did focus on patients with psychotic disorders and reduced admissions and allowing early discharge by providing early follow-up.
The new service also increased networking with other local agencies and there is a suggestion in the qualitative data that the overall development of the service, including the refurbishment of the inpatient unit, improved staff morale and that this had a beneficial effect on patients.
Footnotes
Acknowledgements
Funding was provided by the Commonwealth Department of Health and Family Services – Research and Development Grants Advisory Committee (RADGAC), the University of Tasmania and Curtin University.