Abstract
The primary aim of any mental health service is to improve the health and functioning of people with mental disorders. Emphasis is increasingly being placed on the direct measurement of such health outcomes using standardised measures with proven reliability and validity. Internationally, there is a strong push for outcome monitoring in mental health [1,2]. In Australia, the 1992 National Mental Health Policy argued that such reviews should be an inherent part of regular practice [3]. The ultimate goal of the policy is to establish national outcome standards against which mental health service sectors can be evaluated and assessed. Once such standards are in place, routine outcome measurement will form an integral part of good clinical care as the information will identify the outcome of treatment and can be used to inform and improve service delivery [4].
Ongoing measurement is costly, so the information must be obtained using measures that are reliable, valid and sensitive to change. For routine use, measures should also be multidimensional, feasible in terms of brevity and cost, and require minimal training, administration, scoring and interpretation. At present, outcome is generally assessed by patients and clinicians in an unsystematic fashion and in the absence of definitions for making judgements. Such informal methods of evaluation are unreliable [5]. Patient judgements often rely heavily on current symptomatology and clinicians often perceive the health of their most recently seen patients as indicative of their entire patient load [6].
Although outcome measurement is not new [7], at present there are no universally agreed consumer outcome measures in place. We therefore thought it timely to study routine outcome measurement in the most common service in Community Mental Health in Australia, namely, case management. This form of service delivery has increasingly been seen as the preferred method of caring for the seriously mentally ill [8,9]. Case management offers continuity of care as contact is maintained with the patients for purposes of assessment, monitoring, and coordination of services [10]. The two main approaches to case management in Australia are standard and assertive case management. The former involves case managers maintaining case loads of up to 30 people with low intensity and infrequent contacts. Assertive case management involves intense and frequent patient contacts with more disturbed patients so case loads are generally restricted to 10 patients per team member.
The aim of this study was to trial routine outcome measurement with the standard and assertive case management teams of two mental health services within Sydney. It was expected that reliable and valid measures would demonstrate (i) greater levels of disability and need for patients receiving assertive case management compared with those receiving standard case management and (ii) clinically meaningful mental health outcomes in patients case managed by community mental health services.
Method
Participants
Case managers and patients of two adjoining Mental Health Services within Sydney participated in this study: The Inner City (ICMHS) and Eastern Suburbs Mental Health Services (ESMHS). Both services target adults with a serious mental illness and serve as a reasonable test of outcome measurement in this setting.
The ICMHS services a catchment population of 72 000 in one of the most difficult and deprived areas in Australia. The area is characterised by high levels of transience, homelessness, drug and alcohol misuse and has the highest concentration of street crime, homicide and people living with HIV and AIDS in Australia. The ICMHS provides standard case management and assertive case management [11]. The latter caters for clients who require more intensive care due to factors such as non-compliance, transience or frequent hospitalisation. Both types of case management aim to keep hospital admissions to a minimum and encourage clients to become fully integrated into the community by promoting self care and independent living skills [11,12].
The ESMHS services a catchment population of 278 000 encompassing a wide range of socioeconomic groups [13]. The Case Management Team targets adults living in the eastern suburbs with a serious mental illness or those experiencing severe distress caused by psychological or social problems. The team provide ongoing short- and long-term clinical and psychoeducational support, client assessment and referral services as required.
Design
Fifteen case managers (each having between seven and 32 patients) volunteered to complete a general information sheet and measures on each of their patients. Five workers provided assertive case management for the most disabled patients and 10 provided standard case management. The Health of the Nation Outcome Scales (HoNOS) (4th draft) [14] was administered at baseline, 6 and 12 months. The baseline measure assessed disability and the change in this measure across time assessed outcome. Initially, the Short-Form 36 (SF-36) (Australian Standard Version 1.0) [15,16] was administered as a second measure of disability and outcome. However, due to the poor completion rate on this measure (44%) it was discontinued after 6 months and detailed results are not reported. Both measures were recommended for routine use in an Australian Government review [4].
The patient and staff versions of the Camberwell Assessment of Need (CAN) [17] were introduced at 12 months. It was administered only once as it was not used as an outcome measure, but rather as a measure of need. At the time of this study, the CAN differed from the HoNOS in that it could be rated by both clinicians and patients and therefore potentially provided additional information to the HoNOS. A self-rating version of the HoNOS has since been developed [18]. Although the relationship between measures of need and disability have yet to be clearly defined [19], both are considered important in service planning [17].
All participating case managers received formal training in the use of the HoNOS and the CAN using standardised vignettes.
Sample
Fifteen community mental health clinicians, with a combined caseload of 283 patients, participated in the study. Outcomes measures were completed for 260 clients. Patients receiving standard case management (n = 220) had a mean age of 44 years and 62% were male. Patients receiving assertive case management (n = 40) had a mean age of 41 years and 58% were male. In the standard case management group, 81% had a primary diagnosis of schizophrenia and 19% an affective disorder or personality disorder. In the assertive case management group, 95% had a primary diagnosis of schizophrenia and 5% an affective disorder. The diagnoses in the two groups were not significantly different (χ2 = 1.62, df = 1, p = 0.20).
Measures
The HoNOS is a reliable, clinician-completed measure designed to assess health and social functioning over the past fortnight [14]. A Cronbach's α of 0.76 for the baseline HoNOS total score indicates that the measure has high internal consistency, supporting the use of the total score as a meaningful summary of disability (illness severity) and outcome (change over time) [14]. The measure was developed for use with psychiatric populations. It is reliable, valid and sensitive to change, although no population reference normative data are yet available [4].
To provide an external reference of outcome to assess change, case managers were asked to rate at 12 months whether patients had improved (1), remained stable (2) or deteriorated (3) over the previous 6 months (i.e. since their last assessment). These ratings were made without reference to scores on the outcome measures. Ratings were obtained at 12 months for the final 6 month period as one rating over a year was considered insufficient given the instability of the patient group.
Need was measured using the staff and patient versions of the CAN, a reliable instrument designed for use with people with serious mental illness [17]. It addresses 22 areas of need on a three-point scale of no problem (0), no or only a moderate problem because of continuing intervention (1) or current serious problem (2). A score out of 22, representing total number of needs, was computed for both versions. Ratings of 0 were scored as 0, indicating no need and scores of 1 or 2 were scored as 1, indicating the presence of a need.
Data analysis
Independent t-tests assessed the concurrent validity of the HoNOS and CAN. Correlations compared patient and clinician ratings for the HoNOS and CAN. Paired t-test analyses were used to assess patient disability and outcome over time. Three separate analyses were conducted to reflect outcome based on whether case managers had rated their patients as having improved, remained stable or deteriorated over the previous 6 months (i.e. since the last assessment). An adjusted alpha level of 0.01 was adopted to account for multiple analyses.
Results
Disability and need
To assess disability and need it must first be determined that the measures are feasible in terms of their completion.
Completion
At baseline, the 15 case managers had a total of 283 patients in their caseload. The HoNOS was completed for 92% of these patients. The completion rates were equivalent in the standard and assertive case management groups. Non-completion only occurred if case managers had not had recent contact with a patient. Case managers indicated that the HoNOS addressed areas of importance and was brief and easy to complete.
The SF-36 was completed by 44.5% of patients at baseline. A major barrier was obtaining access to patients for interview. Up to 31% of patients were precluded from interview as they were in the process of engaging with the service; or were too ill, uncooperative or unpredictable. Case managers perceived that contact with these patients could jeopardise an already precarious relationship. Other patients either could not be contacted (18%), refused to be interviewed (12%) or failed to complete the interview (5%). Patients who did complete the SF-36 found the questions repetitive and expressed eagerness for the interview to cease. Because of the low rate of completion, the SF-36 data represent a biased sample, and thus no further data analysis will be conducted on the SF-36.
Case managers completed the staff version of the CAN for 74% of their patients, once, at 12 months. Assessments were not completed because patients had not recently been seen (7%) or because clinicians considered that completing the CAN in addition to the HoNOS would impinge upon their clinical workload (18%). The patient version of the CAN was also administered once, at 12 months, to 49% of patients. As with the SF-36, at case managers' request, over one-quarter of patients were not approached. Of patients approached 16% refused to participate. Both staff and patient versions of the CAN were completed for 78 patients. There were no significant differences between those with complete CAN data and those without in terms of sex, age, diagnosis or disability as measured by the HoNOS.
Concurrent validity of disability and need scores
There are no standards by which to determine the concurrent validity of measures of disability and need, that is, whether they measure what they claim to measure. However, it is possible to determine their relationship to independently assigned service and treatment indicators. In this study, two groups of patients with differing levels of disability and need were identified; those receiving standard case management and a more disabled group receiving assertive case management. It was expected that there would be greater levels of disability and need for patients receiving assertive case management compared with those receiving standard case management. The levels of disability, measured by the HoNOS (baseline measure), and need, measured by the CAN, for these groups are presented in Table 1.
Patient levels of disability and need
Disability
Health of the Nation Outcome Scales scores clearly reflect differences between the two patient groups with standard case management patients rated as significantly less disabled than assertive case management patients at baseline (t = 6.15, df = 259, p < 0.001), 6 months (t = 9.9, df = 212, p < 0.001) and 12 months (t = 7.8, df = 216, p < 0.001).
Need
Patient and staff ratings of need were discrepant. Patients of the different case management groups rated themselves as having similar levels of need (t = 1.69, df = 76, p > 0.05). Staff, however, rated the assertive case management patients as having significantly more needs than standard case management patients (t = 4.03, df = 76, p < 0.001).
Correlation of scales
Measures within the domains of disability and need should share some relationship with each other as should clinician and patient ratings. Clinician ratings on the CAN correlated highly with HoNOS scores (Pearson's r = 0.82, p < 0.001). This indicates a strong relationship between measures of disability and need. There were poorer relationships between patient and clinician ratings. Patient ratings of need correlated only moderately with staff ratings of need (Pearson's r = 0.39, p < 0.01).
Outcome
The patient population receiving case management in the two services studied were clinically heterogeneous. The severity of their problems ranged from recent onset psychosis to chronic schizophrenia and homelessness. It is possible that some chronically ill patients may deteriorate in their functioning over a 6 month period due to circumstances such as becoming homeless or the loss of family support. A valid outcome measure should reflect not only improvement in health status, but also lack of change or deterioration. The final aim of this study was to assess patient outcomes. It was expected that the HoNOS would detect clinically meaningful change in mental health outcome across time.
Using the case manager rating of change, the HoNOS scores for the 6 and 12 month assessments were examined. The patients rated by the case managers as having improved also demonstrated significant improvement in HoNOS score (n = 70, mean (SD) = 9.4 (6.5) and 7.4 (5.6) for 6 and 12 months respectively; t = 2.77, df = 69, p < 0.005), the patients rated as stable by the case managers showed no change in HoNOS score (n = 71, mean (SD) = 10.1 (6.1) and 11.1 (7.4) respectively; t = −1.25, df = 70, p > 0.10) while the patients rated as deteriorating demonstrated significant deterioration on the HoNOS (n = 17, mean (SD) = 9.1 (4.3) and 14.8 (6.3) respectively; t = −4.43, df = 16, p < 0.005).
Discussion
This study trialled measurement of disability, need and outcome with the standard and assertive case management teams of two mental health services within Sydney. As expected, patients receiving assertive case management had greater levels of disability and need than those receiving standard case management. Significant change in mental health status was also demonstrated with the HoNOS.
The HoNOS was acceptable to clinicians and had an excellent completion rate. There was adequate internal consistency for the total score, suggesting that a single summary score provided a meaningful measure of disability. The HoNOS differentiated across varying levels of disability. The outcome for the majority of patients was to remain stable or improve over time. Few became more disabled.
The SF-36 had a poor completion rate and, consistent with previous research [20], patients felt the measure did not address issues relevant to mental health. This indicates that the SF-36 is limited as an outcome measure for routine use in community mental health settings.
The patient version of the CAN was also only completed for approximately half of the registered patients. However, patients who did complete the measure found it acceptable and highly relevant to mental health.
Clinicians completed the staff version of the CAN on three-quarters of their patients and considered the measure appropriate for use in mental health. The staff version of the instrument differentiated between levels of disability. The discrepancy between staff and patient ratings for the CAN highlights the disparity between patient and clinician ratings and reinforces the importance of obtaining both perspectives.
Consistent with previous research [20], this study found that the HoNOS performs well as a routine measure of mental health outcome. Preliminary findings from this study also suggest that the CAN may be useful as a routine measure of need.
There were several limitations associated with the study which should be considered in conjunction with the findings. First, the case managers in this study were volunteers. It is likely that the quality and quantity of data obtained may differ in a conscripted sample. It could also be the case that the assessment of patient outcome for the HoNOS is potentially confounded as both the HoNOS rating and the external indicator of change were completed by the same case manager. However, this method was considered meaningful as case managers did not refer to previous ratings when completing the HoNOS and would be unlikely to have recalled their ratings from 6 months previously.
Despite these limitations, this was a naturalistic study conducted within existing mental health services. The complex nature of the patients' problems, their high mobility and the demanding nature of the inner city area made this a very challenging project to complete. Notwithstanding these difficulties, completion rates for the staff-completed measures were comparable to those found in well-resourced model research programs.
During the course of this study it became evident that several processes must be addressed before routine outcome measurement can become a reality. First, a major barrier lies with the clinicians, for whom outcome measurement is often seen as a process that occurs in addition to their clinical work. Outcome measurement will not be viable until clinicians consider it an essential element of good clinical care. Second, for outcome measurement to be implemented on a routine basis, information systems are required to manage the data and, ideally, allow comparisons across service sectors.
In conclusion, this study has found that routine outcome measurement is feasible. If it is to become a reality the next step must be to extend the focus to staff education and training and to the development and trial of computerised information systems.
Footnotes
Acknowledgements
This work was supported by a Health Outcomes Grant from the New South Wales Department of Health. The authors thank the staff and patients of the Inner City and Eastern Suburbs Mental Health Services for their participation in this project. Cathy Issakidis and Kristy Sanderson are also gratefully acknowledged for their input and assistance.