Inpatient Psychosocial Rehabilitation in Rural NSW: Assessment of Clinically Significant Change for People with Severe Mental Illness

Abstract

Objective: The aim of the present study was to describe an inpatient psychosocial rehabilitation programme in rural New South Wales and to assess the effectiveness of the programme using measures of clinically significant change.

Method: The first 88 patients with a schizophrenia spectrum disorder to enter the Manara Clinic and Turon House, New South Wales, psychosocial rehabilitation programmes were assessed at admission and discharge using the Brief Psychiatric Rating Scale, the Health of the Nation Outcome Scales, and the Kessler-10 self-report measure.

Results: Significant improvements in psychiatric symptomatology, psychosocial functioning, and psychological distress were found over the course of the inpatient stay. Clinical significance analyses using patients in the community as the reference group indicated a reliable and clinically significant improvement for 33% of inpatients on psychiatric symptomatology, 39% of inpatients on psychosocial functioning, and 21% of inpatients on psychological distress.

Conclusions: The psychosocial rehabilitation programme provides clinically significant initial benefits for patients with severe mental illnesses. More attention needs to be paid to evaluating which components of psychosocial rehabilitation contribute most to these benefits. Follow-up evaluation is required to determine whether the benefits of this programme are sustained in the community.

Keywords

clinical significance health services evaluation mental health psychosocial rehabilitation severe mental illness

Severe and persistent mental illnesses result in functional impairments in vocational, social, and/or independent living skills [1]. In the past, the enduring nature of these impairments has necessitated long-term psychiatric hospital-based care. Since the introduction of National Mental Health Strategy [2] in Australia there has been a shift away from prolonged hospitalization and towards community reintegration. As a result, most patients with severe mental illnesses in Australia currently reside in the community rather than in psychiatric hospitals [3].

Responsibility for preparing patients for community care is falling to residential, medium length-of-stay, and day units that provide psychosocial rehabilitation [4]. Psychosocial rehabilitation is typically defined as a process of facilitating an individual's restoration to an optimal level of independent functioning in the community [5] and is one of several models that have the goal of reintegration of people with severe mental illnesses into the community.

Outcome evaluation

There are increasing calls for accountability of economic resources and demonstrations of effectiveness in treatment interventions [1]. Outcome is defined as changes to a patient's health status that are attributable to an intervention [6]. Studies assessing treatment effectiveness are often based on routine outcome data collection throughout the course of treatment. Routine collection of data can inform practice for quality assurance and development of best practice standards [7]. It can also be used within interventions to provide information about the best course of action for individual patients [7].

Outcome studies consistently report positive results for community reintegration programmes [8]. Positive outcomes include reductions in psychiatric symptomatology and the frequency and duration of subsequent hospitalizations, as well as improvements in community tenure, self-esteem and psychosocial functioning [4, 8, 9].

There is a paucity of studies evaluating inpatient psychosocial rehabilitation. Furthermore, the limited numbers of studies that do evaluate outcomes for psychosocial rehabilitation are plagued by methodological and conceptual problems. These problems include high dropout rates, programme fidelity considerations, and the use of statistical significance and neglect of consideration of clinical significance.

Bruseker and O'Halloran conducted a study evaluating outcomes for an inpatient psychosocial rehabilitation programme in New South Wales [4]. Consistent with other community reintegration programmes, patients were found to improve significantly on ratings of psychiatric symptomatology and quality of life [4]. However, a dropout rate of 50% between discharge and follow up, limited the conclusions that could be drawn.

Programme fidelity

Interpretation of psychosocial rehabilitation outcome literature is also limited by the lack of attention to issues of programme fidelity. Programme fidelity is the degree to which a particular programme follows a specified programme model [10]. Many facilities broadly describe their interventions as psychosocial rehabilitation but lack clear definitions of the model being implemented [10]. This is particularly a problem when it comes to comparisons of psychosocial rehabilitation programmes offered at different facilities. Variable estimates of treatment effectiveness can be explained only when the differences between the programmes can be identified. This is next to impossible when detailed programme descriptions are not provided in the literature.

Clinical significance

A third limitation of the inpatient psychosocial rehabilitation outcome literature is sole use of statistical significance techniques to estimate programme effectiveness [11]. Two restrictions of statistical significance techniques are (i) that they identify differences only at the group level rather than at the individual level; and (ii) that they provide no information about the clinical significance of outcomes [12]. Thus, although statistical significance techniques can demonstrate that changes at the group level are not attributable to chance, they provide no indication of how meaningful these changes are at an individual level. Indeed, statistical significance techniques can be misleading if a patient's score on a measure of psychiatric symptomatology improves, but this improvement does not result in any real changes in the patient's level of functioning.

Jacobson and Truax identify two criteria for evaluating a programme's success in achieving rehabilitative goals [12]. A patient's change must (i) be beyond that which can be attributed to chance, otherwise referred to as reliable change, and (ii) reflect the extent to which the patient moves from a score typical of a dysfunctional population to a score typical of a more functional population. Operationally, three possible cut-off points to determine a move from dysfunctional to functional are identified [12]. Clinically significant change has occurred when: (i) the post-treatment score falls more than two standard deviations away from the mean of the dysfunctional population; (ii) the post-treatment score falls within two standard deviations of the mean of the functional population; or (iii) the post-treatment score places the client closer to the mean of the functional population than the mean of the dysfunctional population.

The third cut-off is calculated using the following formula:

where mean_clin and mean_norm are the mean scores of the dysfunctional population and the functional population, respectively, and SD_clin and SD_norm are the standard deviations of the scores in these two groups. This cut-off point is arguably the least arbitrary because it bases calculation of cut-off on both a dysfunctional and a functional population.

Currently, the limited literature surrounding outcomes for inpatient psychosocial rehabilitation is still reliant on statistical significance techniques; there are no studies reporting calculated estimates of reliable and clinically significant change.

Aims

The aim of the current study was to evaluate the outcomes of an inpatient psychosocial rehabilitation programme offered by Manara Clinic and Turon House located in rural New South Wales. This study will add to the sparse inpatient psychosocial rehabilitation outcome literature by evaluating programme effectiveness using clinical significance techniques. The inclusion of a detailed description of the setting, treatment team, admission criteria, guiding principles, as well as structure and activities of the psychosocial rehabilitation programme offered at the Manara Clinic and Turon House, allows for future comparisons between the programme offered at these facilities, and that offered at other psychosocial rehabilitation facilities.

Method

Setting and programme description

Manara Clinic and Turon House are two medium length-of-stay inpatient facilities that operate out of Bloomfield Hospital in Orange, New South Wales, and provide psychosocial rehabilitation services to individuals with persistent and severe mental illnesses. The facilities accept patients from the Greater Western Area Health Service (GWAHS) as well as from elsewhere in New South Wales. The GWAHS covers a rural and remote region of 441 205 km².

Manara Clinic is a unit for men and admitted its first patient in January 2003. Turon House is the unit for women and admitted its first patient in October 2003. Both units have 16 rehabilitation beds each. Both Manara and Turon are medium-secure facilities with private rooms for all residents, communal bathrooms, laundry, kitchen, dining, and entertainment areas, including a large outdoor area. The facilities also have seclusion rooms, a nursing staff office, and group meeting rooms.

Treatment team

The mental health team providing care and programmes for each facility include (i) a psychiatrist, (ii) a psychologist, (iii) a 4 day week^–1 social worker who divides time between the two facilities, (iv) 1.5 diversional therapists per unit, (v) a 3 day week^–1 occupational therapist who divides time between the two facilities, (vi) a clinical nurse consultant, (vii) a nursing unit manager (NUM), and (viii) 15 full-time nursing staff for each unit. All staff received specific induction training in psychiatric rehabilitation with an emphasis on facilitating community-oriented outcomes.

Admission criteria

Criteria for entry into the programme require a DSM-IV disorder, excluding individuals with a primary diagnosis of dementia, delirium, substance abuse, or developmental disorder, although patients with a secondary or tertiary diagnosis of some of the aforementioned disorders are admitted. Also excluded are patients in an acute phase with florid symptoms of their illness, and/or those who have an acute physical illness requiring high levels of nursing care or intensive medical management. Patients who display repeated aggressive or other antisocial behaviours may also be excluded. Only those individuals considered to have the potential to benefit from participation in the programme are admitted.

Programme principles

The Manara Clinic and Turon House operate under the philosophy that all individuals with an enduring mental illness have the potential to learn, grow, contribute to the community, and attain a rewarding quality of life. Programme principles were adapted from the NSW Health Government Action Plan [13].

These principles are operationalized within the units through (i) planning and implementing all services in collaboration with the patient, (ii) supporting, acknowledging and valuing family and carer involvement, (iii) using of targeted, evidence-based programmes and supporting the documentation of outcomes, (iv) building on strengths and lessening functional disabilities to achieve the best possible quality of life, (v) individualized assessment of need, and (vi) a whole-of-person approach to address comorbidity and other compounding problems.

Programme structure and activities

Individual care plans, supported by Mental Health Outcome Assessments Training [14] are developed based on patient and carer interviews, and assessment using the Health of the Nation Outcome Scales (HoNOS) [15], and Kessler-10 Self-Report Measure of Non-Specific Psychological Distress (K-10) [16], which are administered at admission. Individual management plans that provide a day-to-day programme are developed from care plans.

The individual programmes include activities of daily living, such as hygiene and routine household activities to prepare the patient for independent living. Also, patients are encouraged to attend group programmes such as medication education, drug and alcohol groups, anger management groups, and healthy living and fitness education. Patients are assigned to these programmes based on their individual rehabilitation requirements.

Patients have access to an on-campus Living Skills Centre as well as externally run programmes including access to Orange Activity Centre, horticultural programmes, Young People's Group, Consumer Advisory Group, and music groups. Patients can also participate in progressive leave conditional on adherence to a number of programme activities. These conditions include adherence to medication taking, abstinence from drug and alcohol use, an absence of aggressive behaviour, adherence to the individual management plan, adherence to the smoking policy, and full participation in all aspects of the treatment programme.

Participants

The current outcome evaluation study involved the first 56 male patients admitted to Manara Clinic over a 19 month period, and the first 32 female patients admitted to Turon House over an 11 month period. These participants constituted 88 of the first 100 consecutive admissions. There was no difference in age distributions between male and female patients. For the full sample 22% were in the 15–24 year age range, 45% between 25 and 34 years, 22% between 35 and 44 years, and 11% were ≥45 years. The average age at admission of the male participants was 32.5 years (SD = 10 years) with a range of 20–55 years, and the average age at admission for the female participants was 30.5 years (SD = 10 years) with a range of 15–58 years. Although male participants tended to have a longer average length of stay (mean = 148 days, SD = 90 days, range = 30–371 days) than female participants (mean = 117 days, SD = 69 years, range = 29–275 days), this difference was not statistically significant, t(78) = 1.61, p > 0.05. The average length of stay for the full sample was 136 days (SD = 84 days).

Participants were selected for participation in the psychosocial rehabilitation programme based on the admission criteria mentioned previously. Although the primary diagnoses of the sample were schizophrenia (89%, n = 78), and schizoaffective disorders (11%, n = 10), 52% also had a secondary diagnosis, with the majority (43%) having a secondary diagnosis of a drug use disorder. The other problems and needs that were characteristic of this sample were treatment resistance, medication non-compliance, frequent and prolonged periods of hospitalization, a lack of control over symptoms, difficulties with self-care and independent living skills, and deficits of insight into their own illness. All participants in the psychosocial rehabilitation programme were subject to Temporary Patient or Continued Treatment Patient Orders, that is, involuntary admission under the provisions of the Mental Health Act, New South Wales, 1990 as amended.

Measures

The effectiveness of the psychosocial rehabilitation programme is assessed by comparing patient admission and discharge scores on standardized measures of mental health. The standardized measures used in the present study were the Brief Psychiatric Rating Scale (BPRS) [17], HoNOS [15], and the K-10 [16].

The BPRS is a 24-item measure of psychiatric symptomatology with a score of 24 indicating no problem and a score of 168 indicating an extremely severe problem across all symptom sets [17]. The BPRS procedure involves rating the scale following a structured interview process.

The HoNOS measures psychosocial functioning in the domains of behavioural, symptoms, and social functioning as well as the domain of impairment [15]. The HoNOS is a 12-item scale with scores ranging from 0 = no problem to 4 = severe problem for each item.

The K-10 is a 10-item scale that asks the client to indicate how frequently in the last 30 days they experienced symptom distress such as feelings of depression, restlessness, tiredness, worthlessness, and nervousness [16]. Scores range from 1 = none of the time to 5 = all of the time.

Satisfactory reliability and validity have been established for all three measures.

Procedure

The standardized mental health measures were administered at admission, discharge and 3 month reviews. The Manara Clinic and Turon House psychologist or psychology intern administered the BPRS. The HoNOS was administered by nursing staff. The K-10 was completed by the patients.

Admission scores were compared against a repeat score for seven patients on the BPRS, six patients on the K-10, and five patients on the HoNOS due to discharge measures not being completed. The repeat score was the most recent 3 month review prior to discharge.

Means and standard deviations were calculated for admission and discharge total scores on each measure. Paired samples t-tests were conducted comparing admission and discharge scores for initial statistical significance analysis.

Reliable change indices were calculated using the Christensen and Mendoza formula for reliable change (Appendix I) [18]. Clinical significance cut-off scores were calculated for the BPRS, HoNOS, and K-10 for each of the Jacobson and Truax three calculation methods [12]. The first cut-off method required a move away from the dysfunctional population mean by two standard deviations and was calculated from the current study sample statistics. The second cut-off method requires a move to within two standard deviations of a functional population and the third method calculates the midpoint between the dysfunctional and functional populations. One of the major challenges in assessing clinical significance is access to appropriate normative comparison data for various measures. In the context of the current study, a more functional population was considered to be a group with similar diagnostic characteristics who were able to reside in the community. Functional population normative data for the BPRS were taken from the Maastricht Psychosis Study [19] that reported means and standard deviations for 43 patients with schizophrenia currently residing in the community and 43 individuals from the general population. Normative data for the HoNOS were taken from a clinical significance study conducted by Parabiaghi et al. using a classification of severity method for defining functional populations [11]. Means and standard deviations were obtained from 3926 patients with schizophrenia designated as mild and 1669 patients designated as subclinical. The designation of mild and subclinical populations is based on scores on individual items of the HoNOS and gives no reference to the actual functioning level of patients. Normative data for the K-10 were taken from a sample of 150 community residents with mental health problems in a Boston study [20].

Results

Psychiatric symptomatology

There were 82 complete admission and discharge datasets available for analysis on the BPRS. The mean score at discharge (mean = 46.01, SD = 10.76) was significantly lower than the mean score at admission (mean = 62.23, SD = 14.08), t(81) = 9.54, p < 0.001, suggesting reductions in symptoms

The calculated reliable change index was 18.28. The reliable change index was rounded so that an 18 point change reflected that reliable change had occurred at the 95% confidence level. From this reliable change index 35 patients (43%) were designated as reliably improved and only one patient showed a reliable deterioration. The remaining participants were designated either improved (n = 35) or deteriorated (n = 11) but this was not of a magnitude to suggest reliable change.

Using the BPRS, clinical significance cut-off based on moving two standard deviations away from the inpatient sample mean was calculated as 34. Cut-off points for moving within two standard deviations of the comparison community sample of people with schizophrenia was 59. The cut-off point for a move closer to the mean of the community sample than the present inpatient sample was calculated as 49. Cut-off points were calculated for a move from a dysfunctional to a normal population. These cut-off points indicated that no patients demonstrated reliable improvement from a dysfunctional to a normal population, nor did any patients reliably deteriorate from a normal level of functioning to a dysfunctional level. Table 1 presents the number of patients designated as achieving a reliable and clinically significant change in BPRS scores at each of the other cut-offs.

Table 1.

Patients showing reliable and clinically significant change on the BPRS using different cut-off criteria (n = 82)

	Improved		Deteriorated
Cut-off point	n	%	n	%
>2 SD away from inpatient sample mean	6	7.3	0	0
Within 2 SD of community sample with schizophrenia	31	37.8	0	0
Closer to community sample with schizophrenia than inpatient sample	27	32.9	1	1

BPRS, Brief Psychiatric Rating Scale.

Psychosocial functioning

There were a total of 84 subjects with complete admission and discharge datasets on the HoNOS. The mean HoNOS discharge score (mean = 6.93, SD = 4.35) was significantly lower than the mean admission score (mean = 15.82, SD = 6.29), t(83) = 12.82, p < 0.001.

The calculated reliable change index for the HoNOS was 10, which indicated a total of 35 patients (42%) with reliable improvements in scores and no patients with reliable deterioration in scores. The remaining patients either improved (n = 41) or deteriorated (n = 3) at a level that was not of sufficient magnitude to be designated as reliable, or made no change (n = 5).

The clinical significance cut-off point for moving two standard deviations away from the inpatient sample mean was calculated as 3. Cut-off scores of 13 and 6 were calculated for moving within two standard deviations of the comparison mild and subclinical sample means, respectively [11]. Finally, the clinical significance cut-off points based on a score that is closer to the mild or subclinical sample mean than the inpatient sample mean were calculated as 10 and 6 for the mild and subclinical samples, respectively. No patients showed reliable deterioration using any of the cut-off points. The number of patients designated as having a reliable and clinically significant improvement on the HoNOS for each cut-off point is reported in Table 2.

Table 2.

Patients showing reliable and clinically significant improvements on the HoNOS using different cut-off criteria (n = 84)

Cut-off point	n	%
>2 SD away from inpatient sample mean	11	13.1
Within 2 SD of comparison sample with schizophrenia designated as mild disability	34	40.5
Within 2 SD of comparison sample designated as subclinical disability	23	27.4
Closer to mild comparison sample than inpatient sample	33	39.3
Closer to subclinical comparison sample than inpatient sample	23	27.4

HoNOS, Health of the Nation Outcome Scales.

Self-reported psychological distress

There were 70 complete admission and discharge datasets available for analysis on the K-10. The mean K-10 score at discharge (mean = 15.66, SD = 5.61) was significantly lower than the mean K-10 score at admission (21.01, SD = 11.44), t(69) = 4.07, p < 0.001.

The reliable change index for the K-10 was 7.58, indicating a 7 point change being needed to give 95% confidence that a reliable change occurred in the patient over the course of their inpatient stay. A reliable improvement was evident in 39% (n = 27) of the patients and a reliable deterioration was evident in 10% (n = 7) of the patients. A further 15 patients showed some improvements and 12 showed some deterioration, with nine showing no change.

The K-10 clinical significance cut-off point for moving two standard deviations away from the inpatient sample mean was calculated as −2; the cut-off point for moving within two standard deviations of a comparison functional population [20] was calculated as 23, and the final cut-off point that reflects a score that is closer to the mean of the functional sample than the inpatient sample was calculated as 14. Table 3 presents the number of patients designated as having made a reliable and clinically significant change in K-10 scores for each cut-off point.

Table 3.

Patients showing reliable and clinically significant change on the K-10 using different cut-off criteria (n = 70)

	Improved		Deteriorated
Cut-off point	n	%	n	%
>2 SD away from inpatient sample mean	0	0	0	0
Within 2 SD of comparison outpatients residing in the community	13	18.6	1	1.4
Closer to comparison community outpatients than inpatient sample	15	21.4	1	1.4

K-10, Kessler-10 Self-Report Measure of Non-Specific Psychological Distress.

Discussion

Overall, the results indicate positive outcomes for patients with severe mental illnesses over the duration of their stay in the Manara Clinic and Turon House psychosocial rehabilitation programmes. Statistical significance tests revealed significant improvements from admission to discharge on all three of the assessed outcome domains: psychiatric symptomatology, psychosocial functioning, and psychological distress. These results are broadly consistent with the findings for other psychosocial rehabilitation and community reintegration programmes [4, 8, 9].

The inclusion of clinical significance analyses in the study allowed for a more meaningful estimate of programme effectiveness, as the number of patients whose scores suggest they have moved from a more dysfunctional population to scores more consistent with a functional population. In this case the more functional populations were represented by samples of individuals with schizophrenia who were residing in community settings. The different cut-off methods produced different estimates of reliable and clinical significant change in the sample. Perhaps the most appropriate clinical significant cut-off points were those indicating a discharge score that places the patient closer to the mean of the functional population than the mean of the dysfunctional population. Across all three outcome measures the Jacobson and Truax clinical significance cut-off calculation method that required a move of two standard deviations away from the dysfunctional population mean [12] appeared too stringent. This method also appeared to be strongly influenced by large standard deviations in the study sample at admission, making achievement of this criterion extremely difficult. Similarly, the method requiring a move within two standard deviations of the functional population may have provided an overly generous estimate of treatment effectiveness. Taken together, a large difference in estimates provided by these two cut-off points indicates considerable overlap between the dysfunctional and functional populations.

One of the major limitations of analyses of clinical significance is the lack of appropriate normative information by which to compare groups. Furthermore, different estimates of treatment effectiveness were found for different normative populations. This is most evident on the BPRS, on which no individuals achieved reliable and clinically significant improvement when it required a move from a dysfunctional population to a normal population. It is clear that this is an unreasonable criterion for a severely disabled sample that required inpatient care. Thus, improvement to a level of symptomatology experienced by those functioning in the community with severe mental illnesses would provide a more realistic estimate of effectiveness in this context than improvement to a level experienced by those without a psychological disorder [21].

The variable estimates of treatment effectiveness for the different normative comparison groups on the HoNOS are difficult to interpret given the method for differentiating these normative populations. The classification of severity system used to differentiate the mild and subclinical groups was based on somewhat arbitrary scores on the HoNOS itself rather than on the functioning capacity of those completing the HoNOS [11]. The external validity of such a classification system is yet to be established, because the designation of mild and subclinical provides no information on the functional status of the patients.

Given the caveats noted here, the results indicate a reliable and clinically significant improvement by 33% of the patients on psychiatric symptomatology, 39% of the patients on psychosocial functioning, and 21% of the patients on psychological distress. These estimates are consistent with the findings of Parabiaghi et al., who note that analysis of clinical significance for outcome measures generally produces estimates of treatment effectiveness that are below 50% of the sample [11]. A reliable and clinically significant deterioration over the course of the study was seen only in one patient on the BPRS and one patient on the K-10.

Psychosocial rehabilitation is a term that gathers together a wide variety of programmes and interventions under one label. As such, future research should begin identifying the active components or mechanisms of these complex multicomponent interventions. To further demonstrate programme effectiveness, it may also be possible to compare progress of those who are waiting for admission with those who enter the programme, although ethical and practical constraints may limit elements of experimental control.

Although the present study provides preliminary outcomes there is a need for follow up of patients once they return to community care. The present study found that 39% of patients experienced improvements consistent with other patients residing in the community, but there is a need to see whether they are able to maintain these improvements and continue to build on them. Continuity of care arrangements are likely to be critical to future improvements. Ongoing assessment of symptom severity and distress, along with other domains of functioning, should continue. In addition, traditional outcomes such as length of community tenure should be supplemented with more patient-oriented recovery outcome measures that focus on variables such as hopefulness, goal striving, meaning in life, identity, and increased responsibility for self-management [22].

Appendix I

The SE of measurement of a difference is calculated as:

where SD₁ is the standard deviation of the admission total and α is Cronbach's coefficient.

The RC index is calculated as:

Footnotes

Acknowledgements

The authors are grateful to Associate Professor Paul Fanning for his efforts and guidance in establishing this psychiatric rehabilitation service in Bloomfield Hospital. The authors acknowledge the contributions by Dr R Kaneyson and the staff of Manara Clinic and Turon House.

References

1. Srividya

Rothman

Vogler

Spaulding

. Evaluating outcomes of rehabilitation for severe mental illness. Rehabil Psychol 2005; 50: 43–55.

2. Australian Health Ministers. National mental health policy. Canberra: Australian Government Publishing Service, 1992.

3. Newton

Rosen

Tennant

Hobbs

Lapsley

Tribe

. Deinstitutionalisation for long-term mental illness: an ethnographic study. Aust N Z J Psychiatry 2000; 34: 484–490.

4. Bruseker

O'Halloran

. Preliminary results from an Australian psychosocial rehabilitation program for people with serious mental illness. Aust N Z J Mental Health Nurs 1999; 8: 58–64.

5. Cnaan

Blanketz

Messinger

Gardner

. Psychosocial rehabilitation: toward a definition. Psychosoc Rehabil J 1988; 11: 59–77.

6. Andrews

Peters

Teeson

. The measurement of consumer outcome in mental health: a report to the national mental health information strategy committee. Clinical Research Unit for Anxiety Disorders, Sydney 1994.

7. Slade

. What outcomes to measure in routine mental health services and how to assess them: a systematic review. Aust N Z J Psychiatry 2002; 36: 253–269.

8. Brekke

Long

. Community-based psychosocial rehabilitation and prospective change in functional, clinical, and subjective experience variables in schizophrenia. Schizophr Bull 2000; 26: 667–680.

9. Habibis

Hazelton

Schneider

Bowling

Davidson

. A comparison of patient clinical and social outcomes before and after the introduction of an extended-hours community mental health team. Aust N Z J Psychiatry 2002; 36: 392–398.

10.

10. Bond

Evans

Salyers

Williams

Kim

. Measurement of fidelity in psychiatric rehabilitation. Mental Health Service Res 2000; 2: 75–787.

11.

11. Parabiaghi

Barbato

D'Avanzo

Erlicher

Lora

. Assessing reliable and clinically significant change on the Health of the Nation Outcome Scales: method for displaying longitudinal data. Aust N Z J Psychiatry 2005; 39: 719–25.

12.

12. Jacobson

Truax

. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol 1991; 59: 12–19.

13.

13. Mental Health Implementation Group. NSW Health, framework for rehabilitation for mental health: NSW Government Action Plan. Sydney: NSW Department of Health, 2002.

14.

14. NSW Health Department. NSW Mental Health Outcome Assessments and Training (MHOAT) facilitators manual. Sydney: NSW Department of Health, 2001.

15.

15. Wing

Beevor

Curtis

Park

SBG

Hadden

Burns

. Health of the Nation Outcome Scales (HoNOS): Research and development. Br J Psychiatry 1998; 172: 11–18.

16.

16. Kessler

, Andrew

, Colpe

et al . Short screening scales to measure population prevalences and trends in non-specific psychological distress. Psychol Med 2002; 32:959–976.

17.

17. Lukoff

Neutcherlein

Ventura

. Manual for the Expanded Brief Psychiatric Rating Scale. Schizophr Bull 1986; 12: 594–602.

18.

18. Christensen

Mendoza

. A method for assessing change in a single subject: an alteration of the RC index. Behav Ther 1986; 17: 305–308.

19.

19. Janssen

Krabbendam

Jolles

Van Os

. Alterations in theory of mind in patients with schizophrenia and non-psychotic relatives. Acta Psychiatr Scand 2003; 108: 110–117.

20.

20. Kessler

Barker

Colpe

. Screening for serious mental illness in the general population. Arch Gen Psychiatry 2003; 60: 184–189.

21.

21. Myin-Germeys

Krabbendam

Jolles

Dellespaul

Van Os

. Are cognitive impairments associated with sensitivity to stress in schizophrenia? An experience sampling study. Am J Psychiatry 2002; 159: 443–449.

22.

22. Andresen

Oades

Caputi

. The experience of recovery from schizophrenia: towards and empirically validated stage model. Aust N Z J Psychiatry 2003; 37: 586–594.