Predicting clinically significant change in an inpatient program for people with severe mental illness

Abstract

Objective:

The first aim of this study was to assess the proportion of patients who achieved reliable and clinically significant change over the course of treatment in an inpatient psychosocial rehabilitation program. The second aim was to determine whether age, gender, length of stay, and diagnosis and co-morbid diagnosis predicted those who were classified as improved or not improved, using clinical significance criteria.

Method:

Three hundred and thirty-seven patients from inpatient units at Bloomfield Hospital, Orange, New South Wales, Australia were assessed at admission, 3-month reviews and discharge using the expanded Brief Psychiatric Rating Scale, the Health of the Nation Outcome Scales and the Kessler 10.

Results:

Reliable and clinically significant improvement was found for 32.4% of inpatients on psychiatric symptomatology, 19.5% on psychosocial functioning and 20.2% on psychological distress. Logistic regression analyses found that the predictor variables collectively predicted those who made reliable and clinically significant improvement on psychiatric symptomatology, but not on psychosocial functioning or psychological distress. Those with a primary diagnosis of schizo-affective disorder had higher rates of improvement in psychiatric symptomatology compared to those with a diagnosis of schizophrenia. Those with co-morbid substance abuse disorders showed a trend towards greater improvement.

Conclusions:

Inpatient treatment is associated with clinically significant improvements for some patients with a severe mental illness. Patients with schizo-affective disorders are proportionally more likely to make improvement.

Keywords

Inpatient clinical significance severe mental illness co-morbidity

Introduction

Severe and persistent mental illnesses are debilitating for the individual and financially costly for the community (Access Economics and SANE Australia, 2002); however, there is considerable support for the benefits of psychosocial interventions to improve a range of client outcomes (Huxley et al., 2000; Kurtz and Mueser, 2008). Statistical significance and effect size statistics have frequently been used to describe outcome. However, more recently, the usefulness of reporting clinical significance is being recognized (Johnson et al., 2006; Lambert and Ogles, 2009). This is because in contrast to these other methods, clinical significance identifies differences in treatment outcome for individuals (rather than groups) and provides an indication of whether this change is clinically meaningful. While multiple methods of measuring clinical significance exist, the Jacobson and Truax’s (1991) method seems particularly useful for measuring the effectiveness of inpatient programs as it examines whether an individual’s change in their post-treatment score is a reliable change (i.e. unlikely to be attributable to measurement error) and whether the post-treatment score reflects a move from the dysfunctional population to a score more typical of a functional population.

While a number of studies have used clinical significance methods to measure treatment effectiveness (Bowersox et al., 2009; Glick et al., 1993), only a limited number of studies have examined this within a psychiatric inpatient population and have reported calculated estimates of reliable and clinically significant (RCS) change (e.g. Murugesan et al., 2007; Newnham et al., 2007).

Murugesan et al. (2007) used clinical significance methods to assess outcomes for people with a schizophrenia spectrum disorder attending a medium length-of-stay psychosocial rehabilitation program in Australia. RCS improvement was made by 33% of patients on psychiatric symptomatology, 39% on psychosocial functioning and 21% on psychological distress. Murugesan et al. (2007) focused on those with schizophrenia spectrum disorders, but the units treated patients with a range of other diagnoses and psychiatric co-morbidities. The sample size was not sufficiently large to allow identification of potential predictors of RCS improvement. For example, people with co-morbid substance abuse in addition to their Axis I diagnosis have been found to be more difficult to treat and have poorer outcomes (Grella and Stein, 2006), making co-morbidity a potential predictor.

The current study aims to determine the percentage of individuals who show RCS improvement at the time of discharge from an inpatient rehabilitation program. While research has examined predictors of outcome among patients with a schizophrenia spectrum disorder (e.g. Schennach-Wolff et al., 2009), no study appears to have examined demographic and clinical predictors of treatment outcome for psychiatric inpatients with predominantly a schizophrenia spectrum disorder, using clinical significance criteria to measure outcome. Identification of predictors of RCS change has various benefits, such as the potential to assist with refining treatment. Given this, the second aim of this study is to explore whether age, gender, length of stay, diagnosis and co-morbid diagnosis predict whether patients show RCS improvements or not.

Method

Setting and program description

Data were collected from a medium length-of-stay inpatient rehabilitation facility (Bloomfield Hospital in Orange, New South Wales, Australia). Participants attended either the 16-bed male or 16-bed female inpatient unit. The mental health team involved in the program included a psychiatrist, a psychologist, a social worker, diversional therapists, an occupational therapist, a clinical nurse consultant, a manager and nursing staff for each unit. Patients were encouraged to attend groups relevant to them on topics such as drug and alcohol, medication education or anger management. Care plans were tailored to each patient. Discharge was planned, except on those rare occasions where a patient managed to abscond.

Admission criteria

Admission criteria for entry into the program remained unchanged since the previous study (Murugesan et al., 2007). The criteria included a Diagnostic and Statistical Manual of Mental Disorders, fourth edition (DSM-IV) disorder but excluded individuals with a primary diagnosis of dementia, delirium, substance abuse or developmental disorder. Patients who had secondary diagnoses of substance abuse were admitted.

Patients were excluded if they were in an acute phase of their illness with florid symptoms, and/or those who had an acute physical illness requiring high levels of nursing care or intensive medical management. Patients who were frequently aggressive or showed other antisocial behaviours were also excluded. The majority of patients were involuntarily admitted. The principal reason for admission was for psychosocial rehabilitation. Patients were frequently admitted for the purposes of improving their medication adherence and reducing substance abuse.

Participants

The potential participant pool consisted of 371 consecutive patients, 18 years and older, who were admitted and discharged between November 2003 and November 2008.

This was a separate group to those previously reported by Murugesan et al. (2007) in the same programs.

Individuals were excluded from the study if:

it was their second admission during the study period, then data from their second admission were excluded (eight patients); or

admission or discharge data were missing, so change across time could not be calculated (26 patients).

This provided a 91% participation rate overall (n = 337).

There were 170 patients admitted to the male unit and 167 patients admitted to the female unit over the study period. The mean age at admission of the male participants was 33.7 years (SD = 9.2 years), with a range of 18–61 years. The mean age at admission of the female participants was 33.4 years (SD = 10.3 years) with a range of 18–59 years. The mean length of stay for male participants was 109 days, (SD = 61, range 6–397 days). The mean length of stay for female participants was 113 days (SD = 84, range 15–602 days). Of these female patients, only four stayed more than 400 days. The mean length of stay for the full sample was 111 days (SD = 73 days, median = 91 days).

Patient diagnoses were extracted from medical records. The majority of patients had a primary diagnosis of schizophrenia (68.0%, n = 229). Schizo-affective disorder was the next most common diagnosis (20.8%, n = 70). Some patients had another disorder involving psychosis (6.8%, n = 23). The rest of the patients had other diagnoses (4.5%, n = 15), such as a personality disorder, depression and substance use disorder. Calculations of co-morbid diagnoses revealed 38.0% (n = 128) of patients had a secondary diagnosis of substance use disorder, 6.5% (n = 22) had an alternative co-morbid disorder and 55.5% (n = 187) had no co-morbid diagnosis.

Measures

The measures used in the current study are routinely administered as part of the national outcome assessment protocols in mental health by the hospital. They are used for the purposes of planning and monitoring patient care as well as measuring outcome. Patients have the right to refuse to complete any specific measures.

The effectiveness of the program was assessed by comparing patient admission and discharge scores on the measures. These measures were the same as those used in the study by Murugesan et al. (2007) and were as follows: the Brief Psychiatric Rating Scale-Expanded version (BPRS-E; Lukoff et al., 1986), the Health of the Nation Outcome Scales (HoNOS; Wing et al., 1998) and the Kessler 10 (K10; Kessler et al., 2002).

The BPRS-E is a 24-item measure of psychiatric symptomatology that involves a structured interview process. It can be divided into four subscales: thought disturbance, anergia, affect and disorganization (Lukoff et al., 1986). Items are scored on a rating scale from 1 (not present) to 7 (extremely severe) and then a total score is calculated. The BPRS-E has good inter-rater reliability (Ventura et al., 1993).

The HoNOS is a clinician-completed 12-item measure of psychosocial functioning. It covers four domains: behaviour (e.g. aggression), impairment (e.g. cognitive dysfunction), symptoms (e.g. hallucinations) and social functioning (includes problems with relationships and living conditions) (Wing et al., 1998). Scores for each item range from 0 (no problem) to 4 (severe to very severe problem). The HoNOS has good reliability, with an internal consistency of 0.75 (Trauer, 1999).

The K10 is a 10-item self-report measure of psychological symptom distress (Kessler et al., 2003). Respondents are asked: ‘in the past 30 days, about how often did you feel . . .’ followed by various symptoms (e.g. ‘depressed’ and ‘nervous’). Respondents rate each item on a 5-point Likert scale which ranges from ‘none of the time’ to ‘all of the time’. The K10 has satisfactory internal reliability as measured by Cronbach’s coefficient (α = 0.93) (Kessler et al., 2003).

Procedure

The research received ethical review and approval from the University of Wollongong Human Research Ethics Committee. Outcome measures were administered at admission, discharge and 3-month reviews. The HoNOS was administered by nursing staff. The K10 was completed by the patients. Psychologists working at both units administered the BPRS-E.

Where there was some missing data but over 50% of the scale items were completed, a pro-rated score was calculated (proportional average of the available items). Where less than 50% of items were completed for any measure, these data were excluded from the analyses. This resulted in six cases being excluded.

Analytical strategy

Paired samples t-tests determined statistically significant changes over time. Consistent with the previous study (Murugesan et al., 2007), where discharge data were not available, scores from the most recent review prior to discharge were used instead. Means and standard deviations were calculated for admission and the review/discharge total scores on each measure.

Jacobson and Truax’s (1991) method was used to calculate clinically significant change. This method involves calculating reliable change indices (RCIs) using the formula proposed by Christensen and Mendoza (1986):

RCI = 1.96 \times {SE}_{diff}

According to Jacobson and Truax (1991), for reliable change to occur, a patient’s difference in total score from admission to discharge had to be equal to or greater than the RCI. The SE of measurement of a difference is calculated as:

{SE}_{diff} = {SD}_{1} \times \sqrt{2} \times \sqrt{(1 - α)}

SD₁ is the standard deviation of the admission total and α is Cronbach’s coefficient. Percentages that make reliable change (improvement or deterioration) were calculated using these methods. To calculate clinical change, three different cut-off points needed to be calculated. Clinical change is said to occur if an individual’s post-treatment score meets the following cut-off criteria:

Cut-off 1: More than two SDs away from the dysfunctional population mean.

Cut-off 2: Within two SDs of the functional population mean.

Cut-off 3: Closer to the functional population mean than the dysfunctional population mean.

Clinical significance cut-off scores were calculated for the three outcome measures, using the Jacobson and Truax’s (1991) methods. Cut-off 2 and 3 were the most appropriate for the current sample and enabled comparisons with Murugesan et al. (2007). The following formula was used to calculate cut-off 3:

C u t - o f f 3 = \frac{(m e a n c l i n \times S D n o r m) + (m e a n n o r m \times S D c l i n)}{(S D n o r m + S D c l i n)}

Mean_clin and mean_norm are the mean scores of the dysfunctional population and the functional population. A more functional population was considered to be a group with similar diagnostic characteristics who were able to reside in the community. Functional population normative data for the BPRS-E were taken from a study which included 43 patients from a community mental health centre and outpatient clinic (Janssen et al., 2003).

Functional population normative data for the HoNOS were taken from a study which included a subclinical sample of 1669 patients who came into contact with a community mental health service (Parabiaghi et al., 2005). Patients were considered subclinical if they had a score < 2 for all HoNOS items. Normative data used for the BPRS-E and HoNOS were the same as the previous study by Murugesan et al. (2007). For the K10, normative data from a large representative sample (n = 10,641) of individuals living in the Australian community were used (Andrews and Slade, 2001).

Following calculation of clinically significant improvement, binomial logistic regression analyses were conducted using the outcome measures, with the dichotomous dependent variables of ‘reliable and clinically significantly improved’ versus ‘not improved’. Those in the ‘not improved’ category included those who made no change and the few patients who also had RCS deterioration. Patients who were asymptomatic at baseline were not included in regression analyses since they were, by definition, not able to make clinically significant improvement (Bowersox et al., 2009). Age, gender (treatment unit), length of stay, primary diagnosis and co-morbid diagnosis were provided for each patient and comprised the independent variables.

Since the goal was to predict those that showed RCS improvement, we chose cut-off points that produced the largest proportion of improved participants. This was cut-off 2 for the BPRS-E and HoNOS and cut-off 3 for the K10. Primary diagnosis was categorized into three groups: schizophrenia, schizo-affective disorder, and other diagnoses. Co-morbid diagnosis was comprised of three groups, ‘no co-morbid diagnosis’, ‘those with a secondary diagnosis of substance use disorder (drug and/or alcohol)’, and ‘other co-morbid disorders’.

Results

Statistical significance analyses

Paired sample t-tests were computed, comparing means of admission and discharge scores for all outcome measures (see Table 1). Across all three measures, overall patients demonstrated significant improvement. There were variations in sample sizes for the different outcome measures. The main reason the BPRS-E was not completed was due to the unavailability of a psychologist to conduct interviews at the appropriate time. For the HoNOS, non-completion was due to being overlooked by staff. For the K10, this was due to patient refusal.

Table 1.

Statistically significant changes in outcome measures from admission to discharge.

Measure	n^a (%)	Admission M (SD)	Discharge M (SD)	t
BPRS-E	173 (51.3)	70.38 (16.77)	52.32 (13.35)	16.34***

HoNOS	329 (97.6)	13.49 (5.96)	6.51 (4.27)	18.98***

K10	258 (76.6)	20.90 (8.81)	16.51 (6.84)	8.67***

Number of complete admission and most recent review/discharge data sets available for analysis. BPRS-E: Brief Psychiatric Rating Scale-Expanded; HoNOS: Health of the Nation Outcome Scales; K10: Kessler [Psychological Distress Scale] 10. ***p < 0.001.

Predicting RCS change

Psychiatric symptomatology (BPRS-E)

There were 173 complete admission and discharge data sets available for analysis of the BPRS-E. The RCI was 17.8, rounded so that an 18-point change reflected that reliable change had occurred at the 95% confidence level. Based on this, 84 patients (48.6%) were designated as reliably improved and no patients showed reliable deterioration. The remaining participants showed no reliable change.

The BPRS-E normative sample mean was 39. Cut-off 2 was calculated as 59. RCS improvement was made by 62 patients (35.8%). Cut-off 3 was calculated as 52. RCS improvement was made by 56 patients (32.4%). No patient showed RCS deterioration, regardless of the cut-off method used.

For the logistic regression, cut-off 2 was selected and the following groups were obtained: those who made RCS improvement (n = 62) and those who did not (n = 67) (asymptomatic patients were not included in the logistic regressions). The full model containing all predictors was statistically significant, [χ²(n = 129) = 19.7, d.f. = 7, p < 0.05]. The model accounted for 18.9% (Nagelkerke R²) of the total variance, and correctly classified 69.0% of cases (74.6% correctly classified as no RCS improvement, 62.9% correctly classified as making RCS improvement). The estimates of the effects of the predictor variables calculated from the regression are summarized in Table 2. Primary diagnosis was a significant predictor and co-morbid diagnosis approached significance. For primary diagnosis, schizophrenia was the reference since this was the most frequent diagnosis. Those with schizo-affective disorder were 3.5 times more likely to be in the improved group compared to those with a primary diagnosis of schizophrenia [exp(β) = 3.52]. Although co-morbid diagnosis only approached significance, it suggested that those with co-morbid substance abuse disorders were over two times more likely to be in the improved group [exp(β) = 2.29].

Table 2.

Binary logistic regression predicting reliable and clinically significant (RCS) improvement on psychiatric symptomatology (BPRS-E cut-off 2).

Variable	β	SE of β	Exp(β) (OR)	95% CI
Age	−0.02	−0.02	0.98	(0.94, 1.02)

Gender^a	−0.61	0.41	0.54	(0.24, 1.22)

Length of stay (days)	−0.00	0.00	1.00	(0.99, 1.00)

Primary diagnosis^b
Schizo-affective disorder	1.26*	0.59	3.52	(1.12, 11.18)
Other diagnosis	0.39	0.86	1.47	(0.27, 7.95)

Co-morbid diagnosis^c
Substance abuse	0.83^	0.43	2.29	(0.99, 5.29)
Other disorders	0.01	0.77	1.01	(0.22, 4.60)

Constant	0.93	0.93	2.54

p < 0.05. ^ p = 0.053. The dependent variable is 0 = no RCS improvement, 1 = RCS improvement.

0 = female, 1 = male. ^bSchizophrenia is the reference category. ^cNo co-morbid diagnosis is the reference category.

OR: odds ratio; CI: confidence interval.

To check for potential differences between those included in the analysis (n = 173) and those who did not have pre/post BPRS-E measures (n = 164), we compared the groups using a series of chi-square tests. There were no significant differences between the groups for gender, age or co-morbid diagnosis (p > 0.05). There was a significant difference for primary diagnosis, [χ²(2, N = 337) = 7.18, p < 0.05]. The only difference was between those with a diagnosis of schizophrenia versus schizo-affective disorder, with a smaller proportion of those included in the analysis having a diagnosis of schizo-affective disorder (41%) than those excluded from the analysis (59%), [χ²(1, N = 299) = 4.78, p < 0.05].

Psychosocial functioning (HoNOS)

There were 329 complete admission and discharge data sets available for analysis for the HoNOS. The RCI for the HoNOS was 10.12 (rounded to 10). Using this RCI, a total of 100 patients (30.4%) made reliable improvement and three patients (0.9%) reliably deteriorated. The remaining participants showed no reliable change. Cut-off 2 was calculated as 6. RCS improvement was made by 75 patients (22.8%). RCS deterioration occurred for one patient. Using cut-off 3 (a score of 5), RCS improvement was made by 64 patients (19.5%). RCS deterioration occurred for one patient.

Cut-off 2 was selected for the logistic regression and the following groups were obtained: those who made RCS improvement (n = 75) and those who did not (n = 220). The full model was not significant and accounted for only 4.9% of variance [(Nagelkerke R²) χ²(n = 295) = 3.3, d.f. = 7, p > 0.05].

Psychological distress (K10)

There were 258 complete admission and discharge data sets available for analysis for the K10. The RCI on the K10 was 6.9 (rounded to 7). Using this RCI, 83 of the 258 patients (32.2%) were designated as reliably improved and 18 patients (7.0%) showed reliable deterioration. The remaining participants showed no reliable change. Cut-off 2 was calculated as 24.3. Given that this cut-off is higher than the mean of the admission scores of patients on the K10, this was not seen as a useful cut-off. Cut-off 3 was calculated as 16.6 (rounded to 17). RCS improvement was made by 52 of 258 patients (20.2%). RCS deterioration occurred for 12 patients (4.7%).

Cut-off 3 was selected for the logistic regression and the following groups were obtained: those who made RCS improvement (n = 52) and those who did not (n = 94). The full model was not significant and accounted for 9.3% of variance [(Nagelkerke R²) χ²(n = 146) = 10.2, d.f. = 7, p > 0.05].

Discussion

The RCIs suggest that between 32% and 49% of patients made an improvement. Further, between 20% and 32% of patients made RCS improvements across the three outcome measures (using cut-off 3). The RCS method is considered a conservative estimate of change (Clark et al., 2007) and in the context of the current study these rates of improvement may be an underestimate, given that the most recent 3-month review data were used when discharge ratings were not available (up to 18% of cases).

While deterioration was generally infrequent, it occurred most often on the psychological distress measure. Anecdotal observations suggest that for some patients this may occur as a result of increased stress associated with discharge and transition back into the community.

When compared to the Murugesan et al. (2007) study, which was conducted in the same treatment facility, there were similar rates of RCS improvement (range 21–33% using cut-off 3). Despite the current study having a larger and more diagnostically diverse group of patients, RCS improvement rates were very similar on the measures of psychiatric symptomatology and psychological distress. Rates of improvement have previously been reported on the HoNOS using the RCI for a large sample (n = 21,749) of adults attending public sector acute inpatient settings across Australia (Burgess et al., 2009). Using the same criteria as in our study for calculating RCI, it was found that 38% of admission to discharge pairs made improvement. In ambulatory settings, only 11.9% of the 10,354 admission to discharge pairs improved using the RCI. In our study the equivalent improvement rate using the RCI was 30.4%.

The use of the same performance indicators facilitates comparisons between areas. Whether this rate of improvement is ‘satisfactory’, likely depends on the perspective of different stakeholders. Individuals attending this inpatient setting have usually exhausted the capacity of public sector acute inpatient services. It is often these units that refer to the longer-stay facilities. Patients are clearly seen as being in need of further treatment (and supervision) which cannot be provided by other shorter-term services. Around 25% make clinically meaningful improvement, which potentially increases their capacity to live in less restrictive environments in the community. For these individuals and their families such outcomes are likely to be viewed positively. Whether this is ‘good value for money’ cannot be determined from our study. Such questions would ideally be addressed through randomized clinical trials where those referred to medium-term treatment are compared to other forms of rehabilitation (e.g. intensive outpatient services).

Sizeable groups appeared to be ‘asymptomatic’ on the measures at admission when compared to normative samples (HoNOS, 10%; BPRS-E, 25%; K10, 43%). As a result, these participants could not be included in regression analyses that predicted RCS improvements because by definition they cannot attain this change (Bowersox et al., 2009). Whether participants appear asymptomatic depends on the normative comparison group and the outcome variable. For example, the asymptomatic rate for the K10 is relatively high, but psychological distress is more of a state versus trait construct, and is relatively transient, thus ratings are likely to vary dependent on recent experiences. Many patients may not be experiencing acute levels of distress at the time they come into hospital and to some extent these ratings will depend on how long they have been in hospital before the intake assessment is completed.

In the case of the BPRS-E, the normative comparison group was a sample of patients still receiving treatment but on an outpatient basis in the community. Although they could be considered a more ‘functional’ group overall, they are still likely to be experiencing significant psychiatric symptoms. Thus, some in our inpatient sample are likely to appear ‘asymptomatic’ in comparison. It is also possible that some individuals are experiencing specific behavioural difficulties (e.g. self-harm) that mean overall ratings on the measures are not elevated (only specific items). Finally, it is possible that an individual could be considered asymptomatic on one measure, but not on another. Having low levels of severity in one domain (e.g. psychological distress) does not necessarily mean that there is good mental health overall.

In addition to determining rates of improvement, a question of equal or more importance is, ‘How might we begin to increase the proportion of people who improve?’ A step in this direction involves understanding which factors are related to improvement. Burgess et al. argue that: ‘Expectations of change will also vary within settings, depending on individual person-based factors such as diagnosis, level of severity of presenting symptoms and so on’ (2009: 537). Our study is one of the first to determine whether some of the person-based factors are related to RCS change among inpatients.

Regression analyses indicated that for psychiatric symptomatology, a significantly higher proportion of patients with schizo-affective disorder made RCS improvement relative to those with schizophrenia. This finding is consistent with other research indicating a more favourable prognosis for people with schizo-affective disorder (Walker et al., 2004). However, it is unclear why this was only found for psychiatric symptomatology and not for other outcome domains.

First, the relationships between the BPRS-E and other outcome measures at intake and discharge were small to moderate (r < 0.26). Thus, they appear to be capturing different outcome domains which could account for variable prediction. As noted, the transient nature of psychological distress (K10) may mean that prediction is less reliable using relatively static variables such as diagnosis.

With regard to psychosocial functioning, the HoNOS captures a broad range of behavioural and social domains but several of the items are unlikely to show any variability within an inpatient setting (e.g. problems with occupation and activities, problems with living conditions). A lack of variability and low sensitivity to change has led Audin et al. to conclude that the ‘HoNOS is severely limited in its ability to detect and record clinically meaningful change’ (2001: 565), at least in outpatient psychological treatment. All of these factors may have contributed to the difficulty in finding reliable predictors on these measures.

Given that some of the logistic regression analyses did not predict RCS change, further research should examine other potential predictors of RCS change. For example, there is some evidence to suggest that neurocognitive deficits may be related to outcome (Green et al., 2000). In addition, more dynamic treatment variables, such as medication adherence or participation in other treatment activities, may prove fruitful for predicting outcomes in the future.

The presence of a co-morbid diagnosis of substance abuse approached significance for predicting RCS changes in psychiatric symptomatology. The superior improvement rates for those with a co-morbid diagnosis of a substance use disorder are most likely a function of reduced access to alcohol and drugs as a result of being in residential treatment. The result is notable because treatment outcomes for those with co-morbid substance abuse disorders tend to be poor compared to those without co-morbidity (Turkington et al., 2009).

There is a need to determine whether these initial benefits while in residential care are able to be maintained and built on once individuals are discharged into the community or to less restrictive care. This is particularly important because rates of community-based treatment have been found to be lower in those with severe mental illness and co-occurring substance use disorder (Clark et al., 2007). Among substance abusing samples, a co-morbid diagnosis of schizophrenia is associated with lower rates of attendance at Alcoholics Anonymous meetings compared to those without schizophrenia (Tómasson and Vaglum, 1998).

In our study, one-third of those with a diagnosis of schizophrenia also had a co-morbid diagnosis of a substance use disorder. If individuals with severe mental illness and co-morbid substance use disorders are less likely to engage in outpatient treatment, they are potentially at higher risk of relapse. A review of relapse in drug and alcohol use among those with co-occurring mental health and substance use disorders found ‘no studies’ had examined such relapse among those diagnosed with schizophrenia spectrum disorders (Bradizza et al., 2006). Our more recent literature search also failed to find any studies that report these relapse rates.

The present study has several limitations. First, a considerable proportion of patients were considered asymptomatic according to the cut-off points, indicating a high overlap between ‘functional’ populations and the inpatient population. Although patients may have low distress scores, they can still have high symptom severity or functional deficits. They may be asymptomatic on one measure, but this does not automatically mean they have ‘good’ overall mental health. Second, while clinical significance methods have several advantages, a limitation of using this method is the lack of appropriate normative data (Lambert and Ogles, 2009). Third, it is difficult to evaluate the meaning of RCS rates without a control group or a comparison group, such as other forms of care.

Higher proportions of patients with schizo-affective disorders and those with co-morbid substance use disorders showed RCS improvements in psychiatric symptomatology compared to those with schizophrenia and those with no co-morbid diagnosis. This is encouraging, given the difficulty in treating co-morbidity in outpatient settings, but there is an urgent need to follow patients into the community to assess their engagement with treatments that address both their mental illness and substance abuse. Continued assessment of outcomes and drug and alcohol-related relapse would also help to inform treatment program development.

Footnotes

Acknowledgements

The support of Robyn Jeffrey in managing the data collection is greatly appreciated.

Funding

Western NSW Local Health District (WNSWLHD) provided a small grant to the University of Wollongong to support data entry in preparation for statistical analysis.

Declaration of interest

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.

References

Access Economics and SANE Australia (2002) Schizophrenia costs: An analysis of the burden of schizophrenia and related suicide in Australia. Melbourne, Australia: SANE.

Andrews

Slade

(2001) Interpreting scores on the Kessler Psychological Distress Scale (K10). Australian and New Zealand Journal of Public Health 25: 494–497.

Audin

Margison

Clark

. (2001) Value of HoNOS in assessing patient change in NHS psychotherapy and psychological treatment services. British Journal of Psychiatry 178: 561–566.

Bowersox

Saunders

Wojcik

(2009) An evaluation of the utility of statistical versus clinical significance in determining improvement in alcohol and other drug (AOD) treatment in correctional settings. Alcoholism Treatment Quarterly 27: 113–129.

Bradizza

Stasiewicz

Paas

(2006) Relapse to alcohol and drug use among individuals diagnosed with co-occurring mental health and substance use disorders: a review. Clinical Psychology Review 26: 162–178.

Burgess

Pirkis

Coombs

(2009) Modelling candidate effectiveness indicators for mental health services. Australian and New Zealand Journal of Psychiatry 43: 531–538.

Christensen

Mendoza

(1986) A method for assessing change in a single subject: An alteration of the RC index. Behavior Therapy 17: 305#x02013;308.

Clark

Samnaliev

McGovern

(2007) Treatment for co-occurring mental and substance use disorders in five state Medicaid programs. Psychiatric Services 58: 942–948.

Glick

Clarkin

Haas

. (1993) Clinical significance of inpatient family intervention: conclusions from a clinical trial. Hospital & Community Psychiatry 44: 869–873.

10.

Green

Kern

Braff

. (2000) Neurocognitive deficits and functional outcome in schizophrenia: are we measuring the “right stuff”? Schizophrenia Bulletin 26: 119–136.

11.

Grella

Stein

(2006) Impact of program services on treatment outcomes of patients with comorbid mental and substance use disorders. Psychiatric Services 57: 1007–1015.

12.

Huxley

Rendall

Sederer

(2000) Psychosocial treatments in schizophrenia: a review of the past 20 years. The Journal of Nervous and Mental Disease 188: 187–201.

13.

Jacobson

Truax

(1991) Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology 59: 12–19.

14.

Janssen

Krabbendam

Jolles

. (2003) Alterations in theory of mind in patients with schizophrenia and non-psychotic relatives. Acta Psychiatrica Scandinavica 108: 110–117.

15.

Johnson

Dow

Lynch

. (2006) Measuring clinical significance in rehabilitation research. Rehabilitation Counseling Bulletin 50: 35–45.

16.

Kessler

Andrews

Colpe

. (2002) Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychological Medicine 32: 959–976.

17.

Kessler

Barker

PR.

Colpe

. (2003) Screening for serious mental illness in the general population. Archives of General Psychiatry 60: 184–189.

18.

Kurtz

Mueser

(2008) A meta-analysis of controlled research on social skills training for schizophrenia. Journal of Consulting and Clinical Psychology 76: 491–504.

19.

Lambert

Ogles

(2009) Using clinical significance in psychotherapy outcome research: the need for a common procedure and validity data. Psychotherapy Research: Journal of the Society for Psychotherapy Research 19: 493–501.

20.

Lukoff

Nuechterlein

Ventura

(1986) Manual for the expanded BPRS. Schizophrenia Bulletin 12: 594–602.

21.

Parabiaghi

Barbato

D’Avanzo

. (2005) Assessing reliable and clinically significant change on Health of the Nation Outcome Scales: method for displaying longitudinal data. Australian and New Zealand Journal of Psychiatry 39: 719–725.

22.

Murugesan

Jeffrey

Deane

. (2007) Inpatient psychosocial rehabilitation in rural NSW: assessment of clinically significant change for people with severe mental illness. Australian and New Zealand Journal of Psychiatry 41: 343–350.

23.

Newnham

Harwood

Page

(2007) Evaluating the clinical significance of responses by psychiatric inpatients to the mental health subscales of the SF-36. Journal of Affective Disorders 98: 91–97.

24.

Schennach-Wolff

Jäger

Seemüller

. (2009) Defining and predicting functional outcome in schizophrenia and schizophrenia spectrum disorders. Schizophrenia Research 113, 210–217.

25.

Tómasson

Vaglum

(1998) Psychiatric co-morbidity and aftercare among alcoholics: a prospective study of a nationwide representative sample. Addiction 93: 423–431.

26.

Trauer

(1999) The subscale structure of the Health of the Nation Outcome Scales (HoNOS). Journal of Mental Health 8: 499–509.

27.

Turkington

Mulholland

Rushe

. (2009) Impact of persistent substance misuse on 1-year outcome in first-episode psychosis. British Journal of Psychiatry 195: 242–248.

28.

Ventura

Green

Shaner

. (1993) Training and quality assurance with the Brief Psychiatric Rating Scale: ‘The drift busters’. International Journal of Methods in Psychiatric Research 3: 221–244.

29.

Walker

Kestler

Bollini

. (2004) Schizophrenia: etiology and course. Annual Review of Psychology 55: 401–430.

30.

Wing

Beevor

Curtis

. (1998) Health of the Nation Outcome Scales (HoNOS). Research and development. British Journal of Psychiatry 172: 11–18.