Abstract
Demonstrating the effectiveness of mental health services is a priority identified by the Australian Commonwealth Government in its National Mental Health Strategy [1]. In an attempt to promote an evidence-based approach to service delivery, the health department in Queensland has introduced a data collection procedure in which a battery of assessment instruments are routinely administered at intake and discharge with all clients of child and youth mental health facilities throughout the state [2]. The assessment package includes the Health of the Nation Outcome Scale – Child and Adolescent (HoNOSCA) [3], the Children's Global Assessment Scales (CGAS) [4] and the Strengths and Difficulties Questionnaire (SDQ) [5]. The success of these measures in providing meaningful information for evaluating the impact of child and youth mental health services is yet to be determined. In the present paper we report on the results of a trial in which the HoNOSCA was routinely administered to monitor the progress of consumers in an adolescent mental health in-patient setting.
The HoNOSCA measures a range of child and adolescent relevant problems, including aggressive behaviour, self-harm, substance misuse, academic problems, psychotic symptoms, and emotional and relationship problems. At least five studies have investigated the feasibility, acceptability, reliability and validity of the HoNOSCA [3],[6–9]. Generally, the HoNOSCA has been found to be a feasible instrument for clinicians to administer across a range of settings, to be reliable, have good temporal stability across 1 week [9] and to have moderate to good interrater reliability [3], [7], [8]. Evidence of convergent validity has been found, with studies showing moderate to strong correlations between HoNOSCA scores at intake and the complexity of patients' presenting problems and circumstances as measured by the Paddington Complexity Scale (PCS) [6], clinician-rated CGAS score [6] and adolescents' self-reported SDQ scores [9], but correlates less strongly with parent-rated measures of child functioning, such as the SDQ [6].
Although there are relatively few studies looking at the relationship between HoNOSCA score and patients' demographics, there appears to be some differences in HoNOSCA score across treatment settings (outpatient vs in-patient), age groups and gender. Intake HoNOSCA scores for in-patient samples are typically higher than those for outpatient samples [3], [9]. Adolescents tend to score higher than children on suicide ideation, selfinjury, substance abuse and depressive symptoms [3], [8]. Boys have been reported to score higher than girls on disruptive/aggressive behaviour, over-activity and problems with peers, while girls score higher on self-injury, emotional problems, and problems with family members [8]. There is some evidence that HoNOSCA score at intake is predictive of eventual length in treatment, with a moderate correlation between intake score and the number of subsequent treatment sessions attended [7]. HoNOSCA scores have been found to decrease approximately three points at 3 months and almost four points at 6 months in outpatient samples [7], [8] and correlate with clinicians' global impressions of change in functioning [3], [7], [8].
All these studies, with the exception of Gowers et al. [9] have used combined samples of children and adolescents, and either outpatient or mixed in-patient and outpatient samples. Given the differences between children and adolescents, and between in-patients and outpatients, further research is needed on the reliability, validity, and sensitivity to change of the HoNOSCA in a more homogeneous, adolescent-only, in-patient setting [10]. The primary aims of the present study were to: (i) provide preliminary data on the reliability and validity of the HoNOSCA in an adolescent in-patient setting; (ii) compare the HoNOSCA scores of an adolescent psychiatric in-patient sample with reported data on inpatient and outpatient samples; and (iii) report descriptions of the current sample, including changes shown on the HoNOSCA over the course of their admission and the association between HoNOSCA and age, gender and length of stay at the unit. Specific hypotheses were that: (i) HoNOSCA scores would remain relatively stable over a 2-week period of admission (test-retest reliability); (ii) HoNOSCA scores would correlate positively with other measures of psychosocial functioning (convergent validity) and the actual behaviour of consumers during admission (concurrent validity); (iii) consumers in the current inpatient sample would show higher intake HoNOSCA scores than intake scores for outpatient samples; and (iv) that, on average, HoNOSCA scores would decrease over the period of admission (assessed at 3 and 6 months). Given the limited number of previous studies investigating age, gender or length of stay, there were no specific hypotheses regarding age differences, gender differences or length of stay and HoNOSCA score at intake or over time.
Method
Sample
The sample consisted of 51 adolescents admitted to an adolescent psychiatric inpatient unit in south-east Queensland between August 2000 and October 2002. The adolescents had a mean age of 15.05 years at admission (range 12–17) and 41% were female. There was no significant difference in age between boys (mean = 15.87) and girls (mean = 15.57, t (49) = −0.42, ns). The sample was characterized by a range of psychiatric problems including early onset psychosis, depression, anxiety disorders and conduct disorder. Average length of stay at the unit during the course of the study was 24.28 weeks (range 1–81 weeks). On average, intake HoNOSCA scores were obtained 3.1 weeks from the time of admission.
Setting
The unit is a 15-bed, in-patient facility for adolescents that accepts referrals from throughout Queensland, Australia. The unit is a medium to long-term facility with admissions generally lasting at least 3 months and sometimes for more than 12 months. Patients typically present with severe, complex, long-standing and often treatmentresistant problems, which require intensive, coordinated programs of intervention in order to reduce symptoms and improve functioning. The unit includes a school and therapeutic program delivered by a multidisciplinary team, including teachers, nursing staff, occupational therapist, speech pathologist, social worker, psychologists and psychiatrists.
Measures
Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA)
The HoNOSCA [3] is a 15-item clinician-rated scale. Ratings of a participants' functioning over the preceding 2 weeks were made on a 5-point scale reflecting severity of problems from no problem (0) to severe problem (4). A total HoNOSCA score is calculated by summing the first 13 items (items 14 and 15, which reflect parent understanding and knowledge are not calculated in the total HoNOSCA score [3]). It has been suggested that the HoNOSCA derives four subscale scores: (i) a Behaviour score, which assesses disruptive behaviour, inattention, overactivity, self-injury and substance misuse; (ii) an Impairment score assessing scholastic impairment, physical illness and disability; (iii) a Symptoms score, which taps psychotic, somatic, and emotional symptoms; and (iv) a Social score, which measures problems with peer and family relationships, poor self-care, and school attendance. Sub-scale scores are calculated by summing items loading on each scale (see [3]).
Paddington Complexity Scale (PCS)
The PCS [6] is a 16-item measure of clinical and environmental complexity, and focuses on three problem areas: psychiatric, physical/ developmental and environmental problems. It is a clinician-rated instrument and can be rated retrospectively by a range of professionals in a multidisciplinary team. The PCS derives a total score by summing items. Total score can range from 0 to 32 with higher scores indicating greater severity and complexity of the problem.
Clinician Rated Change Scale
A seven-point perceived patient change scale described by Brann et al. [8] was used to provide a retrospective clinician rating of global change. This scale was rated from ‘much worse’ to ‘much better’.
Critical incidents
The average number of critical incidents per month for each patient was recorded. Critical incidents include instances of self-harm, aggression and harm to others, unauthorized absences, accidents and injuries occurring within the unit.
Procedure
All members of the unit's multidisciplinary team were trained in the use of the HoNOSCA. Each patient was rated monthly by a multidisciplinary team at weekly case conferences. A psychologist, trained in its use, completed the PCS for each patient retrospectively after reviewing the patient's psychiatric record. A second psychologist rated 43% of the patients on the PCS to assess interrater reliability. Intraclass correlation coefficients (ICCs) used to assess interrater reliability of PCS scores found a single measure ICC of 0.96 across the two raters, indicating a high level of reliability. To obtain a measure of global change, the unit's psychiatrist, individual therapists (typically clinical psychologists), teachers, and nursing staff were asked to provide a retrospective rating of global change for each patient in their care. The mean rating across informants was used as a summary measure of global change.
Results
Reliability and validity
Test-retest
Test-retest validity was calculated from the initial HoNOSCA score and a subsequent score provided 2- to 4-weeks later. It was expected that within an in-patient sample that functioning would remain relatively stable once the adolescents had had a period to settle into the unit. HoNOSCA scores were found to be stable over this period (r = 0.80, p < 0.001).
Internal reliability
The HoNOSCA is not considered to be a unidimensional scale, with the initial study finding low intercorrelations between items [3]. The current study similarly found poor internal consistency (Cronbach alpha = 0.45). Further, the internal validity of the four subscales suggested in the original validation study (Behaviour, Impairment, Symptoms and Social) was also poor, ranging from −0.10 for Symptoms to 0.48 for Social [3]. Accordingly, only total HoNOSCA score and individual item scores were used in subsequent analyses.
Convergent validity
Convergent validity was assessed by comparing scores on the HoNOSCA and PCS. There was a moderately strong positive correlation between total HoNOSCA and PCS (r = 0.46, p < 0.01) showing higher HoNOSCA scores at intake were associated with more complex presenting problems. The mean PCS scores at intake (11.80, SD = 3.95) were associated with higher scores on several items of the HoNOSCA at intake, including overactivity/aggression (r = 0.66, p < 0.001, one-tail), scholastic/language problems (r = 0.26, p < 0.05, one-tail), psychotic symptoms (r = 0.25, p < 0.05, one-tail), peer relationships (r = 0.33, p < 0.05, one-tail) and self-care and relationship problems (r = 0.33, p < 0.01, one-tail).
Concurrent validity
Evidence of concurrent validity was demonstrated by the finding that the mean number of critical incidents patients were involved in each per month was positively associated with the intake HoNOSCA score (r = 0.34, p < 0.05). At the individual item level, there was a positive correlation between number of incidents and disruptive/aggressive behaviour (r = 0.28, p < 0.05), nonaccidental self-injury (r = 0.43, p < 0.01), and drug and alcohol difficulties (r = 0.28, p < 0.05). Further evidence of concurrent validity was shown by a positive association between changes in HoNOSCA scores over the course of admission and clinicians' retrospective reports of change. Specifically, mean ratings of patient's global change between intake and discharge or between intake and the time of the study for participants who had not been discharged, was 5.12 (SD = 0.82). The mean change in HoNOSCA scores within the subsample of patients clinicians rated as having improved, was 1.96 at 3 months and 3.40 at 6 months, while patients considered not to have changed had HoNOSCA change scores of 0.76 at 3 months and −0.30 at 6 months.
Analysis of HONOSCA scores
Intake ratings
The mean HoNOSCA score at admission was 15.86 (SD = 5.45). Scores ranged between 5 and 28. Scores on individual items for the current sample and the Australian outpatient sample (for comparison) are presented in Table 1. One patient in the current study was missing individual item scores and therefore was not included.
There was no significant association between age at admission with total HoNOSCA score at intake (r = −0.08, ns). Younger patients showed greater disruptive/aggressive behaviour than older patients (r = −0.40, p < 0.01). No significant difference was found between female (mean = 15.57, SD = 5.74) and male patients [mean = 16.07, SD = 5.33; t(49) = 0.32, ns] on total HoNOSCA scores at intake. On individual items boys scored significantly higher on disruptive/ aggressive behaviour (1.77 vs. 85, t(48) = 2.56, p < 0.05) and scholastic or language difficulties than girls [1.47 vs 50, t (48) = 2.65, p < 0.01].
Mean item scores for Australian outpatient sample and inpatient sample at intake, 3 months and 6 months post-intake
Item scores for outpatient sample come from Brann et al. (2001); † significant difference between inpatient and outpatient samples, p < 0.05; ‡trend towards significance between inpatient and outpatient samples, p = 0.06; §significant difference between intake and 3 month score, p < 0.05 (one-tail); ¶significant difference between intake and 6 month score, p < 0.05 (one-tail).
Sensitivity to change at 3 months
Change scores were calculated by subtracting HoNOSCA scores recorded closest to 3 months, or for current patients, the most recent HoNOSCA score, from the intake HoNOSCA score. Positive change scores indicated improved functioning, while negative change scores indicated deterioration. Given the directional hypothesis of overall improvement in HoNOSCA score over time, change score analyses were performed using one-tail tests. Forty-five patients (47% female) had HoNOSCA scores at both intake and 3-months (mean number of weeks from admission was 11.69 weeks, SD = 2.70). The mean HoNOSCA change score from intake was 1.51 (9.5% decrease; range −9–12), which represented a statistically significant improvement in functioning [t (44) = 2.23, p < 0.05]. On individual items a significant decrease in family life and relationship problems was found at 3 months [t(44) = 3.72, p < 0.001]. Younger patients showed greater improvement than older patients on scholastic/language problems (r = −0.31, p < 0.05) at 3 months. There was a trend for older patients to display more improvement than younger patients on psychotic symptoms at 3 months (r = 0.27, p = 0.08). There was a significant time by gender interaction on HoNOSCA total score at 3 months, with boys showing a greater improvement (mean improvement of 2.79 points) at this time than girls [mean improvement of 0.05 points; F1,43 = 4.39, p < 0.05].
There was a moderately strong negative correlation between HoNOSCA change scores at 3 months and PCS scores (r = −0.47, p < 0.01), indicating that less complex cases improved more than more complex cases. In particular, participants showing less complex presenting problems showed greater improvement in emotional problems (r = −0.44, p < 0.01) and psychotic symptoms (r = 0.32, p < 0.05). There was a trend for participants showing less complex presenting problems at intake to show improvement in levels of disruptive/ aggressive behaviour at 3 months (r = −0.29, p = 0.06).
Sensitivity to change at 6 months
A 6-month change score was calculated for patients who stayed at least 20 weeks at the unit. Twenty-five adolescents (60% female) had scores at this time point. The mean change in scores over this 6-month period (mean = 23.8 weeks, SD = 2.06) was 1.92 (12% improvement; range −7–16) and approached significance [t (24) = 1.67, p = 0.06, one-tail]. On individual items there was significant improvement in psychotic symptoms [t (23) = 1.89, p < 0.05, one-tail] and family and relationship problems [t (23) = 1.90, p < 0.05, one-tail]. Older patients showed greater improvement than younger patients on psychotic symptoms (r = 0.39, p = 0.06) and displayed more improvement in overactivity/aggressive behaviour (r = 0.41, p < 0.05) and family life/ relationship difficulties (r = 0.41, p < 0.05) at 6 months. There was no significant difference between boys and girls in change scores at 6 months (2.50 vs 1.53 improvement, respectively).
There was no association between the complexity of a participant's presenting problems on the PCS and total HoNOSCA change score at 6 months (r = −0.05, ns). For individual HoNOSCA items participants showing less complex presenting problems showed improvement in scholastic/language difficulties (r = −0.53, p < 0.01) and improved school attendance (r = −0.48, p < 0.05).
Length of stay
Length of stay was not associated with total HoNOSCA score (r = 0.03, ns) or individual item scores at intake. HoNOSCA change scores at 3 and 6 months were negatively correlated with length of stay (r = −0.36 at 3 months; r = −0.35 at 6 months), showing that, as would be expected, patients who improved more (as rated by the HoNOSCA) stayed at the unit for a shorter time. Longer stays in the unit were associated with higher levels of nonaccidental self-injury (r = −0.34, p < 0.05) and less improvement in self-care and relationship problems (r = −0.44, p < 0.05). There was a trend for continuing problems with peer relationship to be associated with a longer stay on the unit (r = −0.37, p = 0.07). Girls were found to remain at the unit longer than boys [35.12 weeks vs. 17.06 weeks, respectively; t(48) = −3.55, p < 0.001]. Patients who stayed longest on the unit displayed greater variability in their HoNOSCA scores over time. A ‘variability’ score, calculated for each patient by subtracting their lowest HoNOSCA score during their time in the unit from their highest HoNOSCA score, was strongly positively correlated with length of time spent in the unit (r = 0.68, p < 0.001).
Discussion
The aims of the present study were to provide preliminary data on the reliability and validity of the HoNOSCA in an adolescent in-patient setting; to compare intake and HoNOSCA change scores of an adolescent psychiatric in-patient sample with reported data on outpatient samples; and to report the relationship between HoNOSCA scores and age, gender and length of stay. The results of analyses used to determine the reliability and validity of the HoNOSCA were favourable. Test-retest reliability was found to be acceptable although internal consistency for the total scale and subscales was poor. This is not surprising given the instrument taps a number of independent psychosocial and psychiatric domains. Thus, while total HoNOSCA score may be a useful indicator of global functioning, individual items rather than subscales appear to be better indicators of functioning in specific domains.
The moderate correlations between intake HoNOSCA score and PCS provides support for the convergent validity of the HoNOSCA. Further support for concurrent validity was the finding that participants with higher HoNOSCA intake scores were involved in more altercations with unit staff, incidents of self-harming behaviour, and absconding from the unit. Not surprisingly, higher scores on disruptive/aggressive behaviour, self-harm, and drug and alcohol problems at intake were associated with greater numbers of incidents.
Although these relationships provide support for the validity of HoNOSCA scores at intake, for an outcome measure to be accepted by clinicians, there needs to be an association between changes on the HoNOSCA and change in the patients as perceived by the clinicians themselves [8]. In the present study agreement was found between changes in the HoNOSCA scores over three and 6 months, and clinician's global ratings of change. It should be noted, however, that not one patient was considered to have deteriorated during their stay at the unit by the clinicians. This result is similar to Brann et al.'s [8] findings that only three outpatients in their sample were rated by clinicians as having deteriorated in functioning over time. It could be speculated that while there is agreement between global ratings and a more objective rating scale, clinicians are reluctant to report that a patient in their care has deteriorated and that the HoNOSCA provides a more objective account of a patient's progress. However, an alternative explanation may be that clinicians note specific changes in aspects of the individual's functioning not measured by the HoNOSCA. Generally though, HoNOSCA scores appear to be valid and reliable indicators of patients global functioning.
The present study found lower intake HoNOSCA scores at intake than would be expected in the light of previous studies. Specifically, the mean HoNOSCA score at intake was similar to that reported for adolescents in an Australian outpatient setting (15.21, SD = 6.66) [8] and lower than Gowers et al.'s [9] British adolescent in-patient sample (19.6, no SD provided). However, in comparison with the Australian outpatient adolescents [8], the current sample had more severe levels of overactivity and attention difficulties, peer relationship difficulties, family life and relationship problems, and greater levels of psychotic symptoms, while the outpatient sample were characterized by more severe drug and alcohol problems and poorer school attendance than the current sample. Given that the HoNOSCA was rated for the prior 2 weeks, it is possible that the on-site school and limited access to drug and alcohol on the unit may have been responsible for lower scores on these items. Additionally, as the adolescents in this sample had generally been referred after a period in an acute inpatient unit for medium to long-term treatment, their symptomatology may have settled somewhat prior to admission. Finally, it was noted that family and peer relationship problems were high in this sample and that being removed from these environmental stressors may have contributed to lower scores on the HoNOSCA at intake. In addition, the adolscents may not have developed sufficiently trusting relationships with staff to reveal the extent of their psychopathology. Importantly, the relatively low baseline HoNOSCA scores may, at least in part, explain the limited change in HoNOSCA scores noted at 3 months compared with outpatient samples [8] and the British in-patient sample [9]. It should be noted that while the HoNOSCA intake scores were relatively low, PCS scores at intake in the current study were higher in the present study compared to scores reported for the outpatient sample [6], indicating the presenting problems of the current sample were more complex.
The greatest improvement within the present sample appeared to be the patients' relationships with their family at 3 months; an improvement that remained stable at 6 months. By 6 months, there was a significant improvement in psychotic symptoms. Lack of improvement in other domains may be explained in part by (i) the small number of patients in the sample, especially at 6 months, thereby reducing the power to detect change; and (ii) scores at 6 months reflecting the functioning of longer-term patients who showed less improvement and greater variability in functioning. Indeed, in the present study HoNOSCA scores fluctuated considerably for patients staying at least 6 months at the unit. The presenting problems for these adolescents included predominantly psychotic symptoms and self-harming behaviour that justified ongoing management to contain these symptoms. Outcome evaluations using the HoNOSCA need to acknowledge that simply comparing pre- and postintervention differences will obscure the important contribution of a service in stabilizing the functioning of these individuals. In order to determine the impact of a service in stabilizing fluctuating patterns of functioning, regular monthly administration of the HoNOSCA would be recommended. Further the variability in scores over time means postintervention measures for long-term patients cannot be taken as a reliable measure of functioning, limiting the usefulness of the HoNOSCA as an outcome measure for individuals displaying psychotic and self-harming behaviour.
Little association between total HoNOSCA score and age at admission was found, although younger patients showed greater problems with disruptive and aggressive behaviour. These findings are somewhat disparate from previous studies that found younger patients had greater problems with overactivity/inattention whilst older patients had more severe emotional problems, greater substance abuse, psychotic symptoms and poorer school attendance than younger patients [3], [8]. The discrepancy in results between studies may be due to characteristics of the current sample that consisted of only adolescents, rather than including both children and adolescents. There was some degree of association, however, between age and change on HoNOSCA score, with older patients showing greater improvement over the course of their admission, especially in psychotic symptomatology, family life/relationships and disruptive/ aggressive behaviour.
There was no gender difference in total HoNOSCA score at intake, although boys showed greater levels of disruptive/aggressive behaviour and scholastic/language problems than girls. Although not significant, there was a tendency for boys to show higher levels of overactivity/ inattention than girls and for girls to have greater emotional problems and display more self-harm than boys. This is similar to Brann et al.'s outpatient sample [8], however, unlike Brann et al. there was no difference between boys and girls in substance misuse, psychotic symptomatology, family life/relationships or school attendance.
Despite the lack of difference in total HoNOSCA score at intake, boys showed more rapid improvement in global functioning than girls. There were no gender differences in the degree of change for adolescents who stayed 6 months or more. Although HoNOSCA score at intake was not predictive of eventual length of stay in the unit, adolescents who stayed in the unit longer showed less improvement than those who stayed for a relatively short-term. This is contrary to Garralda et al. [7] who found greater improvement in HoNOSCA score at 6 months for those outpatients who attended a greater number of sessions over this time period. The disparate findings may well reflect differences between in- and outpatient programs. Whilst increased attendance to outpatient therapy would be expected to be positively associated with improved psychosocial functioning, inpatients showing improved functioning are more likely to be discharged from the unit thereby, showing a negative association between length of stay and HoNOSCA change score.
Methodological issues concerning the present study should be highlighted. Notably, the sample size of the present study over the 6-month period was small, and the HoNOSCA was not compared with other outcome measures currently used in adolescent settings such as the CGAS or the SDQ. Further, the unit under study was a specialized unit that admits patients who have generally benefited minimally from traditional outpatient treatment programs and acute inpatient units.
In summary, the present study found support for the validity of the HoNOSCA and test-retest reliability, but not the internal reliability of the scale. The total HoNOSCA score provided evidence of high levels of mental health problems within the current in-patient sample. However, individual items rather than the total HoNOSCA score revealed important age, gender and length of stay differences. Similarly, while the total HoNOSCA score was sensitive to change, the total HoNOSCA score obscured important changes across gender, age and length of stay in specific domains of functioning over the course of admission. The study identified a subset of long-term patients suffering serious psychiatric conditions that showed fluctuating levels of change over the course of their admission. Successful outcomes for these individuals may be increased stability in functioning rather than improved functioning. However, further research is needed to clarify the variables that predict which individuals respond to in-patient treatment regime and which individuals would benefit from ongoing management of their mental health problems.