Abstract
The evaluation of outcomes in child psychiatric inpatient care is critical in maximizing the therapeutic impact of treatment. Determining which aspects of inpatient treatment are most and least helpful to clients can instigate changes to service delivery, which in turn, maximizes effectiveness of interventions. In addition, there is increasing financial pressure on both public and private health services to demonstrate the effectiveness of service provision through evaluation studies [1].
Despite the recognition of the need for increased programme effectiveness and accountability, there is a scarcity of research and evaluation efforts regarding child psychiatric inpatient treatment. Relatively few studies have systematically investigated the effects of short-term psychiatric inpatient treatment on children and families [2, 3]. Two of the most recent and frequently cited reviews of the literature describe only 29 [2] and 34 [4] outcome studies, respectively. Of the latter group of investigations only six used a prospective follow-up design, while the remainder were retrospective. The authors noted that differences in methodology, treatment strategies and measurement of outcomes made it difficult to draw conclusions from the available literature regarding the effectiveness of short-term inpatient treatment for children. Indeed, the literature describing psychiatric inpatient treatment of children is characterized by wide variability in most aspects of programme design and implementation. Programmes exist in a variety of settings and offer different levels of treatment intensity and restrictiveness [4].
Further to the difficulties encountered in the design of studies in psychiatric inpatient treatment of children, measurements have been based on single informant reports [4]. More recently, calls for the inclusion of a range of assessments from multiple informants have been made [5], therefore emphasizing the need to increase the sophistication of the outcomes measured.
One common feature of reported child inpatient interventions is the role played by the family in enhancing a child's response to treatment. Research suggests that psychiatric inpatient treatment of children is more likely to be effective when families are positively involved in the admission process, particularly when an effective therapeutic alliance between unit staff, the child and parents is established [4–6]. The family orientated treatment model described by the Finnish investigators, Sourander and Piha [7], is particularly relevant in this regard. This treatment model emphasizes the systematic integration of families in the therapeutic regime, and views the child, family, unit staff and the professional outpatient community as an ‘extended therapeutic system’. The implication of this finding is that current inpatient treatment services may benefit by emulating aspects of this therapeutic model.
Another consistent finding reported in the literature is that outpatient treatment after discharge substantially increases the likelihood of continued improvement in child behaviour and functioning [2, 4, 5, 8–11]. Several investigators have emphasized that short-term inpatient treatment of children may serve several important functions, including crisis intervention and assessment, but should always be viewed as an acute treatment strategy [7]. Integral to this view is the notion that extended, long-term outpatient treatment is inherently necessary for the effective treatment of serious psychiatric disorders in children.
The results of predictive investigations also have implications for the operation of child psychiatric inpatient units. Although some inconsistent findings have been reported in the literature concerning the effect of specific treatment variables on outcome [4, 11], researchers have identified child and family variables associated with both positive and less favourable outcomes at follow-up. Of particular relevance was the finding that children with severe undersocialized, aggressive conduct disorders, psychotic symptoms, or organic conditions, are less likely to respond favourably to inpatient treatment, than children with an emotional disorder. Thus, shortterm psychiatric inpatient treatment is not likely to be sufficient to ameliorate severe antisocial, aggressive behaviour in children [7]. Also, length of treatment, another characteristic of intervention, was not associated with outcome [4, 7], indicating that effective interventions for these children may not simply consist of extended inpatient treatment.
The aim of the present study was to assess the outcomes of brief (4–6 weeks) child and family admissions to the Statewide Child Inpatient Mental Health Unit (SCIMHU), using a range of standardized outcome measures. Current literature in the area highlights the need for additional research examining the outcome of short-term child psychiatric inpatient treatment. In this context, there was a critical need to conduct an outcome study of the SCIMHU to determine the therapeutic effectiveness of admissions to the Unit, and to assess changes in relevant child, parent and family variables over time.
The ‘context, input, process, product’ (CIPP) model of evaluation [12, 13] provided the conceptual framework for the study. The Product (i.e. change on key child, parent, and family variables) component of the model was particularly relevant to the present study. The objective of product evaluations is to ascertain the extent to which a programme has met the needs of the group for which it is designed.
Method
Design
Child, parent and family functioning outcomes were examined using a pre-post-follow-up design. Key variables were measured at three time points, namely (i) prior to admission; (ii) at discharge; and (iii) four months after discharge. Questionnaires were administered to the children and their families, as well as to their teachers and referrers to the Unit.
Description of the unit
The SCIMHU is a 12-bed inpatient facility for children aged up to 12 years with severe behavioural and emotional disturbances. The Unit provides residential facilities for the whole family if required. The Unit provides crisis, acute and short-term treatment, generally up to six weeks, with longer admissions where appropriate. Admissions are seen as a temporary phase to allow for stability to be established in the child's life. Children are required to attend the Unit from Monday to Friday, then return home for weekends with their parents or carers. As the admission progresses, children are reintegrated back with their external schools by ‘schooling out’ for one day per week. While the majority of admissions to the Unit are child admissions, accommodation is available for families, but is typically reserved for parents with younger children, where it would be developmentally inappropriate to separate the child from the parents.
The unit has two principal objectives: (1) to provide each child and family with a comprehensive assessment, depending on the individual needs and circumstances of each child and family; and (2) to achieve sufficient change in the short term, to enable effective treatment to occur in a less intensive outpatient setting. In addition to these objectives, and where appropriate, the unit aims to provide diagnostic clarification to parents and referrers, specialist educational and neuropsychological assessment, and a comprehensive discharge plan to help reintegrate and manage the child in the community. Treatment at the Unit is multimodal, encompassing a range of individual, parent, family, group, social, educational, psychological and physical therapeutic interventions chosen according to the individual needs of the child and the family, and implemented by a multidisciplinary team.
The treatment work conducted at the Unit occurs within the context of a therapeutic ‘milieu’, which refers to the physical and organizational structure, and interaction with staff of the Unit. The overarching aim of the milieu is to reduce the emotional and behavioural problems of children admitted and to provide for each child's personal and psychological needs through the provision of goal directed treatment. The therapeutic milieu consists of a variety of components, including the ward programme; parent and family therapy; individual child therapy; special school; and outpatient treatment and community integration.
Sample
A total of 84 families of children who were consecutively admitted to short-term (4–8 week) psychiatric inpatient treatment at the SCIMHU, agreed to participate in the study. Of the 84 participating families, 63 (75%) returned pre-admission questionnaires, 44 (54%) discharge questionnaires, and 54 (70%) follow-up questionnaires. On average, follow-up questionnaires were returned four months after discharge (mean = 126 days, SD = 18). One parent questionnaire was excluded from analysis because it was returned more than five months (150 days) after discharge. Given the variability in questionnaire returns across time points, two data sets were generated. The first data set consisted of 54 parents (65%) whose returns did not have matching questionnaires across time; while the second data set comprised 29 parents (35%), who returned complete sets of questionnaires, that is, at pre-admission, discharge and four month follow-up.
It is the latter data set that constitutes the final study sample, which consisted of 21 boys (72%) and 8 girls (28%), with a mean age of 9.34. Slightly more than half (52%) of the children were from single parent families, where the mother was the primary carer; 10 children were from two-parent families consisting of both biological parents; three children from families with a biological mother and a step-father, while one child was in his grandparents’ care.
The majority (97%) of admissions were regular (i.e. non-crisis) planned admissions, with a mean length of 30.4 days. Twenty-one (72%) were child admissions, while the remaining 8 (28%) were family admissions. Only one child met the criteria for a ‘crisis’ admission requiring immediate intervention due to the imminent threat of physical harm to themselves or others. Approximately 68% of families resided in the Melbourne metropolitan area, and 15 families (32%) resided in rural Victoria.
Diagnoses at intake followed a comprehensive assessment of the individual child and family, and were based on ICD-10 diagnostic groupings. Due to the severity of presenting problems, diagnostic overlap was particularly common. Behavioural disorders (including Oppositional Defiant Disorder, Attention Deficit Disorder and Mixed Disorder of Conduct and Emotions) were the most frequently reported diagnoses in 65% of the study sample. Parent–child relationship problems were present in 30% of the sample, adjustment disorders and anxiety disorders were reported in 20% of the sample and Pervasive Developmental Disorders were diagnosed in 15% of cases.
In addition to the parent questionnaires, 51 teachers (61%) returned pre-admission questionnaires, and 61 (73%) returned at follow-up. In total, 42 (50%) matching sets of pre-admission and follow-up teacher questionnaires were received and included in the final analyses. Thirtyseven matching Strengths and Difficulties Questionnaires (SDQ) and 46 matching Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA) were returned by referrers at preadmission and at four-month follow-up. A large proportion of referrers (87%) were children and adolescent mental health services (CAHMS) workers.
Procedure
Upon ethics clearance provided by the Austin and Repatriation Medical Centre Human Research Ethics Committee, three groups of informants (parents, teachers and referrers) were invited to complete a series of questionnaires to assess changes in key child, parent and family variables. Parents (or carers) and referrers were approached by a researcher 15 min before the assessment interview conducted at the Unit to discuss the purpose of the evaluation, and were invited to participate in the study. The purpose and aims of the study were explained with an information sheet. Informed consent was obtained from all parents in the study at pre-admission and during the first week of admission. Teachers were also contacted by phone during the first week of the admission and invited to participate in the study.
Following the initial contact between the researcher and families, a number of strategies were used to ensure questionnaire returns from families. These strategies included mailing reminder letters, phone contact, and in one particular case, a home-visit, at each of the data collection time points. Despite these labour-intensive efforts, questionnaire returns were not as high as anticipated. At this point, further inspection of the characteristics of complete and incomplete returns across time points, was necessary. This procedure is described below.
Study sample versus non-study sample characteristics
Prior to conducting analyses of the data, comparisons of the two samples, one sample consisting of 29 complete questionnaire returns (study sample), and the other of 54 incomplete data sets (non-study sample) were made. This analysis was required to determine whether the two groups were comparable along relevant child, parent and family variables, and to exclude the possibility that the results obtained were influenced by a biased sample of returns. Pre-admission comparisons of the two samples regarding family composition, admission characteristics, and key child, parent and family variables, indicated that the two groups of children and families were comparable along all dimensions measured. No statistically significant differences were obtained on any of the pre-admission child, parent or family variables measured. It was on the basis of these findings that the final decision to only include the final sample of 29 families was made.
Measures
Measures used and time points at which the three groups of informants completed the questionnaires are summarized in Table 1. Parents constituted the largest source of information. Instruments used were a combination of both established and relatively new measures.
Questionnaires completed by parents (P), teachers (T), and referrers (R)†
The Strengths and Difficulties Questionnaire (SDQ)[14]
The SDQ is a 25-item measure used to assess change in child behaviour and functioning. It contains five subscales: hyperactivity, emotional symptoms, conduct problems, peer problems, and pro-social behaviour. With the exception of the later subscale, lower scores on the SDQ subscales indicate better behaviour and functioning.
The Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA)[15]
The HoNOSCA is a brief, clinician-rated consumer outcome questionnaire, which is designed to measure consumer health status and treatment outcomes in child and adolescent mental health, over several aspects of health functioning. Higher scores on the HoNOSCA are indicative of more unhealthy child functioning.
The Piers-Harris Children's Self Concept Scale (P-H)[16]
The P-H is an 80-item self-report measure completed by children (with the assistance of a researcher, if necessary), designed to measure children's self-concept across six domains: behaviour, intellectual and school status, physical appearance and attributes, anxiety, popularity, and happiness and satisfaction. In addition the P-H includes a response bias and consistency index which can be used to filter invalid responses. Higher scores on the P-H indicate higher self-esteem than low scores.
The Parenting Scale (PS)[17]
The PS is a 30-item self-report measure completed by parents, designed to facilitate the early detection of dysfunctional discipline and assist in the prevention and early treatment of behaviour problems. The PS has three subscales: laxness, over-reactivity and verbosity. Lower scores indicate more functional parenting practices.
The Parenting Sense of Competence Scale (PSOC)[18]
The PSOC is a 16-item self-report measure designed to assess parenting self-esteem. The scale reflects the degree to which the parent feels comfortable with various aspects of his/her role as a parent. Includes two subscales: satisfaction and efficacy. Higher scores indicate greater parenting efficacy and satisfaction.
Centre for Epidemiological Studies Depression Scale (CES-D)[19]
The CES-D is a 20-item self-report measure designed to assess depressive symptomatology in adults. The measure has no subscales, lower scores indicate fewer depressive symptoms. The CES-D is not a clinical tool, and therefore is included here only as an indicator of depressive symptomatology.
The McMaster Family Assessment Device (FAD)[20]
The FAD is a 60-item self-report instrument that assesses family functioning on several dimensions and distinguishes between healthy and unhealthy families. The FAD has six subscales: problem solving, communication, roles, affective responsiveness, affective involvement, and behaviour control, plus one global functioning scale. Lower scores indicate ‘healthier’ family functioning.
Results
Data analysis
For the purposes of the present study, the final data set consisted of 29 questionnaires returned by parents at pre-admission, discharge and four month follow-up. Prior to analysis, all data were screened to ensure that the underlying assumptions of homogeneity and variability had not been violated. Descriptive statistics (means and standard deviations) were calculated for available data. Unless otherwise specified, MANOVA with univariate within-subjects contrasts were performed on all data to explore change across time. Where significant results were obtained, effect sizes (eta squared) were also calculated and reported. Effect sizes correspond to the following values: 0.0–0.2 = small; 0.3–0.5 = medium; and 0.6–1.0 = large or strong effect size. Results from analyses of parent, teacher and referrer data are presented separately below. Data included in the separate analyses represent all returns; therefore, the teachers’ and referrers’ sample sizes differ to that of the study sample of parents (i.e. 29 matching sets across time).
Parent ratings in key child, parent and family variables from pre-admission to discharge and four month follow-up
Repeated measures MANOVA analyses indicated a significant main effect of time on the SDQ total scale, F(2,27) = 12.90, p < 0.00 (Eta 2 = 0.48). Within subjects contrasts revealed significant improvements on overall child behaviour, including hyperactivity characteristics, emotional symptoms, conduct problems, peer relations, and prosocial behaviour. These changes were significant from preadmission to discharge, and from pre-admission to four month followup. Effect sizes were medium for total difficulties, hyperactivity, and conduct problems.
Significant changes were also obtained for parent satisfaction and efficacy scores across time, F(2,24) = 5.40, p < 0.012 (Eta 2 = 0.31), with significant improvements on the PSOC total scale and subscales (i.e. efficacy and satisfaction) from pre-admission to discharge, and pre-admission to follow-up. The satisfaction subscale only improved significantly from pre-admission to follow-up. Similarly, significant changes were revealed in overall parenting practices as depicted in the PS total scores, F(2,16) = 11.22, p < 0.001 (Eta 2 = 0.58), including the laxness, over-reactivity and verbosity subscales. These findings suggest that most parents used more effective parenting strategies in discipline situations at the point of discharge and four months following discharge. Effect sizes were medium for all parenting behaviours measured by the subscales.
As for levels of parental depression, results indicated that parents’ reported level of depression improved significantly from preadmission to discharge, F(1,28) = 12.47, p < 0.001 (Eta 2 = 0.30), and from pre-admission to four month follow-up, F(1,28) = 4.93, p < 0.035 (Eta 2 = 0.15). Mean parent depression scores at pre-admission to discharge, and follow-up were 21.62 (SD = 14.31), 15.17 (SD = 12.23) and 17.31 (SD = 15.84), respectively. Scores at discharge and followup were close to the 16 point cut-off, suggesting the presence of parental depressive symptoms at these time points. Overall family functioning scores obtained across time indicate no significant change, F(2,25) = 3.15, p < 0.06). However, subscale scores improved significantly in two subscales: roles (from pre-admission to discharge), and behaviour control (from pre-admission to follow-up). Scores on the roles subscale deteriorated from discharge to follow-up, suggesting a decline in patterns of behaviour for handling family tasks.
Child reports
Changes in children's self-concept were assessed during the first and last week of admission with the Piers-Harris Children Self-Concept Scale. A response bias and consistency indices, calculated before analysing the data, were within an acceptable range. A MANOVA was performed to assess changes from pre-admission to discharge, yielding no statistically significant results.
Teacher reports
Teacher ratings over time indicated a significant improvement in child behaviour and functioning from pre-admission (mean = 23.02, SD = 6.20) to follow-up (mean = 20.83, SD = 6.50), F(1,41) = 7.65, p < 0.01 (Eta 2 = 0.15). Within subjects contrasts revealed that scores on one subscale (conduct problems) account for the significant change at follow-up.
Referrer reports
Referrers’ ratings of child behaviour and functioning indicated a significant total scale score improvement from pre-admission (mean = 25.70, SD = 5.03) to four month follow-up (mean = 22.14, SD = 5.86), F(1,36) = 19.59, p < 0.01 (Eta 2 = 0.35). More specifically, Table 2 shows significant improvements on the hyperactivity, emotional symptoms, conduct problems, peer relations and pro-social behaviour subscales.
Results of univariate within subjects contrasts comparing mean pre-admission and follow-up on the SDQ subscales: referrer Data (n = 37)
Referrers’ HoNOSCA reports indicated a significant, overall reduction of clinical symptoms from pre-admission (mean = 21.80, SD = 7.95) to four month follow-up (mean = 15.33, SD = 7.69), F(1,45) = 45.78, p < 0.01 (Eta 2 = 0.50). Specifically, children's scores improved significantly in disruptive, overactivity, emotional and psychotic symptoms, family relationships, peer relationships, self-care and independence.
Cross-informant reports
Differences between parent, teacher and referrer ratings of child behaviour and functioning were examined at pre-admission and fourmonth follow-up. Only matching data across informants were included in the analysis.
As depicted in Table 3, analyses of pre-admission scores revealed that parent SDQ total score ratings were significantly higher than referrer ratings, which in turn were significantly higher than teacher ratings. Specifically, parents rated the children's behaviour on the four SDQ subscales higher than teachers (i.e. parents rated the children's behaviour as worse) but not significantly higher than referrers, except for hyperactivity. Differences between referrer and teacher ratings of behaviour and functioning were not consistent, with referrer ratings significantly higher on the conduct problems and peer relations subscales, but no significant differences on the other subscales were observed.
Parent, teacher, and referrer ratings of SDQ at pre-admission (n = 33)
At four-month follow-up, significant main effects for informants’ ratings were obtained on the SDQ total score, F(2,84) = 6.59, p < 0.01 (Eta 2 = 0.14), and the hyperactivity subscale, F(2,84) = 7.04, p < 0.01 (Eta 2 = 0.13). Consistent with pre-admission results, they revealed that parent SDQ total score ratings of child behaviour and functioning were significantly higher than teacher and referrer ratings alike. An identical pattern emerged on the hyperactivity subscale. There were no significant differences between teacher and referrer SDQ total score ratings at follow-up.
Discussion
The need for additional research exploring the outcome of child psychiatric inpatient treatments has been highlighted [4, 7]. Recent literature emphasizes the importance of examining change across a range of outcome variables [3], as perceived by a range of informants, and across several settings (e.g. home and school). Thus, the principal aim of the present study was to evaluate the impact of intervention on key child, parent and family variables following admissions to an inpatient setting. Parents’ reports constituted a large proportion of the data provided from pre-admission to discharge and through to four-month follow-up. Teachers and referrers also reported on child behaviour at preadmission and follow-up, and children provided reports on a self-concept measure.
Analyses of complete sets of parent data revealed a pattern of change across time. Significant improvements were obtained in child behaviour and functioning, parenting competence and efficacy, parenting practices, and levels of parental depression. These positive changes are consistent with findings reported elsewhere [4, 5], suggesting that interventions that incorporate both child- and parent-level goals yield desirable outcomes. In the present evaluation, parents were actively involved throughout the process of admission and design of goals for intervention. This process was facilitated by an experienced and skilled team of professionals, who were able to engage parents and children in the clinical tasks in a meaningful way.
Specific changes obtained in the present study are not surprising. It can be argued that given the intensity of the programme (i.e. 5 days a week for an average of 6 weeks), the predominantly behavioural nature of the presenting problems on admission, and the emphasis of interventions on the modification of parenting practices to achieve child behaviour change, the positive outcomes were expected. Interestingly, overall changes in other variables, such as family functioning and child self-concept did not vary significantly across time. These findings suggest that short-interventions may not be as effective for those areas of child and family functioning that are more complex in nature, and that may require longer term treatment and/or other intervention modalities, beyond what could be offered within an inpatient setting. Furthermore, the specific characteristics of families whose children are admitted into the unit may be an important variable to consider. In the present study, about half of the families in the study sample were headed by one parent. Thus, the question arises as to whether inpatient treatment benefits all families regardless of their varying backgrounds and presenting problems.
Also reported by teachers and referrers were significant pre-admission to follow-up improvements in child behaviour and functioning, indicating that the positive changes initiated during the admission were maintained over time. The inclusion of multiple perspectives in the reports of treatment outcomes has been seen as highly desirable [4, 5]. In the present study, observations of increased healthier child functioning from perspectives other than the parents’ highlighted not only agreement on the favourable impact of the intervention on child behaviour, but the transfer of such behaviour changes to other settings such as the outpatient clinic and school. Aspects of case management such as coordination with referrers and careful planning at discharge, and outpatient treatment following discharge may account for these positive outcomes. The latter, in particular has been found to be associated with improvements in child functioning. Extended follow-up, beyond the four months used in the present study, however, may be beneficial in further testing the effectiveness of inpatient intervention across time and settings.
Further cross-informant analysis of child behaviour and functioning at pre-admission revealed that parents’ ratings were more problematic than both teachers’ and referrers’ alike. Other studies have revealed moderate correlations, and no significant differences between parents’ and teachers’ ratings of child behaviour on admission to inpatient settings [21]. In the present study, it is possible that parents’ perceptions of difficult behaviour are partly due to increased time spent with their children, and therefore increased encounters with difficult behaviour. Furthermore, the present results highlight the importance of considering specific environments where behaviours arise, and the need to design interventions that address the differences among environments. Clearly, this may be beyond the scope of the inpatient unit, but underscores interventions modalities that may need to be considered in the discharge plan and outpatient unit.
There are three main limitations to the present study, which are important to consider for future evaluation research. First, due to ethical concerns about withholding treatment from families in crisis, we did not include a comparison group, making it difficult to conclusively attribute the positive changes reported by respondents to the admission process alone. Moreover, the study highlights the logistical and funding constraints involved in carrying out evaluation efforts, therefore limiting the service's ability to convincingly test its outcomes. The disappointing number of complete sets of data may also be a manifestation of the difficulties of conducting evaluation research in an inpatient unit. Alternative recruitment strategies, specifically designed for a group of families with multiple problems, may need to be investigated to ensure increased returns. One step in the right direction may be building in evaluation research mechanisms from the outset (i.e. design of intervention programme), and as critical component of service delivery in child inpatient units. Similarly, collaborative efforts with services that provide other treatment modalities may act as appropriate comparison groups, to overcome the methodological problems earlier described.
Second, we did not examine the impact of specific treatment variables on outcome, a feature of research which has been argued for elsewhere [5]. This would be an important objective for future research in order to delineate the critical dimensions of inpatient treatment associated with positive outcome. Third, given the diversity found in child inpatient units and the range of variables involved, our findings may not hold for other services.
Footnotes
Acknowledgements
We thank Lorraine Flynn, Laurel Downey, Anne McLoughlin and Maureen O'Brien at the Austin and Repatriation Medical Centre for their ongoing contributions to the project.