Abstract
Client satisfaction with health services is an obvious aspect of the quality of service delivery and a relevant outcome measure. Most of the data available in this domain refer to physical health services. Nevertheless, there is a growing body of research on mental health services [1, 2, 3, 4, 5, 6, 7, 8, 9]]. The literature suggests that overall client satisfaction is reasonably high in both adult and children's services. For example, Eyers et al. [2] reported that 64% of respondents attending a specialised mood disorders clinic for adults were very satisfied and a further 21% partly satisfied, while Kotsopoulos et al. [6] described that 84% of parents attending the children's outpatient service of the Royal Ottawa Hospital were satisfied or very satisfied. However, less is known about the factors that contribute to client satisfaction. There are suggestions that symptom improvement and other positive outcomes are associated with satisfaction, but the association is weak [1, 8]] or non-existent [7].
More research is needed to understand the factors underpinning client satisfaction, its relationship with outcome and the place of patient satisfaction in the evaluation of services [7]. This is particularly necessary in standard clinical settings, for there is doubt that results obtained in specialised research centres can be generalised to everyday practice [10] or that results of one-off surveys are comparable with results of continuing assessment of satisfaction. In addition, factors influencing satisfaction may vary substantially depending on the type of service provided (e.g. consultation, ongoing treatment) or the setting in which services are delivered (e.g. inpatient vs outpatient) [3]. Parents’ satisfaction with treatment is particularly important in the case of young people because parental attitudes are critical in engagement and continuation of treatment. Parents are often an integral part of the treatment process and meeting the needs of the family is often a major goal of treatment. Therefore, satisfying parents is one of the aims of services for children and adolescents. Learning about facets that alter parental satisfaction may facilitate the design of services that are more effective and/or acceptable to consumers.
The object of this study was to increase our understanding of the factors influencing satisfaction in child and adolescent services by presenting the results of parent satisfaction ratings routinely obtained over a 4-year period in a standard clinical setting. The relationship between clinicians’ ratings of outcome and parental satisfaction and the correlates of satisfaction were also examined.
Method
The service
Rivendell is a mental health facility for children and adolescents affiliated with the University of Sydney, Australia. The unit provides consultation, inpatient, day and outpatient treatment at no cost to the patient. The type of services can be described as secondary level (20% of referrals) for people living in central Sydney and tertiary for the entire state of New South Wales (80% of referrals). Many of the tertiary patients are referred for consultation or advice and subsequently returned to local services for further treatment. Referrals are made by a variety of health professionals, school counsellors and, occasionally, by parents. Inpatient services are restricted to 12–18-years-olds and operate from Monday to Friday, with patients spending weekends with their natural or foster family.
All patients undergo a comprehensive psychiatric evaluation that includes interviews with the child, the family and obtaining information from teachers and other relevant professionals. Each patient receives a DSM-III-R (or DSM-IV from 1994 onwards) diagnostic formulation and a treatment plan or recommendations. The assessment process has been described elsewhere [11].
Subjects
Clinicians were asked 3 months after initial assessment, and at 3-monthly intervals thereafter, whether patients were still in treatment or if the case was closed. A case was considered closed if the patient had been discharged, referred elsewhere or had not been seen for 3 or more months and there was no plan for further appointments. All patients routinely assessed at the unit between mid-1992 and mid-1996 whose cases had been designated closed by the treating clinician were eligible for inclusion in the study. Patients continuing in treatment and those seen for consultation in juvenile justice facilities or outside the unit (e.g. in other hospitals or facilities) were excluded. Data were available for 1278 patients (Fig. 1). The mean age was 13.7 years (range = 3–19); 87% were 12 years of age or older; 58% were male.
Clinicians' rating of outcome
This has been described in detail previously [12]. In summary, clinicians were asked to rate outcome once the episode of treatment had finished, blind to parent's rating of satisfaction which was available several weeks later. An episode of treatment comprised all the treatment provided (both outpatient and inpatient) until the case was closed.
Outcome was dichotomised as ‘positive’ (if in the opinion of the treating clinician the goals of assessment/treatment had been substantially achieved), versus ‘doubtful’ or ‘negative’. Examples of circumstances that would result in a negative outcome were: dropping out, failing to show improvement and inability to engage the patient or family. However, a positive outcome did not necessarily mean symptom reduction. Since a large number of patients were referred for consultation, a positive outcome could mean, for example, that the goal of the consultation, in the clinician's opinion, had been achieved. Outcome was not operationalised further, giving clinicians considerable freedom to make their judgement.
Flow chart describing the number of patients included in the different stages of the study
A substantial number of patients were treated by two clinicians. In these cases, both were asked to rate outcome and the poorest rating was considered. Examination of 106 consecutive cases treated and independently rated by two clinicians showed moderate agreement in outcome rating (K = 0.58).
Cases with a negative outcome were reviewed in a monthly meeting with all clinicians, where possible reasons for the poor result and appropriate action were considered. Among the former, reasons often mentioned were: dropped out, did not respond to treatment, etc. Examples of the latter included changes in policies or procedures, re-contact the family offering further services, etc.
Parent satisfaction
Satisfaction was measured using a parent satisfaction questionnaire (PSQ) [9], slightly modified to meet the needs of this setting and population. The questionnaire (copy available on request) had eight items which were rated on a four-point scale from 1 (least satisfied) to 4. It also included questions about further use of services and asked for specific comments or criticisms of the service. Internal consistency of the PSQ in this population was high (α = 0.96).
The PSQ, together with a self-addressed, prepaid envelope, was mailed to parents or guardians once the case was closed. Parent Satisfaction Questionnaires were not mailed in 9% of the cases (Fig. 1), largely because those patients were not living with their parents (e.g. living in refuges or institutions). Copies of the returned PSQ were given for perusal to the clinician involved and negative PSQs were reviewed in a monthly meeting with all clinicians, where possible reasons for poor satisfaction or criticisms and corrective measures were considered.
Statistical analysis
All reported percentages were rounded to the nearest unity. Categorical variables were analysed using Chi-squared analysis. When association was significant and the size of the effect was of interest, odds ratios (OR) rather than significance is reported. Differences between ordinal variables (e.g. age) were tested using the Mann-Whitney U-test. Continuous variables were analysed using independent samples t-test (two-tailed). Agreement between dichotomous ratings was computed using kappa. Only results with p < 0.05 are reported.
Results
The number of patients included in each stage of the study is presented in Fig. 1. Of the 1252 patients whose outcome was rated by clinicians, 14% (n = 175) had been inpatients (residential or day-patient). Inpatient status could not be ascertained in 35 cases (excluded from analysis when this variable was relevant).
Clinicians’ rating of outcome
The majority of patients (69%) were rated by clinicians as having had a positive outcome. Inpatients and females were 1.7 times as likely to have been given negative ratings as outpatients and males, respectively. The most frequent reasons given by clinicians for negative outcome were (some had more than one): dropout (36%); failure to improve with treatment (26%); patients being forced to attend against their will (10%); family having unrealistic expectations of treatment (10%); family wanting the patient to be admitted as an inpatient but this was not recommended (7%); and family being pressured to attend (e.g. by school or social services) (4%).
Parents’ ratings of satisfaction in various items of the Parent Satisfaction Questionnaire (PSQ) and clinicians’ rating of outcome according to number of sessions attended and inpatient status
Among outpatients, clinicians’ rating of outcome changed according to the number of sessions the patient had attended (bottom of Table 1). Rates of positive outcome were highest (74%) among those who had attended only once and lowest (64%) among those who attended two or three sessions (χ2 = 8.63, df = 2, p < 0.05).
Parent satisfaction
The PSQ was not mailed to 9% of the eligible patients. This group was significantly older than those who were sent the PSQ (14.2 vs 13.6 years; U = 32075.0; p < 0.05). This is not surprising, for older adolescents are more likely to live independently or in institutions. Of the 1149 PSQs mailed, 41% were returned. The gender and age distribution of the patients from the returned and not-returned groups were similar. However, those who did not return the PSQ were more likely to have been rated by clinicians as having had a negative outcome than those who returned it (OR = 1.7; 95% confidence interval [CI] = 1.3–2.3).
Specific aspects of satisfaction are presented in Table 1. Overall, 76% of parents felt mostly or very satisfied with the service. There were no differences between inpatients and outpatients on any of the items with the exception that parents of inpatients felt the service had met most or all the family's needs more often than parents of outpatients (69% vs 56%, respectively; χ2 = 4.0; df = 1; p < 0.05). Table 1 also shows that, among outpatients, there was a significant increase in all aspects of satisfaction as the number of treatment sessions increased, being consistently higher across items in patients who had attended more than three treatment sessions.
When ratings on the eight PSQ items were added, the mean score was 24.6 (range = 8–32). The distribution was skewed towards high satisfaction. For example, while 2% had the lowest possible score (eight, a rating of 1 on each item), 14% had the highest. A score < 21 represented the lowest quartile. Parents were considered dissatisfied if their overall PSQ score was lower than 21.
There were no differences between total satisfaction scores of inpatients (mean = 24.8, SD = 6.9) and outpatients (mean = 24.1, SD = 6.8). Changes in satisfaction scores according to the number of sessions attended are presented in Fig. 2. Rates of very dissatisfied parents (PSQ score of 12 or lower) decreased rapidly as the number of sessions increased, while rates of the very satisfied (PSQ score of 28 or higher) moved in the opposite direction. Differences were small in the middle range of PSQ scores. The profile for inpatients was similar to outpatients who had attended more than one session with the difference that there was a small group (9%) who were very dissatisfied.
Parent Satisfaction Questionnaire (PSQ) according to the number of sessions attended: —, one;—-, two or three;—, four or more
Satisfaction and outcome
There were 397 patients who had a valid PSQ (a PSQ with all eight questions answered) and a clinician's outcome rating (Fig. 1). Those rated by clinicians as having had a negative outcome had lower PSQ scores than those rated positive (mean = 20.3 vs 25.7; t-test = 5.82; df = 138.9; p < 0.001). There was a significant association between positive outcome and dichotomised satisfaction (PSQ scores > 20). However, the extent of the agreement was small (K = 0.27).
Clinicians’ rating of outcome agreed with parents’ dichotomised rating of satisfaction in 73% of the cases. Clinicians rated the outcome as negative in 12% of cases when parents were satisfied, while the reverse occurred in 15%. Since the proportion of false positives and false negatives was similar, and small, this suggests that clinicians were not overly biased.
As shown in Table 1, parents’ rating of satisfaction increased with the number of sessions attended among outpatients, while clinicians outcome ratings followed a U-curve, being highest in those who had attended once, or more than three sessions. To further examine this, agreement between dichotomised parental satisfaction and clinicians’ outcome was computed according to the number of sessions: kappa was 0.17 for those who attended one session, 0.36 for those attending two or three and 0.13 for those who attended four or more. Outcome ratings across these three groups identify parent satisfaction with a sensitivity of 0.87, 0.82, 0.85 and a specificity of 0.27, 0.55 and 0.30, respectively. That is, sensitivity remained quite constant while specificity was higher in those who attended two or three sessions. This suggests that clinicians’ rating of outcome and parents’ satisfaction may have been influenced by different factors depending on the number of sessions.
Findings among inpatients were different. As already noted, clinicians were more likely to rate inpatients as having had a negative outcome when compared with outpatients. Agreement between dichotomised parental satisfaction and clinicians’ outcome rating was moderate (kappa = 0.48).
Dropout
Clinicians identified dropout from treatment as one of the main reasons for negative outcome in 34% of cases. There was a significant association between subsequent dropping out and the number of outpatient sessions attended (χ2 = 17.52; df = 2; p < 0.001). Dropout rates were lower in those who attended one session (7%) than two or three (17%) and four or more sessions (14%). This also shows that about 7% of the patients seen once had agreed to further treatment but did not follow this through. Surprisingly, there were no differences in any aspect of parental satisfaction between those who dropped out of treatment and those who did not.
Clinical improvement
Patients identified by clinicians as not having shown improvement with treatment as a reason for the negative outcome were eight times as likely to be inpatients. When inpatient status was controlled through hierarchical logistic regression, those who dropped out were six times as likely to have been considered by clinicians as not having shown clinical improvement. This suggests that clinical improvement might have been the main factor considered by clinicians when rating outcome among inpatients and by parents when rating satisfaction (given the better agreement between them) but other factors appear to be more important in clinicians’ and parents’ minds in the case of outpatients. However, lack of improvement was perceived by clinicians as an important factor in those who dropped out.
Changes of satisfaction and outcome over time
Because the period covered by the study was quite long (4 years) it was possible to establish whether there had been changes in ratings of satisfaction or outcome over time. When the group was dichotomised into those treated before (n = 594) and after (n = 658) 1994, clinician ratings of outcome were slightly more likely to have been positive after 1994 (OR = 1.3; 95% CI = 1.1–1.7) but there were no differences in parent ratings of satisfaction.
Discussion
A positive feature of this study is that it was based on data collected routinely in a standard clinical setting over a considerable period of time. As such, it is unlikely that the study itself could have biased clinicians’ behaviour, a possibility in shorter-term investigations. Because treatment had already finished, it is unlikely that parents were influenced by considerations related to continuing treatment. Also, the number of patients involved was reasonably large. The main weakness is the low response rate: 60% of parents did not return PSQs, comparable with those of similar studies in which further reminders were not sent [2]. There is evidence that levels of dissatisfaction are higher among non-responders [05, 13]]. Although response rates up to 80% can be achieved with follow-up letters and phone calls [2, 5]], this is impractical in a normal clinical setting when satisfaction is examined on an ongoing basis. Also, the wish to obtain client satisfaction ratings from all patients must be balanced with other work demands and respect for client's wants and privacy. Mailing the PSQ gave parents the opportunity to make comments about the service if they wished. Making further intrusions (e.g. by sending reminders or telephone calls if they did not answer) was avoided. Although low return rates can diminish generalisabilty of the results, the validity of responses obtained after pressure may also be questionable [5]. Another shortcoming was that few variables were measured. This was largely the consequence of the naturalistic design of the study.
Overall, ratings of satisfaction were comparable with those obtained in other services. Typically, these range between 60% and 90% [2, 3, 8]]. The lowest level was obtained in response to the rather ambiguous question of whether the service had met none, some, most or all of the family's needs.
Comparison among settings is difficult because the type of services provided may vary. A comparison of parent satisfaction between two groups of services on intake-assessment, outpatient and inpatient treatment was performed in the Fort Bragg Evaluation Project [3]. The demonstration project group were provided with an exceptionally well resourced, seamless array of services with a single point of entry. Patients needing more intensive services were also allocated a case manager who ensured that the most appropriate services were provided. The comparison group included two sites that offered traditional mental health services without case management. Significantly more parents at the demonstration site (94–98%) reported being satisfied or very satisfied with the intake-assessment phase than at the comparison sites (79–86%). Similar results were found for the outpatient treatment phase, although there were fewer differences. Finally, there were no differences in overall satisfaction among inpatient services between the sites. In contrast with our results (of a small but significant agreement between clinicians’ rating of outcome and parental satisfaction) further analysis of the Fort Bragg data has shown that there was no relationship between clinical improvement and parental satisfaction [7].
It is likely that different factors influence parent satisfaction depending on the type of service provided. This may largely reflect parent expectations. While easy access, convenience of appointments, etc., are relevant for intake/assessment or outpatient treatment, these are probably of lesser importance for satisfaction with inpatient treatment. Alternatively, (as suggested by Heflinger et al. [3]) it may be more difficult to effect change in practices of inpatient services. Also, inpatient services are likely to treat more severe cases.
Rates of clinician judgements of positive outcome were highest for patients seen only once while these patients’ parents were the least satisfied. This may reflect the fact that the service had a large proportion of tertiary referrals and parents might have not been informed adequately by the referring agent about the nature and aims of the assessment, or that clinicians were rating outcome on the basis of referring agent requirements rather than patient and family needs, among possible explanations. The increase in agreement and satisfaction observed in those who attended two or three sessions may reflect a convergence or a better understanding of assessment/treatment issues by both parent and clinician.
Astriking finding was the increase in the level of satisfaction with increasing length of treatment for outpatients. A possible explanation for this is that parent satisfaction may largely depend on the amount of time and dedication clinicians give to the patient and family. In this case, satisfaction may reflect the quality of the relationship, whether parents felt supported, etc., rather than whether the child improved [14].
These results are consistent with other data about treatment decisions. For example, families and clinicians are involved in making decisions about termination of treatment in most cases. Nevertheless, clinicians tend to play a dominant role in terminating more restrictive services, such as inpatient admission, while families seem to play a more central role in terminating outpatient care, often by just dropping out. Yet, by whom or how this decision is made does not seem to be related to outcome [15] or satisfaction (as in these data).
Implications
Different settings will require different instruments. We concur with other researchers [8] that measures need to enquire specifically about aspects parents consider important when rating satisfaction. Only that information will be of benefit when identifying problems or needs and in implementing change. The current global scales are of limited use in this regard. As a result of this study we have developed a different questionnaire in which parents are asked whether they are satisfied or dissatisfied and explain the reasons by choosing items from a list (copy available on request). There also is a need to ask children and, particularly, adolescents about the service received since parent's and the child's needs will not necessarily be the same.
Obtaining a high degree of client satisfaction is possible [3]. However, parent satisfaction means just that. Although it might be tempting, satisfaction cannot be used to obtain proxy for clinical improvement [07, 14]]. Improving access, making convenient appointment times and having a single point of entry are the type of measures that might result in higher satisfaction and should be implemented, particularly if no extra costs are involved [16]. However, these will need to be complemented by independent efforts to improve outcomes. This is likely to prove a bigger challenge.
Footnotes
Acknowledgements
We thank Rivendell staff and Dr Garry Walter for their help.