Abstract
Consumer satisfaction surveys are increasingly important in mental health services. Obtaining such perspectives is a requirement in the USA and Canada and is also becoming an area of focus in Australia and New Zealand [1, 2]. While consumer satisfaction and consultation are crucial elements of ‘quality assurance’, increasingly, the views of consumers are also seen as an important component in the development of clinical practice guidelines in diverse psychiatric settings, and for good reason. Expectations and views of consumers potentially influence the therapeutic relationship, satisfaction and overall outcomes [3]. The provision of services for eating disorders is no exception; the development of clinical practice guidelines for anorexia nervosa commissioned by the Royal Australian and New Zealand College of Psychiatry is being conducted concurrently with consumer guidelines [Beumont P: personal communication].
In addition to the wider consumer movement in mental health service provision, the field of eating disorders poses additional imperatives for the analysis of patient perspectives. First, it has been reported that up to half of the patients prematurely cease contact following assessment with an eating-disorders service [4, 5]. Consumer perspectives may illuminate reasons for such high rates of treatment disengagement [6] for disorders associated with such disturbing morbidity and mortality rates [7]. Second, unlike most other psychiatric populations, core features of an eating disorder can be highly valued by the patient [8]. Accordingly, contact with treatment services can be a highly ambivalent prospect [9]. Identification of positive aspects of service provision may go some way towards enhancing motivation for change. Third, while provocative patient perspectives on treatment are readily and graphically represented in anecdotal accounts [10, 11] and self-help books [12], relatively little is known about the wider breadth and diversity of opinions following contact with a specialist eating-disorders service. Kopec-Schrader et al. surveyed inpatients [13], whereas outpatient care constitutes the more common treatment experience for people with eating disorders [14]. Views on recovery are even scarcer and recently this has been described as a neglected area of outcome research [15]. Finally, even experienced staff underpredict the value placed on core treatment components by current eating disorder patients they know well [16]. This finding is a feature of many, if not all, general adult psychiatric services [17, 18], and suggests very strongly that relying on staff interpretations of consumer perspectives may be less than ideal.
In the past a significant problem with satisfaction surveys has been a positive response bias [19]. That is, the conclusion from almost all surveys of mental health services has been that most clients are mostly satisfied. The inclusion of open-ended questions goes some way towards addressing this problem. For example, Kopec-Schrader et al. found high satisfaction on scale-like questions, but when further prompted, 35% of parents made criticisms [13]. However, open-ended questions are difficult to analyse. Comments need to be categorized and this adds subjectivity. Previous studies concerned with this issue advise a number of strategies including the formulation of categories to ‘fit’ the population under investigation [2, 19, 20].
In summary, using both structured and open-ended formats, the purpose of this study was to systematically analyse patient consumer perspectives following contact with a specialist eating-disorders service within the public health system. The study served the additional purpose of reviewing the utility of a satisfaction questionnaire developed by the service and in use for 7 years without previous evaluation.
Method
Setting
The Christchurch Eating Disorders Service was established in the mid-1970s and includes the sole specialist, hospital-based inpatient unit for the treatment of eating disorders within the New Zealand Public Health System [21]. Most patients are referred from general practitioners, and all undergo comprehensive assessments, including DSM-IV diagnosis [22], at the time of first presentation to the service. Though most proceed to receive outpatient treatment, a smaller number receive more intensive day patient or inpatient treatment.
Elicitation of consumer opinions following contact with the service is thoroughly embedded within the philosophy of the service, and at point of first contact, consent to obtain such opinions is sought routinely. As suggested by Watson and Leathem [2], to reduce problems of positive bias and power differentials, an independent member of the team who had no clinical responsibilities conducted this process.
Recruitment of participants
Three months following first contact with the service, a consumer satisfaction survey was posted out as part of other standard questionnaires (not part of this study) to consenting participants regardless of whether the consumer proceeded with further treatment or not. A postpaid return envelope was provided. As participation was anonymous, no additional demographic or clinical information which could be linked to individual participants was sought. However, standard descriptive information about the participant cohort during the time of the study was known, and this provided a general profile of all possible participants responding to the questionnaire. Although invitation to complete the consumer satisfaction questionnaire remains an ongoing aspect of service provision, this study reports on all data from questionnaires posted during the 2 years ending in December 1998, a time chosen for reasons of recency and service provision stability.
Consumer satisfaction questionnaire
In the absence of an existing measure for this setting, clinical and research staff of the service developed a consumer satisfaction questionnaire to reflect factors relevant to the culture of the service. Such factors included the organization of the service, the stipulated priorities of the government funding body and aspects identified by consumers in previous research as important when consulting mental health services. The questionnaire consisted of 13 questions (questionnaire available from corresponding author upon request). Two initial questions sought information about treatment utilization following the index assessment, eight questions elicited responses in a mix of yes/no and rating scale, and three questions used an open-ended format inviting commentary about improvements and the best/worst aspects.
Results
Respondents
In the time period chosen for analysis, 278 questionnaires were sent out, of which 120 were returned with no further prompting (43%). Responders and non-responders did not differ significantly with respect to age, occupation, living situation, ethnicity or marital status. However, respondents with anorexia nervosa were overrepresented (18% of responders, 5% of non-responders; ?2 = 6.5, p < 0.05) and ‘eating disorder, not otherwise specified’ were underrepresented (23% of responders, 35% of non-responders; ?2 = 8.8, p < 0.05) in the sample. Twenty-three per cent of respondents had bulimia nervosa, while the remaining 39% of respondents had presented with a mixture of other disorders. Although questionnaires were sent to 11 men, none of these were returned.
In summary, respondents were typically single (55%) women aged in their mid-twenties (mean = 27 years), of European ethnicity (94%), who were offered outpatient treatment following their assessment (52%). Of those respondents seen for appointments subsequent to the index assessment (76%), 61% had been seen for ≤?10 appointments.
Responses
Descriptive statistics were calculated for all five questionnaire items with fixed-response choices. Rating aspects of assessment and treatment, over 75% of respondents endorsed these components as either good or excellent. The highest individual rating of satisfaction was for ‘How respectfully were you treated by staff?’ with 59% of respondents endorsing this as excellent, while ‘How adequate were the explanations given to you by the Eating Disorders Service staff?’ attracted the lowest satisfaction rating with 23% of respondents denoting this as poor or fair.
Ninety-four per cent of respondents indicated that they would agree to be seen by the service again, and an equal number affirmed that they would advise a friend with an eating disorder to do so as well. Consistent with this trend, 81% of respondents indicated that their family/friends were satisfied with the treatment they had received from the service. Overall, there was an 85% approval rating (good or excellent) for the aspects of service provision measured by the forced-choice part of the questionnaire.
Open-ended responses
Consumers were invited to list the three best aspects of the assessment and treatment, and the three worst aspects. The volume of responses between these two categories was significantly different (t = 5.34, df = 128, p < 0.01) with respondents providing more best aspects (mean = 2, SD = 1.4) than worst aspects (mean = 1.4, SD = 1.3). A more in-depth analysis of these responses was then conducted following the manner of Griffiths et al. [20]. First, all three authors independently perused all comments to generate meaningful categories. These categories were then refined and defined at collective meetings at which, through a process of consensus, all comments were then assigned to one of five categories.
The category of ‘therapeutic alliance’ was defined as any comments referring to commonly accepted components of therapeutic alliance: namely aspects of bonds and trust, collaboration and shared commitment within the confines of a confidential professional relationship [23]. A typical positive comment included non-specific aspects such as ‘I felt valued and cared about’ while a typical negative comment within this category included ‘I guess I felt like a statistic’. The category of ‘outcome’ was defined as any comment referring to commonly described psychotherapeutic changes targeted by treatment for an eating disorder [24, 25]. Such comments as ‘not being preoccupied with food’ and ‘gaining weight’ were classified under this category. The category of ‘treatment type’ referred to specific modalities and activities of treatment commonly subsumed under a specialist eating-disorders service. For example, ‘exposure sessions’ and ‘being weighed’ along with ‘having to stay in hospital’ and ‘one-to-one treatment’. The category of ‘competency’ referred to specific references made regarding perceived staff competency and skill. Examples include ‘they knew what they were doing’ and ‘bad communication’. The final category of ‘service provision’ referred to administrative and environmental aspects of service provision such as the location and layout of the service, access and waiting times.
Different trends emerged for best and worst aspects of treatment. Overwhelmingly, the most frequently reported best feature of the service related to positive components of therapeutic alliance (51%). The remaining positive comments were evenly dispersed among the other categories of treatment type (14%), outcome (16%), competency (10%) and service provision (9%).
Specific treatment modalities and components prompted one-third of negative comments. Respondents disliked the structure of treatment, including being weighed, along with the perceived pressure to change and trial new behaviours that they felt ill-equipped to manage. Just under one-quarter of negative comments (22%) related to outcomes specifically targeted by mainstream treatment modalities. These included weight gain, loss of a compensatory strategy such as purging and the resultant increased sense of personal responsibility for monitoring food intake/routines. The remaining negative comments were relatively evenly dispersed among therapeutic alliance (16%), competency (9%) and service provision (19%).
Forty-three respondents provided 62 comments regarding how the service could be improved, and these were coded using the same categories developed from the other open-ended questions. Most comments (42%) requested changes in treatment modality or components: for example, individualized rather than group treatment, more support groups and more lenient rules while on the weight gain programme. Thirty-two per cent of comments requested changes in service provision such as improved decor, provision of treatment outside normal office hours and improved quality of meals. The remaining comments were spread between the categories of therapeutic alliance (11%), assessment/treatment outcome (10%) and competency (5%).
Discussion
While some health researchers continue to question the utility and validity of consumer satisfaction surveys [26], public and government pressure for consumer voice in the mental health sector has been increasing. Consumers of all psychiatric services have opinions and these need to be incorporated routinely into care [18]. Further, we would argue that consumer opinion forms an important insight into the unique predicaments of people with eating disorders. The methods by which such opinions are collected remains a subject of debate and will inevitably bias the outcome [27, 28]. However, by using both a structured and open-ended format specifically designed for the setting, and specifically inviting negative commentary, this study sought to explore the opinions of a large group of consumers of an eating-disorders service. The participation rate (43%) was modest but consistent with, or better than, other reported return rates of similar unprompted postal surveys [19,29–33], although less than that reported by a recent New Zealand study [2]. The latter study posted questionnaires within 1 week of obtaining consent and this time frame was not appropriate for this study. Demographic and clinical characteristics of the sample were consistent with other public-sector eating-disorder services [4], and this along with the close match between respondents and nonrespondents, suggests that a good representation of consumers was achieved. As a small group of consumers in this study (4%), it was not surprising that men did not feature among respondents. However, if it transpires that men increasingly present to services [34] special efforts may need to be made to secure such opinions.
In general, participants overwhelmingly reported high rates of satisfaction with multiple components of service provision, and even when asked to list negative aspects through an open-ended format, were less able to do so than cite positive aspects. By itself, this suggests the need for few major changes to service provision. However, while traditional rating scales for consumer satisfaction studies are useful, the inclusion of an open-ended format in this study provided a more detailed account of consumer perspectives, and in doing so, identified predictable and unforeseen sources of satisfaction and dissatisfaction.
A therapeutic alliance has long been seen as a key to successful therapy, irrespective of therapeutic modality [35], and it therefore not surprising that this features so prominently among positive comments. In addition to knowledge and skills relevant to eating disorders, it is clearly important that staff are reminded of the potency of client perceptions of the working alliance as a source of satisfaction. This finding compliments recent studies citing the considerable importance of examining therapeutic alliance from the clients' perspectives [36, 37]. While the largest source of dissatisfaction referred to treatment type, such comments were generally directed towards difficult, yet essential, treatment components such as structured assistance with eating, confronting body image disturbances, and self-disclosure of core difficulties. Accordingly, although recognizing the distress induced by such treatment, the expressed dissatisfaction alone does not substantiate grounds for seriously reviewing the inclusion of such core components.
However, responses describing the worst aspects of outcome warrant further comment. Though weight gain and cessation of bingeing and purging are seen by clinicians as indications of treatment effectiveness, patients may not perceive this as such. Indeed, a significant number of people with eating disorders attach great value to their symptomatology [8] and remain unconvinced about the need to change [9]. Such incongruencies between clinician and patient expectations may be an important cause of premature treatment dropout. Furthermore, this poses unique problems in constructing and evaluating consumer satisfaction questionnaires in the field of eating disorders. Unlike many other psychiatric disorders, for eating disorders, heightened consumer dissatisfaction may be an endemic feature of service provision for eating disorders.
Categories developed from responses to the openended questions overlapped with those devised by Kopec-Schrader et al. [13] and Nguyen et al. [19]. The absence of a complete fit emphasizes the importance of constructing categories relevant to the setting, culture and type of consumer [26, 38]. For example, no responses were received regarding parents' needs and involvement [13] or support staff [19]. The change experienced by all mental health services suggests that surveys need to be repeated. Finally, the question regarding how the service could be improved tended to elicit more commentary on treatment components and modalities and service provision than was elicited by the probe of ‘What were the three best/worst aspects?’. Clearly, when asked for suggestions regarding improvements, consumers focus on day-to-day treatment. However, it could be that generation of suggestions within these two categories was easier than the more abstract categories of therapeutic alliance or competency.
Greenwood et al. suggest interviewing any respondent who voices dissatisfaction [39]. However, this is unsuitable for a busy specialist service. Consumers in this study demonstrated their ability to communicate views through a written format alone. This may not generalize to other psychiatric populations. Responses to open-ended questions can be difficult to analyse, yet they are an essential part of meaningful satisfaction questionnaires. Without this, response bias increases the chance of closed-format questionnaires reinforcing the status quo. Future research should be directed towards investigating the impact of perspectives on outcome, and eliciting the opinions of nonprofessional caregivers.
Footnotes
Acknowledgements
We thank consumers of the Christchurch Eating Disorders Service and Jenny Fear and Jill Rau for collection of data.