Abstract
Vaughan J. Carr and Terry J. Lewin, Centre for Mental Health Studies, University of Newcastle, Callaghan, New South Wales, Australia:
Recent correspondence in this journal [1–5], following publication of our article reporting that a consultation-liaison (C-L) psychiatry service in general practice had no demonstrable effect on patient outcomes at 6 months [6], has examined statistical, methodological and clinical significance issues in our orginal report. In broad terms, the status of this work is that, at present, C-L psychiatry in general practice is of unproven effectiveness. Our article and the subsequent correspondence have highlighted some of the practical and methodological difficulties in evaluating the effectiveness of this type of clinical service. Since outcome evaluation is expected to be an increasing government and third-party payer requirement of clinical services, we would like to make some further comments of a cautionary nature using, as an example, supplementary data on the perceived effectiveness of treatment for psychological morbidity among typical general practice attenders.
We evaluated by postal questionnaire the 6-month outcome of 451 patients attending their general practitioner (GP) who had not been referred to our C-L service (indeed, this subject pool was used to select demographically and symptom-matched control subjects for the C-L evaluation study [6]). They were divided into four groups on the basis of their recent and past history of emotional problems and whether or not they had recently received treatment for their emotional problems. Initial assessment and outcome evaluation involved data collection on symptoms (SCL-90-R), global ratings of health and relationships, and frequency of visits to a doctor; the instruments were similar to those reported in our C-Levaluation [6]. One-quarter (25.3%) of the sample reported psychological ill-health during the 6 months between questionnaires, of whom one-half (50.0%) reported receiving treatment, mostly (63.2%) from their GP. Patients with no past or recent (i.e. in the last 6 months) emotional problems had consistently lower symptoms and higher subjective health ratings throughout the study. Patients with past, but no current, emotional problems tended to improve in symptomatology and subjective health over the 6-month follow-up, relative to those with recent emotional problems. There were no differences on any of the outcome variables examined between those receiving and not receiving treatment for their recent emotional problems, indicating that the global effectiveness of treatment received could not be substantiated. Copies of the data analyses on which these findings were based are available from the authors on request.
Several interpretations of the data, other than that treatment is ineffective, are possible. First, self-reported symptoms and subjective ratings of health may be insufficiently sensitive to detect treatment-related effects. However, the self-report methods used have face validity and yielded rates of psychological morbidity and GP treatment that compare favourably with rates determined by more elaborate methods used by others in primary care settings. The patient groups were distinguishable from each other in terms of background characteristics (e.g. age, social support, recent life events, neuroticism, extraversion) and differed in degree of psychological morbidity in the expected directions. These lines of evidence suggest that the methods of measurement chosen, and the outcome measures used, would have been sufficiently sensitive to detect significant change in morbidity over time and any global positive effect of treatment that may have occurred. In addition, the numbers of subjects in each group (ranging from 57 to 228) were sufficient to detect a change given the sensitivity of the instruments chosen.
It may be that the subjects were diagnostically too heterogeneous and the forms of treatment offered too varied to obtain main effects for time and treatment. The 6-month follow-up period may have been too brief to detect significant change in morbidity. At the relatively minor levels of psychological morbidity found in this study, it may be that most subjects were suffering from chronic low grade neurotic symptoms, consistent with a relatively enduring or stable background level of such symptoms, which are resistant to treatment. Finally, the general practitioners may not have had the knowledge or skills to effect change in these subjects.
We believe that it is premature and probably erroneous to take the findings of this supplementary study, and those of our related evaluation of C-L psychiatry in general practice [6], at face value and conclude that treatment of psychological morbidity in the primary care setting is ineffective. Rather, we would prefer that the failure to find evidence of effectiveness be taken as a warning that outcome evaluation is an undertaking that is fraught with complexities. Outcome evaluation is, however, an undertaking that will become increasingly required of service providers, will inevitably acquire funding implications, and may become vulnerable to misuse. Therefore, it is important that some of the lessons learned in our work be taken up by those engaged in outcome evaluation in the field of psychiatry generally.
Future outcome evaluations will need to include a mixture of generic and specific measures. Under the Second National Mental Health Plan [7], there will be an increased emphasis on measuring consumer outcomes and, hopefully, the implementation and documentation of individualised service plans. The identification of a core, but necessarily generic, set of measures for use across all mental health settings to assess patient outcomes systematically (e.g. Health of the Nation Outcome Scales, Life Skills Profile and Mental Health Inventory), while welcome in many respects, may convey the misleading impression that ‘one size fits all.’ In other words, one or more of these core outcome measures could become the standard(s) by which to measure effectiveness of mental heath services, regardless of the psychiatric conditions or disabilities being treated. Notwithstanding the demonstrable reliability, validity and flexibility of the proposed generic instruments, their undiscriminating use across the spectrum of mental health problems may yield null effects of treatment, as in our studies in general practice.
It is difficult enough to evaluate the efficacy of different treatments for a condition as homogeneous as depression, in which the relative superiority of one treatment over another very much depends on which outcome measure is chosen from among a range of appropriate outcome measures [8], so that the widespread application of ‘all-purpose’ outcome measures will run the risk of yielding null or contradictory findings open to various interpretations depending on the agenda of who is making the interpretation.
We suggest that the selection and application of reliable, valid, and specific outcome measures in service evaluation be based on the following principles. First, the psychiatric disorders in question should be relatively homogeneous in terms of diagnosis and the nature and severity of the disabilities with which they are associated. Second, the outcome measures should be appropriate for the disorders and their associated disabilities; that is, they must measure factors that are relevant to the particular disorders and disabilities in question, as well as measure some aspects of these disorders and disabilities that are amenable to change. Third, outcomes should be measured at appropriate time intervals after commencing treatment; that is, at times which are compatible with the nature and severity of the disorders and disabilities in question as well as the type, quantity and duration of the treatment/rehabilitation being provided. Finally, there should be some means of determining that treatment/rehabilitation is actually being given by trained personnel in the manner required.
In the medium-term, the inclusion of both generic and specific outcome measures across a variety of evaluation studies and settings, and calibration of relationships between these measures, should facilitate an overall improvement in our ability to quantify and compare the benefits of mental health treatments and services.