Abstract
Psychiatry often fails to get the message across that we have many good treatments. Despite the increasing evidence of the efficacy of many treatments, we are overwhelmed by criticism from consumers, funders or the media that psychiatric treatment does not relieve or cure. At the 1998 Royal Australian and New Zealand College of Psychiatrists Meeting, Professor Kalucy was sufficiently concerned to ask four speakers to respond to an article in the Scientific American of December 1996 in which John Horgan told readers that he questioned whether there was evidence for the effectiveness of treatments in psychiatry. This article is based on a contribution to that seminar.
Everyone knows the characteristics of good treatment: it should, for a defined condition, reduce symptoms, disability and, after recovery, the risk of relapse. Such progress should be easily and reliably measured. The improvement should be shown to be clinically and statistically better than that due to an ineffective or placebo treatment in randomised controlled trials (RCT). These trials should control for biases in measurement of the characteristics and in selection of patients. The trials should also control for changes in symptoms, disability and risk factors that are due to spontaneous remission, regression to the mean, and the effect of being in good treatment, commonly called ‘the placebo effect’. Furthermore, good treatment must be able to be manufactured or taught to others, so that it can be delivered to all who need it. Good treatment, widely applied, should have a demonstrable effect on the prevalence of a disorder, not just by preventing recurrence but by shortening the duration and severity of illness in those who are ill.
The randomised placebo-controlled trial remains the benchmark for evaluating efficacy and informing the process of constant improvement. It is 50 years since the first randomised controlled trial in medicine and 25 years since Cochrane wrote: ‘that no new treatments should be introduced into medicine unless they have been shown in RCTs to be more effective than existing treatments, or as effective, but safer or less expensive’ [1]. Nevertheless, a randomised controlled trial is not always necessary. If the treatment is strong enough, as streptomycin was in tuberculous meningitis, then everyone can see the benefit and the usual confounds and non-specific factors could not possibly explain the change. If the benefit applies to large numbers, like the drop in motor vehicle accident fatality rates with the introduction of seat belts, then the possibility of confounding the evidence by selection or measurement bias recedes, and again no trial is needed. In fact, human progress is seldom based solely on certainty; it usually progresses via probabilities. We obtain the best information we can from scientific evidence like RCTs, and supplement that with the opinions of experts, then look at the resources at our disposal, and finally decide what to do before the money runs out.
As always, it is the level of proof required that is the problem, and people who find the results of a trial distasteful will always ask for more and better measures in the hope that the distasteful result will not be substantiated. Jerome Frank believed that there are no specific remedies, that healing is due to persuasion and the non-specific effects of being in treatment, that the placebo effect is the primary active ingredient. If this is so, then one hardly needs much training provided one is a plausible purveyor of hope. Truax and Carkhuff [2] made the logical deduction when they sought to train lay people as mental health workers. They used Rogerian therapy that emphasised genuineness, accurate empathy and non-possessive warmth as the way to maximise improvement. In our 1981 meta-analysis [3], we found such counselling to be as effective as other placebos and significantly less effective than the specific therapies. This simple treatment, although loved by clinician and patient, was not of specific benefit for even simple disorders. But efficacy, the potential effectiveness of the treatment, was not the reason that Rogerian counselling became so popular. Efficiency was; why use expensively trained social workers, psychologists and psychiatrists when your friendly local caring adult, working part-time for little money, could do nearly as well at much less cost? A strong counsellor movement was created. In Australia at present, as many people go to counsellors as go to psychiatrists or clinical psychologists [4]. And patients and the media do not distinguish between the non-specific help from counsellors and the specific treatment to be expected from mental health professionals. They should.
When discussing the benefits of treatment, it is useful to distinguish between efficacy (the ability of a treatment to produce benefit if applied ideally), effectiveness (the benefit that actually occurs when a treatment is used in practice) and efficiency (the resources required to produce a unit of health gain). See Turnock [5] for a discussion of these terms.
Efficacy
Efficacy is the potential effectiveness of a treatment, probably best measured from the results of RCTs conducted by experts. A recent book Treatments that Work by Nathan and Gorman [6] is representative of a number of publications. It is more important than most because it was initially written on behalf of the American Psychological Association and the American Psychiatric Association. The authors asked experts to review the evidence for the efficacy of each treatment in each disorder and rated the quality of the evidence into six grades ranging from type 1 studies in which RCTs met all the criteria mentioned above; type 2 studies in which one of the necessary criteria was omitted from the RCTs; type 3 studies which included open trials and case control studies; and types 4–6, studies of even lesser veracity.
Restricting comment to the type 1 or best evidence, then proceeding through the disorders in alphabetical order, one would, they say, have to conclude that both cognitive-behavioural therapy (CBT) and naltrexone have been shown to be efficacious in alcohol use disorders. Lithium, valproate and carbemazepine are all efficacious in acute mania, and lithium can reduce the frequency of episodes. Noradrenergic drugs are efficacious in borderline personality disorder. Antidepressants produce short-term gains, and CBT short- and long-term gains in bulimia. Stimulants are markedly efficacious in attention deficit hyperactivity disorder (ADHD). Four structured therapies are efficacious with conduct-disordered children, either directly or via parents or families. Acetylcholinesterase inhibitors are efficacious in dementia. Antidepressants, electroconvulsive therapy (ECT) and psychosocial interventions are efficacious in depression in the elderly. Cognitive-behavioural therapy, benzodiazepines, buspirone and the tricyclic antidepressants (TCA) are all efficacious in generalised anxiety disorder.
To continue the list from Nathan and Gorman [6], TCA, selective serotonin re-uptake inhibitors (SSRI) and monoamine oxidase inhibitors (MAOI) and CBT are efficacious in major depression. Behaviour therapy and serotonin re-uptake inhibitors (SRI) are of benefit in obsessive-compulsive disorder. Behaviour therapy and CBT, TCA and MAOI, benzodiazepines and SRI and SSRI are all efficacious in panic disorder, with or without agoraphobia. Monoamine oxidase inhibitors, TCA, SSRI and behaviour therapies are efficacious in posttraumatic stress disorder (PTSD). Conventional and atypical antipsychotics are efficacious in schizophrenia. Behaviour therapy, social learning, social skills training and structured family intervention programs also are efficacious in schizophrenia. The benzodiazepines and zolpidem improve sleeping in the short term, behavioural interventions are efficacious in the long term. Exposure therapies, MAOI, SSRI, and benzodiazepines are efficacious in social phobia. Antidepressants decrease pain in somatoform pain disorder. Exposure therapies are efficacious in specific phobias. Nicotine gum or patch is efficacious in smoking cessation. Methadone and naltrexone are both efficacious in opiate dependence.
The Nathan and Gorman evidence is important, and a psychiatrist, asked by a patient what other treatments work in their disorder, would be ethically required to mention the treatments for which there is type 1 evidence and then proffer appropriate advice. To do less would be negligent. And all candidates for the College exam should know about, and be able to carry out all these treatments.
Effectiveness
That a treatment has been shown, in a randomised placebo-controlled trial, to be efficacious is one thing, but the real question is whether such treatments are actually effective in practice when applied by the average doctor to the average person who seeks treatment. Does an efficacious treatment for a particular group still work under the usual conditions of clinical care or, if it doesn't, have the barriers like professional competency, organisational inertia and social attitudes been too powerful to allow it to deliver the potential noted in the research study. The matter is important and will become more important when the College seeks to implement the treatment guidelines that are presently in development.
How can we derive evidence that mental health care is effective under the usual conditions of clinical practice if we have to be there to observe the benefit? One school of thought holds that we can evaluate effectiveness only through analysing the outcome of a cohort of routinely treated individuals but this presumes that outcome measurement was routine in that environment (see [7, [8]] for two examples in which routine outcome measurement was part of the usual conditions of clinical care). Other quasi-experimental designs can be used, but the problem is that patients and doctors know they are being assessed, which can alter the way treatment is delivered. Whether something is effectiveness research can be hotly disputed. The report of the UK Prism study [9] contended it was the first to address effectiveness of community care for psychosis. It was followed by an impassioned editorial claiming that it was but the most recent in a long line of such research [10]. In fact, in none of the studies mentioned were patients being treated under the usual conditions of clinical care; all were taking part in carefully assessed, and quite mer-itorius, demonstration projects. It seems that there is a continuum between an experimental study of efficacy, a demonstration project, evaluations of sample cohorts, and the retrospective analysis of routinely treated patients. Wells [11] agrees that the situation is complex, and to work at the interface between clinical trials and effectiveness research suggests methods like cost-utility and econometric analyses, and special hybrid research designs.
It is as though there is no method, apart from the routine measurement of outcome, whereby a concerned practitioner might check that his or her practice was effective. Yet it is important that they do, for otherwise the clinician's illusion will too easily demoralise. Clinicians commonly believe that they are less effective than they are, because they judge their effectiveness from the patients they are currently seeing. Clinicians spend the majority of their time with their difficult and slow-to-improve patients and quite forget the patient who responded quickly and did not need to be seen again. Thus, a routine review of outcome will show the effective clinician that while their typical workload is with people who are chronically ill, their typical patient is someone who responds more quickly. Routine outcome measurement, even that using one brief instrument like the SF12, can therefore be both rewarding and essential.
In the absence of routine outcome measurement or quasi-experimental studies, one needs repeated prevalence or mortality data to estimate effectiveness. In what used to be known as a phase four study, after the multicentre RCTs were completed and the new treatment was used widely, one would discover that the prevalence of the target disorder was diminishing. New cases should be occurring at the same rate but, because they were being treated effectively and quickly, the duration of illness would be shortened and thus the prevalence would decline. There is an argument that this is happening in acute myocardial infarction due to the introduction of thrombolysis. The reason one can be certain is that the end-point, death, is routinely and reliably measured, an example of routine outcome measurement at its best. One might hope that the effectiveness of the current youth suicide prevention project will be apparent for the same reason. But mostly mental disorders result in disability not death, and disability is not routinely and reliably measured. It should be.
In regard to the effectiveness of mental health services generally, there are no phase four studies that we know of anywhere. The Burden of Disease methodology [12] might well be of value in assessing effectiveness for it is sensitive, not just to changes in prevalence, but also to changes in disability. At one level this is a concern, because such studies show that, while the burden of physical diseases is 60% lower in developed countries, the burden of mental disorders is similar in developed and developing countries, despite the marked disparity in the available services. This was originally considered to be an artifact of the scarcity of prevalence data in the developing world but it is increasingly likely that the prevalences and disabilities are similar: that is, despite the considerable activity by mental health staff in the developed world, there has been no demonstrable gain in mental health. It is as though treatments which have been shown to be efficacious in the research setting are simply not proving to be effective in practice.
Clearly this cannot be true, unless of course psychiatrists spend more time on the long-term care of the incurable than on the potentially effective treatment of those who could be cured, for then a high level of activity would not affect the burden of disease. There are indicators that this could be so. This emphasis on the chronic mentally ill is common in the developed world, and is not at all peculiar to Australia.
Perhaps we need to shift the emphasis. Specialist psychiatrists should be supporting primary care services to care for people with intractable conditions, and spend their own professional time, and train others to do likewise, with people who can be relieved or cured with skilful treatment.
Health services are merit goods (goods with high elasticities of demand) that people feel entitled to, and as such, are to some extent immune from the full constraints of the economic rationalists. The Netherlands is considering establishing separate budgets for prevention, cure and care [13], and accepts that each of these activities could have quite different levels of effectiveness and still be funded, a different solution to that we previously proposed [14]. Psychiatry may be obsessed with care, and the section of this article on efficacy highlighted the possibility of cure, but prevention in psychiatry is seldom a priority. The US Public Health Service [15], in trying to estimate the relative contribution of public health initiatives for prevention of physical disease, as compared to clinical initiatives to relieve and cure, thought that most of the reduction in the burden of physical disease in developed countries like the US could be attributed to improved standards of living and to public health initiatives like clean air and water, safer environments and the control of communicable disease. Only a minor part was attributed to the advances in clinical medicine.
Even if the leverage of public health measures in mental disorders is not so profound, public health initiatives might be as effective as clinical care in reducing the burden of mental disorders. Income support during sole person pregnancy, preschool educational programs for the disadvantaged, programs to delay teenage pregnancies, reducing sexual abuse and violence, and encouraging school retention, are all programs that have the potential to improve both the mental health and the longevity of the recipients, and hence reduce their dependence on health services during their productive years [16]. While mental disorders may be reduced by such societal approaches, there is now evidence that specific programs can prevent the emergence of disabling anxiety and depressive disorders in susceptible children [17], and strategies for the prevention of substance abuse are constantly being mooted. It is important to remember that in the National Survey of Mental Health and Wellbeing [4] the 18–24-year-old age group were those with the highest prevalence of mental disorders. Perhaps it is time that, like the Netherlands, we considered establishing separate budgets for prevention, cure and care, in order to reduce the burden of disease attributable to mental disorders. The second National Mental Heath Strategy has liaison with consumers as one of its goals, a synonym for emphasis on care. Perhaps the political imperatives of satisfying the electorate are getting in the way of the purpose of medicine, which is to prevent and cure disease.
To return to clinical practice. Is there direct evidence of effectiveness in specific settings in which outcome is measured as a routine? In physical conditions like diabetes, where measurement of an indicator is integral to treatment, it is easy to establish evidence for effectiveness (benefit attributable to routine treatment), but in other chronic disorders like arthritis, where outcome measurement is not routine, the task is more difficult. In psychiatry, where outcome measurement in routine clinical practice is rare, the situation is similar. For example, there are four studies of specific clinical environments, two negative, two positive in regard to effectiveness. The first, by Weisz et al. [18] concluded, on the basis of meta-analyses of psychotherapy for children and adolescents, that the effects of treatment diminished as the environment in which the studies were conducted increasingly resembled routine clinic practice, and that once they did, effectiveness was no longer evident. Following that, Shadish et al. [19] reanalysed 56 studies that seemed representative of adult clinic practice to a greater or lesser degree. The effectiveness in the research-type clinics was consistent with the experimental studies, but there was a paucity of data about outcomes in sites most like the average clinical setting.
There are a number of positive studies of the effectiveness of drug therapy, for the pharmaceutical industry has realised the importance of demonstrating that their treatments are effective. For example, MacKay et al. [20] studied SSRI in general practice and obtained data on four SSRI in 10 000 patients. Three SSRI, but not the fourth, were regarded as effective. If the general practitioners' routine rating of outcome was unbiased, this result rules out some placebo or spontaneous remission effect as an explanation for the difference and so is evidence of effectiveness of three of the drugs. Hunt and I [8] have reported on the long-term results of treatment at our clinic, treatment of patients who were referred, treated routinely and were not the focus of research. The results, effectiveness data from a specialised tertiary referral clinic, were marginally less strong than the usual experimental studies, but even so subjects made a profound change in symptoms, disability and risk factors by the end of a brief period of treatment, and then continued to improve without additional treatment over the 2-year follow-up period. People who refused or dropped out of treatment were also followed up. They made no improvement in the 2 years after being assessed, despite receiving treatment as usual from their general practitioner or psychiatrist. Treatment in our clinic was effective when the experimenters departed. Treatment as usual was not.
The National Mental Health Strategy specified outcome management as one of its goals, and suggested that services could eventually be funded according to their outcome, provided that the outcome could be attributed to the intervention being paid for. If a service shows a drop in disability scores with treatment, one could presume that a treatment effect had been achieved, provided there is some evidence of an association between being treated and getting better, like a dose response curve: that is, people who had more treatment did better than people who received less treatment. There are three substantial Australian studies [21, [22], [23]], two inpatient, one in community care, that used the Health of the Nation Outcome Scales (HoNOS) or the Life Skills Profile, and showed that improvement follows both treatment and illness phase transition (e.g. admission to discharge), but failed to show that the change was associated with the intensity or duration of treatment. In all of these three studies, one would presume that the sicker people required longer admissions or more occasions of service to produce the same improvement, so to some extent the ability of these studies to show a dose-response curve was compromised. But, without a dose-response curve in a pre-post trial or equivalent methodological safeguard, one could not be certain that improvement was the direct result of treatment; after all it could be another example of the benefits of non-specific therapy as advocated by Frank. In another community study [24], an RCTthat compared intensive with routine case management, it was clear that the group who received intensive case management did better. But again the number or duration of visits was not a significant determinant of improvement, showing the need for detailed data on service delivery if we are to draw the correct conclusion. Perhaps Wells [11] is right and we do need new and more powerful research designs to demonstrate what we believe is happening.
Effectiveness studies ask how many of the target population responded to treatment, whereas efficacy studies ask how many of the people who met the inclusion criteria and agreed to be treated responded to the experimental treatment. The difference is important. ‘Is the treatment effective for the average patient?’ is the question, not ‘Has it been shown to be better than placebo in randomised controlled trials?’. The data from our clinic refusers and drop-outs lead me to the conclusion that treatment-as-usual can, in some disorders, lead to disease-as-usual, a conclusion that can also be drawn from three of the outcome studies mentioned above. This type of information reinforces the need to implement evidence-based medicine and routine outcome measurement, if only to ensure that we have evidence that patients are getting better as we think they should, and society is getting health gains for its health dollars.
Efficiency
Before one can talk about efficiency, and therefore begin to ask whether mental health services are cost-effective, one has to have robust evidence of effectiveness. We do not have this at present but we may be able to estimate it from the focal studies of effectiveness that are beginning to appear. The cost-effectiveness of 500 life-saving interventions in medicine was recently published. No psychiatric interventions were listed. The median medical intervention cost US$19 000 per year of life saved [25]. Preliminary data suggest that psychiatric treatment will do quite well against this benchmark. Cowley and Wyatt [26] developed a treatment protocol for developing countries for both schizophrenia and manic depressive psychosis. The estimated cost benefits, $233 per disability adjusted life year (DALY) gained in schizophrenia and $268 per DALY gained in manic depressive illness, were regarded as affordable compared with other adult chronic disease interventions in similar countries. In developed countries like Australia, the costs are considerably greater, but modelling procedures like those used by Cowley and Watts should show most psychiatric treatments to be competitive with other medical interventions. Psychiatric treatment is not intrinsically expensive and such modelling strategies might inform changes in health service planning better than reliance on usage data like case mix which tends to reinforce the status quo.
In Australia, people with mental disorders account for 20% of the total burden of disease [27], mainly because of the many years they live with disability. If we have treatments that are potentially effective, we should be able to do something to reduce this burden. People with mental disorders receive services that, in total, cost only 5% of the health budget [28]. Clearly, there is an imbalance in Australia between the burden of mental disorders and the funds being expended on them. The World Bank notion [29] is that funds should be distributed according to the burden of a disease, measured in disability adjusted life years lost, and the cost-effectiveness of treatments for that disease, measured in dollars per disability adjusted life years gained. There is little point, they say, in investing in health if there is not going to be any health gain. Caiman [30] has argued similarly but adds an equity proviso, that while people in the UK are entitled to shelter, primary care and emergency services when ill, specialist services will have to be purchased on cost-effectiveness grounds with any money that remains. If we want the budget for mental health to be increased, we will have to gather the data to show that the services we can provide are cost-effective in reducing the burden of disease. And while that will not be easy, it is attainable, given that we have so many treatments of proven efficacy. The difficulty is the need to change the emphasis, and redeploy some mental health specialists from the care of people with chronic and intractable psychosis, in whom few gains can be expected, to the care of people with anxiety, depressive and substance use disorders who generate a greater cumulative burden of disease, and in whom gains can be expected. Perhaps we should look seriously at the Netherlands model of having separate budgets for prevention, cure and care [13], for only a strategy like that would free us from the procrustian bed we now inhabit.