Abstract
Scarcity of funds dedicated to health care means that choices of what to fund are inevitable. Currently choices are driven largely by historical patterns of expenditure and the influence of professional, industrial and community interest groups. Cost-effectiveness analyses provide additional information that can help decision-makers set funding priorities that will improve the effectiveness and efficiency of health services [1].
In the national and Victorian burden of disease (BOD) studies, mental disorders ranked third behind cardiovascular disease and cancer, accounting for 13% of the overall burden of disease [2], [3]. This raises the question as to why such a large burden persists while there is subsidized access to potentially effective treatments for most disorders [4], [5]. To answer this question the Mental Health Branches of the Australian Department of Health and Ageing and the Department of Human Services, Victoria commissioned the Assessing Cost-Effectiveness – Mental Health (ACE–MH) study. The research question for the ACE–MH study is to assess from a health sector perspective, whether there are options for change that could improve the effectiveness and efficiency of Australia's current mental health services by directing available resources toward ‘best practice’ cost-effective services.
Overview of methods
The ACE–MH study uses a macro-economic evaluation approach to aid decision-making in mental health care in Australia [6]. This approach aims to give policy makers and health experts in Australia a greater involvement in both the study design and conclusions, as recommended by the Panel on Cost Effectiveness in Health and Medicine [1]. Recently the Centre for Health Program Evaluation and the Department of Human Services, in a collaborative effort, trialled a similar approach in the analysis of cancer control priorities in Australia [7]. Concurrently, a further study, ACE–Heart Disease, is being conducted using the same methods.
The key characteristics of the ACE–MH approach are: a clear rationale for selection of options for change; an evidence-based approach; adoption of a two-stage approach to the assessment of benefit, involving both health gain (i.e. cost per disability-adjusted life year [DALY] saved) and ‘judgement’ aspects which are included as second stage filters (‘equity’, ‘strength of evidence’, ‘feasibility’ and ‘acceptability to stakeholders’); use of an economic protocol specifically developed for the study, which ensures transparency and consistency of methods; and extensive uncertainty testing.
The DALY was chosen as the measure of health gain for this study because it is able to capture both morbidity and mortality effects across a wide range of disorders and intervention types and because baseline information on health status was available using the DALY for Australia [2], [3].
ACE–MH was guided by a steering committee of mental health experts, policy-makers and representatives of community mental health organizations, working to the following terms of reference: To select up to 30 major interventions for mental disorders, based on: the size of the problem addressed; importance in terms of current expenditure; relevance to current policy decision-making; availability of evidence to support analyses; indications that additional expenditure would lead to significant health gain or conversely, that decreased expenditure would lead to little or no reduction in health outcomes; and ability to specify the intervention in clear concrete terms. To define ‘benefit’ and the associated criteria by which the interventions are judged. To critically examine the evidence and analyses presented by the researchers. To formulate conclusions based on the presented evidence.
The interventions chosen for analysis are shown in Table 1. The comparator to the interventions selected as options for change in the ACE studies is ‘current practice’. To determine ‘current practice’ we utilize the Australian National Surveys of Mental Health and Wellbeing: Adult Component [8], the Child and Adolescent Component [9], the Low Prevalence Component [10] and expert advice. Current practice includes a mixture of no treatment, treatment with evidence-based medicine (EBM) and treatment with non-EBM. For most interventions in the ACE–MH study we model the effect of switching those currently on non-EBM onto an evidencebased intervention. An intervention was classed as evidence-based if it was currently recommended in clinical practice guidelines and supported by randomised controlled trial evidence. Current contact with EBM was defined from self-reported receipt of interventions.
Interventions selected for analysis in ACE–Mental Health
The perspective in ACE–MH is that of the health sector. This includes the government as health service provider, as well as the impact on patients and their families/carers. Discounting at 3% is applied to both costs and benefits to match the rate chosen in the Australian BOD studies [2], [3].
The target audience for the interventions is Australians with prevalent disease in the year 2000. The time horizon, both for the provision of the interventions and for tracking the associated costs and consequences, is specified in the individual reports. Due to the heterogeneity in the illness/intervention combinations, it is not possible to have a uniform time horizon. Time horizons were chosen to realistically reflect how the interventions would be applied in real life; and to ensure tracking of all relevant costs and benefits. Regardless of the time horizon, it is assumed that all interventions are in ‘steady-state’, that is they work at their full potential and that trained practitioners are available. Thus, the analyses do not address implementation and ‘learning curve’ issues or costs.
Assessment of benefit
Benefits are calculated by a two-stage process. The first stage involves the estimation of the health gain that could be attributed to each intervention using the DALY. The second stage involves the assessment of issues that either influence the degree of confidence that can be placed in the cost-effectiveness ratios (such as the level of available evidence), or broader issues that need to be taken into account in decision-making about resource allocation (such as equity and acceptability to stakeholders).
Stage one: measurement of the health gain
In comparison to other health areas such as cancer and heart disease, the emphasis in mental health services is largely on improvement in quality of life of those with the mental disorder, as the current repertoire of interventions provides limited opportunities for primary prevention and improved survival. This puts a heavy emphasis on the ability to translate measures of impact described in the literature into a change in the DALY measure, and in particular, the years lived with disability component. Years lived with disability are determined by the incidence, duration and severity of disease. Severity is measured by the DALY disability weights on a scale from 0 to 1, with zero being full health and one being death. In the ACE–MH study we use the Dutch disability weights [11], which are also used in the Australian burden of disease studies [2], [3]. Most interventions included in ACE–MH impact on disease severity, with prevention or delay of relapse also being measured in some interventions (e.g. maintenance treatment of depression and family interventions for schizophrenia).
The first step in measuring the health gain is to determine the efficacy of an intervention, that is the impact of the intervention on severity, duration or risk of relapse. When the intervention affects the severity of the mental disorder we use the effect size as our main measure of efficacy. The second step is to transfer the measure(s) of effect into a change in the DALY disability weight. These two steps are described below.
Effect size
Effect sizes are used to measure the benefit of an intervention in relation to a placebo or other control group. In the psychiatric literature outcomes are most commonly expressed as continuous measures but many different scales are used. Thus, the most relevant measure of effect size for mental health interventions is the standardized mean difference. It quantifies the magnitude of the difference between two groups in a metric-free unit, by expressing the mean difference in standard deviation units. We use Hedges' g [12] because it includes an adjustment to correct for small sample bias and is used in Cochrane Collaboration systematic reviews.
If there is a systematic review or meta-analysis of the intervention, the effect size can often be obtained from this. However, for many of the interventions we have conducted our own meta-analysis because systematic reviews are not available, do not include all relevant studies or present the results in a form that is different from that required for ACE–MH. Wherever possible we use randomised controlled trials to calculate the effect size, as this is the best methodology for determining treatment efficacy [13].
We pool effect sizes using the random effects method of DerSimonian and Laird [12]. Firstly, an effect size is calculated for each study by averaging across selected outcome measures within the study. This differs from Cochrane systematic reviews but is consistent with meta-analyses of the psychiatric literature [14], [15] and allows inclusion of all relevant outcome measures that reflect health-related quality of life in calculating the effect size. In practice, much of the efficacy data presented for mental health interventions is limited to symptom-specific measures which can affect various aspects of health-related quality of life, to varying extents. Following meta-analysis, any heterogeneity between trials is investigated according to established methods [16].
A ‘placebo effect’ is not included in our modelling of effectiveness in ACE–MH due to controversy regarding its existence [17], [18]. Further, its inclusion has little or no impact on the relative effectiveness or cost-effectiveness of treatments because patients in the non-evidence-based treatment group or the non-adherent group also get a placebo effect (but no treatment effect).
Transferring the effect size into a change in the disability weight (DW)
Translating the effect size into a change in the DALY DW is the most difficult methodological issue in ACE–MH, as there is no well-established and accepted method available. From our exploration of available methods we settled on the ‘conversion factor method’ and the ‘survey severity method’. The results of both methods are used as a range to determine the total years lived with disability, costs and cost-effectiveness ratios for the interventions. Uncertainty from each of the methods is included in the results.
The ‘conversion factor method’ uses a DW ‘conversion factor’ to translate an effect size into a change in the DW [19]. We multiply the effect size by the DW conversion factor for the particular mental disorder. This conversion factor is an average change in DALY disability weights for the equivalent of a standard deviation change in severity for the particular mental disorder.
An assumption of this method is that the degree of change in effect size units in clinical trials reflects the degree of change in disability weights from application of the translation factor. While the relationship between disability weight and effect size change was defined for symptoms and the overall level of disability, the effect size from meta-analyses predominantly summarizes changes in symptoms. While there is a close correspondence between symptoms and disability in anxiety and depression [20], and in schizophrenia greater severity elicits less favourable preference values [21], it is not known if the magnitude of the change is comparable.
For the ‘survey severity method’ a health status measure from the mental health survey is used to classify the severity of respondents to calculate an average disability weight. The effect size is then applied directly to this health status measure, severity of respondents is reclassified and a new average disability weight calculated. The difference in average disability weights is the anticipated change due to treatment.
For anxiety and depression we use the Mental Component Score of the SF-12 in the adult component of the National Survey of Mental Health and Wellbeing [8] and for ADHD we use the Psychosocial Summary Score from the Child Health Questionnaire in the child and adolescent component of the Survey [9]. Both measures are designed to have a mean population value of 50 and a standard deviation of 10. The specific steps are illustrated in Table 2. A similar approach is used for schizophrenia but more assumptions are required (see Appendix).
The survey severity method: specific steps to calculate a change in the disability weight due to an intervention
An assumption of the ‘survey severity method’ is that the effect size calculated from clinical trials can be applied directly to a general health status measure such as the Mental Component Score or Psychosocial Summary Score. This may be a significant issue when the effect size from meta-analyses is calculated from a predominance of symptom measures, as these are usually more sensitive to outcome than generic measures.
Non-adherence
The non-adherence rate is important to the costeffectiveness ratio because patients who do not adhere to treatment would be expected to incur costs at no or reduced health benefit. For ACE–MH, we use both the dropout rate from trials and from longitudinal studies to determine the adherence rate. We used the uncertainty analysis to incorporate possible differences between trial results and ‘real-life’ by using a uniform distribution between the observations from trials and longitudinal studies. In the absence of longitudinal studies we assumed a lower value of 50%.
Stage two: the second filter criteria
While evidence on cost-effectiveness is the main focus of activity in ACE–MH, recognition is also given to broader aspects where decisions rest heavily on judgement and notions of ‘due process’. These additional criteria function as a ‘second filter’ by which each of the interventions is judged before recommending allocation of more or less resources. One of the roles of the steering committee was to select and apply the second filter criteria. The filters chosen are described in Table 3. The main outcome of the second stage analysis was a table for each intervention in which the issues were flagged and a qualitative judgement made explicit about each criterion and its impact.
The second stage filter criteria
Assessment of costs
A common convention [22] in costing is to describe the analysis in three steps: Identification – which activities/services and which cost impacts are included in the analysis? Measurement – what is the extent of resource usage associated with these activities/services? and Valuation – what is the monetary value of this resource usage?
For ‘identification’ the health sector perspective means that costs (and cost offsets) impacting on government, together with costs (and cost offsets) impacting on patients and their families/carers are included. Costs to government are essentially the resources involved in organizing and operating the services. Costs to patients and their families/carers are primarily identified as outof-pocket expenses associated with visits to health professionals and associated treatments. The steering committee also expressed an interest in time costs (i.e. travelling, waiting and treatment time). These costs were calculated, where available and presented separately. With the exception of supported employment for people with schizophrenia, we did not include production losses/gains associated with ill health itself (i.e. premature death, absence due to morbidity, reduced productivity while at work), nor did we include non-health care goods and services. Pathway analysis is used to identify component activities of the various interventions and their current practice comparator, based on the published literature and supplemented with expert advice.
The ‘measurement’ of resource usage is also facilitated by the pathway analysis. Component activities are identified and quantities estimated. For example, a contact for CBT therapy usually entails a standard 60-minute consultation with a psychologist. The number and type of medical visits, drugs and so on are estimated during this measurement step, with references documented.
When attempting to cost non-adherence to treatment, we could find no information on the likely subsequent health-seeking behaviour and associated costs. Thus, we assume that the non-adherers have the same healthseeking behaviour, and the same costs, as those currently not receiving evidence-based treatment. When estimating the cost of non-evidence-based treatment, we have utilized the National Surveys of Mental Health and Wellbeing to determine the average number of visits made to different health professionals in the previous 12 months (in the case of the adult component) or 6 months (in the case of the child and adolescent component). We have not included the cost of non-evidencebased drugs or natural therapies in the cost of nonevidence-based treatment. The conservative approach to the estimation of non-evidence-based care will produce conservative estimates of the economic merit of options for change.
In the ‘valuation’ step, a unit price for each of the activities, together with the data source, is specified. Costs to the government and to the patient (including family/carers) are reported separately. Costs are measured in real prices for the reference year (2000). Where necessary, the Australian Institute of Health and Welfare health sector deflators [23] are used to adjust prices to the reference year. Unit costs for doctor visits (GP, paediatrician and psychiatrists) and drugs are obtained from the Australian Department of Health and Ageing. For doctor visits costs to government (i.e. average benefit paid) and to patient (i.e. average difference between fee charged and benefit paid) are calculated from Medicare services that were processed in the 1999/2000 financial year. Both patient-billed and bulk-billed services are included but services provided in hospital are excluded, due to unreliability of the fee charged data. For drugs, costs are for scripts filled under the Pharmaceutical Benefits Scheme and the Repatriation Pharmaceutical Benefits Scheme for the 1999/2000 financial year. The cost to the patient is averaged over ‘general’ and ‘concession’ patients and ‘safety net’ and ‘non-safety net’ patients, in proportion to the number of prescriptions for each item (i.e. brand, form and quantity of the drug) within a class. Cost per script is converted to an average cost per day for each quantity and form using the recommended dose per day. When costing a class of drugs we weight the cost by current usage of the individual brands, rather than using the cheapest brand in the class.
Incremental cost-effectiveness
The cost-effectiveness ratio is determined as the incremental cost of the intervention divided by the incremental benefit and presented as cost (A$) per DALY saved. The incremental cost is defined as the cost of the intervention minus the cost of current practice. Likewise, the incremental benefit is the benefit of the intervention minus the benefit associated with current practice.
Uncertainty analysis
Simulation-modelling techniques (with Monte Carlo sampling) using @Risk software [24] are used to allow the presentation of an uncertainty range around the benefits, costs and cost-effectiveness ratios. This approach is recommended by the Canadian Coordinating Office for Health Technology Assessment [25] and is also mentioned as one of a number of methods of exploring uncertainty in the 1996 US Consensus Panel on Cost-Effectiveness in Health and Medicine [1].
The probability distributions around the input variables are based on: standard errors quoted in, or calculated from, the literature; the range of parameter values quoted in, or calculated from, the literature; and from expert advice on the likely scenarios under Australian conditions. In addition to the uncertainty range, the @RISK analysis can show the input parameters with the greatest influence on the final results and hence is an indication of research priorities if greater accuracy of results is desired.
Strengths and limitations of the methods
Strengths of ACE–MH include the use of a common economic protocol to ensure comparability of the results, extensive uncertainty testing, interpretation of costeffectiveness ratios within a broader decision-making framework that includes consideration of second filter criteria and use of Australian data for demography, health system costs and offsets, disease incidence/ prevalence, risk factors and disease burden.
The main limitation in ACE–MH methods is the measurement of the health benefit due to the intervention. The measurement of the effect size is based on accepted methods and uses the best study design for measuring efficacy, that is systematic reviews of randomised controlled trials [13] (Table 4). However, adjustment needs to be made to transfer trial results into real-life settings (effectiveness) where results may vary due to the motivation of clinicians and patients, the availability of skilled clinicians and the capacity to vary the intervention to suit the needs of the patient, among other things. We attempt to take these factors into account using the adherence rate. However, this may not explain all of the variation likely to be seen in real-life practice, nor does it fully address the issue of generalizability of trial results, which are often conducted in a highly selected patient group.
Classifying the strength of the evidence [7]
Translation of the effect size into a change in the DALY disability weight is another key issue. This required the use of new methods – the ‘conversion factor method’ and ‘survey severity method’ – each of which has its own strengths and limitations. The advantage of the ‘conversion factor method’ is that it allows measurement of health gain in a comparable manner for most mental disorders. The disadvantages of the method are that: the health state descriptions of severity states for psychosis were less formally derived than those for depression and anxiety; there is no DW conversion factor for attention deficit hyperactivity disorder (ADHD); and it cannot be used when outcomes from the trials are primarily dichotomous.
Like the ‘conversion factor method’, the ‘survey severity method’ also has the advantage of allowing measurement of health benefits in a comparable manner for most mental disorders. An important advantage is that it allows dichotomous outcome data to be incorporated when only data on relapse are available. This is done by calculating the odds ratio or relative risk of relapse and converting it to a risk difference [26], which is then used to change the proportions in each of two or more severity states. The disadvantages of the method are that: for most disorders only three or four severity states are specified by Dutch disability weights (this reduces the precision of continuous measurements of impact, although applying the effect size to a large number of survey respondents to some degree ‘evens out’ the discontinuous steps); and it relies on the accuracy of the survey instruments for measuring severity, such as the SF-12, DIP-DIS [27] (for schizophrenia) and Child Health Questionnaire. Both methods have the disadvantage that they do not allow measurement of the impact of side-effects, which is particularly important for drug interventions for psychosis.
ACE–MH was also limited by the lack of healthrelated quality of life outcome data from trials and the availability of only two to four DALY disability weights, which coarsely describe severity for each mental disorder. Future work would be enhanced by the use, in trials, of a combination of utility-based economic instruments that facilitate comparisons across quite different interventions and disorders (a multiattribute utility instrument) and which are sensitive to small changes (a disease-specific instrument) [28]. The accuracy of translating change in severity into DALY units would benefit from having disability weights specified for a greater number of different health states within each disorder. Also the valuation study from which the DALY ‘conversion factor’ by severity level is derived for mental disorders [19] would need to be replicated to increase confidence in its validity.
The limitation in the methods for translating an effect size into a change in the DALY disability weight mainly influences comparisons of interventions across disorders. However, it is important to note that the main driver of cost-effectiveness ratios is the effect size. Further, the effect sizes for different interventions for a particular disorder are comparable as the outcome measures are consistent. Thus, comparisons of costeffectiveness ratios within disorders are more valid than across disorders.
Conclusions
Despite the limitations in methods the ACE–MH study is providing useful information for policymakers. To our knowledge there have been no similar attempts at determining cost-effectiveness of interventions across a wide range of mental health interventions in a comparable manner. Despite considerable uncertainty around key input variables, clear distinctions in cost-effectiveness between mental health interventions (particularly within disorders) are apparent. Nevertheless, until there is greater consensus on how to quantify health benefits in mental disorders our estimates should be considered provisional, though indicative of the relative magnitude of the health gain. Results of the study will be published in upcoming papers in this and other journals. We hope the publication of our results will both encourage debate about future directions for mental health policy, and encourage further research to clarify those issues where current knowledge is lacking.
Caveat
The ACE-MH project was jointly funded by the Australian Department of Health and Ageing, Mental Health and Suicide Prevention Branch and the Department of Human Services, Mental Health Branch, Victoria in recognition of the importance of research into the costeffectiveness of interventions in mental health treatment and care. This work draws upon, but is also limited by the available research and the assumptions necessary to complete the work.
The results of the analyses provide valuable material, likely to contribute to future policy deliberations by all service providers. Conclusions drawn from the economic evaluations should be considered within the context of the second stage filter process, which qualifies the results taking into account issues of equity, feasibility, strength of evidence and acceptability to stakeholders. This second stage filter process addresses some of the practical considerations required for changes in actual service practice.
Footnotes
Acknowledgements
Principal investigators for the project are: Theo Vos, Rob Carter and Gavin Andrews. We thank members of the ACE–MH steering committee for their project input: David Barton, Graham Burrows (Chair), Sue Caleo, Vaughan Carr, Dermot Casey, Joy Easton, William Hart, Helen Herrman, Barbara Hocking, Assen Jablensky, Anthony Jorm, Lyn Littlefield, Patrick McGorry, John McGrath, Paul Morgan, Lorna Payne, Deb Podbury, Kristy Sanderson, Suzy Saw, Bruce Singh, Bruce Tonge, Ruth Vine, Harvey Whiteford. We also thank the other researchers who have worked on the project: Maturot Chalamat, Marie Donnelly, Justine Corry, Louise Heuzenroeder and Ruth Rossell.