Abstract
Introduction
Economic evaluation (EE) is still a historically young discipline in healthcare. Originally started as a technique for assessing investments in the public sector, EE has been applied to healthcare since the last decades of the previous century and disseminated rapidly to many countries, thanks partly to a successful English manual, 1 which was later translated into many other languages. The first edition of the manual mainly referred to EE on health procedures, services or programmes, and not to products like drugs. Then, very soon, the Canadian State of Ontario and Australia issued the first pharmacoeconomic guidelines for drugs reimbursement,2,3 followed later by many Western European countries (e.g. The Netherlands, Norway, Portugal and Sweden).
Last year the Dutch National Health Care Institute issued new guidance for EE in healthcare. 4 Although the latest guidelines (Box 1) go beyond drugs (differently from the past), covering five further areas (prevention, diagnostics, medical devices, long-term care and forensics), pharmaceuticals are still the main field of application. Several elements of novelty reflect the methodological developments in EEs, two of which have been judged the most important recommendations for drugs: 5 value of information analysis and indirect medical costs. Both concern EE ‘borderline’ subjects coping with uncertainty. The first is a statistical tool intended to be useful for assessing uncertainty related to the consequences of inappropriate decision-making in healthcare; 6 the second concerns the unrelated future medical costs induced during the life years gained indirectly from a technology which prolongs the life of patients. 7
In general, although we do not dispute that the new Dutch guidelines can be considered a scientific step forward in improving the requirements for EEs in healthcare, 5 moving from good to better recommendations and still open to further improvement in the future, we still feel that health decision-makers should be fully aware of the general intrinsic limits of EE that have been underlined in the last two decades, moving from theory to practice. Thus, here we critically analyse the main key points for conducting an EE, underlining the main weaknesses of EE as pragmatic tools for public decision-making, particularly in the perspective of pharmaceutical pricing and reimbursement.
Efficacy/effectiveness
The need for a properly designed randomised clinical trial must be fully acknowledged before any EE is conducted on a specific drug. 8 Lacking clinical evidence based on ‘head-to-head’ trials, an EE can hardly add valuable information for health decision-makers by definition, since the comparative efficacy of overlapping therapies can only be inferred through arguable estimates and uncertain assumptions. The manual mentioned at the beginning 1 clearly stated that EE is most useful and appropriate when preceded by evidence of efficacy and/or effectiveness. Nevertheless, the literature is full of EEs9–11 that are likely to add only: (1) further uncertainty when based on sophisticated long-term models populated by doubtful data; and (2) poor information when based on short-term ‘decision trees’ derived from heterogeneous and inconclusive clinical studies. In general, on the basis of clinical studies not adequately supported by rigorous methodology, EEs can only speculate on uncertainty and overemphasise economic results of uncertain pharmacological treatments.
Quality of life
Main features of the Dutch guidelines.
ICER: incremental cost-effectiveness ratio; LYG: life year gained; QALY: quality-adjusted life year; RCT: randomised clinical trial.
Source: Zorginstituut Nederland. Guideline for economic evaluations in healthcare (2016), https://www.zorginstituutnederland.nl/binaries/content/documents/zinl-www/documenten/publicaties/publications-in-english/2016/1606-guideline-for-economic-evaluations-in-healthcare/Guideline+for+economic+evaluations+in+healthcare.pdf.
More specifically, a still crucial issue – already mentioned in the editorial on Dutch guidelines 5 as a topic to be addressed in the future – is that the evaluation of health states by patients can differ substantially from that of the general public (on whom quality-adjusted life year gains are assessed), who have not necessarily experienced the same impairments. The likelihood of adaptation to a condition is a well-known and debated problem which can bias the findings. 14 For instance, after individuals needing a wheelchair assigned higher scores to worse health states than those still walking, 15 we concluded that the former might have accepted their present situation better than the latter.
Viewpoint
According to best practice, an EE should take a societal perspective to include all costs and benefits of a new technology. Different from the third-party payer’s viewpoint, the societal perspective (recommended by Dutch guidelines) also includes indirect costs, i.e. the labour costs resulting from the loss of patients’ and/or carers’ productivity due to illness. This is a controversial and unsettled area since methods to quantify indirect costs still suffer major limitations and their application is open to discretion in practice. 16
Two methods are currently mentioned most for evaluating indirect costs. The traditional ‘human-capital approach’, by far the most frequently used in published EEs, is based on the idea that an individual’s value is her/his production potential, so mortality and/or morbidity reduce this. Assuming a state of full employment in the long run, per capita average wages are mainly used to estimate the earnings lost during the absence from work and this is likely to lead to overestimates, 17 particularly in this period of economic crisis when full employment is hardly ever the case in the EU. More realistically, the ‘friction-cost method’ assumes that, in the absence of full employment, indirect costs occur only during what is called a ‘friction period’, 18 i.e. the time organisations need to restore the production lost due to having to replace an ill worker. Adopted since the first edition of the Dutch guidelines, it is seldom applied in other jurisdictions, requiring a huge amount of administrative information at country level to estimate the length of the friction period and the labour productivity. Yet, for the sake of methodological consistency, the friction method should imply a different approach for the evaluation of direct costs too. 19
In general, a societal perspective can hardly fulfil healthcare decision-makers’ expectations, as they can only manage their own budget and other potential savings fall outside their control. 20 Actually, even an EE limited to the healthcare system’s perspective does not necessarily overcome this concern, since the ‘silos effect’ (i.e. the difficulty of redeploying costs and savings from one budget to another) can persist in healthcare management as well (e.g. from healthcare to health prevention). 21
Discounting
Discounting 22 is part of another never-ending debate in EE, probably hard to understand for non-economists. The economic theory recommends discounting as necessary to correctly compare therapeutic interventions whose costs and consequences occur at different times. 23 According to the ‘time preference’ concept (i.e. better to spend money later than sooner, in order to be able to use it in the meantime), discounting should be applied to costs borne in different years and the discount rate adopted should be a proxy of the long-term rate prevailing in financial markets (i.e. the opportunity cost of an investment). Though harder to grasp, discounting should be applied to benefits too for the sake of consistency with ‘time preference’, although not necessarily using the same rate. 24 Until recently the dominant theory was to apply the same rate to both costs and benefits, but the Dutch guidelines have prescribed differential discounting (4% for costs and 1.5% for benefits) since the previous version, in line with the latest developments in the literature. 25
Time horizon
A long enough time horizon is recommended for any EE to include all the economic and health consequences. This often implies a very lengthy horizon, especially for chronic therapies when lifetime is recommended (like in Dutch guidelines). In theory, a long-term horizon raises the question whether to start spending immediately on a new treatment that might lead to savings in the future, 20 since a new technology can only be introduced by additional funding or by disinvesting from alternative treatments. In practice, the need for extensive reliance on assumptions is an intrinsic limit of long-term models dealing with clinical efficacy and cost evaluation, particularly for new drugs where only short-term experimental data are available. 11 Of course, this is likely to lead to great within- and between-study variability generated by authors’ choices of information sources and assumptions. For instance, the unrelated future medical costs 7 – mentioned as a major element of novelty in the new Dutch guidelines – are likely to become only as substantial as they are uncertain to predict in the long term.
Costing
Costs should be the mainstay of any EE and its major ‘added value’ in the field of health technology assessment. Different from efficacy, costs are dramatically affected by domestic features (e.g. clinical practice patterns and healthcare system frameworks) so can hardly be extrapolated from one country to another. 26 It is worth recalling that the estimate of each cost item is made up of both resource use and unit cost.
Resource consumption
The volumes of all healthcare cost items estimated in an EE are rarely available from a single source and their collection often results in a piecemeal, patchy exercise,11,27,28 with data extracted from a heterogeneous mix of more (e.g. clinical and administrative records) or less (e.g. foreign data, expert panels and assumptions) credible sources.
Unit costs
Ideally representing the ‘opportunity cost’ of the product/service considered (i.e. the value of the benefit foregone without purchasing its best alternative use), 1 unit costs are the second cost component and vary in practice from micro-costs to prices and tariffs according to the cost item considered. 25 Apparently easier to evaluate and thus less debated, they can actually be rough proxies of real healthcare costs, starting from drug prices, which are becoming increasingly uncertain in many EU countries both for new drugs under confidential agreements 29 and for mature drugs purchased through tenders whose awarded prices are kept confidential. Yet, the unit costs of hospital services (by far the main healthcare cost) are usually sourced from tariff lists. Although tariffs provide consistency and comparability of EEs at a system level, in European countries like Italy national lists stem from limited domestic surveys and are seldom updated. 30 Micro-costing studies on a representative sample of hospitals should be the most reliable source to estimate real costs; 31 however, their results can vary a lot depending on the cost accounting method adopted (e.g. ‘top down’ or ‘bottom up’) and largely also on the reliability of the local accounting systems.
Incremental cost-effectiveness ratio
Unless a new technology dominates the current one (i.e. it is at the same time more effective and less expensive), the result of an EE where a new technology produces greater effects but only with greater costs is usually expressed through an incremental cost-effectiveness ratio. 1 The Incremental cost-effectiveness ratio is the cost per additional unit of benefit, mainly expressed in cost per quality-adjusted life year. The underlying question (hardly made explicit) that public decision-makers have to deal with in this period of unprecedented economic crisis is whether it is worth spending an increasing share of budgets on very expensive new drugs. Clearly the Incremental cost-effectiveness ratio can hardly contribute to address this question, since it simply represents the average cost of an additional health gain. 32 In fact, as was paramount in the case of the new sky-high priced drugs for hepatitis C, 33 their reimbursement for prescriptions to all eligible individuals requires massive extra budgets that an Incremental cost-effectiveness ratio cannot capture by definition. This is also why, with today’s widespread cuts to health expenditure, budget-impact analyses have become increasingly recommended in national guidelines and popular among public decision-makers. 20
Sensitivity analysis
Sensitivity analysis is the last recommended step needed before concluding an EE, aimed at testing the confidence of the estimated Incremental cost-effectiveness ratio. 1 Since any EE contains some (often high) degree of uncertainty, the ‘baseline’ Incremental cost-effectiveness ratio is considered more solid if it does not change dramatically after varying the main assumptions and estimates used for calculating it. Beyond the more widely used types of deterministic (e.g. univariate and extremes) and probabilistic (e.g. Monte Carlo simulation and bootstrapping) sensitivity analyses, 34 the value of information analysis 6 adopted in the latest Dutch Guidelines (see above) is one of the ‘frontier’ tools, intended to estimate the potential benefits of additional data to avoid taking inappropriate decisions. In general, any kind of sensitivity analysis, even the most sophisticated, cannot support the results of an EE based on incorrect clinical evidence but can only perpetuate these errors (i.e. ‘garbage in–garbage out’). 35
Funding
Last but not least, since financial support can play a crucial role in choosing EE methodology and assumptions, 36 we have to add the issue of funding in order to end our critical evaluation on behalf of public decision-makers. Obviously, industry is interested in supporting EEs that show some economic advantage for the sponsored product37,38 and extensive recourse to modelling techniques (especially Markov chains) is the most flexible way to achieve it, requiring many assumptions whose choice is wide open to authors’ discretion. For instance, subjective health professionals’ opinions (typically ‘expert panels’) are a widely used ‘shortcut’ to fill in missing information.39,40 As recent examples further confirm,34,41 consultants can always manage to select the most favourable inputs to feed models and/or pick subgroups of patients in order to show an acceptable Incremental cost-effectiveness ratio for sky-high priced drugs. Even the thresholds set by public agencies like the English NICE might help target sponsored models, the desired favourable result being known ex ante.42,43
Some years ago we reported that the presence of pharmaceutical sponsorship is highly predictive of a positive conclusion in EEs based on Markov models, with a risk of error less than 1%. 44 So, trying to resolve a very old dilemma raised in the literature, 45 we dare to say that ‘time has already told’ that marketing prevails over science in pharmacoeconomic models funded by industry, and disclosure of conflicts of interest is only a ‘palliative’ to deter these exercises of scant utility for public decision-making. At the same time, it is fair to recognise that most EE drawbacks are still there even when public agencies (like the English NICE) conduct these studies, the main difference being that the public interests of health systems should prevail over the private ones of the pharmaceutical industry in methodological choices and assumptions.
Comment
We have tried to summarise point by point all the limits that EE suffers moving from theory to practice. Even with more restrictive choices (e.g. adopting the third-party payer’s perspective, excluding non-healthcare costs and limiting the time horizon to the short term), most drawbacks of EE as a tool for pharmaceutical pricing and reimbursement still persist, being intrinsic and insurmountable. Even though part of health technology assessment, EE itself actually implies a multidisciplinary approach, requiring clinical, epidemiological, economic and administrative inputs.
The main EE pragmatic limit for health policy decisions on new technologies is the lack of vital information at the early stage of market authorisation. The latest European Medicines Agency tendency to fast-track drugs to marketing through the so-called adaptive licensing is likely to undermine further the clinical evidence on relative efficacy of new drugs compared to those already marketed. 46 As a consequence, most EEs on new drugs risk becoming mere forecasting exercises which lead to speculating on future economic ‘trade-offs’, where insufficient clinical evidence is projected into the future on the basis of heterogeneous assumptions and estimates, adding prices as an important source of variation, quite possibly misleading public decision-makers.
In conclusion, although we share EE as a ‘philosophical’ approach to assessing healthcare interventions and the new Dutch guidelines can be considered the ‘forefront’ of EE as an experimental technique, at the same time we find it hard to believe that EE is the pragmatic tool that public decision-makers should look at for answers to pricing and reimbursing new technologies, particularly drugs, in view of the huge underlying economic interests.