Dutch guidelines for economic evaluation: ‘from good to better’ in theory but further away from pharmaceuticals in practice?

Abstract

Introduction

Economic evaluation (EE) is still a historically young discipline in healthcare. Originally started as a technique for assessing investments in the public sector, EE has been applied to healthcare since the last decades of the previous century and disseminated rapidly to many countries, thanks partly to a successful English manual,¹ which was later translated into many other languages. The first edition of the manual mainly referred to EE on health procedures, services or programmes, and not to products like drugs. Then, very soon, the Canadian State of Ontario and Australia issued the first pharmacoeconomic guidelines for drugs reimbursement,^2,3 followed later by many Western European countries (e.g. The Netherlands, Norway, Portugal and Sweden).

Last year the Dutch National Health Care Institute issued new guidance for EE in healthcare.⁴ Although the latest guidelines (Box 1) go beyond drugs (differently from the past), covering five further areas (prevention, diagnostics, medical devices, long-term care and forensics), pharmaceuticals are still the main field of application. Several elements of novelty reflect the methodological developments in EEs, two of which have been judged the most important recommendations for drugs:⁵ value of information analysis and indirect medical costs. Both concern EE ‘borderline’ subjects coping with uncertainty. The first is a statistical tool intended to be useful for assessing uncertainty related to the consequences of inappropriate decision-making in healthcare;⁶ the second concerns the unrelated future medical costs induced during the life years gained indirectly from a technology which prolongs the life of patients.⁷

In general, although we do not dispute that the new Dutch guidelines can be considered a scientific step forward in improving the requirements for EEs in healthcare,⁵ moving from good to better recommendations and still open to further improvement in the future, we still feel that health decision-makers should be fully aware of the general intrinsic limits of EE that have been underlined in the last two decades, moving from theory to practice. Thus, here we critically analyse the main key points for conducting an EE, underlining the main weaknesses of EE as pragmatic tools for public decision-making, particularly in the perspective of pharmaceutical pricing and reimbursement.

Efficacy/effectiveness

The need for a properly designed randomised clinical trial must be fully acknowledged before any EE is conducted on a specific drug.⁸ Lacking clinical evidence based on ‘head-to-head’ trials, an EE can hardly add valuable information for health decision-makers by definition, since the comparative efficacy of overlapping therapies can only be inferred through arguable estimates and uncertain assumptions. The manual mentioned at the beginning¹ clearly stated that EE is most useful and appropriate when preceded by evidence of efficacy and/or effectiveness. Nevertheless, the literature is full of EEs^9–11 that are likely to add only: (1) further uncertainty when based on sophisticated long-term models populated by doubtful data; and (2) poor information when based on short-term ‘decision trees’ derived from heterogeneous and inconclusive clinical studies. In general, on the basis of clinical studies not adequately supported by rigorous methodology, EEs can only speculate on uncertainty and overemphasise economic results of uncertain pharmacological treatments.

Quality of life

Quantifying quality of life is very hard and uncertain by definition. Even when restricted to aspects specifically related to health, quality of life is always a complex, abstract, multidimensional concept that defines an individual’s satisfaction or happiness with life s/he considers important. That is why any general scale to assess it – on which quality-adjusted life years are based, by far the most widely used variable in EE – necessarily stems from simplistic items¹² and immediately raises scepticism.¹³

Box 1.

Main features of the Dutch guidelines.

Key points
Efficacy/effectiveness	Systematic reviews based on RCTs are preferred. When RCTs are not available or they have downsides and limitations, non-randomised or non-comparative studies will suffice or may even be preferred. It is strongly recommended to perform the evidence synthesis on the basis of studies using direct comparisons. When direct comparisons are not available indirect comparative studies may be used; these studies should be carefully selected to prevent bias as much as possible. Placebo-controlled studies, in principle not suited for economic evaluations, may be included in a mixed treatment comparison or in a network meta-analysis to complete the network.
Quality of life	Expressed in QALYs, at least the EQ-5D-5L with Dutch valuation and, whenever relevant, also life years gained.
Viewpoint	Societal perspective. All relevant societal costs and benefits, irrespective of who bears the costs or to whom the benefits go, should be taken into account in the evaluation and reporting. In addition to the societal perspective the results can be presented from other perspectives (such as the healthcare perspective).
Discounting	Costs at a constant discount rate of 4%; future effects at a constant discount rate of 1.5%.
Time horizon	Lifetime. A different time horizon may be chosen if well-argued reasons can be provided that a lifetime horizon is not relevant.
Costing	Costs within the healthcare system (all medical costs directly related to or influenced by the prevention, diagnostics, therapy, rehabilitation and management of the disease or treatment, costs of the treatment of side effects of an intervention; all medical costs which may arise during gained life years, also referred to as indirect medical costs). Costs for patient and family (travel expenses, own contributions/co-payments). Costs incurred in sectors outside the healthcare system, productivity losses (paid or unpaid work). The friction cost method is recommended to calculate productivity losses as a result of paid work absenteeism. Both the volumes and the cost-prices per unit should be based on national data wherever possible. Volume data derived from international studies must be validated for the Netherlands.
ICER	Both an ICER with the QALY as outcome and an ICER with LYGs as outcome should be presented. When more than two interventions are compared, the results and ICERs of the different interventions should be presented in a well-structured manner.
Sensitivity analysis	The influence of parameter uncertainty should be examined using probabilistic sensitivity analyses in which all uncertain parameters are analysed. Univariate and multivariate sensitivity analyses should be performed to provide insight in the relative influences of input parameters on the ICER and in the consequences of fixed values in the model such as discount rates and prices. Value of information analyses can define whether and what kind of additional study is efficient and they can indicate whether the definitive decision should be postponed while awaiting the results of additional studies.

ICER: incremental cost-effectiveness ratio; LYG: life year gained; QALY: quality-adjusted life year; RCT: randomised clinical trial.

Source: Zorginstituut Nederland. Guideline for economic evaluations in healthcare (2016), https://www.zorginstituutnederland.nl/binaries/content/documents/zinl-www/documenten/publicaties/publications-in-english/2016/1606-guideline-for-economic-evaluations-in-healthcare/Guideline+for+economic+evaluations+in+healthcare.pdf.

More specifically, a still crucial issue – already mentioned in the editorial on Dutch guidelines⁵ as a topic to be addressed in the future – is that the evaluation of health states by patients can differ substantially from that of the general public (on whom quality-adjusted life year gains are assessed), who have not necessarily experienced the same impairments. The likelihood of adaptation to a condition is a well-known and debated problem which can bias the findings.¹⁴ For instance, after individuals needing a wheelchair assigned higher scores to worse health states than those still walking,¹⁵ we concluded that the former might have accepted their present situation better than the latter.

Viewpoint

According to best practice, an EE should take a societal perspective to include all costs and benefits of a new technology. Different from the third-party payer’s viewpoint, the societal perspective (recommended by Dutch guidelines) also includes indirect costs, i.e. the labour costs resulting from the loss of patients’ and/or carers’ productivity due to illness. This is a controversial and unsettled area since methods to quantify indirect costs still suffer major limitations and their application is open to discretion in practice.¹⁶

Two methods are currently mentioned most for evaluating indirect costs. The traditional ‘human-capital approach’, by far the most frequently used in published EEs, is based on the idea that an individual’s value is her/his production potential, so mortality and/or morbidity reduce this. Assuming a state of full employment in the long run, per capita average wages are mainly used to estimate the earnings lost during the absence from work and this is likely to lead to overestimates,¹⁷ particularly in this period of economic crisis when full employment is hardly ever the case in the EU. More realistically, the ‘friction-cost method’ assumes that, in the absence of full employment, indirect costs occur only during what is called a ‘friction period’,¹⁸ i.e. the time organisations need to restore the production lost due to having to replace an ill worker. Adopted since the first edition of the Dutch guidelines, it is seldom applied in other jurisdictions, requiring a huge amount of administrative information at country level to estimate the length of the friction period and the labour productivity. Yet, for the sake of methodological consistency, the friction method should imply a different approach for the evaluation of direct costs too.¹⁹

In general, a societal perspective can hardly fulfil healthcare decision-makers’ expectations, as they can only manage their own budget and other potential savings fall outside their control.²⁰ Actually, even an EE limited to the healthcare system’s perspective does not necessarily overcome this concern, since the ‘silos effect’ (i.e. the difficulty of redeploying costs and savings from one budget to another) can persist in healthcare management as well (e.g. from healthcare to health prevention).²¹

Discounting

Discounting²² is part of another never-ending debate in EE, probably hard to understand for non-economists. The economic theory recommends discounting as necessary to correctly compare therapeutic interventions whose costs and consequences occur at different times.²³ According to the ‘time preference’ concept (i.e. better to spend money later than sooner, in order to be able to use it in the meantime), discounting should be applied to costs borne in different years and the discount rate adopted should be a proxy of the long-term rate prevailing in financial markets (i.e. the opportunity cost of an investment). Though harder to grasp, discounting should be applied to benefits too for the sake of consistency with ‘time preference’, although not necessarily using the same rate.²⁴ Until recently the dominant theory was to apply the same rate to both costs and benefits, but the Dutch guidelines have prescribed differential discounting (4% for costs and 1.5% for benefits) since the previous version, in line with the latest developments in the literature.²⁵

Time horizon

A long enough time horizon is recommended for any EE to include all the economic and health consequences. This often implies a very lengthy horizon, especially for chronic therapies when lifetime is recommended (like in Dutch guidelines). In theory, a long-term horizon raises the question whether to start spending immediately on a new treatment that might lead to savings in the future,²⁰ since a new technology can only be introduced by additional funding or by disinvesting from alternative treatments. In practice, the need for extensive reliance on assumptions is an intrinsic limit of long-term models dealing with clinical efficacy and cost evaluation, particularly for new drugs where only short-term experimental data are available.¹¹ Of course, this is likely to lead to great within- and between-study variability generated by authors’ choices of information sources and assumptions. For instance, the unrelated future medical costs⁷ – mentioned as a major element of novelty in the new Dutch guidelines – are likely to become only as substantial as they are uncertain to predict in the long term.

Costing

Costs should be the mainstay of any EE and its major ‘added value’ in the field of health technology assessment. Different from efficacy, costs are dramatically affected by domestic features (e.g. clinical practice patterns and healthcare system frameworks) so can hardly be extrapolated from one country to another.²⁶ It is worth recalling that the estimate of each cost item is made up of both resource use and unit cost.

Resource consumption

The volumes of all healthcare cost items estimated in an EE are rarely available from a single source and their collection often results in a piecemeal, patchy exercise,^11,27,28 with data extracted from a heterogeneous mix of more (e.g. clinical and administrative records) or less (e.g. foreign data, expert panels and assumptions) credible sources.

Unit costs

Ideally representing the ‘opportunity cost’ of the product/service considered (i.e. the value of the benefit foregone without purchasing its best alternative use),¹ unit costs are the second cost component and vary in practice from micro-costs to prices and tariffs according to the cost item considered.²⁵ Apparently easier to evaluate and thus less debated, they can actually be rough proxies of real healthcare costs, starting from drug prices, which are becoming increasingly uncertain in many EU countries both for new drugs under confidential agreements²⁹ and for mature drugs purchased through tenders whose awarded prices are kept confidential. Yet, the unit costs of hospital services (by far the main healthcare cost) are usually sourced from tariff lists. Although tariffs provide consistency and comparability of EEs at a system level, in European countries like Italy national lists stem from limited domestic surveys and are seldom updated.³⁰ Micro-costing studies on a representative sample of hospitals should be the most reliable source to estimate real costs;³¹ however, their results can vary a lot depending on the cost accounting method adopted (e.g. ‘top down’ or ‘bottom up’) and largely also on the reliability of the local accounting systems.

Incremental cost-effectiveness ratio

Unless a new technology dominates the current one (i.e. it is at the same time more effective and less expensive), the result of an EE where a new technology produces greater effects but only with greater costs is usually expressed through an incremental cost-effectiveness ratio.¹ The Incremental cost-effectiveness ratio is the cost per additional unit of benefit, mainly expressed in cost per quality-adjusted life year. The underlying question (hardly made explicit) that public decision-makers have to deal with in this period of unprecedented economic crisis is whether it is worth spending an increasing share of budgets on very expensive new drugs. Clearly the Incremental cost-effectiveness ratio can hardly contribute to address this question, since it simply represents the average cost of an additional health gain.³² In fact, as was paramount in the case of the new sky-high priced drugs for hepatitis C,³³ their reimbursement for prescriptions to all eligible individuals requires massive extra budgets that an Incremental cost-effectiveness ratio cannot capture by definition. This is also why, with today’s widespread cuts to health expenditure, budget-impact analyses have become increasingly recommended in national guidelines and popular among public decision-makers.²⁰

Sensitivity analysis

Sensitivity analysis is the last recommended step needed before concluding an EE, aimed at testing the confidence of the estimated Incremental cost-effectiveness ratio.¹ Since any EE contains some (often high) degree of uncertainty, the ‘baseline’ Incremental cost-effectiveness ratio is considered more solid if it does not change dramatically after varying the main assumptions and estimates used for calculating it. Beyond the more widely used types of deterministic (e.g. univariate and extremes) and probabilistic (e.g. Monte Carlo simulation and bootstrapping) sensitivity analyses,³⁴ the value of information analysis⁶ adopted in the latest Dutch Guidelines (see above) is one of the ‘frontier’ tools, intended to estimate the potential benefits of additional data to avoid taking inappropriate decisions. In general, any kind of sensitivity analysis, even the most sophisticated, cannot support the results of an EE based on incorrect clinical evidence but can only perpetuate these errors (i.e. ‘garbage in–garbage out’).³⁵

Funding

Last but not least, since financial support can play a crucial role in choosing EE methodology and assumptions,³⁶ we have to add the issue of funding in order to end our critical evaluation on behalf of public decision-makers. Obviously, industry is interested in supporting EEs that show some economic advantage for the sponsored product^37,38 and extensive recourse to modelling techniques (especially Markov chains) is the most flexible way to achieve it, requiring many assumptions whose choice is wide open to authors’ discretion. For instance, subjective health professionals’ opinions (typically ‘expert panels’) are a widely used ‘shortcut’ to fill in missing information.^39,40 As recent examples further confirm,^34,41 consultants can always manage to select the most favourable inputs to feed models and/or pick subgroups of patients in order to show an acceptable Incremental cost-effectiveness ratio for sky-high priced drugs. Even the thresholds set by public agencies like the English NICE might help target sponsored models, the desired favourable result being known ex ante.^42,43

Some years ago we reported that the presence of pharmaceutical sponsorship is highly predictive of a positive conclusion in EEs based on Markov models, with a risk of error less than 1%.⁴⁴ So, trying to resolve a very old dilemma raised in the literature,⁴⁵ we dare to say that ‘time has already told’ that marketing prevails over science in pharmacoeconomic models funded by industry, and disclosure of conflicts of interest is only a ‘palliative’ to deter these exercises of scant utility for public decision-making. At the same time, it is fair to recognise that most EE drawbacks are still there even when public agencies (like the English NICE) conduct these studies, the main difference being that the public interests of health systems should prevail over the private ones of the pharmaceutical industry in methodological choices and assumptions.

Comment

We have tried to summarise point by point all the limits that EE suffers moving from theory to practice. Even with more restrictive choices (e.g. adopting the third-party payer’s perspective, excluding non-healthcare costs and limiting the time horizon to the short term), most drawbacks of EE as a tool for pharmaceutical pricing and reimbursement still persist, being intrinsic and insurmountable. Even though part of health technology assessment, EE itself actually implies a multidisciplinary approach, requiring clinical, epidemiological, economic and administrative inputs.

The main EE pragmatic limit for health policy decisions on new technologies is the lack of vital information at the early stage of market authorisation. The latest European Medicines Agency tendency to fast-track drugs to marketing through the so-called adaptive licensing is likely to undermine further the clinical evidence on relative efficacy of new drugs compared to those already marketed.⁴⁶ As a consequence, most EEs on new drugs risk becoming mere forecasting exercises which lead to speculating on future economic ‘trade-offs’, where insufficient clinical evidence is projected into the future on the basis of heterogeneous assumptions and estimates, adding prices as an important source of variation, quite possibly misleading public decision-makers.

In conclusion, although we share EE as a ‘philosophical’ approach to assessing healthcare interventions and the new Dutch guidelines can be considered the ‘forefront’ of EE as an experimental technique, at the same time we find it hard to believe that EE is the pragmatic tool that public decision-makers should look at for answers to pricing and reimbursing new technologies, particularly drugs, in view of the huge underlying economic interests.

Footnotes

Declarations

References

Drummond

Stoddart

Torrance

. Methods for the economic evaluation of health care programmes, Oxford: Oxford University Press, 1987.

Ontario, Ministry of Health. Guidelines for preparation of economic analysis to be included in submission to drug programs branch for listing in the Ontario Benefit Formulary/Comparative Drug Index, Toronto: Ontario Ministry of Health, 1991.

Department of Health, Housing and Community Services. Guidelines for the pharmaceutical industry on preparation of submissions to the Pharmaceutical Benefits Advisory Committee, Canberra: Commonwealth of Australia, 1992.

Nederland Zorginstituut. Richtlijn voor het uitvoeren van economische evaluaties in de gezondheidszorg, Diemen: Zorginstituut Nederland, 2015.

Versteegh

Knies

Brouwer

. From good to better: new Dutch guidelines for economic evaluations in healthcare. Pharmacoeconomics 2016; 34: 1071–1071.

Bindels

Ramaekers

Ramos

Mohseninejad

Knies

Grutters

. Use of value of information in healthcare decision making: exploring multiple perspectives. Pharmacoeconomics 2016; 34: 315–322.

Rappange

van Baal

van Exel

Feenstra

Rutten

Brouwer

. Unrelated medical costs in life-years gained: should they be included in economic evaluations of healthcare interventions? Pharmacoeconomics 2008; 26: 815–830.

Garattini

Grilli

Scopelliti

Mantovani

. A proposal for Italian guidelines in pharmacoeconomics. Pharmacoeconomics 1995; 7: 1–6.

Garattini

Ghislandi

Padula

. Older anti-TNF-α agents: why not group them for common indications in the EU? Expert Rev Pharmacoecon Outcomes Res 2016; 16: 1–4.

10.

Garattini

van de Vooren

Curto

. Cost-effectiveness of trastuzumab in metastatic breast cancer: mainly a matter of price in the EU? Health Policy 2015; 119: 212–216.

11.

Garattini

Ghislandi

Da Costa

. Cost-effectiveness modeling in multiple sclerosis: playing around with non-healthcare costs? Pharmacoeconomics 2015; 33: 1241–1244.

12.

Kane

. The futility of utility weighting. J Clin Epidemiol 2008; 61: 1195–1196.

13.

Rawles

. Castigating QALYs. J Med Ethics 1989; 15: 143–147.

14.

Dolan

. The effect of the experience of illness on health state valuations. J Clin Epidemiol 1996; 49: 551–564.

15.

Ghislandi

Apolone

Garattini

Ghislandi

. Is EQ-5D a valid measure of HRQoL in patients with movement disorders? A comparison with SF-36 and FIM questionnaires. Eur J Health Econ 2002; 3: 125–130.

16.

Garattini

Tediosi

Ghislandi

Orzella

Rossi

. How do Italian pharmacoeconomists evaluate indirect costs? Value Health 2000; 3: 270–276.

17.

Koopmanschap

Rutten

. Indirect costs in economic studies: confronting the confusion. Pharmacoeconomics 1993; 4: 446–454.

18.

Koopmanschap

Rutten

van Ineveld

van Roijen

. The friction cost method for measuring indirect costs of disease. J Health Econ 1995; 14: 171–189.

19.

Johannesson

Karlsson

. The friction cost method: a comment. J Health Econ 1997; 16: 249–255.

20.

Garattini

van de Vooren

. Budget impact analysis in economic evaluation: a proposal for a clearer definition. Eur J Health Econ 2011; 12: 499–502.

21.

van de Vooren

Curto

Garattini

. Curing cervical cancer or preventing it: a case of opportunity cost in the long run? Vaccine 2014; 32: 6867–6869.

22.

Keeler

Cretin

. Discounting of life-savings and other non-monetary effects. Manage Sci 1983; 29: 300–306.

23.

Katz

Welch

. Discounting in cost-effectiveness analysis of healthcare programmes. Pharmacoeconomics 1993; 3: 276–285.

24.

Claxton

Paulden

Gravelle

Brouwer

Culyer

. Discounting and decision making in the economic evaluation of health-care technologies. Health Econ 2011; 20: 2–15.

25.

Weinstein

Stason

. Foundations of cost-effectiveness analysis for health and medical practices. N Engl J Med 1977; 31: 716–721.

26.

Pagano

Brunetti

Tediosi

Garattini

. Costs of diabetes: a methodological analysis of the literature. Pharmacoeconomics 1999; 15: 583–595.

27.

Koleva

De Compadri

Virgili

Nobili

Garattini

. A critical review of the full economic evaluations of pharmacological treatments for glaucoma. J Med Econ 2008; 11: 719–741.

28.

Garattini

Compadri

Koleva

Pasina

Nobili

. A critical review of the full economic evaluations of pharmacological treatments for colorectal cancer. J Med Econ 2008; 11: 177–197.

29.

van de Vooren

Curto

Freemantle

Garattini

. Market-access agreements for anti-cancer drugs. J R Soc Med 2015; 108: 166–170.

30.

Garattini

Curto

Padula

Freemantle

. Real-world evidence in economic evaluations: really realistic? J R Soc Med 2016; 109: 404–407.

31.

Garattini L, Guiuliani G and Pagano E. A model for calculating costs of hospital wards: an Italian experience. J Manage Med 199; 13: 71–82.

32.

Birch

Gafni

. Information created to evade reality (ICER). Things we should not look to for answers. Pharmacoeconomics 2006; 24: 1121–1113.

33.

van de Vooren

Curto

Garattini

. Pricing of forthcoming therapies for hepatitis C in Europe: beyond cost-effectiveness? Eur J Health Econ 2015; 16: 341–345.

34.

Brennan

Akehurst

. Modelling in health economic evaluation. What is its place? What is its value? Pharmacoeconomics 2000; 17: 445–459.

35.

Eddy

Assessing medical technology. In: Eddy

(ed). Technology assessment: the role of mathematical modelling, Washington, DC: National Academy Press, 1985, pp. 144–175.

36.

Krimsky

. Conflict of interest and cost-effectiveness analysis. J Am Med Assoc 1999; 282: 1474–1475.

37.

Friedberg

Saffran

Stinson

Nelson

Bennett

. Evaluation of conflict of interest in economic analyses of new drugs used in oncology. J Am Med Assoc 1999; 282: 1453–1457.

38.

Cornago

Li Bassi

De Compadri

Garattini

. Pharmacoeconomic studies in Italy: a critical review of the literature. Eur J Health Econ 2007; 8: 89–95.

39.

Morgan

Barer

Evans

. Health economists meet the fourth tempter: drug dependency and scientific discourse. Health Econ 2000; 9: 659–667.

40.

Rennie

Luft

. Pharmacoeconomic analysis: making them transparent, making them credible. J Am Med Assoc 2000; 283: 2158–2160.

41.

Garattini

Curto

Freemantle

. Personalized medicine and economic evaluation in oncology: all theory and no practice? Expert Rev Pharmacoecon Outcomes Res 2015; 15: 733–738.

42.

Burls

Sandercock

. How to make a compelling submission to NICE: tips for sponsoring organizations. BMJ 2003; 327: 1446–1448.

43.

Bell

Urbach

Ray

Bayoumi

Rosen

Greenberg

. Bias in published cost effectiveness studies: systematic review. BMJ 2006; 332: 699–703.

44.

Garattini

Koleva

Casadei

. Modeling in pharmacoeconomic studies: funding sources and outcomes. Int J Technol Assess Health Care 2010; 26: 330–333.

45.

Drummond

. A reappraisal of economic evaluation of pharmaceuticals. Science or marketing? Pharmacoeconomics 1998; 14: 1–9.

46.

Garattini

Curto

. European Medicines Agency: leave the UK but remain the same? J R Soc Med 2016; 109: 324–325.