Abstract
Introduction
Real-world evidence is an emerging worldwide paradigm that claims to join the forces of academia, research institutes and industry to advance in the field of data science, particularly pharmaceuticals. 1 Although appealing, the concept is still controversial and so a common, technical definition is currently lacking in the literature. According to the ISPOR task force report published almost 10 years ago, 2 real-world evidence can be defined as that drawn from ‘data used for decision making that are not collected in conventional randomized controlled trials (RCTs)’. In practice, while randomised controlled trials are the acknowledged ‘gold standard’ for efficacy, their selected populations, idealised conditions and limited time horizons may be considered intrinsic limits to assess effectiveness and costs.
The sources of real-world data can be various, 2 the main ones, in order of rigour, being: (i) registries (prospective observational cohort studies); (ii) electronic health records (mainly e-medical charts); and (iii) administrative databases (typically retrospective data). Real-world evidence is expected to support rational decision-making, especially after market approval of drugs. 3 This has recently encouraged regulatory authorities to fast-track drugs to market as soon as possible, e.g. the European Medicines Agency through its ‘adaptive licensing’. 4 Ideally, once preliminary efficacy and safety have been assessed, the evaluation of relative effectiveness and cost-effectiveness is postponed after marketing approval, relying on real-world data for evidence. Thus, real-world sources are potentially vital for economic evaluations.
To explore the current state of the art of the subject, we first conducted a literature review of European full economic evaluations which claimed to be based on real-world data and then discussed the policy implications from a third-party payer’s perspective.
Real-world and economic evaluations
Literature search
Main characteristics of the selected studies and real-world data sources.
COPD: chronic obustrictive pulmonary disease; CUA: cost-utility analysis; FP: fluoropyrimidine; HIV: human immunodeficiency virus; Qol: quality of life; RW: real world; TPP: third-party payer.
Unknown sample size.
Study-by-study analysis
In the most recent Dutch study (on follicular lymphoma), 5 the cost-effectiveness of rituximab (the sponsored drug) was assessed in different scenarios to match randomised controlled trials with real-world evidence. While randomised controlled trial efficacy and volumes of healthcare services came from the long-term follow-up of a European trial (334 patients), real-world effectiveness and resource consumption were mainly sourced from two national haematological registries, from which a sample of 113 patients was selected. To compare two subgroups of patients (treated and untreated with rituximab), the ‘propensity score’ method was applied, and eventually only 86 patients (43 per subgroup) were included in the analysis. Utility values were sourced from a British observational study (cited only as a congress abstract).
The second Dutch study (the only one funded by a public authority) 6 focused on oxaliplatin in therapeutic regimens for treating patients in stage III colon cancer. The authors matched efficacy from a large multicentre international randomised controlled trial (1347 patients) with real-world effectiveness from a national population-based observational study to obtain different scenarios, by virtually splitting real-world patients (391) too as eligible or ineligible according to the randomised controlled trial inclusion criteria. Utility values were entirely derived from the literature. Resource consumption for estimating costs was taken from the registry mentioned for all scenarios (including that based on randomised controlled trial efficacy). Retrospective real-world data led to two unbalanced arms (281 with oxaliplatin vs. 110 without). This was the only study that estimated micro-costs in a sample of Dutch hospitals to cost hospital services.
In the Italian study (on HIV infection), 7 the real-world data to assess the effectiveness of two alternative antiretroviral regimens were derived from a clinical database of a big hospital in Lombardy region, but the sample size and patients’ characteristics were not reported. Mortality rates were based on national statistics; quality of life was sourced from American literature and validated for Italy by an expert panel of 10 infectious disease specialists. Real-world resource consumption was taken from the Lombardy region administrative database (unknown number of patients in this case too), except for two main side effects from national clinical guidelines.
The British study 8 compared indacaterol (the sponsored drug) with tiotropium and salmeterol in patients with chronic obstructive pulmonary disease. Efficacy and utility were derived from multi-centre international randomised controlled trials on indacaterol, chronic obstructive pulmonary disease-related mortality rates from a Spanish economic evaluation. Real-world data were limited to the resource use of the main healthcare services and taken from a large national survey of 20,001 subjects. One clinical expert validated all resource consumption, including that from foreign literature and assumptions for chronic obstructive pulmonary disease exacerbations. Despite the short time horizon (three years), efficacy and costs were both discounted.
Policy implications
Real world is a fashionable term that still finds scant application in European economic evaluations according to our review. A few recent studies claimed to refer to real-world evidence, mostly based on mixed data sources and small real-world samples. The major apparent contradiction was that, despite reviewed-real claims, models (mainly long-term) populated by a mix of sources (including expert opinions and authors’ assumptions) underpinned all studies reviewed – real ‘patchworks’ like many other published economic evaluations. 9
In theory, it is obvious to insist that randomised controlled trials (especially those for market approval) cannot prove effectiveness so economic evaluations based on them should be called cost-efficacy rather than cost-effectiveness analyses. 6 In practice, however, it is hard to demonstrate effectiveness in real-world by means other than randomised controlled trials, because of the many potential biases mainly generated by lack of randomisation. 3 Health policy makers have relied on the randomised controlled trial design for information on efficacy with good reason since allocating patients by chance to alternative treatment conditions permits an unbiased comparison of treatment differences. In a properly designed randomised controlled trial, any difference observed between the randomised conditions at the end of the trial must be due to either the treatment itself or the play of chance, and statistics can assess the extent to which the differences have arisen by chance or not. As a consequence, real-world evidence-based effectiveness is still scant in literature. For instance, an attempt to replicate the findings of landmark randomised controlled trials in heart failure, using a sophisticated propensity score approach in real-world data, ended in failure with a massively biased estimate providing a qualitatively opposite (and incorrect) result to that found in the randomised controlled trials. 10
For costs, i.e. the real ‘added value’ of economic evaluations, it is worth recalling that the estimate of each cost item is made up of both resource use and unit cost. Real-world data can only contribute to assessing the former, while the latter require different sources by definition. 11 In addition, volumes of all cost items are hardly ever available from a single source so more than one is usually necessary and models populated with real-world data can hardly be an exception.
Unit costs are the second cost component, as influential as resource consumption in estimating real costs and far from realistic in many economic evaluations, starting from drug prices, which are becoming increasingly uncertain for both new drugs under confidential agreements 12 and mature drugs purchased through tenders. 13 Then too, drastic price reductions thanks to generics and biosimilars after patent expiry are hardly ever assumed for already marketed drugs, even in long-term models. Besides drugs, the unit costs of hospital services (by far the main cost from a third-party payer’s perspective) are usually sourced from (DRG-like) tariffs – not from micro-costs – which are often rough proxies of real costs in many settings, although their use provides consistency and comparability of models at a system level. This is particularly true in European countries like Italy, where national tariffs are seldom updated (twice during the last decade).
Comment
To conclude, we are afraid that expectations raised by real-world evidence will be unlikely fulfilled in the short run for effectiveness and are even compromised in the longer run for costs. European regulatory authorities must be aware of these limits and should reconsider the present tendency to rely on preliminary efficacy and safety for market approval and on cost-effectiveness for pricing and reimbursement after launch. We believe they would do better to push the pharmaceutical industry from the very start of the approval procedure for new drugs to produce evidence on comparative efficacy with those already marketed and therapeutically overlapping, then setting prices according to their incremental efficacy (if shown). 14 Otherwise, it is easy to predict that pharmaceutical expenses will become more and more unsustainable in most EU countries (wealthy Western ones included) during this (never-ending) period of economic crisis.