The Expanding Role of Real-World Evidence Trials in Health Care Decision Making

Abstract

Real-world evidence (RWE) is the clinical evidence about benefits or risks of medical products derived from analyzing real world data (RWD), which are data collected through routine clinical practice. This article discusses the advantages and disadvantages of RWE studies, how these studies differ from randomized controlled trials (RCTs), how to overcome barriers to current skepticism about RWE, how FDA is using RWE, how to improve the quality of RWE, and finally the future of RWE trials.

Keywords

FDA observational pragmatic randomized controlled trial real-world evidence

Definitions of Real-World Data and Real-World Evidence

Although the terms real-world data (RWD) and real-world evidence (RWE) are frequently used interchangeably, they have different meanings. RWD are information gathered through observations of routine clinical practice from multiple sources that can be linked together to provide meaningful patterns. RWD is based on patients and their clinicians choosing treatments according to the patients’ clinical characteristics and preferences.¹

If the RWE trial design and conduct closely approximate testing the effectiveness of the product in real-world clinical practice, then the trial is known as a pragmatic trial² (as opposed to an explanatory trial, which aims to evaluate the efficacy of an intervention in a well-defined controlled setting)³ RWE is the analysis of RWD from a study designed with a high degree of pragmatism.⁴ The 21st Century Cures Act defines RWE as evidence derived from “data regarding the usage, or potential benefits or risks, of a drug derived from sources other than randomized clinical trials.”⁵ RWE represents conclusions derived from observations of patients in health care settings (rather than subjects in a research environment).

Why More RWE Is Now Available

The US health care system in the past few years has become able to generate and interpret large new sources of data from electronic health records, billing claims, disease registries, wearable devices, and patient generated data on social media. For example, 99% of hospitals across the country now use electronic health record systems (EHRs), compared to about 31% in 2003.⁶ In the US, 11% of office-based doctors used electronic records in in 2006,⁷ whereas by 2017, 90% of office-based doctors used electronic records.⁸ Every day, billions of user-specific data points are generated by wearable devices and smartphones, but over 99% of this newly created digital data remains unanalyzed.⁹ New open data policies, such as Open FDA¹⁰ and academic data sharing initiatives^11,12 are expanding the pool of searchable data for RWE investigators. New software for interpreting the emerging tsunami of data based on emerging capabilities of artificial intelligence, machine learning, and natural language processing are simplifying the tasks of combing through large databases to assess correlations between patient features, health care professional features, diseases, and treatments.

Advantages and Disadvantages of Traditional Randomized Controlled Trials

A randomized controlled trial (RCT) in medicine is an experiment whereby the subjects participating are randomly allocated to either the group receiving the intervention under investigation or to a group receiving standard treatment (or placebo treatment) as the control. This approach to assessing the benefit of a treatment is intended to reduce bias by balancing both known and unknown confounders into the two groups. In an RCT, subjects with particular traits, who are more or less likely to result in benefit from the intervention, are distributed roughly equally through random allocation into one treatment arm or the other. Randomization allows researchers to determine the effects of the intervention versus no intervention while other variables between groups are kept constant. The RCT is often considered the gold standard for evidence¹³ to evaluate the efficacy of an intervention.

RCTs have at least six disadvantages for guiding health care decision making. These include: (1) high cost, (2) long time to complete, (3) comparison against only placebo or few alternate treatments, (4) impractical eligibility criteria, (5) underrepresentation of some types of people, and (6) trial effect, which can unduly inflate the measured benefit of an intervention.¹⁴ Furthermore the patient’s underlying condition can affect the results of a RCT in way that may be difficult to account for with the randomization process.¹⁵ An example is the tendency of diabetes subjects with very elevated A1C levels to often benefit more from interventions in RCTs than subjects with only slightly elevated A1C levels.¹⁶ If only the former subjects are randomized, then the study will likely show a sizable benefit of the intervention but the results will not necessarily be generalizable for patients with diabetes whose control is not as poor as that of the trial subjects. If the study emphasizes generalizability and both types of subjects are recruited, then likely the benefit will become diluted and the trial organizers risk reaching a conclusion that the same intervention will fail to demonstrate efficacy in a diabetes population with diverse levels of glycemic control.

Advantages of RWE

RWE is derived from analyzing data collected from a health care setting that is not part of a context of an RCT. RWE trials are useful for generating hypotheses about relationships between real-world phenomena and outcomes. RWE trials can be classified as observational or pragmatic. Observational data can be defined as data generated from experience with routine medical care that has been systematically recorded, for example as electronic medical records, registries, or administrative claims in a format that can be used for research.¹⁷ In an observational study the researcher does not control the intervention being studied. Observational research provides a mechanism to determine risk factors and mechanisms of activities or interventions that could serve as potential targets in future RCTs.¹⁸ Pragmatic trials seek to answer the question of whether an intervention works under usual conditions. Randomized trials tend to be considered as either pragmatic or explanatory. Explanatory randomized controlled trials address whether an intervention works under ideal conditions.¹⁹ Features of an optimal pragmatic RWE trial are listed in Table 1.²⁰ RWE provides a different type of information than RCTs do. RWE is useful for comparing the effectiveness of various interventions in the real world, where patients are not being closely followed or encouraged to adhere to treatment like they are in clinical trials, especially when the goal of a study is to assess the durability of effect, generalizability, and long-term safety of an intervention. For low event rate endpoints, RWE can identify rare unanticipated complications better than RCTs, which generally study fewer subjects for shorter time periods. RWE studies can also provide information about real-world adherence to an intervention and the threshold of disease severity at which point the intervention is prescribed. The cost of RWE trials is much less than RCTs (especially for RWE pertaining to a long time period), and retrospective RWE studies do not present difficulties with recruitment—the appropriate subjects are already in the database. RWE is currently being used for some regulatory approval decisions, postapproval monitoring of safety signals, payer coverage decisions, and outcomes-based contracting.

Table 1.

Features of an Optimal Pragmatic RWE Clinical Trial.²⁰

1	There are no inclusion or exclusion criteria
2	Practitioners are not constricted by guidelines on how to apply the experimental intervention
3	The experimental intervention is applied by all practitioners, thus covering the full spectrum of clinical settings
4	The best alternative treatments are used for comparison with no restrictions on their application
5	The comparative treatment is applied by all practitioners, covering the full spectrum of clinical settings
6	No formal follow-up
7	The primary outcome is a clinical meaningful one that does not require extensive training to assess
8	There are no plans to improve or alter compliance for the experimental or the comparative treatment
9	No special strategy to motivate practitioner’s adherence to the trial’s protocol
10	The analysis includes all participants in an intention-to-treat fashion

Disadvantages of RWE

In RWE studies, biases can be introduced by unrecognized confounders. For example, patient and physician factors that determine the selection of a treatment may also affect health outcomes. Confounding by indication of treatment (selection bias) means that a physicians has a tendency to prescribe more effective treatments to patients who are most likely to benefit by them. Also, patients who adhere to a prescribed treatment may be more likely to engage in other healthy behaviors (adherence bias), which can create the false conclusion that the intervention is the sole cause of an improved outcome.²¹ In some datasets, there might be an imbalance of risk factors or protective factors with patients receiving one intervention compared to another intervention because the two groups were not randomized to have equal incidences of this factor, and the lower risk patients might appear to have a better outcome because of the intervention (confounding bias), but the difference in outcomes was actually due to the confounding factors. Some potential confounding factors, (such as obesity, sedentary lifestyle, alcohol use, or drug use) as well as some outcomes (such as mortality or change in symptoms) might not be consistently captured in electronic health records or medical claims. Adherence can be high in a clinical trial that measures efficacy (which is performance under ideal conditions); whereas in the real world adherence may be low and outcomes may be less favorable. Reporting bias can occur if some outcomes or datasets in the health record are selectively revealed or withheld. Whereas a RWE trial can inform on the overall performance of a treatment, it will be difficult to identify the specific components or biases that account for its effectiveness. Therefore, a RWE study is said to suggest a hypothesis that can be tested with a RCT.

RWD datasets reviewed by researchers after they have been created by clinicians are vulnerable to systematic omissions or misclassification. For electronic health records, secular trends in coding criteria and reimbursement policies for hospital claims can impact data registration.²² Data mining occurs when analysts reexamine existing datasets to generate new information. There is a concern that an organization can repeatedly analyze RWD with various models until a desired outcome is identified. Therefore, RWD can be manipulated via repeat analyses with nondisclosure of unhelpful results.²³ Data sharing between different health care organizations might be restricted because of privacy laws, which can limit the generalizability of a dataset to the patients from a single health care organization. At this time there are no generally agreed upon standards for the design, conduct, analysis, and reporting of RWE trials, which can undermine confidence in such studies and create a suspicion that the RWD has been “cherry picked” to prove a predetermined conclusion.²⁴

According to a Cochrane Systematic Review in 2014 about results of health care outcomes assessed with observational study designs compared with those assessed in randomized trials, little evidence was found for significant effect differences between observational studies and RCTs.²⁵ In the literature however, there are examples when effects identified in observational studies were not be reproduced in RCTs.^26-29 Given advantages and disadvantages of both RCTs and RWE studies, it is likely that the two types of trials in combination can provide evidence that neither type alone can readily provide.

The FDA RWE Program

In December 2018, FDA announced that Pursuant to the 21st Century Cures Act (Cures Act) signed into law on December 13, 2016, the agency is launching a Real-World Evidence Program, to apply across its drug and biologic review process.³⁰ This framework will allow for evaluation of drug or biologic products (but not devices) using RWE to help support the approval of a new indication, an expansion of the indication to a new population,³¹ or a post approval study.³² This program will cover both prospective noninterventional clinical trials where the population receiving the intervention is identified at the start of the study, such as in pragmatic trials, and retrospective observational studies where the population receiving the intervention is studied from historical data. Although FDA uses RWE for assessing postmarket product safety, the application of RWE to support effectiveness determinations has previously been limited to a few instances.³² A risk of this new policy will be that a RWE trial with unrecognized biases could lead to approval of an ineffective or unsafe new indication whereas previously, new indications each required an RCT.³³

How to Improve the Value of RWE

Six methods for improving the value of RWE have been discussed by clinical trial experts. First, stakeholders should develop methodology standards for RWE trials to improve the quality of these studies and increase the confidence of study users. The FDA plans to encourage this type of activity. A consensus panel of database experts recently recommended disclosure of a set of inputs to define a population from a database in any RWE study to improve the study’s transparency and reproducibility.³⁴ The “active comparator, new user” study design mitigates biases in RWE observational studies that compare interventions.³⁵ Second, since about one fourth of FDA-mandated postmarket studies are currently not being registered,^36,37 a mandatory national registry for RWE trials should be created. Currently an investigator can easily data mine by changing assumptions and repeating the RWD analysis until a desired outcome is generated. Such a practice is deceptive and would be inhibited by a registry that could be established to resemble www.clinicalatrials.gov. Third, since results of about one fourth of FDA-mandated studies are currently not being reported,^36,37 formal restrictions should be established that data from an unregistered RWE study could not be being published or submitted for regulatory purposes.³⁸ Fourth, for a fatal disease like cancer, a synthetic control arm accurately predicting mortality outcomes in patients not receiving an investigational intervention in a RWE prospective study should be built so that every patient in a pragmatic trial can receive active treatment, and no one would have to be randomized to the control treatment.³⁹ Such a high-quality EHR-derived mortality dataset for retrospective and prospective RWE generation was recently reported by Flatiron Health.⁴⁰ Fifth, RWE researchers should collaborate with (a) The Patient-Centered Outcomes Research Institute (PCORI) and its partner networks to link claims data with EHR data⁴¹; and (b) PCORI’s spin-off initiative, the National Patient-Centered Clinical Research Network (PCORNET), which is a network of research networks with a common data model and query system that is standardizing EHR data collected by more than 100 health institutions.⁴² Sixth, researchers should collect data from patients with the secure FDA mobile app that was released in November 2018.⁴³

Conclusion

RWE will become an increasingly important concept for determining the performance of drugs and biologics because of the exponential increase in electronic data available to researchers and the improved analytics tools that are also becoming available. RWE trials are much less costly than RCTs. For this reason, the pharmaceutical industry might want to replace many types of postmarketing RCT safety studies with RWE studies. Postmarket multiyear cardiovascular RCT outcomes trials for diabetes drugs cost in the range of $250 million dollars,⁴⁴ and RWE studies cost far less than that. RWE can measure outcomes of interventions for all patients, which is the most important measure of an intervention to patients, health care professionals, and payers.³⁹ Although, compared to RCTs, RWE has many advantages, but this approach to data collection also has potential problems with data quality, consistent methodology, and a risk of not accounting for confounding factors. Both RCTs and RWE trials have their specific weaknesses and strengths, and in combination the data from both types of trials can supplement each other.

RCTs will remain necessary for the foreseeable future for initial clearance of investigational treatments, which must initially be demonstrated to be effective in a core target group and for a core indication in an RCT before they can be introduced to clinical practice (where there is less oversight than in a RCT). However, rare complications of therapies might not be detected in the general population from an RCT with a small number of subjects, tight inclusion criteria, and a short duration of 6-12 months. RWE postmarketing studies will become increasingly used to (1) assess safety and effectiveness in real-world populations for regulatory purposes;⁴⁵ (2) compare clinical outcomes in real-world observational trials of clinical interventions (which might not necessarily require regulatory approval) to determine optimal treatment strategies;^46,47 (3) identify prescribing patterns for drugs with similar indications for population health analyses;⁴⁸ (4) measure resource utilization by patients in the real world;¹ and (5) compare economic outcomes in real-world observational trials to determine the most economically attractive treatment strategies.^49,50

In conclusion, it appears that RWE will increasingly supplement post approval RCTs, but not replace them. It is also highly unlikely that RWE trials could replace phase 1-3 trials for most therapies. Based on new RWE policies recently announced by FDA, it is expected that RWE studies will allow drugs and biologics to be approved more efficiently for new indications and to have their indications expanded if they demonstrate good postmarket performance. As the flaws in the structure of RWE trials become corrected with standards, rules, more reliable databases, and better methods for eliminating confounding factors, then the importance of this research method will continue to grow in the regulatory, clinical, and economic arenas.

Footnotes

Abbreviations

EHRs, electronic health record systems; PCORI, Patient-Centered Outcomes Research Institute; RCTs, randomized controlled trials; RWD, real-world data; RWE, real-world evidence.

Declaration of Conflicting Interests

The author declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: DCK is a consultant to EOFlow, Lifecare, Merck, Novo, Roche, and Voluntis.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

References

Blonde

Khunti

Harris

Meizinger

Skolnik

NS.

Interpretation and impact of real-world clinical data for the practicing clinician. Adv Ther. 2018;35:1763-1774.

Nicholls

Carroll

Brehaut

, et al. Stakeholder views regarding ethical issues in the design and conduct of pragmatic trials: study protocol. BMC Med Ethics. 2018;19:90.

Thorpe

Zwarenstein

Oxman

, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. CMAJ. 2009;180:E47-E57.

Jarow

LaVange

Woodcock

Multidimensional evidence generation and FDA regulatory decision making: defining and using “real-world” data. JAMA. 2017;318:703-704.

21st Century Cures Act, HR 34, 114th Cong, 2nd Sess, 2016.

Landi

. Survey: nearly all U.S. hospitals use EHRs, CPOE systems. Available at: https://www.healthcare-informatics.com/news-item/ehr/survey-nearly-all-us-hospitals-use-ehrs-cpoe-systems. Accessed February 20, 2019.

Vestal

. Some states lag in using electronic health records. Available at: https://www.usatoday.com/story/news/nation/2014/03/19/stateline-electronic-health-records/6600377/. Accessed February 20, 2019.

Practice Fusion I. EHR adoption rates: 20 must-see stats. Available at: https://www.practicefusion.com/blog/ehr-adoption-rates/. Accessed February 20, 2019.

Habib

Rehman

Liew

Wah

Shuja

Daghighi

Mining personal data using smartphones and wearable devices: a survey. Sensors (Basel). 2015;15:4430-4469.

10.

US Food and Drug Administration. OpenFDA. Available at: https://open.fda.gov/. Accessed February 20, 2019.

11.

Data sharing and the future of science. Nat Commun. 2018;9:2817.

12.

National Academy of Sciences. Sharing Clinical Trial Data. Available at: http://www.nationalacademies.org/hmd/Activities/Research/DrugForum/Sharing-Clinical-Trial-Data.aspx. Accessed February 20, 2019.

13.

Backmann

What’s in a gold standard? In defence of randomised controlled trials. Med Health Care Philos. 2017;20:513-523.

14.

Krause

Saver

RS.

Real-world evidence in the real world: beyond the FDA. Am J Law Med. 2018;44:161-179.

15.

Faraoni

Schaefer

ST.

Randomized controlled trials vs. observational studies: why not just live together?

BMC Anesthesiol. 2016;16:102.

16.

Sherifali

Nerenberg

Pullenayegum

Cheng

Gerstein

HC.

The effect of oral antidiabetic agents on A1C levels: a systematic review and meta-analysis. Diabetes Care. 2010;33:1859-1864.

17.

Heikinheimo

Bitzer

Rodríguez

LG.

Real-world research and the role of observational data in the field of gynaecology—a practical review. Eur J Contracet Reproduct Health. 2017;22:250-259.

18.

Barnish

Turner

The value of pragmatic and observational studies in health care and public health. Pragmat Obs Res. 2017;8:49-55.

19.

Thorpe

Zwarenstein

Oxman

, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62:464-475.

20.

Patsopoulos

NA.

A pragmatic view on pragmatic trials. Dialogues Clin Neurosci. 2011;13:217-224.

21.

Zuidgeest

MGP

Welsing

PMJ

van Thiel

, et al. Series: pragmatic trials and real world evidence: paper 5. Usual care and real life comparators. J Clin Epidemiol. 2017;90:92-98.

22.

Varkila

MRJ

Cremer

. Is research from databases reliable? Not sure [published online ahead of print December 14, 2018]. Intensive Care Med. doi:10.1007/s00134-018-5498-9.

23.

Hampson

Towse

Dreitlein

Henshall

Pearson

. Real-world evidence for coverage decisions: opportunities and challenges [published online ahead of print November 9, 2018]. J Comp Eff Res. doi:10.2217/cer-2018-0066.

24.

Schwartz

JL.

Real-world evidence, public participation, and the FDA. Hastings Cent Rep. 2017;47:7-8.

25.

Anglemyer

Horvath

Bero

Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. Cochrane Database Syst Rev. 2014;4:MR000034.

26.

Bhandari

Tornetta

III Ellis

, et al. Hierarchy of evidence: differences in results between non-randomized studies and randomized trials in patients with femoral neck fractures. Arch Orthop Trauma Surg. 2004;124:10-16.

27.

Freemantle

Marston

Walters

Wood

Reynolds

Petersen

Making inferences on treatment effects from real world data: propensity scores, confounding by indication, and other perils for the unwary in observational research. BMJ. 2013;347:f6409.

28.

Gaudino

Di Franco

Rahouma

, et al. Unmeasured confounders in observational studies comparing bilateral versus single internal thoracic artery for coronary artery bypass grafting: a meta-analysis. J Am Heart Assoc. 2018;7:pii:e008010.

29.

Hemkens

Contopoulos-Ioannidis

Ioannidis

JP.

Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey. BMJ. 2016;352:i493.

30.

US Food and Drug Administration. Statement from FDA Commissioner Scott Gottlieb, M.D., on FDA’s new strategic framework to advance use of real-world evidence to support development of drugs and biologics. Available at: https://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/ucm627760.htm. Accessed February 20, 2019.

31.

Khosla

White

Medina

, et al. Real world evidence (RWE)—a disruptive innovation or the quiet evolution of medical evidence generation? F1000Res. 2018;7:111.

32.

US Food and Drug Administration. Framework for FDA’s real world evidence program. 2018. Available at: https://www.fda.gov/downloads/ScienceResearch/SpecialTopics/RealWorldEvidence/UCM627769.pdf. Accessed February 20, 2019.

33.

Lieberman

. With media watchdogs on the sidelines, pharma-funded advocacy groups pushed Cures Act to the finish line. Available at: http://www.healthnewsreview.org/2016/12/with-media-watchdogs-sidelined-pharma-funded-advocacy-groups-pushed-cures-act-to-the-finish-line/. Accessed February 20, 2019.

34.

Wang

Schneeweiss

Berger

, et al. Reporting to improve reproducibility and facilitate validity assessment for healthcare database studies V1.0. Value Health. 2017;20:1009-1022.

35.

Lund

Richardson

Sturmer

The active comparator, new user study design in pharmacoepidemiology: historical foundations and contemporary application. Curr Epidemiol Rep. 2015;2:221-228.

36.

Miller

Korn

Ross

JS.

Clinical trial registration, reporting, publication and FDAAA compliance: a cross-sectional analysis and ranking of new drugs approved by the FDA in 2012. BMJ Open. 2015;5:e009758.

37.

Wallach

Egilman

Dhruva

, et al. Postmarket studies required by the US Food and Drug Administration for new drugs and biologics approved between 2009 and 2012: cross sectional analysis. BMJ. 2018;361:k2031.

38.

Berger

Sox

Willke

, et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: recommendations from the joint ispor-ispe special task force on real-world evidence in health care decision making. Value Health. 2017;20:1003-1008.

39.

Shaywitz

. Will real world performance replace RCTs as healthcare’s most important standard? Available at: https://www.forbes.com/sites/davidshaywitz/2018/05/11/will-real-world-performance-replace-rcts-as-healthcares-most-important-standard/#55f35bf43557. Accessed February 20, 2019.

40.

Curtis

Griffith

Tucker

, et al. Development and validation of a high-quality composite real-world mortality endpoint. Health Serv Res. 2018;53:4460-4476.

41.

Institute P-COR. Improving methods for linking secondary data sources for CER/PCOR. Available at: https://www.pcori.org/research-results/2014/improving-methods-linking-secondary-data-sources-cerpcor. Accessed February 20, 2019.

42.

Pencina

Rockhold

D’Agostino

Sr . Deriving real-world insights from real-world data: biostatistics to the rescue. Ann Intern Med. 2018;169:401-402.

43.

US Food and Drug Administration. FDA in brief: FDA launches new digital tool to help capture real world data from patients to help inform regulatory decision-making. Available at: https://www.fda.gov/NewsEvents/Newsroom/FDAinBrief/ucm625228.htm. Accessed February 20, 2019.

44.

Hirshberg

Katz

Cardiovascular outcome studies with novel antidiabetes agents: scientific and operational considerations. Diabetes Care. 2013;36(suppl 2):S253-S258.

45.

Ponzani

Berra

Di Lelio

, et al. Impact of insulin degludec in type 2 diabetes: real-world data on effectiveness and safety. Diabetes Ther. 2018;9:2209-2218.

46.

O’Brien

Karam

Wallia

, et al. Association of second-line antidiabetic medications with cardiovascular events among insured adults with type 2 diabetes. JAMA Network Open. 2018;1:e186125.

47.

Wysham

Lefebvre

Pilon

, et al. An investigation into the durability of glycemic control in patients with type II diabetes initiated on canagliflozin or sitagliptin: a real-world analysis of electronic medical records. J Diabetes Complications. 2019;33(2):140-147.

48.

Wei

Buysman

Grabner

, et al. A real-world study of treatment patterns and outcomes in US managed-care patients with type 2 Diabetes initiating injectable therapies. Diabetes Obes Metab. 2017;19:375-386.

49.

Roberts

Dalal

AA.

Clinical and economic outcomes in an observational study of COPD maintenance therapies: multivariable regression versus propensity score matching. Int J Chron Obstruct Pulmon Dis. 2012;7:221-233.

50.

Tarricone

Boscolo

Armeni

What type of clinical evidence is needed to assess medical devices?

Eur Respir Rev. 2016;25:259-265.