Awareness,Knowledge,and Utility of RCT Data vs RWE: Results From a Survey of US Cardiologists: Real-world Evidence in Clinical Decision Making

Abstract

Real-world evidence (RWE) provides a potential rich source of additional information to the body of data available from randomized clinical trials (RCTs), but there is a need to understand the strengths and limitations of RWE before it can be applied to clinical practice. To gain insight into current thinking in clinical decision making and utility of different data sources, a representative sampling of US cardiologists selected from the current, active Fellows of the American College of Cardiology (ACC) were surveyed to evaluate their perceptions of findings from RCTs and RWE studies and their application in clinical practice. The survey was conducted online via the ACC web portal between 12 July and 11 August 2017. Of the 548 active ACC Fellows invited as panel members, 173 completed the survey (32% response), most of whom were board certified in general cardiology (n = 119, 69%) or interventional cardiology (n = 40, 23%). The survey results indicated a wide range of familiarity with and utilization of RWE amongst cardiologists. Most cardiologists were familiar with RWE and considered RWE in clinical practice at least some of the time. However, a significant minority of survey respondents had rarely or never applied RWE learnings in their clinical practice, and many did not feel confident in the results of RWE other than registry data. These survey findings suggest that additional education on how to assess and interpret RWE could help physicians to integrate data and learnings from RCTs and RWE to best guide clinical decision making.

Keywords

Cardiologists clinical decision making randomized clinical trials real-world evidence survey

Introduction

Clinical practice guidelines are largely based on published results from carefully designed randomized controlled trials (RCTs)^1-4 as they provide for control of bias, meticulous data collection, and careful selection of patient populations, which reduce variation that could influence study results.^5,6 Regardless of the therapy being studied, patients who meet the strict inclusion and exclusion criteria for RCTs tend to be younger, have fewer comorbidities or less severe/advanced disease, and may be more motivated to be compliant with treatment protocols than patient populations encountered during routine clinical care,⁷ indicating that results may not be fully generalizable to patient populations in broader health care settings.⁸ Demographic groups with high burden of disease (eg, elderly patients, African Americans) may thus be under-represented in RCTs;^9-12 extrapolation of clinical trial data to broader populations should be performed with caution.¹³ Once approved for clinical use, new drugs are prescribed to a broader range of patients, who could be less adherent to the dosing regimens and may require treatment follow-ups compared with patients studied in RCTs.^5-7,14

Real-world evidence (RWE) relates to the evaluation of the usage and potential benefits or risks of a drug using real-world data (RWD), including patient health status, health care, and outcomes routinely collected from a variety of sources such as electronic health records, payer claims data, new technologies for patient-generated data, and dedicated registries.⁸

RWE is useful in clinical decision making because it is representative of diverse patient populations encountered by health care providers (HCPs). RWE helps to closely analyze the safety results of RCTs in a diverse population and uncover rare outcomes associated with long-term therapy. However, there are limitations associated with RWEs including the design of the analysis and inability to obtain information due to incomplete databases or lack of quality RWD.^8,15 While RWE adds to the body of data from RCTs, it is imperative that HCPs understand the strengths, limitations, and methodological details when assessing and applying RWE to their clinical practice.

Oral anticoagulation (OAC) in patients with nonvalvular atrial fibrillation (NVAF) was selected as a suitable therapy area to assess the impact of RWE in clinical practice. Guidelines support the use of direct-acting oral anticoagulants (DOACs) (alternative to vitamin K antagonists) in patients with NVAF with history of stroke or transient ischemic attack, or with CHA₂DS₂-VASc score ⩾2 because of their predictable pharmacological properties and limited food-drug interactions.^1,16 However, patient- and drug-related factors determine the suitability of DOACs in patients with NVAF.¹⁷ A wealth of RWE is available to guide clinical decision making, but how cardiologists perceive the clinical utility of this information is unknown.

The American College of Cardiology Foundation (ACCF) Market Intelligence team surveyed US cardiologists to gain insight into current thinking in clinical decision making and utility of different data sources. The survey evaluated the perception of findings from RCTs and RWE studies and their application in clinical practice and included use of DOACs in patients with NVAF as a specific example. Key points included the perceived utility of study results, relevant sources considered when extrapolating to clinical practice, and whether cardiologists view data sources as complementary when treating patients. The survey assessed whether—and, if so, how—the survey panel respondents integrate data derived from RCTs and RWE studies into their clinical practice decision making.

Methods

A survey was developed specifically for use with the ACCF CardioSurve panel, a group of 548 cardiologists selected from the current, active Fellows of the American College of Cardiology (“ACC”) (ACC and ACCF shall be referred to hereinafter collectively as “ACC”) by a stratified random selection technique to obtain a representative sampling of US cardiologists based on key demographics. Survey questions were developed as a collaboration between the ACC and an advisory team of cardiologists, funded by Boehringer Ingelheim Pharmaceuticals, Inc.; the authors and Boehringer Ingelheim were provided a courtesy copy to review for medical accuracy. Questions used neutral language and, to avoid biasing the responses, the funding support for the survey was not disclosed to participants during the survey. The survey was conducted by ACC staff, including survey programming, fielding, and analysis, and was in no way influenced by Boehringer Ingelheim, the authors, or any other party. The full survey (Supplemental Material- CardioSurve Survey) included 14 questions, requesting respondents to rank their answers in order from most to least important, to indicate their degree of confidence or familiarity with various information sources using a Likert-type scale (1 = not at all confident/familiar to 5 = extremely confident/familiar) or to indicate their agreement/disagreement with specific statements (“strongly disagree” to “strongly agree”). RWE was defined as evidence “derived from very large datasets, such as insurance claims databases and electronic medical records (EMR), representing medical practice among heterogeneous sets of patients in real-life settings, in order to determine how new pharmaceutical treatments, perform beyond the scope of clinical trials.” No differentiation was made between insurance claims databases and EMR.

The survey was conducted online via the ACC web portal between 12 July and 11 August 2017, and all panel members were invited to participate via email. Responses were collated and analyzed descriptively by the ACC Market Intelligence office. Statistical analysis of the responses was performed using SPSS Statistics 23. Responses were also grouped by practitioner experience level, defined as early-career (⩽10 years), mid-career (11–20 years), or late-career practitioner (>20 years). Pearson correlation coefficient tests were used to calculate significant associations between variables, with statistical significance defined as P < .01 (2-tailed).

Results

Overall, 32% (173/548) of the ACC panel members completed the survey, and most were board certified in general cardiology (n = 119, 69%) or interventional cardiology (n = 40, 23%). For experience level, 54 (31%) were early-career, 39 (23%) were mid-career, and 65 (38%) were late-career practitioners; 11 (6%) did not specify the level. The representativeness of the ACC CardioSurve panel on these measures was within ±5% of the active US Fellows of the ACC membership.

RWE ranking among treatment-decision tools: The most commonly selected high-tier tools included clinical practice guidelines (71%), followed by clinical trials (43%) and appropriate-use criteria (24%) (Figure 1a). RWE (mid-tier tool) was ranked fifth in importance by 21% respondents and first or second in importance by only 16% respondents.

Figure 1.

(a) Ranking of clinical tools used to inform patient treatment decisions: overall panel responses. Respondents were asked to rank the 9 data sources from the most important to least important tool (or to “not sure”) when considering patient treatment decisions. (b) Confidence in various RWE sources: overall panel responses among those who were familiar with the term RWE. Participants who responded that they were familiar with RWE (n = 147) were asked, “How confident are you in these RWE sources?” Options ranged from “1—not at all confident” through “5—extremely confident,” as well as the option “not sure” for each data source. Respondents did not need to rank sources in order of confidence (ie, the same answer could be given for all sources).

Experience level stratification was concordant for early- and late-career practitioners (Table 1).

Table 1.

Ranking of clinical tools used to inform patient treatment decisions, according to practice tenure.

Top 2 picks (most important [1] / [2] scores)	Early-career (⩽10 years)(n = 54)	Mid-career (11–20 years)(n = 39)	Late-career (>20 years) (n = 65)
Clinical practice guidelines	72%	82%	68%
Clinical trials	48%	36%	48%
Appropriate use criteria	22%	33%	20%
Expert consensus decision pathways	17%	13%	20%
Real-world evidence	15%	5%	18%
Journal articles	13%	5%	12%
CME roundtable review of peer-reviewed published evidence	4%	8%	6%
Health-system protocols	4%	3%	2%
Performance measures	2%	5%	3%

Abbreviation: CME, continuing medical education.

Respondents were asked to rank the 9 data sources from the most important to least important tool (or to “not sure”) when considering patient treatment decisions.

RWE familiarity and confidence: Majority of participants (85%) were familiar with RWE, with 21% mid-career, 15% late-career, and only 4% early-career practitioners (Supplementary Table 1) considering themselves “extremely familiar” with RWE.

Participants were assessed on their confidence on RWE sources such as administrative claims data, EMR, registry data, pharmacy data, clinician notes, patient forums, and social media. Approximately 61% of cardiologists, who were familiar with RWE were very or extremely confident in registry data as an RWE source, followed by EMR (29%), pharmacy data (28%), clinician notes (26%), or administrative claims data (17%). Very few respondents were very or extremely confident in-patient forums or social media (Figure 1b). Across tenures, respondents had more confidence in registry data than any other RWE source listed (Table 2).

Table 2.

Confidence in various RWE sources, according to practice tenure.

Top 2 picks (extremely confident [5] / [4] scores)	Early-career(⩽10 years)(n = 44)	Mid-career(11–20 years)(n = 32)	Late-career(>20 years)(n = 57)
Registry data	64%	75%	54%
EMRs	32%	41%	21%
Pharmacy data	18%	31%	32%
Clinician notes	27%	41%	23%
Administrative claims data	9%	25%	18%
Patient forums	5%	13%	7%
Social media	5%	3%	2%

Abbreviation: EMR, electronic medical record; RWE, real-world evidence.

Application to practice: Participants familiar with RWE were assessed on frequency of applied learnings gathered from RWE into practice. According to the survey results, about 23% frequently, 48% sometimes, and 21% rarely or never applied RWE information into practice, with 7% being unsure. Cardiologists who most likely applied RWE findings in practice were those who believed in RWE and RCT data as guidance for clinical decision making (Table 3a) and demonstrated strong confidence in RWE findings (Table 3b and Figure 1a). Cardiologists were further assessed on clinical utility of data and insights from RWE studies versus RCTs (Table 3C).

Table 3.

Perspectives of cardiologists who are more likely to apply RWE findings into their practices, by agree/disagree statements (a), confidence ratings (b), and general perceptions in comparison to RWE (c).

(a)

Agree/disagree statements	Cardiologists who “Always” or “Often” apply learnings from RWE (n = 34)	Pearson correlation coefficient*
RWE and RCT data can provide useful guidance for clinical decision making	100% agree	0.401
RWE enables more effective clinical assessments and decision making over a broader patient population	85% agree	0.481
RWE allows clinicians to tailor health care decisions more closely to the characteristics of individual patients	79% agree	0.586

Abbreviations: RCT, randomized controlled trial; RWE, real-world evidence.

All correlations cited are significant at the P < .01 level (2-tailed by test).

(b)

Confidence ratings	Cardiologists who “Always” or “Often” apply learnings from RWE (n = 34)	Pearson correlation coefficient*
Confidence in registry data as a source of RWE	71% Extremely/Very Confident	0.413
Overall confidence in findings gathered from RWE	62% Extremely/Very Confident (26% Depends on Research)	0.462
Confidence in EMRs as an RWE source	56% Extremely/Very Confident	0.506

Abbreviations: EMR, electronic medical record; RWE, real-world evidence.

All correlations cited are significant at the P < .01 level (2-tailed by test).

(c)

General perception	Familiar cardiologists who “Agreed” with the following perceptions(n = 147)
RWE and RCTs can each provide useful guidance for clinical decision making	89%
Results from RCTs have greater validity than data from RWE studies for clinical decision making	61%
RWE can be used to tailor health care decisions more closely to the characteristics of individual patients	58%
RWE studies enable more effective clinical assessments due to broader patient population	57%

Abbreviations: RCT, randomized controlled trial; RWE, real-world evidence.

The Dabi-Riva study: To assess awareness of RWE, participating cardiologists were asked about their familiarity with a recent, independently conducted large-scale retrospective analysis of elderly patients with NVAF in the Medicare claims database on dabigatran or rivaroxaban.¹⁸

Only 44% of the participants were familiar with this study (Supplementary Table 2) and of these 64 cardiologists, 58% would incorporate the findings into their treatment plans.

Perceived benefits of RWE: The respondents were asked to select benefits of RWE from a provided list (Supplemental Material-CardioSurve Survey). Approximately 75% of cardiologists indicated that RWE provides clinical insights on patient populations not tested in RCTs. Two additional benefits included that clinical outcomes could be determined from a larger patient population than studied in other sources (56%), and that RWE studies allowed outcome comparisons across different therapies and between competing agents (50%).

Similarly, participants were asked to consider potential concerns with RWE (Supplemental Material-CardioSurve Survey). Most respondents (94%) identified ⩾1 concern, with 5% selecting “not sure” and only 1% selecting “none/no concerns.” The top 3 concerns selected were: “more difficult to recognize confounding variables in non-randomized studies” (73%), “possible generation of incorrect or unreliable conclusions” (70%), and “data quality” (69%). The results were concordant across career tenure categories.

Education opportunities: Opportunities for “postgraduate” education in topics related to RCTs and RWE may help practitioners to better understand and integrate information. In this survey, all participants (n = 173) were asked about their level of interest; approximately 64% of respondents indicated they were very or extremely interested in ACC educational programs regarding RWE. The preferred educational formats for the majority of respondents were online or print publications (Supplemental Table 3).

Discussion

This survey identified several insights into the opinions of cardiologists regarding RWE. Majority of the respondents ranked RWE as less important than RCTs in treatment decision making and considered clinical practice guidelines, as the most important source of information. Inconsistent use of RWE in clinical practice guidelines indicate lack of familiarity with how the data are obtained, how patient groups are defined and controlled for, and how various analytical methods (case-control groups, propensity score matching [PSM], Cox proportional hazards regression models, etc.) impact study outcomes in the available literature.^8,19,20 Regulatory bodies, including the US Food and Drug Administration, have issued guidance on the value of RWD, its generation, and use in regulatory decisions.²¹

Along with general questions on RWE, the survey used an independently conducted, large, retrospective study (The Dabi-Riva study) published within the year prior to the survey in JAMA Internal Medicine¹⁸ as a specific example to gauge awareness of RWE and whether the participants felt it had clinical utility or would influence their clinical practice. While a sizable minority reported they were familiar with this large RWE study, the majority of cardiologists were not, suggesting additional education from professional societies on the potential utility of RWE to clinical practice could be useful. Of those who were familiar with the study, the majority indicated they were likely to incorporate the learnings from this report into their practice, confirming that RWE can impact patient care, at least in this field.

In this survey, most cardiologists agreed that results from RCTs are more valid for informing clinical decisions than findings from RWE studies; however, the majority also agreed that findings from both RWE studies and RCTs could provide useful guidance for clinical decision making.

Confidence in the reported findings and the underlying sources of the “raw data” that serve as the basis for the analysis is essential to the uptake of information generated by RWE studies. The cardiologists in this survey indicated varying degrees of confidence in the findings of RWE studies; 26% respondents indicated they felt their level of confidence would depend largely on the study itself, presumably reflecting appraisal of the study design, although the survey question did not draw out what these respondents looked for in these cases. The concerns of survey respondents were that RWE is solely retrospective and generally observational in nature, with confounding variables in nonrandomized studies leading to possible generation of unreliable conclusions. However, RWE studies could also be performed prospectively and the use of PSM could allow for a more precise estimation of average effect of treatment by reducing bias due to confounding variables, such as population and group differences.^19,22 Pragmatic randomized clinical trial designs and randomized registry trials provide suggestive RWE while maintaining the strength of randomized treatment, they identify correlations between treatments and outcomes without the strict inclusion and exclusion criteria, and combine a randomization module with unselected consecutive enrollment, enabling inclusion of a broader study population that could reflect real-world clinical practice and help bridge the gap between traditional RCTs and RWE. Moreover, pragmatic clinical trials (PCTs) and randomized registry trials are cost effective.^23-25 Tools such as the Pragmatic Explanatory Continuum Indicator Summary 2 (PRECIS-2) may be useful as a guide for selection of the appropriate study design for the intended purpose and help confirm validity of RWE databases.²⁶

It is hoped that appropriate curation and validation techniques in conjunction with methodological and technical advances will lead to improvements in data collection and standardization in RWE studies.²⁷ Greater use of the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines²⁸ should improve the robustness of observational studies and thereby HCPs confidence in the results.

Moreover, specialists indicated the benefit of practical guidance and education in integrating information from RCTs and RWE. When considering RWE, a clear understanding of the strengths, limitations, and boundaries of each dataset would help the reader to identify the appropriate use of the information arising from these studies. The ultimate goal would provide a more complete picture of how individual medication options impact patient health and overall health care costs, and their relative roles in the health care armamentarium.^29-31

However, the survey results are a reflection of the knowledge and comfort levels that cardiologists have with RWE, and the role that these data play in clinical decision making. The survey only provides a “snapshot” from this therapeutic area and the responses are limited to the opinions of the ACC cardiologists included in this survey, in 2017. These findings cannot be extrapolated to other groups within the health care professions and may have changed since the performance of this survey. In addition, the survey assessed cardiologists’ understanding of RWE on a broad level and confidence with data sources (eg, registry data, claims data), but did not assess awareness or confidence on PCTs. In fact, opinions were gathered on RWE in general; however, respondents in the survey had very different views of varied sources of RWE. Nevertheless, as a targeted survey, the results provided useful feedback for those communicating RWE to cardiologists and could be informative when considering RWE and RCTs in other therapeutic areas where drugs have patient-specific risk-benefit profiles.

In conclusion, the results indicate a wide range of knowledge and utilization of RWE. Most cardiologists were familiar with and considered RWE to be a mid-tier data source, useful in clinical practice. However, a significant minority of survey respondents had rarely or never applied RWE learnings into practice, and many did not feel confident in the results of RWE other than registry data. This suggests the clinical community may welcome PCTs, which could provide valuable data on patient groups under-represented in RCTs, while also addressing some concerns about study designs employing retrospective database analysis. Yet, it must be acknowledged that EMR and similar analyses help in understanding treatment in the real world and providing education could enable physicians to maximize the use of available RWE by improving clinicians’ confidence in interpreting study design and understanding the study population better. Taken together, the survey findings may help guide achieve the ultimate goal of integrating data and learnings from RCTs and RWE to best guide clinical decision making.

Supplemental Material

supplementary_material – Supplemental material for Awareness, Knowledge, and Utility of RCT Data vs RWE: Results From a Survey of US Cardiologists: Real-world Evidence in Clinical Decision Making

Supplemental material, supplementary_material for Awareness, Knowledge, and Utility of RCT Data vs RWE: Results From a Survey of US Cardiologists: Real-world Evidence in Clinical Decision Making by Todd C Villines, Mark J Cziraky and Alpesh N Amin in Clinical Medicine Insights: Cardiology

Footnotes

Acknowledgements

The authors meet the criteria for authorship as recommended by the International Committee of Medical Journal Editors (ICMJE), were fully responsible for all content and editorial decisions, and were involved at all stages of manuscript development. The authors received no direct compensation related to the development of the manuscript, the conduct of the survey, or for performing the analyses. Paul Theriot from ACC Market Intelligence assisted in the survey design, performed the survey programming, managed the data collection process, and conducted the data processing and analysis of the findings.

This work was supported by Boehringer Ingelheim Pharmaceuticals, Inc. (BIPI). Writing assistance was provided by José L. Walewski, PhD, and Geraldine Thompson of Envision Scientific Solutions, and formatting and editorial assistance was provided by Shaleen Multani, PhD, and Suchita Nath-Sain, PhD, of Cactus Life Sciences (part of Cactus Communications), which was contracted and compensated by BIPI for these services

Funding:

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Boehringer Ingelheim Pharmaceuticals, Inc. (BIPI) provided financial support to ACC to conduct the survey. Survey programming, data collection, and data analyses were conducted completely independent of BIPI.

The identification of specific products or scientific instrumentation is considered an integral part of the scientific endeavor and does not constitute endorsement or implied endorsement on the part of the author, DoD, or any component agency. The views expressed herein are those of the author and do not reflect the official policy of the Department of the Army, Department of Defense, or US Government.

Declaration of conflicting interests:

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Todd C Villines has previously served as a consultant and has previously been on the speakers’ bureau for Boehringer Ingelheim; he received no compensation related to this study.

Mark J Cziraky has no conflicts of interest to declare and has received no compensation related to this study.

Alpesh N Amin has served as a consultant and/or speaker for Boehringer Ingelheim, Portola, Bristol-Myers Squibb, and Pfizer and has not received funding for this manuscript.

Ethical standards

The manuscript does not contain clinical studies or patient data.

ORCID iD

Todd C Villines

Supplemental material

Supplemental material for this article is available online.

References

January

Wann

Alpert

, et al. 2014 AHA/ACC/HRS guideline for the management of patients with atrial fibrillation: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines and the Heart Rhythm Society. J Am Coll Cardiol. 2014;64:e1-e76.

Camm

Pinto

Hankey

Andreotti

Hobbs

, Writing Committee of the Action for Stroke Prevention Alliance. Non-vitamin K antagonist oral anticoagulants and atrial fibrillation guidelines in practice: barriers to and strategies for optimal implementation. Europace. 2015;17:1007-1017.

Russo

Rago

Proietti

, et al. Efficacy and safety of the target-specific oral anticoagulants for stroke prevention in atrial fibrillation: the real-life evidence. Ther Adv Drug Saf. 2017;8:67-75.

Kirchhof

Benussi

Kotecha

, et al. 2016 ESC Guidelines for the management of atrial fibrillation developed in collaboration with EACTS. Europace. 2016;18:1609-1678.

Mitchell

Harrison

Walker

George

Abernethy

Hirsch

BR.

Clinical trial participants with metastatic renal cell carcinoma differ from patients treated in real-world practice. J Oncol Pract. 2015;11:491-497.

Westgeest

Uyl-de Groot

van Moorselaar

RJA

, et al. Differences in trial and real-world populations in the Dutch castration-resistant prostate cancer registry. Eur Urol Focus. 2018;4:694-701.

Kennedy-Martin

Curtis

Faries

Robinson

Johnston

A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results. Trials. 2015;16:495.

Sherman

Anderson

Dal Pan

, et al. Real-world evidence - what is It and What Can It Tell Us? N Engl J Med. 2016;375:2293-2297.

US Food and Drug Administration. Collection, analysis, and availability of demographic subgroup data for FDA-approved medical products. 2013; https://www.fda.gov/downloads/RegulatoryInformation/LawsEnforcedbyFDA/SignificantAmendmentstotheFDCAct/FDASIA/UCM365544.pdf. Accessed April 12, 2018.

10.

Ranganathan

Bhopal

Exclusion and inclusion of nonwhite ethnic minority groups in 72 North American and European cardiovascular cohort studies. PLoS Med. 2006;3:e44.

11.

Geller

Koch

Pellettieri

Carnes

Inclusion, analysis, and reporting of sex and race/ethnicity in clinical trials: have we made progress?

J Womens Health (Larchmt). 2011;20:315-320.

12.

Hoel

Kayssi

Brahmanandam

Belkin

Conte

Nguyen

LL.

Under-representation of women and ethnic minorities in vascular surgery randomized controlled trials. J Vasc Surg. 2009;50:349-354.

13.

Marietta

Direct oral anticoagulants in atrial fibrillation: can data from randomized clinical trials be safely transferred to the general population? No. Intern Emerg Med. 2015;10:647-650.

14.

Cziraky

Pollock

Real-world evidence studies. Appl Clin Trials. 2015;2017:1-2.

15.

Mahajan

Real world data: Additional source for making clinical decisions. Int J Appl Basic Med Res. 2015;5:82.

16.

Stacy

Richter

Practical considerations for the use of direct oral anticoagulants in patients with atrial fibrillation. Clin Appl Thromb Hemost. 2017;23:5-19.

17.

Hanley

Kowey

PR.

Are the novel anticoagulants better than warfarin for patients with atrial fibrillation?

J Thorac Dis. 2015;7:165-171.

18.

Graham

Reichman

Wernecke

, et al. Stroke, bleeding, and mortality risks in elderly Medicare beneficiaries treated with dabigatran or rivaroxaban for nonvalvular atrial fibrillation. JAMA Intern Med. 2016;176:1662-1671.

19.

Corrao

Building reliable evidence from real-world data: methods, cautiousness and recommendations. Epidemiol Biostat Public Health. 2013;10:40.

20.

Oyinlola

Campbell

Kousoulis

AA.

Is real world evidence influencing practice? A systematic review of CPRD research in NICE guidances. BMC Health Serv Res. 2016;16:299.

21.

US Food and Drug Administration. Framework for FDA’s real-world evidence program. 2018; FDA White Paper on RWD. Available at: https://www.fda.gov/scienceresearch/specialtopics/realworldevidence/default.htm. Accessed December 12, 2018.

22.

Austin

PC.

An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res. 2011;46:399-424.

23.

Lauer

D’Agostino

Sr . The randomized registry trial–the next disruptive technology in clinical research? N Engl J Med. 2013;369:1579-1581.

24.

Anderson

Califf

Sugarman

Ethical and regulatory issues of pragmatic cluster randomized trials in contemporary health systems. Clinical Trials. 2015;12:276-286.

25.

Tashkin

Amin

Kerwin

Comparing randomized controlled trials and real-world studies in chronic obstructive pulmonary disease pharmacotherapy. Int J Chron Obstruct Pulmon Dis. 2020;15:1225-1243.

26.

Loudon

Treweek

Sullivan

Donnan

Thorpe

Zwarenstein

The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147.

27.

Berger

Daniel

Frank

A framework for regulatory use of real-world evidence. 2017; White paper. Available at: https://www.researchgate.net/publication/319650416_A_FRAMEWORK_FOR_REGULATORY_USE_OF_REAL-WORLD_EVIDENCE. Accessed February 11, 2018.

28.

von Elm

Altman

Egger

Pocock

Gotzsche

Vandenbroucke

JP.

The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61(4):344-349.

29.

Beyer-Westendorf

Camm

Coleman

Tamayo

Rivaroxaban real-world evidence: Validating safety and effectiveness in clinical practice. Thromb Haemost. 2016;116:S13-S23.

30.

Jacobs

May

Bair

, et al. Long-term population-based cerebral ischemic event and cognitive outcomes of direct oral anticoagulants compared with warfarin among long-term anticoagulated patients for atrial fibrillation. Am J Cardiol. 2016;118:210-214.

31.

Zirlik

Bode

Vitamin

antagonists: relative strengths and weaknesses vs. direct oral anticoagulants for stroke prevention in patients with atrial fibrillation. J Thromb Thrombolysis. 2017;43:365-379.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.14 MB