Uses and abuses of real-world data in generating evidence during a pandemic

On 11 March 2020, the World Health Organization declared COVID-19 a pandemic and called for immediate collaborative initiatives for faster access to available data, with a view to generating robust research evidence informing global and local public health policy. ¹ This urgency has helped a number of national bodies to secure data and their linkages and to provide safe analytical environment for researches to ask important questions, including pseudonymised data linkages, high-throughput computing environment, and access and authentication processes with clear information governance.^2,3 Linkages of multiple sources of clinical data within a trusted environment are being granted at a rapid pace and there is a greater provision of access to COVID-19 studies, improved collaboration, expedited governance and ethical approval of studies. ⁴ Some organisations have also been proactive in getting groups together to work collaboratively on relevant research questions which will rapidly benefit clinical care and public health alike.

In view of the pandemic unfolding in the big data era and the urgency of actionable information, the potential consequences and responses to this pandemic have been mainly using ‘real-world data. Although disputed, real-world data may be defined as data collected primarily for purpose other than research (‘secondary data’) and the evidence generated from real-world data as ‘real-world evidence’. The fast growth of real-world data translating into real-world evidence has been made possible due to the rapid availability of semi-structured and unstuctured data from a network of resources. For example, the Johns Hopkins Coronavirus Resource Center and the Worldometer are updating global real-world data on a daily basis in real time providing live statistics on the number of cases and fatalities from COVID-19.^5,6 Along with summary statistics, real-world data have provided valuable information regarding patient characteristics, treatment pattern, and clinical outcomes and risk factors for hospitalised patients and mortality (for example, elderly, males, obesity or individuals of minority ethnic groups).^2,7 Real-world data have also been used to develop risk prediction models for the diagnosis and prognosis of COVID-19 and pragmatic trials of therapies using routine data to capture outcomes.^7,8

The rapid deployment of and the exponential rise in real-world data and real-world evidence precipitated by the COVID-19 pandemic has been, however, criticised for conducting and reporting studies of poor quality, which have been amplified during COVID-19.^4,5 The necessity of a rapid while proper peer-review process for the submitted articles to be quickly published in journals and gain citations has been problematic, and likely contributed to the 400% increase in preprint publications compared to mainstream journals in relation to COVID-19.^4,9 This urge to expedite the peer-review process may also have contributed to the publication of poor-quality research as well as fraud, including the recent alarming high-profile retractions of COVID-19-related research papers. ¹⁰ Most of COVID-19 publications have not been registered (i.e. National Library of Medicine) as, in most cases, there are no requirements to register obeservational studies while only a few randomised controlled trials, for which registration is mandatory, have been registered.^4,11 While it is recognised that the rapid access to data and new findings are crucial in the context of an emerging epidemic, it can also result in poorly designed studies gaining media attention, influencing policy, wasting resources, and overall loss of trust by the public and policy makers. There have also been duplicated publications often finding different answers using the same databases, with implications for quantitative evidence synthesis (i.e. meta-analysis) where duplicate publications have included the same cohort of patients.

Research during a pandemic obviously needs to be conducted and published at pace to inform global policy and help save lives. There is, however, potentially a massive waste that has been exacerbated by the pandemic. If we are to continue having trust in real-world data and the quality of real-world evidence publications, the international scientific community needs to give immediate attention to retain credibility of real-world data and real-world evidence. It is fundamentally important that in the rush to publish, scientific quality and high standards for research are retained. For example, greater effort is required to ensure that all published real-world evidence conform to the basic established requirements for observational research. The process of pre-print and journal submission and peer review may also need to be considered to provide greater clarity and transparency. Indeed, some funders have already invested in newer models of publishing that could be used to better effect during a pandemic. For example, the Welcome Open Research uses the F1000Research platform to provide a transparent hybrid open access pathway between preprint archive, open peer review and author responses and subsequent manuscript revision history.

This is a call for the academic community to learn from the COVID-19 pandemic and develop guidelines and processes for real-world data and real-world evidence that work to provide an appropriate balance between rapid access to new data and findings during public health emergencies, while also retaining quality and reducing the risk of fraud.