COVID-19 human challenge trials and randomized controlled trials: lessons for the next pandemic

Abstract

The COVID-19 pandemic touched off an unprecedented search for vaccines and treatments. Without question, the development of vaccines to prevent COVID-19 was an enormous scientific accomplishment. Further, the RECOVERY and Solidarity trials identified effective treatments for COVID-19. But all was not success. The urgent need for COVID-19 prevention and treatment fueled an embrace of risks—to research participants and to the reliability of the science itself—as allegedly necessary costs to speed scientific progress. Scientists and (even) ethicists supported overturning longstanding norms protecting healthy volunteers in human challenge trials to speed vaccine development, but these trials led to no vaccines. Physicians, with the approval of research ethics committees, designed hundreds of unblinded, single-center clinical trials at high risk of bias to speed the identification of new treatments. But these clinical trials led to no treatments. The lesson for future pandemics is that the acceptance of greater risks to participants or science does not reliably lead to progress. We are better served by science that upholds the highest ethical and methodological standards.

Introduction

The COVID-19 pandemic was a global health catastrophe. According to the World Health Organization, there were more than 750M cases of COVID-19 globally and almost 7M deaths (World Health Organization [WHO], 2023). As grim as they are, these figures are very likely underestimates. Lack of testing, particularly in resource poor countries, has meant that COVID-19 cases and deaths go uncounted. Indeed, the excess mortality attributable to the pandemic may be as high as 33.5M deaths (Economist Intelligence Unit, 2023). With the declaration of the end of the COVID-19 global public health emergency in May 2023, it is timely to reflect on trials conducted in the pandemic and draw lessons for the future.

But the lessons we draw, depend on the story we tell. In one telling, the science conducted during the pandemic was successful to a historic degree. Developing a vaccine for a new infectious agent may take 10–15 years (Kalinke et al., 2022). COVID-19 vaccines were developed, tested, and manufactured in 1 year. To date, 13.5B doses of vaccine have been administered globally (WHO, 2023). The search for treatment for COVID-19 has also had notable successes. The RECOVERY Trial resulted in the first proven-effective treatment for COVID-19, namely, dexamethasone (Horby et al., 2021). Equally important, it proved conclusively that other drugs that were used to treat patients, such as hydroxychloroquine, are ineffective (Horby et al., 2020).

But focusing solely on these successes would miss the broader story of ill-conceived, poor-quality, and in some cases unethical science spawned by the pandemic. Meyerowitz-Katz and colleagues describe the problem well:

Early in the COVID-19 pandemic ethicists warned researchers against “pandemic research exceptionalism”—lowering ethical standards because of the urgency of the crisis. Despite these warnings, there have been many examples of researchers treating COVID-19 as exactly that: an exception to the rigorous standards to which we should hold medical research (Meyerowitz-Katz et al., 2022).

In this paper, I will focus on two domains of COVID-19 science. The first involves the proposal to conduct COVID-19 challenge trials on healthy volunteers to speed vaccine development. The second involves randomized controlled trials to identify urgently needed treatments. These two stories are united in my telling by an embrace of risks—to research participants and to the reliability of the science itself—as allegedly necessary costs to speed scientific progress.

Human challenge trials to accelerate the development of COVID-19 vaccines

Human challenge trials

Human challenge trials (also known as controlled human infection trials) are studies in which healthy volunteers are exposed to an infectious agent for a variety of scientific purposes. These purposes include unpacking its mechanism of infection, understanding modes of transmission, assessing preventive behaviors (e.g. mask wearing), and obtaining preliminary measures of the efficacy of drugs or vaccines.

Human challenge trials are without doubt an important scientific tool in the investigation of infectious diseases. These studies have played a role in developing our understanding of numerous diseases. Indeed, at least 22 different infectious agents have been used in modern human challenge trials (Roestenberg et al., 2018). These studies played a key role in the development of vaccines to prevent cholera, dengue, influenza, and malaria (Roestenberg et al., 2018). Perhaps most remarkable is the safety record of human challenge trials. A recent systematic review of human challenge trials published since 1980, found that only two in 1000 research participants experienced a serious adverse event (Adams-Phipps et al., 2023). No research participant has died in a modern human challenge trial.

The safety record of human challenge trials reflects the lengths to which researchers go to protect research participants. Human challenge trials are generally limited to well understood infectious agents and, in most cases, strains used are known to cause disease that is both self-limiting and mild. When an infectious agent is associated with illness that is not self-limiting or potentially severe, the availability of highly-effective curative treatment—known as “rescue therapy”—is essential. For instance, malaria challenge strains have been genetically modified to be exquisitely sensitive to treatment with a single anti-malarial drug (Spring et al., 2014). By these means, “researchers ensure that volunteers will under no circumstances be exposed to the risks of irreversible, incurable or possibly fatal infections” (Bambery et al., 2016).

Calls for COVID-19 human challenge trials

Early in the COVID-19 pandemic, human challenge trials were touted as a promising way to develop urgently needed vaccines. Advocates of COVID-19 challenge trials were confident these studies could speed vaccine development. A prominent ethicist wrote: “In terms of vaccine discovery, there are plenty of ways authorities could speed up the process. One is a human challenge study” (Savulescu, 2020). The advocacy group 1DaySooner, which collected the names of tens of thousands of people willing to participate in COVID-19 challenge trials, asserted that these studies “can provide information much faster than conventional [vaccine] efficacy trials” (1Day Sooner, 2020). As a result, COVID-19 challenge trials were “an important way to accelerate vaccine development and, ideally, to save the lives of millions around the world” (1Day Sooner, 2020).

How could human challenge trials speed vaccine development? The most detailed proposal was put forth by Eyal and colleagues in March 2020 (Eyal et al., 2020). In their proposal, vaccine candidates would be tested as usual in dose finding (phase 1) and immunogenicity (phase 2) trials. However, once a promising vaccine candidate was identified, two smaller trials would replace the usual phase 3 field trial. A human challenge trial would expose healthy volunteers who had received the vaccine or placebo to the SARS-CoV-2 virus and measure vaccine efficacy. If the vaccine candidate was shown to be efficacious, its safety would then be evaluated in a placebo-controlled field trial involving 3000 people. If both the efficacy and safety trials were successful, the vaccine would be rolled out for widescale use. Thus, “[b]y replacing conventional phase 3 testing of vaccine candidates, such trials may subtract many months from the licensure process, making efficacious vaccines available more quickly” (Eyal et al., 2020).

Proponents of COVID-19 human challenge studies were aware that their proposals were not consistent with existing norms governing challenge trials. Most obviously, wildtype SARS-CoV-2 can cause severe disease and death and there is no rescue therapy. Eyal and colleagues argue that the risks in COVID-19 challenge studies should be permitted nonetheless because they are consistent with other risks we do accept. They say: “we actually ask people to take such risks for others’ direct gain every time we ask volunteer firefighters to rush into burning buildings [and] relatives to donate a live organ to loved ones” (Eyal et al., 2020). Savulescu sees permitting more risk as needed to speed scientific progress against the pandemic. He says: “How quickly we develop a treatment or a cure will depend on the risks we are prepared to take. . . We need to run the race by the rules in order to sufficiently protect human participants. But there is a balance to be struck: let’s not run it with one hand tied behind our back” (Savulescu, 2020).

Indeed, these arguments influenced some bioethicists and policy makers at the time. For example, the World Health Organization released ethical guidance for COVID-19 challenge trials in May 2020 that omitted any upper limit to the risk to which research participants may be exposed (WHO, 2020). Further, the document set aside the requirement for rescue therapy saying:

Although treatment is one important way of reducing risk, the existence of specific, curative treatments is not a necessary condition for the ethical acceptability of challenge studies; however, if or when proven specific treatments are developed, these should be administered to participants as required (WHO, 2020).

These changes paved the way for COVID-19 challenge trials to proceed in the midst of a pandemic.

What happened with human challenge trials?

The rapid development of highly effective vaccines to protect against COVID-19 is one of the great scientific success stories of the last 100 years (Ball, 2021). The complete genome of the SARS-CoV-2 virus was published on January 10, 2020. Just 3 days later, the first vaccine candidate was developed. Dose finding trials in human volunteers began 63 days later, and, by July 2020, phase 3 field trials evaluating vaccine candidates were enrolling tens of thousands of volunteers. Trial results were first released in November 2020 and showed that the two mRNA vaccine candidates (Moderna and Pfizer/ BioNTech) were highly effective and safe. The first COVID-19 vaccines were rolled out in the United Kingdom and the United States in December 2020.

The rapid development of COVID-19 vaccines was the result of traditional vaccine development. How did this happen? The vaccine development process was accelerated through unprecedented investment and collaboration among the private sector, government, and academia. The ACTIV (Accelerating COVID-19 Therapeutic Interventions and Vaccines) public-private partnership is illustrative (Corey et al., 2020). The ACTIV partnership brought together the U.S. National Institutes of Health, U.S. Food and Drug Administration, European Medicines Agency, academics, philanthropic organizations and 15 pharmaceutical companies to provide a common infrastructure for COVID-19 vaccine trials. The infrastructure includes access to independent biostatisticians to design and analyze trials, a common research ethics committee to review trials, independent laboratories to process samples, and a single data and safety monitoring board to oversee ongoing trials.

Did human challenge studies accelerate the development of COVID-19 vaccines? No, not at all. While researchers in several countries discussed openly the possibility of COVID-19 challenge trials, ultimately many decided not to proceed. However, two COVID-19 challenge trials did go ahead in the United Kingdom (Zarley, 2023). These COVID-19 challenge trials were reviewed and approved by the Health Research Authority Specialist Research Ethics Committee in accord with guidance from the World Health Organization (Davies, 2023; WHO, 2020). To date, only the results of the COVID-19 challenge study conducted by Imperial College London have been published (Killingley et al., 2022). What is striking about this study is the length of time from inception to publication—2 years. This is the result, in large part, of the time to manufacture a stable viral strain and to determine the dose and route of administration of the virus (the “challenge model”). It is noteworthy that the published COVID-19 challenge model is the original wildtype SARS-CoV-2, which has been superseded by a succession of variants of concern, from alpha to omicron. Consequently, the challenge model could not be used to screen current vaccine candidates.

Lessons learned for human challenge trials

It is humbling to look back at the predictions of scientists and ethicists about COVID-19 human challenge trials. To put it plainly, events have proven their predictions wrong. Several lessons must be learned for the conduct of human challenge trials in future pandemics.

First, human challenge trials are unlikely to accelerate vaccine development in future pandemics. Contrary to claims of proponents, these trials are cumbersome (Deming et al., 2020). Challenge trials require 1–2 years of background work before a vaccine candidate can even be tested. The first step in the process is the production of the challenge stock. The challenge stock must be free of mutations acquired during the manufacturing process and reliably produce infection in animal models. The second step is the development of the challenge model. The challenge model is the protocol that sets out step-by-step instructions for how healthy volunteers will be infected. Determining the “dose” of virus requires cohorts of healthy volunteers to be exposed to increasing amounts of the infectious agent, and each step takes a month or more. The completion of both steps takes considerable time. For example, the most recent influenza human challenge models took about one-and-a-half years to develop (Han, 2019; Memoli et al., 2015). Further, human challenge trials are unlikely to be able to keep up with a dynamic pandemic in which new variants emerge every few months.

Second, ethical standards for human challenge trials must be upheld—not eroded— during a pandemic. The willingness of some experts, including ethicists, to set aside ethical protections for research participants in the COVID-19 pandemic should give us pause. Many seemed in the grip of a pandemic-induced consequentialism, the idea that a few could legitimately be sacrificed so that many could be saved. Human challenge studies should not be conducted with infectious agents known to cause severe illness or death in the absence of rescue therapy. This ethical standard would likely prohibit human challenge trials in the early days of a pandemic. Once an attenuated strain is developed (or highly effective treatment is identified), human challenge trials might play a useful role in furthering our understanding of how the agent infects people, causes disease, and is transmitted.

Randomized controlled trials to identify urgently needed treatments

Is it ethical to randomize patients in a pandemic?

Randomized controlled trials (RCTs) are an essential method for the rigorous evaluation of medical treatments. But the urgency of treatment in the pandemic raised questions about how trials could be conducted ethically. The physician has a duty of care to her patient, requiring that she act and advise to promote the patient’s medical interests (Weijer et al., 2015). The ethical problem is: how can the physician offer a patient enrollment in an RCT while upholding the duty of care?

Clinical equipoise is widely regarded as answering the ethical problem of randomization. According to Freedman, the author of the concept, the duty of care requires the physician to abide by professional norms established by the community of expert practitioners (Freedman, 1987). Thus, competent care is established by the medical community and not by the opinion of the individual practitioner. When there is an honest, professional disagreement in the community of expert practitioners as to the preferred treatment for an illness, the physician may ethically offer their patient enrollment in an RCT evaluating competing treatments.

A recent review mapped discussion early in the pandemic of the ethics of RCTs evaluating putative treatments for COVID-19 (Nix and Weijer, 2021). The review documents several potential impediments to the ethical randomization of patients. First, it was claimed that physician preference for an unproven COVID-19 treatment may disrupt clinical equipoise (Nix and Weijer, 2021). Raschke explains that “many clinicians are not able to maintain such equipoise in the face of catastrophe” (Raschke, 2020). But clinical equipoise refers to uncertainty in the expert community and it is not disturbed by the opinion of a single physician (Nix and Weijer, 2021). Professionalism demands that the physician recognize when a preferred treatment lacks a substantial evidence base and allow trials to proceed. In such circumstances, however, the physician’s preference may be disclosed to the patient (Freedman, 1987).

Second, some worried that the widespread use of an unproven COVID-19 treatment may be incompatible with clinical equipoise (Nix and Weijer, 2021). Waterer says that the “routine use of unproven agents for SARS-CoV-2. . .[means] clinical equipoise is lost and an experimental agent becomes de facto standard of care” (Waterer et al., 2020). But this fails to recognize that treatment is professionally validated by evidence and not mere use. If a treatment is widely used despite the lack of evidence, then, as Hey and colleagues argue, clinical equipoise supports an RCT “when the effectiveness of the standard of care has been called into question. . .[by] doubts about the supporting body of existing evidence” (Hey and Weijer, 2016).

Third, it was asserted that patient preference for an unproven COVID-19 treatment may disrupt clinical equipoise (Nix and Weijer, 2021). Keane observes that if “recruitment is difficult because of placebo-arm aversion, this should be a signal as to the study’s lack of equipoise” (Keane, 2020). But this conflates clinical equipoise with the ethics of consent. The ethical problem of randomization asks how offering RCT enrollment can be consistent with the physician’s duty of care to the patient. Clinical equipoise shows that RCT enrollment is consistent with the duty of care when there is uncertainty in the clinical community about the preferred treatment. “The patient has the freedom to accept or decline enrollment, but neither decision throws equipoise or the ethics of the trial into question” (Nix and Weijer, 2021). Thus, clinical equipoise supports the ethical conduct of well designed RCTs evaluating putative treatments for COVID-19.

Research ethics committee review of trials evaluating COVID-19 treatments

Research ethics committee review of COVID-19 trials generally seems to have proceeded efficiently in Canada and Europe. For instance, in the Canadian Treatments for COVID-19 (CATCO) trial (Canada’s contribution to the World Health Organization’s Solidarity trial), the median time from research ethics committee submission to approval was only 4.5 days (Teo et al., 2023). The rapid approval of the CATCO trial is likely the result of mechanisms to speed research ethics committee approval of COVID-19 trials (Clinical Trials Ontario, 2023). Most European countries also have “fast-track” mechanisms in place for research ethics committee approval (European Network of Research Ethics Committees, 2023).

However, this experience may not generalize to resource-poor settings. The George Institute for Global Health initiated three trials evaluating COVID-19 therapeutics in 42 sites in India (Bassi et al., 2022). The median time from protocol submission to research ethics committee approval was 59.5 days—13 times the Canadian period. According to the authors, “substantial delays [were] introduced by infrequent meetings, adherence to the traditional format of initial scientific subcommittee review before full ethics committee review, and the mandate for legal approval for clinical trial agreements before considering the ethics application” (Bassi et al., 2022). Once the study was approved at a site, researchers faced additional obstacles including “persistent electricity outages, poor internet connectivity, and the large digital divide” (Bassi et al., 2022).

Risk of bias in trials of COVID-19 treatments

Given the urgent need to identify effective treatments for COVID-19, many randomized controlled trials were designed and subsequently approved by research ethics committees. In April of 2020, a total of 580 trials evaluating treatment for COVID-19 were registered in ClinicalTrials.gov, 394 of which were RCTs (Mainoli et al., 2021). By May 2020, that figure had grown to 1551 trials registered, of which 664 were RCTs (Pundi et al., 2020). The drive to accelerate scientific progress was associated with an apparent acceptance of risks of bias. Of the 664 randomized trials registered in May 2020, many lacked design features to protect from bias: 300 trials (45.2%) lacked allocation concealment (blinding), 461 trials (69.4%) lacked a placebo control, and 461 trials (69.4%) were conducted at a single site (Pundi et al., 2020). Conversely, only 75 trials (11.3%) reported blinding, a placebo control, and enrollment at two or more sites.

Single-center trials are at a risk of failing to recruit adequate numbers of participants. In the April 2020 cohort of trials, few were completed and published in a timely manner. Five months after registration, only 11.4% of trials had been completed, and only 7.9% had been published (Mainoli et al., 2021). Of the remaining trials, many reported problems with recruitment. “One hundred forty-four of 580 trials (24.8%) either had the status ‘Not yet recruiting’ or ‘Suspended’, and 18 (3.1%) trials were prematurely stopped (‘Terminated’ or ‘Withdrawn’). The number of completed trials and trials with results [were] much lower than anticipated, considering the planned follow-up” (Mainoli et al., 2021).

Published RCTs evaluating COVID-19 treatments were frequently assessed to have a high risk of bias. Kudhail and colleagues applied the Risk of Bias 2.0 tool to a random sample of 40 randomized controlled trials of COVID-19 therapy (Kudhail et al., 2022). They concluded that 19 of the 40 trials (47%) were at high risk of bias. Most trials (67%) lacked blinding and, as a result, participants and health providers were aware of assigned interventions (Kudhail et al., 2022). This in turn led to deviations from the assigned intervention that may have impacted the study outcome in 27% of trials. Bias from selective reporting of results was a concern in 15% of trials.

The failure to publish trials and the publication of trials with high risks of bias is not only a scientific problem, but also an ethical concern. The social value of research plays a central role in justifying the risks and burdens to which research participants are exposed. Trials that fail to recruit enough participants and are not published produce no social value. Published trials with a high risk of bias have unreliable results and hence, substantially diminished social value. In both cases, research participants may be exposed to risks and burdens that are not justified by the social value to be gained. Further, trials with high risk of bias can also harm patients outside of research. Kudhail and colleagues caution that “poorly conducted trials, which generate biased results, can ultimately translate into ineffective or even harmful treatments” (Kudhail et al., 2022).

Reliance on trials with a high risk of bias led to patient harm

Research conducted on a putative treatment for COVID-19 called ivermectin is an unfortunate example of high risk of bias research and its downstream impact on patients (Meyerowitz-Katz et al., 2022). Interest in ivermectin as a potential treatment for COVID-19 was sparked by a report claiming that it inhibits the replication of the SARS-CoV-2 virus in the laboratory (Meyerowitz-Katz et al., 2022). This was followed by a series of observational studies and RCTs. A meta-analysis of the trials concluded that ivermectin treatment reduces mortality in COVID-19 (Hill et al., 2022). But scrutiny of the trials included in the meta-analysis revealed serious problems. In one trial reporting a statistically significant reduction in mortality, there was evidence of outright fraud; in another trial that reported positive results, there was evidence of serious methodological problems (Lawrence et al., 2021). A revised analysis excluding these trials demonstrated that ivermectin has no impact on mortality in patients with COVID-19. Further investigation revealed that one-third of ivermectin studies had “serious errors or signs of potential fraud” (Schraer and Goodman, 2021).

In several countries, people were needlessly given ivermectin either to treat or prevent COVID-19. In May 2020, health providers in Bolivia distributed 350,000 ivermectin doses (Mega, 2020). In Mexico City, 200,000 medical kits containing ivermectin were distributed (Meyerowitz-Katz et al., 2022). In the United States, human prescriptions for ivermectin increased 24-fold in 2021 (Temple et al., 2021). Many people who received ivermectin were harmed. In one case, four Arkansas prisoners “suffered side effects including vision problems, diarrhea, and bloody stools after the physician at Washington County jail treated their COVID with high doses of ivermectin without their knowledge” (Dyer, 2022). According to the US National Poison Data System, there were 1143 cases of ivermectin poisoning in the first 8 months of 2021, an increase of 163% from the same period in 2020 (Romo, 2021).

Lessons learned for RCTs

As we have seen, high risk of bias RCTs were not only scientifically flawed, but ethically problematic. Unreliable science led to the promulgation of ineffective treatment, and, in some cases, patients were harmed. Several lessons must be learned for the conduct of RCTs in future pandemics.

First, the design of RCTs evaluating putative treatments must prioritize scientific rigor. The COVID-19 pandemic unleashed a wave of poorly designed RCTs with a high risk of bias. These studies did little to advance our understanding of how best to treat patients. Instead, scarce resources were spent on trials that produced unreliable results. Several steps should be taken to promote scientific rigor. RCTs should be led by researchers experienced in the design and conduct of high-quality RCTs. RCTs should include in the research team a qualified biostatistician who has responsibility for the scientific design and analysis of the trial. The RCT must be registered, and the study protocol published in a timely manner.

Second, infrastructure to support high-quality multi-center RCTs is an investment priority now, in advance of the next pandemic. The considerable success of the RECOVERY trial has taught us that the creation of research platforms across multiple institutions would both establish research infrastructure in advance of the next pandemic and allow multiple RCTs to share that infrastructure. Core data sets could be agreed upon and included in routinely collected health records. Data sharing agreements could be put in place to allow data to flow to centralized statistical centers. And master protocols could be designed and approved by research ethics committees.

Third, RCTs of putative treatments in a pandemic must uphold the highest ethical standards. Research ethics principles support the conduct of well-designed research during a pandemic. Specifically, clinical equipoise supports the conduct of RCTs to ensure treatment is evidence based. Research ethics committees have a key role to play in ensuring that informed consent procedures are adequate, study intervention and control conditions are justified, and participant selection is equitable. Research ethics committees should ensure that RCTs are well designed and adequately resourced. They should, therefore, be reluctant to approve single-center RCTs led by inexperienced investigators.

Conclusion

The urgent need for COVID-19 prevention and treatment fueled an embrace of risks as allegedly necessary costs to speed scientific progress. As we have seen, scientists and (even) ethicists supported overturning longstanding norms protecting healthy volunteers in human challenge trials to speed vaccine development, but these trials led to no vaccines. Physicians, with the approval of research ethics committees, designed hundreds of unblinded, single-center clinical trials at high risk of bias to speed the identification of new treatments. But these clinical trials led to no treatments. The lesson for future pandemics is that the acceptance of greater risks to participants or science does not reliably lead to progress. We are better served by science that upholds the highest ethical and methodological standards.