Examining the bias-efficiency tradeoff from incorporation of nonconcurrent controls in platform trials: A simulation study example from the adaptive COVID-19 treatment trial

Abstract

Background:

Platform trials typically feature a shared control arm and multiple experimental treatment arms. Staggered entry and exit of arms splits the control group into two cohorts: those randomized during the same period in which the experimental arm was open (concurrent controls) and those randomized outside that period (nonconcurrent controls). Combining these control groups may offer increased statistical power but can lead to bias if analyses do not account for time trends in the response variable. Proposed methods of adjustment for time may increase type I error rates when time trends impact arms unequally or when large, sudden changes to the response rate occur. However, there has been limited exploration of the degree of type I error inflation one can plausibly expect in real-world scenarios.

Methods:

We use data from the Adaptive COVID-19 Treatment Trial (ACTT) to mimic a realistic platform trial with a remdesivir control arm. We compare four strategies for estimating the effect of interferon beta-1a (the ACTT-3 experimental arm) relative to remdesivir (data from ACTT-1, ACTT-2, and ACTT-3) on recovery and death by day 29: utilizing concurrent controls only (the prespecified analysis), pooling all remdesivir arm data without adjustment (the “unadjusted-pooled” analysis), adjusting for time as a categorical variable, and a Bayesian hierarchical model implementation which adjusts for time trends using smoothing techniques (the “Bayesian time machine”). We compare type I error rates and relative efficiency of each method in simulation settings based on observed ACTT remdesivir arm data.

Results:

The unadjusted-pooled approach provided substantially different estimates of the effect of interferon beta-1a relative to remdesivir compared with the concurrent-only and model-based approaches, indicating that changes in recovery and death rates over time were not ignorable across different stages of ACTT. The model-based approaches rely on an assumption of constant treatment effects for each arm in the platform relative to control; error rates more than doubled in settings where this was not satisfied. Relative efficiency of the model-based approaches compared with the concurrent-only analysis was moderate.

Conclusions:

In simulation settings where key model assumptions were not met, potential efficiency gains from incorporation of nonconcurrent controls were outweighed by the risk of substantial type I error rate inflation. This leads us to advise against these strategies for primary analyses in confirmatory clinical trials, aligning with current FDA guidance advising against comparisons to nonconcurrent controls in COVID-19 settings. The model-based adjustment methods may be useful in other settings, but we recommend performing the concurrent-only analysis as a reference for assessing the degree to which nonconcurrent controls drive results.