Estimating counterfactual placebo HIV incidence in HIV prevention trials without placebo arms based on markers of HIV exposure

Abstract

Introduction

Developing alternative approaches to evaluating absolute efficacy of new HIV prevention interventions is a priority, as active-controlled designs, whereby individuals without HIV are randomized to the experimental intervention or an active control known to be effective, are increasing. With this design, however, the efficacy of the experimental intervention to prevent HIV acquisition relative to placebo cannot be evaluated directly.

Methods

One proposed approach to estimate absolute prevention efficacy is to use an HIV exposure marker, such as incident rectal gonorrhea, to infer counterfactual placebo HIV incidence. We formalize a statistical framework for this approach, specify working regression and likelihood-based estimation approaches, lay out three assumptions under which valid inference can be achieved, evaluate finite-sample performance, and illustrate the approach using a recent active-controlled HIV prevention trial.

Results

We find that in finite samples and under correctly specified assumptions accurate and precise estimates of counterfactual placebo incidence and prevention efficacy are produced. Based on data from the DISCOVER trial in men and transgender women who have sex with men, and assuming correctly specified assumptions, the estimated prevention efficacy for tenofovir alafenamide plus emtricitabine is 98.1% (95% confidence interval: 96.4%–99.4%) using the working model approach and 98.1% (95% confidence interval: 96.4%–99.7%) using the likelihood-based approach.

Conclusion

Careful assessment of the underlying assumptions, study of their violation, evaluation of the approach in trials with placebo arms, and advancement of improved exposure markers are needed before the HIV exposure marker approach can be relied upon in practice.

Keywords

Counterfactual placebo HIV prevention randomized controlled trial rectal gonorrhea trial design

Introduction

The last decade has seen dramatic success in HIV prevention¹ with effective pre-exposure prophylaxis (PrEP) products.^2–8 Despite these successes, HIV remains a major threat to global health.⁹ As considerable challenges to implementing existing prevention interventions exist,^10,11 additional biomedical prevention interventions are needed.

A variety of new preventive interventions (e.g. alternative PrEP agents, vaccines, etc.) are in development.¹² Placebo-controlled randomized trials that enroll individuals without HIV and follow them for incident HIV acquisition have historically been required for regulatory approval of new interventions. For new interventions in the same “class” as an intervention already proven effective, future trials will likely be “active-controlled”;¹³ participants without HIV are randomized to the experimental intervention or an existing “active-control” intervention already proven effective. Even for new interventions in as-yet-unproven classes, for example, vaccines, an active-controlled design may be necessary.

The fundamental challenge of an active-controlled trial is that absolute prevention efficacy, that is, the reduction in HIV incidence for the intervention relative to placebo, cannot be evaluated based on the trial data alone. Instead, relative efficacy of the experimental and active-control interventions is assessed. Yet absolute efficacy is arguably the parameter of most interest.^14,15 A traditional approach to estimating efficacy is using data from a historical placebo-controlled trial of the active control to set a “margin” for establishing non-inferiority or superiority of the experimental intervention, based on the assumption that efficacy established in the historical trial can be carried over to the new trial.^16,17 This approach is challenging in HIV prevention, since many interventions are highly user-dependent,^18–20 and efficacy of vaccines and monoclonal antibodies depends on properties of the exposing virus;^21–23 thus, efficacy in the historical trial may not apply to the current trial. In addition, non-inferiority trials generally require larger sample sizes than placebo-controlled trials, especially if the active control is highly effective. Therefore, developing alternative approaches to evaluating absolute efficacy of new HIV prevention interventions is a priority.

One approach proposed in concept^24,25 and widely discussed in the HIV prevention field^{14,15,25–30} is to use a marker of HIV exposure as a proxy to infer “counterfactual placebo” HIV incidence, that is, the incidence observed had a placebo arm been included in the active-controlled trial. This requires establishing the association between incidence of HIV and an HIV exposure marker in the absence of intervention, estimated based on historical data. Provided the intervention does not affect the HIV exposure marker, incidence of the marker in the active-controlled trial can be used to estimate counterfactual placebo HIV incidence. Figure 1 illustrates this concept. Incident rectal gonorrhea has been proposed as the HIV exposure marker for men who have sex with men, based on observational data suggesting that the incidence rates of these two sexually transmitted infections are highly correlated.²⁴ U.S. Food and Drug Administration (FDA)³¹ advisory committees reviewing new PrEP agents support this approach, and the FDA endorsed the approach in guidance to industry. Yet, a formal statistical framework is lacking.

Figure 1.

Estimation of counterfactual HIV incidence based on an HIV exposure marker. Green solid and dashed curves correspond to the fitted model associating HIV and exposure marker incidences with an associated pointwise 95% confidence interval (CI), based on a set of external cohorts reporting HIV and exposure marker incidence rates (dark blue dots). Given the exposure marker incidence in the active-controlled trial (yellow dot), counterfactual placebo HIV incidence is estimated with use of the fitted model (red dot). The 95% CI for the counterfactual placebo incidence captures uncertainty due to the model fit and uncertainty in the exposure marker incidence.

Here, we (1) articulate a statistical framework for inferring counterfactual placebo HIV incidence for an active-controlled trial using a marker of HIV exposure; (2) describe two estimation approaches and articulate the assumptions under which they produce unbiased estimates; (3) conduct a simulation study designed to closely mimic data on HIV and rectal gonorrhea and evaluate the performance of the methods under idealized conditions, that is, when all assumptions are satisfied; and (4) apply the methodology to data from a recently conducted active-controlled HIV prevention trial⁶ and highlight the limitations of the approach and implications for its use in future HIV prevention trials.

Methods

Setting and notation

Let $X$ indicate the HIV prevention intervention where $X = 0$ denotes placebo, $X = 1$ denotes experimental intervention, and $X = 2$ denotes active-control intervention.

For an active-controlled trial, $Y$ denotes the HIV diagnosis event time and $λ_{k}^{Y}$ denotes the HIV incidence rate for subjects randomized to $X = k$ for $k = 1, 2$ . Let $λ_{0}^{Y}$ denote the counterfactual HIV incidence for the trial population under placebo. Incidence rates are assumed constant over time. The quantity of primary interest is the prevention efficacy of the experimental intervention, given by

P E_{1} = 1 - \frac{λ_{1}^{Y}}{λ_{0}^{Y}}

(1)

We formulate a general approach for evaluating $P E_{k} = 1 - λ_{k}^{Y} / λ_{0}^{Y}$ , $k = 1, 2$ .

Remark 1

Prevention efficacy is evaluated against a backdrop standard of HIV prevention for the target population, consisting of proven and available HIV prevention products.¹³ Therefore, $λ_{0}^{Y}$ , hereafter “placebo incidence,” is the counterfactual HIV incidence where trial participants are randomized to receive a placebo in addition to the standard of HIV prevention. Interpretation of $λ_{0}^{Y}$ and $P E_{k}$ are specific to this standard of prevention.

In a randomized placebo-controlled trial for intervention $k$ , $λ_{k}^{Y}$ and $λ_{0}^{Y}$ are estimated directly. However, in an active-controlled trial, only $λ_{k}^{Y}$ can be estimated, and $λ_{0}^{Y}$ , and therefore $P E_{k}$ , cannot be estimated directly. Instead, data from external cohorts are used to establish a relationship between HIV and an exposure marker, so as to infer $λ_{0}^{Y}$ .

Assumptions

Let $λ_{k}^{Z}$ denote the incidence of HIV exposure marker $Z$ in the active-controlled trial population, given randomization to intervention $X = k$ . We assume $λ_{k}^{Z}$ is constant in time. We establish the relationship between $λ_{0}^{Y}$ and $λ_{0}^{Z}$ based on data from $M$ external cohorts, each of which is conducted under a specific standard of HIV prevention. We refer to these as “placebo” incidence rates for simplicity. For external cohort $m = 1, \dots, M$ , let $λ_{m 0}^{Y}$ and $λ_{m 0}^{Z}$ be the incidences of HIV and the exposure marker, respectively.

We parameterize the relationship between HIV and the exposure marker incidences as

g (λ_{m 0}^{Y}) = f {h (λ_{m 0}^{Z})} + ϵ_{m}

(2)

for $m = 0, \dots, M$ , where $λ_{00}^{Y} = λ_{0}^{Y}$ and $λ_{00}^{Z} = λ_{0}^{Z}$ are the incidence parameters for the active-controlled trial population, $g (\cdot)$ and $h (\cdot)$ are known link functions appropriate for non-negative incidence parameters, for example, logit or log links, $g (\cdot)$ is an invertible function, $f (\cdot)$ is an unknown regression function that can be either parametric or nonparametric, and $ϵ_{m}$ is an i.i.d. mean-zero error term. Importantly, $λ_{m 0}^{Y}$ and $λ_{m 0}^{Z}$ are random variables: there is variability in the placebo incidence rates across cohorts due to different compositions of risk factors and differences in the standard of prevention for both HIV and the exposure marker.

We state the following assumptions.

Assumption 1

Model equation (2) describes a general relationship between placebo HIV and exposure marker incidence rates that holds across external cohorts and the active-controlled trial population.

While HIV and exposure marker incidences may vary, the association between the incidence rates is assumed constant. To evaluate Assumption 1, one must consider carefully the background standard of HIV prevention for the active-controlled trial population, and whether any element of this prevention package influences the relationship between HIV and the exposure marker. For example, oral PrEP is known to reduce HIV but does not have a biological effect on rectal gonorrhea or other non-HIV sexually transmitted infections,⁶ even though it may have an effect in terms of behavioral “risk disinhibition.”³² Therefore, if the trial standard of HIV prevention does not include oral PrEP, the external cohorts should be drawn from populations without access to oral PrEP. Elements of the standard of HIV prevention (i.e. condoms and risk reduction counseling) may influence HIV and rectal gonorrhea incidences but not to modify their association, and therefore may not be critical to consider in evaluating external cohorts. Effective biomedical prevention of non-HIV sexually transmitted infections is another potential effect modifier. Other potential effect modifiers include subject demographics, behaviors, and features of the local HIV epidemic, that is, population prevalence of HIV and level of viral suppression for those living with HIV. Blinding may also influence the relationship between HIV and the exposure marker. While the counterfactual placebo arm is (conceptually) blinded, the external cohorts may not be. While Assumption 1 can be evaluated for the external cohorts, whether it holds for the trial population cannot be tested, given the absence of a placebo arm for the trial population.

Assumption 2

An unbiased estimate of the parameters in $f$ can be obtained based on estimated incidences $({\hat{λ}}_{m 0}^{Y}, {\hat{λ}}_{m 0}^{Z})$ , $m = 1, \dots, M$ , and their associated variances in the external cohorts. The estimate of $f$ after plugging in those parameter estimates is denoted by $\hat{f}$ .

Assumption 2 indicates that the relationship between HIV and exposure marker incidences can be consistently estimated using the observed incidence rates from the external cohorts. This assumption is specific to the estimation approach and will be discussed below.

Assumption 3

The exposure marker incidence is not modified by randomization to active intervention $X = k$ , that is, $λ_{k}^{Z} = λ_{0}^{Z}$ .

Assumption 3 stipulates that the incidence of the HIV exposure marker under $X = k$ is the same as that under placebo. In evaluating this assumption, one must consider whether the intervention, or elements of the standard of HIV prevention, may modify the incidence of the exposure marker. Whether or not the active-controlled trial is blinded is also relevant, since knowledge of receipt of intervention may modify behavior. This assumption can be partially evaluated in the context of an historical randomized, placebo-controlled trial of the intervention with the exposure marker collected as an endpoint. However, the historical data do not inform on whether Assumption 3 holds for the trial population.

Under Assumptions 1–3, counterfactual placebo HIV incidence can be consistently estimated by

{\hat{λ}}_{0, k}^{Y} = g^{- 1} \hat{f} (h ({\hat{λ}}_{k}^{Z})),

where ${\hat{λ}}_{k}^{Z}$ is the observed exposure marker incidence in the trial arm randomized to intervention $k$ . Importantly, uncertainty in the estimated counterfactual placebo HIV incidence is comprised of the uncertainty due to fitting the regression model, $\hat{f}$ , and the uncertainty in the exposure marker incidence, ${\hat{λ}}_{k}^{Z}$ , as illustrated in Figure 1. Prevention efficacy, $P E_{k}$ , can then be estimated by

{\hat{PE}}_{k} = 1 - \frac{{\hat{λ}}_{k}^{Y}}{{\hat{λ}}_{0, k}^{Y}},

where ${\hat{λ}}_{k}^{Y}$ is the estimated HIV incidence among those randomized to intervention $k$ in the active-controlled trial.

Remark 2

If the exposure marker incidence is not modified by either intervention in the active-controlled trial, the exposure marker incidence among all trial participants may be used to estimate counterfactual placebo HIV incidence. This provides a more precise estimate, relative to the estimate based on the exposure marker incidence among participants that received intervention $k$ .

Bivariate linkage model

To estimate $P E_{k}$ , we focus on a special case of model equation (2) for which we discuss validity and violation of Assumption 2. We assume a bivariate normal distribution for the log-transformed HIV and exposure marker incidence rates, $U_{m} = \log (λ_{m 0}^{Y})$ and $V_{m} = \log (λ_{m 0}^{Z})$ . In particular,

(\begin{matrix} U_{m} \\ V_{m} \end{matrix}) ~ MV N_{2} ((\begin{matrix} μ_{U} \\ μ_{V} \end{matrix}), (\begin{matrix} σ_{U}^{2} & ρ σ_{U} σ_{V} \\ ρ σ_{U} σ_{V} & σ_{V}^{2} \end{matrix}))

(3)

for $m = 1, \dots, M$ , where $MV N_{p} (μ, Σ)$ denotes a $p$ -dimensional multi-normal distribution with mean $μ$ and variance matrix $Σ$ , $μ_{U}$ and $μ_{V}$ are means of $U_{m}$ and $V_{m}$ , respectively, $σ_{U}^{2}$ and $σ_{V}^{2}$ are variances of $U_{m}$ and $V_{m}$ , respectively, and $ρ \in (0, 1)$ is the correlation of $U_{m}$ and $V_{m}$ that measures the magnitude of association.

We assume the estimated incidence rates from the external cohorts, $({\hat{λ}}_{m 0}^{Y}, {\hat{λ}}_{m 0}^{Z})$ , are conditionally independent, given the true incidence rates $(λ_{m 0}^{Y}, λ_{m 0}^{Z})$ , for $m = 1, \dots, M$ . The joint conditional distribution is given by

(\begin{matrix} {\hat{U}}_{m} \\ {\hat{V}}_{m} \end{matrix}) | (\begin{matrix} U_{m} \\ V_{m} \end{matrix}) \sim M V N_{2} ((\begin{matrix} U_{m} \\ V_{m} \end{matrix}), (\begin{matrix} s_{U, m}^{2} & 0 \\ 0 & s_{V, m}^{2} \end{matrix}))

(4)

where ${\hat{U}}_{m} = \log ({\hat{λ}}_{m 0}^{Y})$ , ${\hat{V}}_{m} = \log ({\hat{λ}}_{m 0}^{Z})$ , and $s_{U, m}^{2}$ and $s_{V, m}^{2}$ are the conditional variances of ${\hat{U}}_{m}$ and ${\hat{V}}_{m}$ given $(U_{m}, V_{m})$ , respectively. Conditional independence is a common assumption in bivariate outcome meta-analysis,^33,34 in part because conditional dependence is not commonly evaluated or reported in studies. We evaluate bias due to violation of the conditional independence assumption in simulations. Note that equation (4) is a model for the log-transformed incidence rates, but it does not stipulate a model for individual-level HIV and exposure marker outcomes.

Under the bivariate linkage model, the parameters $(μ_{U}, μ_{V}, σ_{U}^{2}, σ_{V}^{2}, ρ)$ can be estimated using a maximum likelihood approach given the observed external cohort data $({\hat{U}}_{m}, {\hat{V}}_{m})$ $(m = 1, \dots, M)$ . Write $({\hat{μ}}_{U}, {\hat{μ}}_{V}, {\hat{σ}}_{U}^{2}, {\hat{σ}}_{V}^{2}, \hat{ρ})$ as the estimates. The counterfactual placebo HIV incidence $λ_{0}^{Y}$ can therefore be estimated by

{\hat{λ}}_{0, k}^{Y} = \exp {{\hat{μ}}_{U} + \frac{\hat{ρ} {\hat{σ}}_{U} {\hat{σ}}_{V}}{{\hat{σ}}_{V}^{2} + {\hat{s}}_{V, k}^{2}} (\log {\hat{λ}}_{k}^{Z} - {\hat{μ}}_{V}))}

(5)

and prevention efficacy can be estimated by

{\hat{PE}}_{k} = 1 - \frac{{\hat{λ}}_{k}^{Y}}{{\hat{λ}}_{0, k}^{Y}}

(6)

See Supplemental Materials for details.

While maximum likelihood estimation yields consistent and efficient parameter estimates under correct model specification, it may not be stable when the number of external cohorts is small, for example, $M < 20$ , as suggested by our simulations. When $M$ is small, we suggest fitting a working regression model or “working model”:

{\hat{U}}_{m} = α^{*} + β^{*} {\hat{V}}_{m} + ϵ_{m}^{*}, ϵ_{m}^{*} ~ N (0, σ^{* 2})

(7)

In general, the working model is mis-specified. However, the estimated regression function based on working model estimates $({\hat{α}}^{*}, {\hat{β}}^{*}, {\hat{σ}}^{*})$ and denoted by ${\hat{f}}^{*}$ , may approximate $f$ enough to provide adequate inference about counterfactual placebo HIV incidence. Using the working model, the counterfactual placebo HIV incidence is estimated by

{\hat{λ}}_{0, k}^{*, Y} = \exp ({\hat{α}}^{*} + {\hat{β}}^{*} \log {\hat{λ}}_{k}^{Z})

(8)

and prevention efficacy is estimated by

{\hat{PE}}_{k}^{*} = 1 - \frac{{\hat{λ}}_{k}^{Y}}{{\hat{λ}}_{0, k}^{*, Y}}

(9)

See Supplemental Materials for details.

In summary, assuming the bivariate linkage model equation (3), the procedure for estimating counterfactual placebo HIV incidence and prevention efficacy is as follows:

Step 1. Given estimated incidences $({\hat{λ}}_{m}^{Y}, {\hat{λ}}_{m}^{Z})$ from $m = 1, \dots, M$ external cohorts, fit the bivariate linkage model using either the maximum likelihood or working model approaches;

Step 2. Given the estimated incidence rate of the exposure marker ${\hat{λ}}_{k}^{Z}$ from the active-controlled trial, estimate the counterfactual placebo HIV incidence $λ_{0}^{Y}$ , using either equation (5) for maximum likelihood, or equation (8) for the working model approaches;

Step 3. Given the estimated HIV incidence rate ${\hat{λ}}_{k}^{Y}$ in the active-controlled trial, estimate $P E_{k}$ using either equation (6) for the maximum likelihood, or equation (9) the working model approaches.

R code for implementation is available on Github (https://github.com/feigao1/CF_Exposuremarker).

Simulation studies

To evaluate the numerical performance of the counterfactual placebo incidence and prevention efficacy estimates, we examine the ideal scenario when all assumptions hold (with maximum likelihood estimation) and when Assumption 2 holds approximately (with working model estimation).

External cohorts

Incidences in the external cohort $(λ_{m 0}^{Y}, λ_{m 0}^{Z})$ are generated from the bivariate linkage model equation (3) with log link functions where the parameter values $(μ_{U}, μ_{V}, σ_{U}^{2}, σ_{V}^{2}, ρ) = (- 3.189, - 2.245, 0.537, 0.814, 0.98)$ are the maximum likelihood estimates based on published studies reporting both HIV and rectal gonorrhea incidence for men who have sex with men, summarized in Supplemental Table S1. The interquartile range of HIV and marker incidences in the cohorts are (2.5, 6.8) and (5.8,19.4) cases per 100 person-years, respectively. We also consider a bivariate linkage model equation (3) with logit link functions (see Supplemental Materials). Since the maximum likelihood estimate of the correlation $ρ$ is high $(0.98)$ , we also consider a moderate correlation scenario wherein $ρ = 0.5$ . We set the number of external cohorts to $M = 10$ or $20$ , as generally only a small number of external studies will be available with the required data. The person-times for the external cohorts, $P Y_{m}$ , are uniformly distributed between 200 and 5000 person-years. The number of HIV and exposure marker events, $n_{case, m}^{Y}$ and $n_{case, m}^{Z}$ , follow binomial distributions with expectations $P Y_{m} λ_{m 0}^{Y}$ and $P Y_{m} λ_{m 0}^{Z}$ , respectively. In each cohort $m$ , we estimate incidences by ${\hat{λ}}_{m 0}^{Y} = n_{case, m}^{Y} / P Y_{m}$ , ${\hat{λ}}_{m 0}^{Z} = n_{c a s e, m}^{Z} / P Y_{m}$ ; standard deviations are estimated by $s_{U, m}^{2} = (1 - {\hat{λ}}_{m 0}^{Y}) / ({\hat{λ}}_{m 0}^{Y} P Y_{m})$ and $s_{V, m}^{2} = (1 - {\hat{λ}}_{m 0}^{Z}) / ({\hat{λ}}_{m 0}^{Z} P Y_{m})$ .

Active-controlled trial

We consider a single arm trial for conciseness, with a follow-up time of $n_{x} = 2000$ or $4000$ person-years. Placebo HIV incidence $λ_{0}^{Y}$ is assumed to be 3, 4.5, or 6 cases per 100 person-years, and exposure marker incidence $λ_{k}^{Z}$ is generated based on the conditional distribution of $λ_{k}^{Z}$ given $λ_{0}^{Y}$ (see Supplemental Materials). Prevention efficacy is assumed to be $0.3$ , $0.6$ or $0.75$ and $λ_{k}^{Y} = (1 - PE) λ_{0}^{Y}$ . The number of HIV and exposure marker events, $n_{case}^{Y}$ and $n_{case}^{Z}$ , follow binomial distributions with expectation $n_{x} λ_{k}^{Y}$ and $n_{x} λ_{k}^{Z}$ , respectively. Incidences are estimated by ${\hat{λ}}_{1}^{Y} = n_{case}^{Y} / n_{x}$ , ${\hat{λ}}_{1}^{Z} = n_{case}^{Z} / n_{x}$ .

Estimation methods and performance measures

We apply maximum likelihood and working model estimation approaches, following the procedure listed at the end of the Methods section. We evaluate the average bias, empirical standard deviation, and coverage probability of nominal 95% confidence intervals (CIs) for counterfactual placebo incidence and prevention efficacy estimates across 5000 simulations.

Results

Simulation

Table 1 summarizes the performance of counterfactual placebo HIV incidence estimates across simulation scenarios. We show results with $M = 10$ and $20$ for the working model, and $M = 20$ for likelihood-based estimation, as likelihood-based estimation requires sufficiently large $M$ to ensure numerical stability. For both estimation approaches, we find high correlation between HIV and the exposure marker $(ρ = 0.98)$ yields accurate and precise estimation, as evidenced by low bias, reasonably low standard deviation, and CIs with close to nominal coverage. Even with modest correlation $(ρ = 0.5)$ , low bias and nominal coverage rates are seen, although the standard deviation is larger than in the high correlation scenario. Performance is minimally impacted by the size of the trial’s active arm. Performance of the working model is comparable to that of likelihood-based estimation in settings with $M = 20$ , while it performs worse for large $M$ , with CIs that are overly conservative (results not shown). Bias and coverage rates are similar under a logit link bivariate linkage model (see Supplemental Table S2).

Table 1.

Bias, standard deviation, and empirical coverage for estimated counterfactual placebo HIV incidence, based on $M$ external cohorts used to estimate the association between HIV and an exposure biomarker with correlation $ρ$ . A total of $n_{x}$ person-years follow-up accrue in the active arm of the trial. Counterfactual placebo HIV incidence varies. Performance is shown for working model and likelihood-based estimation approaches, assuming log link function for marginal incidences.

				$ρ = 0.98$			$ρ = 0.5$
HIV incidence (cases per 100 person-years)			$3$	$4.5$	$6$	$3$	$4.5$	$6$
Exposure marker incidence (cases per 100 person-years)			$7.1$	$11.8$	$17.0$	$4.8$	$13.2$	$26.7$
Working model approach
$M = 10$	$n_{x} = 2000$	Bias × 100	–0.01	–0.02	–0.03	0.13	0.07	0.26
		Standard deviation × 100	0.33	0.38	0.53	1.03	1.05	2.22
		Coverage (%)	96.1	97.5	97.2	95.8	95.5	94.5
	$n_{x} = 4000$	Bias × 100	–0.01	–0.02	–0.02	0.11	0.09	0.25
		Standard deviation × 100	0.30	0.36	0.49	1.00	1.09	2.12
		Coverage (%)	95.4	96.5	96.8	95.4	95.4	95.7
$M = 20$	$n_{x} = 2000$	Bias × 100	–0.01	–0.02	–0.03	0.05	0.02	0.11
		Standard deviation × 100	0.27	0.31	0.39	0.64	0.71	1.39
		Coverage (%)	95.2	96.0	97.3	95.0	95.9	95.1
	$n_{x} = 4000$	Bias × 100	–0.02	–0.02	–0.03	0.04	0.03	0.07
		Standard deviation × 100	0.24	0.27	0.34	0.63	0.73	1.36
		Coverage (%)	94.6	96.2	97.4	95.0	94.8	95.2
Likelihood-based approach
$M = 20$	$n_{x} = 2000$	Bias × 100	0.03	0.02	0.04	0.11	0.08	0.20
		Standard deviation × 100	0.26	0.31	0.40	0.65	0.71	1.39
		Coverage (%)	95.5	95.4	94.4	94.5	94.2	93.7
	$n_{x} = 4000$	Bias × 100	0.02	0.03	0.05	0.10	0.10	0.21
		Standard deviation × 100	0.22	0.26	0.35	0.64	0.71	1.42
		Coverage (%)	95.1	95.2	93.5	94.3	94.3	93.1

The performance of estimates of prevention efficacy based on an active-controlled trial with $n_{x} = 2000$ are shown in Table 2. When $ρ$ is large, prevention efficacy can be estimated with low bias and CIs with near-nominal coverage, even with $M = 10$ external cohorts. For example, when PE is 0.6 against a 4.5 cases per 100 person-years placebo HIV incidence, with 10 external cohorts the bias is less than 0.01 and the nominal 95% CI for prevention efficacy has 93.2% coverage. However, with modest $ρ$ , prevention efficacy is estimated with larger bias and CIs slightly under-cover with $M = 10$ . Fixing $M$ , modest $ρ$ generally yields prevention efficacy estimates with larger standard deviation. With high $ρ$ , prevention efficacy can be estimated with better precision when the placebo HIV incidence rate is higher, because incidences can be more precisely estimated when there are more events. With modest $ρ$ , however, the standard deviation is larger when the placebo HIV incidence is 3 or 6 cases per 100 person-years compared to 4.5. This is because, with moderate $ρ$ , the variability of the prevention efficacy estimate is largely dominated by the variability of the estimated counterfactual placebo HIV incidence, which is larger when the placebo HIV incidence is further from the mean HIV incidence across the external cohorts (illustrated in Figure 1). With $M = 20$ , the standard deviations are similar for the working model and likelihood-based estimation approaches, although the CIs from the working model approach have slightly lower coverage in some cases. Similar performance is observed with logit link functions (see Supplemental Table S3). Performance based on a smaller active-controlled trial with $n_{x} = 1000$ person-years follow-up is shown in Supplemental Table S4; standard deviations are larger but coverage rates and bias are only minimally worse.

Table 2.

Bias, standard deviation, and empirical coverage for estimates of prevention efficacy (PE) based on $M$ external cohorts used to estimate the association between HIV and an exposure biomarker with correlation $ρ$ . A total of $n_{x} = 2000$ person-years follow-up accrue in the active arm of the trial. Counterfactual placebo HIV incidence and true PE vary. Performance is shown for working model and likelihood-based estimation approaches, assuming log link function for marginal incidences.

	$M$	PE estimate		$ρ = 0.98$			$ρ = 0.5$
HIV incidence (cases per 100 person-years)			$3$	$4.5$	$6$	$3$	$4.5$	$6$
Exposure marker incidence (cases per 100 person-years)			$7.1$	$11.8$	$17.0$	$4.8$	$13.2$	$26.7$
			Working model approach
$PE = 0.3$ $n_{x} = 2000$	10	Bias × 100	–1.42	–0.83	–0.96	–3.72	–2.28	–4.88
		Standard deviation × 100	14.49	11.45	10.20	28.78	19.50	28.72
		Coverage (%)	93.0	93.4	93.2	92.2	92.7	90.1
	20	Bias × 100	–1.09	–0.77	–0.63	–1.93	–1.20	–1.88
		Standard deviation × 100	13.55	10.58	9.20	19.89	14.38	18.24
		Coverage (%)	92.5	93.4	93.1	93.0	94.4	92.7
$PE = 0.6$ $n_{x} = 2000$	10	Bias × 100	–0.70	–0.23	–0.53	–2.08	–1.11	–2.81
		Standard deviation × 100	10.03	7.97	7.03	17.51	12.04	16.99
		Coverage (%)	93.3	93.1	93.8	92.6	93.6	90.8
	20	Bias × 100	–0.46	–0.33	–0.42	–1.04	–0.44	–1.05
		Standard deviation × 100	9.49	7.47	6.56	12.63	9.45	11.1
		Coverage (%)	93.3	93.8	93.9	93.8	93.7	93.3
$PE = 0.75$ $n_{x} = 2000$	10	Bias × 100	–0.33	–0.21	–0.22	–1.14	–0.62	–1.19
		Standard deviation × 100	7.48	5.95	5.29	11.80	8.30	10.38
		Coverage (%)	93.3	93.9	93.6	93.0	92.9	91.9
	20	Bias × 100	–0.17	–0.17	–0.20	–0.53	–0.36	–0.48
		Standard deviation × 100	7.18	5.80	5.03	8.91	6.65	7.49
		Coverage (%)	93.4	93.7	93.7	94.0	94.3	93.6
			Likelihood-based approach
$PE = 0.3$ $n_{x} = 2000$	20	Bias × 100	0.08	–0.11	0.04	–0.62	–0.64	–1.04
		Standard deviation × 100	12.36	9.90	8.90	18.55	13.83	18.02
		Coverage (%)	94.9	95.0	93.9	93.8	94.3	92.8
$PE = 0.6$ $n_{x} = 2000$	20	Bias × 100	–0.20	–0.29	0.10	–0.31	0.04	–0.26
		Standard deviation × 100	8.83	7.27	6.30	11.80	9.06	10.92
		Coverage (%)	95.3	94.8	94.5	94.3	94.6	93.0
$PE = 0.75$ $n_{x} = 2000$	20	Bias × 100	–0.16	0.00	–0.05	–0.20	–0.03	–0.54
		Standard deviation × 100	6.81	5.48	4.86	8.44	6.55	7.38
		Coverage (%)	95.3	95.5	95.1	94.7	94.3	94.4

We evaluate power for testing prevention efficacy with this approach compared to a placebo-controlled trial (see Supplemental Figures S1 and S2 for simulation results). Surprisingly, we find power for the counterfactual approach may exceed that obtained from a placebo-controlled trial with the same active arm size. For example, 74% power to detect prevention efficacy of 0.6 can be obtained with 3 cases per 100 person-years placebo HIV incidence, active arm size of 2000 person-years, a highly correlated marker $(ρ = 0.98)$ and 10 external cohorts, while a standard 1:1 placebo-controlled trial with sample sizes of 2000 person-years for both experimental and placebo arms has less than 70% power. The reason is incidence of the exposure marker is much higher than that of HIV, so it can be estimated with better precision than HIV incidence. Therefore, with a highly correlated exposure marker, HIV incidence can be estimated more precisely by leveraging information in the higher-incidence exposure marker.

We evaluate scenarios where the conditional independence assumption in equation (4) is violated; the estimated incidences are correlated conditional on the true incidences. Performance is similar to that under the conditional independence model equation (4) (see Supplemental Materials). Furthermore, we assess performance with external cohort data analyzed at the sub-cohort level, reflecting that site-level data may be available for multi-center studies (Supplemental Table S7). Given a fixed total sample size across external cohorts, more cohorts of smaller sizes are preferred to fewer cohorts of larger size.

Application

We apply the estimation to the DISCOVER trial, a randomized, double-blinded, double-dummy, active-controlled trial that compared the efﬁcacy of coformulated tenofovir alafenamide plus emtricitabine and tenofovir disoproxil fumarate plus emtricitabine for preventing HIV in men and transgender women who have sex with men.⁶ The US FDA approved tenofovir alafenamide plus emtricitabine for men and transgender women who have sex with men based on the trial results.³⁵ Rectal gonorrhea infections were captured in both arms.⁵ Collectively, 1313 rectal gonorrhea cases were observed over 6243 person-years, implying a rectal gonorrhea incidence of 21.0 cases per 100 person-years. Historical data suggest that oral anti-retrovirals do not have biological effects on rectal gonorrhea incidence.³⁶

Table 3 contains point estimates and 95% CIs for counterfactual placebo HIV incidence using likelihood-based and working model estimation, assuming log and logit link functions in the bivariate linkage model equation (2), based on previously reported cohorts reporting both HIV and rectal gonorrhea incidence for men who have sex with men²⁴ (see Supplemental Table S1). The estimated counterfactual placebo HIV incidences are approximately 7 cases per 100 person-years for both estimation approaches and link functions. A naive analysis that assumes an identity link and treats estimated HIV and rectal gonorrhea incidence rates as ﬁxed and known, similar to what is done in the applied literature, gives a lower counterfactual HIV incidence estimate of 6.6 cases per 100 person-years.

Table 3.

Estimated counterfactual placebo HIV incidence (cases per 100 person-years), and corresponding 95% confidence intervals (CIs) for the DISCOVER study. Uncertainty is quantified by 95% confidence intervals except for Bayesian estimates where ^* 95% and ⁺ 80% credible intervals (CrIs) are reported.

Link function	Estimation approach	Est.	95% CI (CrI ^*+)
Log	Likelihood-based	7.10	(5.02, 10.03)
Log	Working model	7.06	(5.25, 9.49)
Logit	Likelihood-based	6.94	(4.82, 9.88)
Logit	Working model	6.87	(5.08, 9.23)
Identity	Naive approach²⁴	6.60	n.a.
	Bayesian Gamma-Copula¹⁵	4.51	(2.06, 7.36)^*
	Bayesian case-cohort sampling³⁷	3.4	(1.9, 5.9)⁺

We compare the results with those from Glidden et al.,^15,37 who applied Bayesian approaches with Gamma-Copula models and case-cohort sampling adjustment to the DISCOVER study. Posterior estimates of counterfactual placebo HIV incidence from the two Bayesian approaches are much lower at 4.51 and 3.4 cases per 100 person-years. We conjecture that the lower estimates are due in part to the chosen prior HIV incidence rate (mean of 2.9 cases per 100 person-years) in Glidden et al.,¹⁵ which was lower than the average incidence across the external cohort studies.

Another difference in the latter estimate is its reliance on an additional data source, namely historical estimates of efﬁcacy.

Given the estimated 0.16 HIV diagnosis cases per 100 person-years from the tenofovir alafenamide plus emtricitabine arm,⁶ the estimated prevention efﬁcacy for tenofovir alafenamide plus emtricitabine versus counterfactual placebo is 98.1% (95% CI: 96.4%–99.4%), based on the working model and log link, and 98.1% with the likelihood-based estimation (95% CI: 96.4%–99.7%). This prevention efﬁcacy inference is simple to interpret and supports tenofovir alafenamide plus emtricitabine effectiveness.

Conclusion

Advancing HIV prevention, and ultimately stemming the HIV pandemic, requires additional biomedical interventions. While active-controlled trials will likely be used in future trials evaluating candidate interventions, absolute efﬁcacy of the experimental intervention cannot be evaluated based on the trial data alone. If a marker of HIV exposure is measured in the trial, and external data are leveraged to model the association between HIV and the exposure marker, under Assumptions 1–3 HIV incidence in a counterfactual placebo arm, and prevention efﬁcacy of the experimental intervention relative to the counterfactual placebo, can be estimated reliably and precisely.

Importantly, we considered performance of the approach when Assumptions 1 and 3 hold, and Assumption 2 either holds or is slightly violated. These are strong and not fully testable assumptions that deserve careful attention. For one, correct speciﬁcation of the model linking HIV incidence with the exposure marker is challenging. Mis-speciﬁcation may be due to omission of covariates that modify the association, incorrect model form, or measurement error of variables. Recent work demonstrates that the rectal gonorrhea and HIV incidence association may differ across populations³⁸ and is difﬁcult to model accurately across cohorts.^14,28–30

While standard statistical methods may check for speciﬁc types of model mis-speciﬁcation, with few external cohorts the power to detect model mis-speciﬁcation is low. Given some of the assumptions are not fully testable, further research is needed into methods for incorporating uncertainty due to violation of these assumptions.

Our ﬁndings suggest that more cohorts of smaller size provide more precise inference than fewer cohorts of larger size. Accuracy and precision may be further improved with individual-level data. As well, with only study-level data from external cohorts the correlation between reported HIV and the exposure marker incidences in the external cohorts is rarely available. Accordingly, our estimation approaches assume conditional independence of HIV incidence and the exposure marker. Our simulation study suggests a degree of robustness to violation of this assumption, mainly because between-study variation dominated with-in study variation. Similar results were found for bivariate meta-analysis.³⁹ However, as discussed by Riley,³⁹ ignoring within-study correlation is expected to yield estimates with inferior statistical properties. Given individual-level data from external cohorts, estimation of the conditional dependence parameter would be feasible and performance improved.

We call for additional research, with application held until such research is conducted. Evaluation of the approach’s performance in HIV prevention trials that included placebo arms is needed to gauge the “real-world” accuracy of the counterfactual placebo estimation. Individual-or trial-sitelevel data from recent HIV prevention trials, with incidence of other sexually transmitted infections captured, should be made public to enable further evaluation of the correlation between HIV and other sexually transmitted infections as potential exposure markers. Finally, HIV exposure markers that more readily satisfy the assumptions we detail should be pursued; markers more fundamentally linked to HIV exposure may be needed to realize the potential.

We did not ﬁnd existing statistical frameworks that provided a good ﬁt for our problem.^40–42 The exposure marker we considered is different from a surrogate marker for which the effect of the intervention on the surrogate reﬂects the effect of the intervention on the primary endpoint.^40,43 The framework we developed may have application to other clinical contexts where a proxy outcome is associated with the clinical outcome under the control condition but is not impacted by the intervention, and a body of data is available for estimating the association between proxy and clinical outcome under the control condition.

Supplemental Material

sj-pdf-1-ctj-10.1177_17407745231203327 – Supplemental material for Estimating counterfactual placebo HIV incidence in HIV prevention trials without placebo arms based on markers of HIV exposure

Supplemental material, sj-pdf-1-ctj-10.1177_17407745231203327 for Estimating counterfactual placebo HIV incidence in HIV prevention trials without placebo arms based on markers of HIV exposure by Yifan Zhu, Fei Gao, David V Glidden, Deborah Donnell and Holly Janes in Clinical Trials

Footnotes

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: D.V.G. has accepted fees from Gilead Sciences. The remaining authors had nothing to disclose.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Institutes of Health/National Institute of Allergy and Infectious Diseases (NIH/NIAID) through grants R01CA152089, R56AI143418, and UM1AI068635 to H.J., and R01AI143357 to D.V.G.

ORCID iD

Fei Gao

Supplemental material

Supplemental material for this article is available online.

References

Saag

Gandhi

Hoy

, et al. Antiretroviral drugs for treatment and prevention of HIV infection in adults. JAMA 2020; 324: 1651.

Chou

Evans

Hoverman

, et al. Preexposure prophylaxis for the prevention of HIV infection. JAMA 2019; 321: 2214.

Baeten

Palanee-Phillips

Brown

, et al. Use of a vaginal ring containing dapivirine for HIV-1 prevention in women. New Engl J Med 2016; 375: 2121–2132.

Nel

Niekerk

van Kapiga

, et al. Safety and efficacy of a dapivirine vaginal ring for HIV prevention in women. New Engl J Med 2016; 375: 2133–2143.

Hare

Coll

Ruane

, et al. The phase 3 DISCOVER study: daily F/TAF or F/TDF for HIV preexposure prophylaxis, https://www.croiconference.org/abstract/phase-3-discover-study-daily-ftaf-or-ftdf-hiv-preexposure-prophylaxis/

Mayer

Molina

Thompson

, et al. Emtricitabine and tenofovir alafenamide vs emtricitabine and tenofovir disoproxil fumarate for HIV pre-exposure prophylaxis (DISCOVER): primary results from a randomised, double-blind, multicentre, active-controlled, phase 3, non-inferiority trial. Lancet 2020; 396: 239–254.

Landovitz

Donnell

Clement

, et al. Cabotegravir for HIV prevention in cisgender men and transgender women. New Engl J Med 2021; 385: 595–608.

Delany-Moretlwe

Hughes

Bock

, et al. Long acting injectable cabotegravir is safe and effective in preventing HIV infection in cisgender women: interim results from HPTN 084. J Int AIDS Soc 2021; 24(Suppl. 1): 8.

The Joint United Nations Programme on HIV/AIDS. Global HIV and AIDS statistics—2021 fact sheet, http://www.unaids.org/en/resources/fact-sheet

10.

Sugarman

. Bioethical challenges with HIV treatment as prevention. Clin Infect Dis 2014; 59(Suppl. 1): S32–S34.

11.

Knight

Small

Thomson

, et al. Implementation challenges and opportunities for HIV treatment as prevention (TasP) among young men in Vancouver, Canada: a qualitative study. BMC Public Health 2016; 15: 262.

12.

AIDS Vaccine Advocacy Coalition. The future of ARV-based prevention and more, https://www.avac.org/infographic/future-arv-basedprevention

13.

World Health Organization. Ethical considerations in HIV prevention trials, https://www.unaids.org/sites/default/files/media_asset/ethical-considerations-hiv-prevention-trials_en.pdf

14.

Glidden

. Advancing novel PrEP products—alternatives to non-inferiority. Stat Commun Infect Dis 2019; 11(1): 20190011.

15.

Glidden

Stirrup

Dunn

. A Bayesian averted infection framework for PrEP trials with low numbers of HIV infections: application to the results of the DISCOVER trial. Lancet HIV 2020; 7(11): e791–e796.

16.

Hung

HMJ

Wang

Tsong

, et al. Some fundamental issues with non-inferiority testing in active controlled trials. Stat Med 2003; 22: 213–225.

17.

Fleming

. Current issues in non-inferiority trials. Stat Med 2008; 27: 317–332.

18.

Grobler

Abdool Karim

. Design challenges facing clinical trials of the effectiveness of new HIV-prevention technologies. AIDS 2012; 26: 529–532.

19.

Cutrell

Donnell

Dunn

, et al. HIV prevention trial design in an era of effective pre-exposure prophylaxis. HIV Clin Trials 2017; 18(5–6): 177–188.

20.

Hanscom

Hughes

Williamson

, et al. Adaptive non-inferiority margins under observable non-constancy. Stat Meth Med Res 2019; 28(10–11): 3318–3332.

21.

Corey

Gilbert

Juraska

, et al. Two randomized trials of neutralizing antibodies to prevent HIV-1 acquisition. New Engl J Med 2021; 384: 1003–1014.

22.

Corey

Gilbert

Tomaras

, et al. Immune correlates of vaccine protection against HIV-1 acquisition. Sci Transl Med 2015; 7: 310rv7.

23.

Gilbert

Huang

deCamp

, et al. Neutralization titer biomarker for antibody-mediated prevention of HIV-1 acquisition. Nat Med 2022; 28(9): 1924–1932.

24.

Mullick

Murray

. Correlations between HIV infection and rectal gonorrhea incidence in men who have sex with men: implications for future HIV pre-exposure prophylaxis trials. J Infect Dis 2020; 221: 214–217.

25.

Murray

. Regulatory perspectives for streamlining HIV prevention trials. Stat Commun Infect Dis 2019; 11(1): 12.

26.

Janes

Donnell

Nason

. Designing the next generation of HIV prevention efﬁcacy trials: synopsis of a 2018 symposium. Stat Commun Infect Dis 2019; 11: 4.

27.

Follmann

. Tomorrow’s HIV prevention trials of vaccines and antibodies. Stat Commun Infect Dis 2019; 11: 7.

28.

Cohen

Donnell

. Novel approaches for development of human immunodeﬁciency virus preexposure prophylaxis agents. J Infect Dis 2019; 221: 172–174.

29.

Glidden

. Statistical approaches to accelerate the development of long-acting antiretrovirals for HIV pre-exposure prophylaxis. Curr Opin HIV AIDS 2020; 15(1): 56–60.

30.

Public meeting webinar 3: protocol design considerations: analyses for efﬁcacy: in-depth webinar report, https://forumresearch.org/storage/documents/HIV_Forum/Public_Meeting_Webinar_3_Report_Final_211027.pdf

31.

U.S. Food and Drug Administration. Final summary minutes of the antimicrobial drugs advisory committee meeting, https://www.fda.gov/media/131002/download

32.

Traeger

Cornelisse

Asselin

, et al. Association of HIV preexposure prophylaxis with incidence of sexually transmitted infections among individuals at high risk of HIV infection. JAMA 2019; 321: 1380.

33.

Van Houwelingen

Arends

Stijnen

. Advanced methods in meta-analysis: multivariate approach and meta-regression. Stat Med 2002; 21: 589–624.

34.

Reitsma

Glas

Rutjes

, et al. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol 2005; 58: 982–990.

35.

Voelker

. PrEP drug is approved for some patients but not for others. JAMA 2019; 322: 1644.

36.

Grant

Lama

Anderson

, et al. Preexposure chemoprophylaxis for HIV prevention in men who have sex with men. N Engl J Med 2010; 363: 2587–2599.

37.

Glidden

Das

Dunn

, et al. Using the adherence-efﬁcacy relationship of emtricitabine and tenofovir disoproxil fumarate to calculate background hiv incidence: a secondary analysis of a randomized, controlled trial. J Int AIDS Soc 2021; 24: e25744.

38.

Donnell

Zewdie

Ratna

, et al. Association between rectal gonorrhoea and HIV incidence in men who have sex with men: a meta-analysis. Sex Transm Infect 2022; 98(7): 492–496.

39.

Riley

. Multivariate meta-analysis: the effect of ignoring within-study correlation. J Roy Stat Soc Ser A Stat Soc 2009; 172: 789–811.

40.

Fleming

Powers

. Biomarkers and surrogate endpoints in clinical trials. Stat Med 2012; 31: 2973–2984.

41.

Tchetgen

ETT

Ying

Cui

, et al. An introduction to proximal causal learning, https://arxiv.org/pdf/2009.10982.pdf

42.

Arnold

Ercumen

. Negative control outcomes. JAMA 2016; 316: 2597.

43.

Temple

. A regulatory authority’s opinion about surrogate endpoints. Clin Meas Drug Eval 1995; 1: 3–22.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.25 MB