Comparison between treatment-placebo difference and saved time measures to assess treatment effect in Alzheimer's disease

Abstract

Background

The estimated treatment-placebo difference from fitted statistical models is often used to evaluate the treatment effect. In addition, we may use time saving as an alternative measure to assess the treatment benefit to patients. Two methods were developed to estimate saved time based on the placebo or treatment disease progression curve.

Objective

In this article, we are interested in comparing the performance of the traditional model and saved time measures with regards to the statistical hypothesis testing.

Methods

By mimicking data from the Phase 2 and 3 donanemab trials, we studied three different treatment disease progression trajectories.

Results

The traditional treatment-placebo difference model often has higher statistical power values than the saved time methods when sample size is small to medium. As sample size goes up, saved time methods have similar statistical power and they could be more powerful than the traditional model in some cases.

Conclusions

Saved time methods may be considered in future Alzheimer's disease trials to improve the efficiency in drug discovery.

Keywords

Alzheimer's disease effect size hypothesis testing saved time treatment-placebo difference

Introduction

To demonstrate the effectiveness of a new treatment as compared to the standard intervention, the effect size of the treatment has to be estimated in a clinical trial. In Alzheimer's disease trials with disease-modifying therapy (DMT) agents, the effect size can be calculated from the expected reduced disease progression in the treatment group (e.g., 25% reduction in the primary outcome) and the standard deviation (SD) of the outcome.^1–3 In DMT studies, the disease progression in the treatment group is frequently assumed to be proportional to that in the placebo group. Then, their difference increases as time goes on if that assumption is satisfied. In another type of AD treatment, symptomatic drug, the treatment-placebo difference is assumed to be a constant over time, in which the effect size is relatively stable when the SD of the outcome is independent of time.

In addition to the treatment-placebo difference, one may use the saved time measure to assess the treatment benefit. Saved time estimates can be computed by using the placebo disease progression (PDP) curve and the expected outcome of the treatment group at the last visit, or the treatment disease progression (TDP) curve and the treatment-placebo difference at the last visit. We illustrate the saved time calculation based on the PDP method and the TDP method in Figure 1, as the length of the dotted lines measured in time by months. The PDP method has been applied to several completed AD trials, such as the donanemab trial,³ and the IMM-AD04 Phase 2 study.⁴ The TDP method was recently developed to better utilize the treatment disease progression curve as saved time has the focus on the time saving for patients treated with a new drug. The PDP method and the TDP method were previously known as the backward projection to placebo (BPP) method and the backward projection to treatment (BPT) method, respectively.⁵ We chose to use the new names to emphasize the disease progression curve that is used in saved time calculation. In general, saved time provides an intuitive interpretation of the drug effectiveness in time which is easy to communicate with patients, family members, and their caregivers.^6–9

Figure 1.

Illustration of saved time calculation based on the PDP method (length of the horizontally dotted purple line) and the TDP method (length of the horizontally dotted orange line).

When we have multiple test statistics for comparing treatment and placebo, it is important to study which test statistic is more powerful than others. We do not expect that one test statistic is uniformly more powerful than the remaining test statistics in this setting. The statistical power is more likely to depend on the trajectory of disease progression. For that reason, we compared the statistical power of the studied statistics for several possible disease progression curves including the aforementioned proportional treatment effect in DMT trials. The second topic we would like to explore is the expected treatment effect size. The reduced declining rate in the treatment group could be slower for a study with a large effect size than that in a study with a small effect size. The findings from this research could provide meaningful guidance on the test statistic for comparing treatment-placebo difference.

Methods

In AD trials, the visits are often pre-specified. In the aforementioned lecanemab phase 3 trial, patients have follow-up visits every 3 months from baseline to 18 months with a total of 6 follow-up visits. Suppose $T_{j}$ is the time for the j-th visit, where $j = 0, 1, \cdot \cdot \cdot, J$ . To investigate the treatment effect as compared to the control, one needs to estimate the treatment-placebo difference at the pre-specified follow-up visit (e.g., $T_{J}$ = 18 months) and standard deviation ( $δ$ ) or standard error (SE) of the outcome change from baseline to the follow-up visit. For example, in the lecanemab phase 3 trial,¹ the primary outcome measure was the change of the Clinical Dementia Rating Scale Sum of Boxes (CDR-SB) at 18 months from baseline. Its standard deviation was $δ$ = 2.031 points from the previous phase 2 trial,¹⁰ and the treatment-placebo difference was expected to be 0.373 points at 18 months. Then, the expected treatment effect size is estimated as d = 0.373/2.031 = 0.184.

In addition to the traditional treatment-placebo comparison based on the two-sample t-test or the linear mixed- effect model (e.g., ${\hat{Δ}}_{D i f f}$ as the estimated mean difference), we consider using the recently introduced concept of saved time to estimate the treatment effectiveness. Two time saving estimates were proposed in the literature by using the PDP curve or the TDP curve. In the PDP method,¹¹ saved time is estimated as the time difference between the last visit and the time on the PDP with the same estimated change from baseline in clinical outcome. Recently, Wang et al.⁵ proposed to include the treatment-placebo difference at the last visit in estimating saved time by using the TDP curve.

Suppose $μ_{j}$ and $λ_{j}$ are the mean of the outcome change at visit j from baseline in the placebo group and the treatment group, where $j = 1, 2, \cdot \cdot \cdot, J$ . In practice, we often don’t have observations between visits, and we use the linear interpolation method to connect the mean values with $β_{j}$ as the slope for the outcome change between $T_{j - 1}$ and $T_{j} : β_{j} = \frac{μ_{j} - μ_{j - 1}}{T_{j} - T_{j - 1}}$ . Then, we have the PDP curve for the outcome change:

F (t) = μ_{j - 1} + β_{j} (t - T_{j - 1}), t \in (T_{j - 1}, T_{j}] .

In computing the PDP saved time, we first calculate the time on the PDP curve whose outcome change equals to the outcome change at the last visit from baseline in the treatment group: $F (τ) = λ_{T_{J}}$ . The value of $\hat{τ}$ can be obtained by solving the equation $\hat{F} (τ) = T_{T_{J}}$ by plugging in the quantities with the sample values. Thus, the PDP saved time is ${\hat{Δ}}_{P D P} = T_{J} - \hat{τ}$ .

In the TDP method, the TDP curve for the outcome change can be expressed as:

G (t) = λ_{j - 1} + ω_{j} (t - T_{j - 1}), t \in (T_{j - 1}, T_{j}],

where

ω_{j}

is the slope in the treatment group when

t \in (T_{j - 1}, T_{j}]

. The TDP saved time is

{\hat{Δ}}_{T D P} = T_{J} - \hat{v}

, where

\hat{v}

is the solution of the equation

\hat{G} (v) = 2 λ_{T_{J}} - μ_{T_{J}}

In this article, we studied three treatment disease progression trajectories

Constant treatment effect: $F (t) - G (t) = c \times F (T_{J})$ , where c is a constant value,

Proportional treatment effect: $G (t) / F (t) = r$ , where r is between 0 and 1,

Constant delayed treatment effect: $F (t) = G (t + t_{d})$ where $t_{d}$ is the value of months.

In the next section, we compare the performance of these three measures: ${\hat{Δ}}_{D i f f}$ from the traditional treatment- placebo difference model, the PDP saved time ${\hat{Δ}}_{P D P}$ , and the TDP saved time ${\hat{Δ}}_{T D P}$ .

Results

We first used the published data from the Phase 3 donanemab trial in the simulation studies to compare the test statistics for investigating the treatment-placebo difference. Patients in that trial were scheduled to have follow-up visits every 3 months with a total of 6 follow-up visits. The change of integrated Alzheimer Disease Rating Scale (iADRS) score at 76 weeks from baseline was the primary outcome measure. From their reported summary data, the mean changes at each follow-up visit from baseline in the placebo group were: (−1.33, −2.75, −4.89, −7.76, −9.90, −13.22). For the mean changes in the treatment group, we studied three treatment disease progression trajectories as discussed in Methods section. The first pattern is often assumed for symptomatic agents, and the second pattern is the trajectory for DMT treatments. The variance of the outcome change at each visit for the placebo group was obtained from the published article.³ The first-order autoregressive, AR(1), correction structure is assumed with the correlation coefficient of 0.5. In the data simulation, we use the same variance-covariance matrix for both groups, and we use the follow-up visits at 3, 6, 9, 12, 15, 18 months.

We may approximately calculate the SE of the treatment-placebo difference from the SD of the outcome at the last visit. However, it is not straightforward to compute SD of saved time estimates as the disease progression curves were used in computing the saved time estimates instead of only the mean values in the treatment-placebo difference. For that reason, we use simulation based approach to obtain the SE of the treatment-placebo difference and the SE of saved time. For each simulated data, we calculate the mean outcome change at each visit and the estimated variance-covariance matrix for each group. Then, this information are used to simulate 10,000 data sets to obtain the SE of each method: $\hat{S E_{1}}$ for the SE of the treatment-placebo difference, and $\hat{S E_{2}}$ and $\hat{S E_{3}}$ for the SE values of ${\hat{Δ}}_{P D P}$ and ${\hat{Δ}}_{T D P}$ . Together with the estimated treatment-placebo difference and saved times, we calculate three wald-type test statistics: $T_{D i f f} = \frac{{\hat{Δ}}_{D i f f}}{\hat{S E_{1}}}, T_{P D P} = \frac{{\hat{Δ}}_{P D P}}{\hat{S E_{2}}}$ , and $T_{T D P} = \frac{{\hat{Δ}}_{T D P}}{\hat{S E_{3}}}$ .

The wald-type test statistics may not have satisfactory performance with regards to type I error (TIE) when the asymptotic distribution is used. Instead, we use simulation studies to obtain the 95% threshold of each test statistic to have a fair comparison between these methods. In the threshold value calculation, we simulate 10,000 data sets by using the placebo group data from the Phase 3 donanemab trial as mentioned above. By using the information from each simulated data set, we simulate another 10,000 data to estimate the SE values. From these simulations, we compute the 95% upper quartile of the 10,000 test statistics for each method. We determine the threshold values for each possible sample size (e.g., 50 to 800 per arm). For statistical power values under each scenario (e.g., sample size, disease progression trajectories), we simulate 2000 data sets. For each data set, its SE is calculated from 10,000 simulations. The threshold value for each method is used to compute the statistical power.

In Figure 2, we presented the statistical power of the three methods when a new drug has a constant treatment effect. The treatment-placebo difference method often has higher statistical power values than the saved time methods. The average statistical power of these cases for the treatment-placebo difference method is 47.6%, which is 3.32% higher than the average statistical power of the PDP method. These two methods have similar statistical power when the sample size is large (e.g., 500 per group). The treatment-placebo difference method could have more than 7% statistical power as compared to the PDP method when sample size is between 100 and 200 per group. The TDP method is often not as powerful as the other two methods. Similar results are observed when a new drug has a proportional treatment effect and a constant delayed treatment effect in Figure 2.

Figure 2.

Statistical power comparison between the PDP method, the TDP method, and the treatment-placebo difference method, as a function of sample size per group, by mimicking phase 3 donanemab trial data. Top: a new drug has a constant treatment effect: c = 0.18 (solid lines) and c = 0.10 (dashed lines). Middle: a new drug has a proportional treatment effect: r = 0.71 (solid lines) and r = 0.83 (dashed lines). Bottom: a new drug has a constant delayed treatment effect: t_d = 2.70 (solid lines) and t_d = 1.50 (dashed lines).

Missing data from patients may affect these models. In the phase 3 donanemab trial, the missing rate was close to 20% for the primary outcome. The missing rate in the phase 3 lecanemab trial was slightly lower, closing to 15% for the primary outcome.¹ Based on these two missing rates, we simulated data based on the phase 3 donanemab trial with the missing rates at the 6 follow-up visits: 2%, 4%, 4%, 6%, 3%, and 1% from month 3 to month 18. In this simulation, we assumed missing completely at random, see Figure 3 for a study with sample size of 200 per arm. The findings are largely similar to the ones observed in Figure 2 with complete data.

Figure 3.

Assuming missing completely at random, statistical power comparison between the PDP method, the TDP method, and the treatment-placebo difference method, as a function of sample size per group by mimicking phase 3 donanemab trial data, when a new drug has a new drug has a proportional treatment effect with r = 0.83 (top) and a constant delayed treatment effect with t_d = 1.50 (bottom).

In the phase 3 donanemab trial, the largest decline of the primary outcome occurred during the last three months (month 15 to month 18), while the phase 2 donanemab trial had the largest decline between 9 months and 12 months which is almost double the decline during the last three months. By using the simulated threshold value based on the phase 2 placebo group data, we present the statistical power between the three methods in Figure 4 when a new drug has the proportional treatment effect (top) and the constant delayed treatment effect (bottom). When sample size is small (e.g., 100 per group), the PDP method has a higher statistical power than the TDP method. That trend is reversed as sample size goes up. The treatment-placebo difference method is more powerful than the TDP method for the proportional treatment effect cases. But, that trend is reversed for a study with the constant delayed treatment effect and medium to large sample sizes (e.g., 200 per group or more).

Figure 4.

Statistical power comparison between the PDP method, the TDP method, and the treatment-placebo difference method, as a function of sample size by mimicking phase 2 donanemab trial data. Top: a new drug has a proportional treatment effect: r = 0.71 (solid lines) and r = 0.83 (dashed lines). Bottom: a new drug has a constant delayed treatment effect: t_d = 2.70 (solid lines) and t_d = 1.50 (dashed lines).

Discussion

In this article, we studied one important statistical question on which test statistic is associated with a higher statistical power after controlling for the TIE rate. We investigated the case with the follow-up time of 18 months which are common in DMT trials. It should be noted that the treatment effect changes over time, starting with a small treatment effect early in a trial which may be due to the placebo effect. In the last few visits, the treatment effect may be affected by other factors, such as drop out, treatment side effects. In a typical phase 2 DMT study, the follow-up time is often shorter than 18 months.^2,12,13 It is still an open question in the AD research community on how to compare saved time from studies with different follow-up durations.

From the simulation studies, the SE values of the treatment-placebo difference and the two saved time measures can be computed. In a simple model (e.g., one-sample t-test), the SD of the parameter of interest can be directly calculated from the SE value and the sample size. Meanwhile, it is a challenge to calculate the SD of saved time from the SE value and other information due to the fact that disease progression curve is used in estimating saved time.¹⁴ This becomes a methodology project to derive the SD of saved time under some mild assumptions, such as the interpolation method to connect the mean values.

We primarily used data from the phase 3 donanemab trial in simulation studies. In addition, we compare the performance of these measures by using data from the phase 2 donanemab trial.² As sample size in phase 2 is relatively small, the placebo disease progression curve is not as smooth as the one from the phase 3 data, especially their difference in the time window having the largest primary outcome decline. The phase 2 had the largest decline close to the middle of the study, while the phase 3 trial had the largest decline at the end of the study. When the largest decline occurs earlier, the decline rate in the treatment group could be faster than that in the placebo group during the last few visits. That could be the reason for the increased statistical power by using the TDP method.

In this article, we investigated the effect of missing completely at random on the three methods to compare the treatment-placebo difference. In addition to that, one may consider other missing data mechanisms, such as missing at random or missing not at random (e.g., missing due to amyloid-related imaging abnormalities (ARIA)). For a study with missing at random data, a longitudinal mixed-effect model and other computationally intensive models may be used to fit the data.¹⁵ It could be very computationally intensive when a mixed-effect model is used to fit each simulated data. We would consider this future work after we can address the computational challenge.^16–18

Footnotes

ORCID iDs

Guogen Shan

Yahui Zhang

Author contributions

Guogen Shan (Conceptualization; Funding acquisition; Investigation; Methodology; Validation; Writing – original draft; Writing – review & editing); Yahui Zhang (Formal analysis; Methodology; Software; Writing – original draft; Writing – review & editing); Guoqiao Wang (Conceptualization; Formal analysis; Methodology; Writing – original draft; Writing – review & editing).

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Shan’s research is partially supported by grants from the National Institutes of Health: R03AG083207, and R01AG070849.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data availability statement

The dataset used in this study was obtained from published articles.

References

van Dyck

Swanson

Aisen

, et al. Lecanemab in early Alzheimer’s disease. N Engl J Med 2023; 388: 9–21.

Mintun

Duggan Evans

, et al. Donanemab in early Alzheimer’s disease. N Engl J Med 2021; 384: 1691–1704.

Sims

Zimmer

Evans

, et al. Donanemab in early symptomatic Alzheimer disease. JAMA 2023; 330: 512.

Dickson

Haaland

Mallinckrodt

, et al. “Time saved” calculations to improve decision-making in progressive disease studies. J Prev Alzheimer’s Dis 2024; 11: 529–536.

Wang

Cutter

Oxtoby

, et al. Statistical considerations when estimating time-saving treatment effects in Alzheimer’s disease clinical trials. Alzheimer’s Dement 2024; 20: 5421–5433.

Raket

. Progression models for repeated measures: estimating novel treatment effects in progressive diseases. Stat Med 2022; 41: 5537–5557.

Stijven

Mallinckrodt

Molenberghs

, et al. Meta TCT: Towards In- terpretable Treatment Effects in Clinical Trials for Progressive Diseases. medRxiv 2024. 2024–2025.

J¨onsson

Ivkovic

Atri

, et al. Progression analysis versus traditional methods to quantify slowing of disease progression in Alzheimer’s disease. Alzheimer’s Res Ther 2024; 16: 1–11.

Dickson

Solomon

Kivipelto

, et al. Evaluation of clinical meaningfulness of fortasyn connect in terms of “time saved”. J Prev Alzheimer’s Dis 2024; 11: 992–997.

10.

Swanson

Zhang

Dhadda

, et al. A randomized, double-blind, phase 2b proof-of-concept clinical trial in early Alzheimer’s disease with lecanemab, an anti-Aβ protofibril antibody. Alzheimer’s Res Ther 2021; 13: 1–14.

11.

Dickson

Wessels

Dowsett

, et al. ‘Time saved’ as a demonstration of clinical meaningfulness and illustrated using the donanemab TRAILBLAZER-ALZ study findings. J Prev Alzheimer’s Dis 2023; 10: 595–599.

12.

Cummings

Zhou

Lee

, et al. Alzheimer’s disease drug development pipeline: 2023. Alzheimer’s Dement: Transl Res Clin Interv 2023; 9: e12385.

13.

Shan

Banks

Miller

, et al. Statistical advances in clinical trials and clinical research. Alzheimer’s Dement: Transl Res Clin Interv 2018; 4: 366–371.

14.

Zhang

Song

, et al. Predicting conversion time from mild cognitive impairment to dementia with interval-censored models. J Alzheimer’s Dis 2024; 101: 147–157.

15.

Coley

Gardette

Cantet

, et al.

How should we deal with missing data in clinical trials involving alzheimers disease patients?

Curr Alzheimer Res 2011; 8: 421–433.

16.

Shan

Zhang

, et al. Blood biomarkers as surrogate endpoints in Alzheimer’s disease research. Front Aging Neurosci 2025; 17: 1576373.

17.

Shan

Zhang

Tang

, et al. Disease progression trajectory curves to estimate saved time in Alzheimer’s disease trials. Contemp Clin Trials 2025; 151: 107814.

18.

Zhang

Tang

, et al. Conversion to Alzheimer’s disease dementia from normal cognition directly or with the intermediate mild cognitive impairment stage. Alzheimer’s Dement 2025; 21: e14393.