Sage Journals: Discover world-class research

Abstract

Blinded sample size reassessment is a popular means to control the power in clinical trials if no reliable information on nuisance parameters is available in the planning phase. We investigate how sample size reassessment based on blinded interim data affects the properties of point estimates and confidence intervals for parallel group superiority trials comparing the means of a normal endpoint. We evaluate the properties of two standard reassessment rules that are based on the sample size formula of the z-test, derive the worst case reassessment rule that maximizes the absolute mean bias and obtain an upper bound for the mean bias of the treatment effect estimate.

Keywords

Adaptive design interim analysis internal pilot study point estimate sample size reassessment

1 Introduction

In clinical trials for the comparison of means of normally distributed observations, the sample size to achieve a specific target power depends on the true effect size and variance. For the purpose of sample size planning, the effect size is usually assumed to be equal to a minimally clinically relevant effect while the variance is often estimated from historical data. For situations where only little prior knowledge on the variance is available, clinical trial designs with sample size reassessment based on interim estimates of this nuisance parameter have been proposed. Stein¹ developed a two-stage procedure, where the second stage sample size is calculated based on a first stage variance estimate aiming to achieve a pre-specified target power. In the context of clinical trials, several extensions of Steins two-stage procedure have been considered^2–8 that all require an unblinding of the interim data for the computation of the variance estimate. However, regulatory agencies generally prefer blinded interim analyses as they entail less potential for bias.^9–13

Gould and Shih¹⁴ proposed to estimate the variance from the blinded interim data by computing the variance from the total sample (pooling the observations from both groups), instead. Although this interim estimate is not consistent¹⁵ and has a positive bias if the alternative hypothesis holds, the bias is negligible for effect sizes typically observed in clinical trials.¹⁶ Furthermore, sample size reassessment based on the total variance has no relevant impact on the type I error rate in parallel group superiority trials¹⁷ (see also the literature^18–21) and achieves the target power well. Similar results on sample size reassessment based on blinded nuisance parameter estimates were obtained for binary data,²² count data^23–25 longitudinal data,²⁶ and for fully sequential sample size reassessment.²⁷ For non-inferiority trials with normal endpoints, a minor inflation of the type I error rate for small sample sizes has been observed.^28,29 However, if the sample size reassessment rule is not only based on the primary endpoint but also on blinded secondary or safety endpoint data, the type I error rate for the test of the primary endpoint may be substantially inflated.³⁰

While for adaptive clinical trials with sample size reassessment based on the unblinded interim treatment effect estimate, it is well known that unadjusted point estimates of the effect size and confidence intervals may be biased;^31,32 the properties of point estimates and confidence intervals computed at the end of an adaptive clinical trial with blinded sample size reassessment have received less attention. In this paper, we investigate the bias and standard error of the treatment effect point estimate and compare the absolute mean bias due to blinded sample size reassessment to upper boundaries derived for adaptive designs with unblinded sample size reassessment.³³ We also investigate the bias of the final variance estimate under sample size reassessment rules that are based on the blinded interim variance estimate. We also investigate the bias of the final variance estimate under sample size reassessment rules that are based on the blinded interim variance estimate. Previously, this bias had been investigated for a corresponding unblinded reassessment rule (which had no upper sample size bound), sharp bounds for the bias had been derived and an additive bias correction had been suggested.⁸ Here, in this paper, we derive corresponding bounds for the bias of the final variance estimate in the blinded case. Furthermore, we assess the coverage probability of one- and two-sided confidence intervals.

In Section 2, we introduce adaptive designs with blinded sample size reassessment. Theoretical results on the bias of estimates are presented in Section 3. In Section 4, we report a simulation study quantifying the bias and coverage probabilities in a variety of scenarios. The impact of the results is discussed in the context of a case study in Section 5. Section 6 concludes the paper with a discussion and recommendations. Technical proofs are given in Appendix 1.

2 Blinded sample size reassessment

Consider a parallel group comparison of the means $μ_{a}, μ_{b}$ of a normally distributed endpoint with common unknown variance $σ^{2}$ . The one-sided null hypothesis $H_{0} : δ \leq 0$ , is tested against $H_{1} : δ > 0$ at level α, where $δ = μ_{b} - μ_{a}$ denotes the true effect size. Let $δ_{0}$ denotes the alternative for which the trial is powered and assume that in the planning phase a first stage per group sample size $n_{1} \geq 2$ is chosen based on an a priori variance estimate $σ_{0}^{2}$ . Note, however, that all results below depend only on the chosen n₁ and not the way it has been determined. As a consequence, for given n₁, they also apply if it has been determined with other justifications. After the endpoints of the first stage subjects are observed, in a blinded interim analysis, the one-sample variance estimate is computed, given by

S_{1, OS}^{2} = \frac{1}{2 n_{1} - 1} [\sum_{i = a, b} \sum_{k = 1}^{n_{1}} (X_{i 1 k} - {\bar{X}}_{\cdot 1 \cdot})^{2}]

where X_ijk is the observation

k = 1, \dots, n_{j}

in stage j = 1, 2 for treatment

i = a, b

and

{\bar{X}}_{\cdot 1 \cdot}

is the mean of the pooled first stage samples. Based on

S_{1, OS}^{2}

, the second stage sample size is chosen with a pre-specified sample size function

n_{2} (S_{1, OS}^{2})

. Below we drop the argument of the function n₂ for convenience if the meaning is clear from the context. Then, the overall sample size is

n = n_{1} + n_{2}

per group. After the study is completed and unblinded, we assume that the point estimates for the mean difference and the variance

\bar{Δ} = {\bar{X}}_{b \cdot \cdot} - {\bar{X}}_{a \cdot \cdot} S^{2} = \frac{1}{2 n - 2} [\sum_{i = a, b} \sum_{j = 1}^{2} \sum_{k = 1}^{n_{j}} (X_{ijk} - {\bar{X}}_{i \cdot \cdot})^{2}]

are computed, where

{\bar{X}}_{i \cdot \cdot} = \sum_{j = 1}^{2} \sum_{k = 1}^{n_{j}} X_{ijk} / n, i = a, b

. Furthermore, (i) the standard fixed sample lower confidence bound corresponding to the one-sided null hypothesis H₀ given by

\bar{Δ} - t_{2 n - 2, 1 - α} S \sqrt{2 / n}

, where

t_{2 n - 2, 1 - α}

denotes the

1 - α

quantile of the central t-distribution with

2 n - 2

degrees of freedom, (ii) the upper confidence bound—corresponding to the complementary null hypotheses

H_{0}^{-} : δ \geq 0

—given by

\bar{Δ} + t_{2 n - 2, 1 - α} S \sqrt{2 / n}

and (iii) the two-sided confidence interval given by

\bar{Δ} \pm t_{2 n - 2, 1 - α / 2} S \sqrt{2 / n}

is determined.

As example consider a sample size reassessment rule aiming to control the power under a pre-specified absolute treatment effect. It is derived from the standard sample size formula for the comparison of means of normally distributed observations with a one-sided z-test using the variance estimate $S_{1, OS}^{2}$

n_{2}^{u} (S_{1, OS}^{2}) = min {n_{2 max}, max {n_{2 min}, 2 (z_{1 - α} + z_{1 - β})^{2} \frac{S_{1, OS}^{2}}{δ_{0}^{2}} - n_{1} + 1}}

(1) where

1 - β

is the desired power and

n_{2 min}

and

n_{2 max}

are pre-specified minimal and maximal second stage sample sizes with

0 \leq n_{2 min} < n_{2 max}

. The +1 in formula (1) was added for mathematical convenience to derive a lower bound for the variance estimate in Theorem 4 below. Given that we anyway have to round the sample size to an integer, this modification is of minor practical importance. While

S_{1, OS}^{2}

is an unbiased variance estimator for

σ^{2}

when the true effect size is zero, it is positively biased by

δ^{2} n_{1} / (4 n_{1} - 2)

for effect sizes

δ \neg = 0

.⁵ A sample size reassessment rule that is based on an adjusted variance estimate which is unbiased under the effect size δ₀ for which the trial is powered is obtained from equation (1) by replacing

S_{1, OS}^{2}

S_{1, OS}^{2} - δ_{0}^{2} n_{1} / (4 n_{1} - 2)

and is given by

n_{2}^{a} (S_{1, OS}^{2}) = min {n_{2 max}, max {n_{2 min}, 2 (z_{1 - α} + z_{1 - β})^{2} (\frac{S_{1, OS}^{2}}{δ_{0}^{2}} - \frac{n_{1}}{4 n_{1} - 2}) - n_{1} + 1}}

(2)

3 Theoretical results on the bias of the final mean and variance estimators

In this section, we consider adaptive two-stage designs with general sample size reassessment rules where the second stage sample size is given as some non-constant function $n_{2} : ℝ^{+} \to ℕ_{0}$ of the blinded interim variance estimate $S_{1, OS}^{2}$ . We derive the bias of the final effect size and variance estimators $\bar{Δ}$ and S².

Theorem 1

Consider an adaptive two-stage design with a general sample size reassessment rule $n_{2} (S_{1, OS}^{2})$ .

The bias $E (\bar{Δ} - δ)$ is given by

\begin{matrix} \int_{0}^{\infty} \int_{0}^{\infty} \frac{t (2 n_{1} - 2) (r (q, δ - t) - r (q, δ + t))}{σ^{2} [1 + r (q, δ + t)] [1 + r (q, δ - t)]} \times \\ ϕ_{0, \sqrt{2 σ^{2} / n_{1}}} (t) χ_{2 n_{1} - 2}^{2} (\frac{q (2 n_{1} - 2)}{σ^{2}}) d t d q \end{matrix}

where

r (x, y) = n_{2} (\frac{2 (n_{1} - 1)}{2 n_{1} - 1} x + \frac{n_{1}}{2 (2 n_{1} - 1)} y^{2}) / n_{1}

Under the null hypothesis δ = 0, the final mean estimator $\bar{Δ}$ is unbiased.

If $n_{2} (S_{1, OS}^{2})$ is increasing and $δ > 0$ ( $δ < 0$ ) the estimator $\bar{Δ}$ is negatively (positively) biased, respectively. Furthermore, the absolute bias is symmetric in δ around 0.

If the sample size reassessment rule is increasing and ${lim}_{S_{1, OS}^{2} \to \infty} n_{2} (S_{1, OS}^{2}) = \infty$ , the bias of $\bar{Δ}$ converges to 0 for $δ \to \pm \infty$ .

Note that the sample size functions $n_{2}^{u}, n_{2}^{a}$ are increasing such that the property (c) applies. If $n_{2 max} = \infty$ , $n_{2}^{u}, n_{2}^{a}$ tend to infinity for $S_{1, OS}^{2} \to \infty$ such that the property (d) applies.

In Theorem 2, we give an upper bound for the bias of the mean in adaptive two-stage designs with a general sample size reassessment rule $n_{2} (S_{1, OS}^{2})$ such that $n_{2 min} \leq n_{2} (S_{1, OS}^{2}) \leq n_{2 max}$ , for lower and upper bounds where $n_{2 min}, n_{2 max}$ , where $n_{2 max}$ may be infinite (corresponding to the case of unrestricted sample size reassessment). For general unblinded sample size reassessment rules that may depend on the fully unblinded interim data, the upper bound for the bias is³³

(\frac{1}{n_{1} + n_{2 min}} - \frac{1}{n_{1} + n_{2 max}}) 0.4 σ \sqrt{2 n_{1}}

(3) and is realized for the sample size reassessment rule that sets

n_{2} = n_{2 min}

{\bar{Δ}}_{1} \leq δ

and

n_{2} = n_{2 max}

otherwise. This upper bound obviously also applies for the case of blinded sample size reassessment based on the total variance estimate because the set of unblinded sample size reassessment rules for which n₂ may depend on the unblinded interim data in any way, includes all blinded sample size reassessment rules. This bound is not sharp if the sample size depends on

S_{1, OS}^{2}

, only. In the theorem below, we derive the sample size reassessment rule that maximizes the bias and compute the corresponding maximal bias which is a sharp upper bound for the bias that can occur under any sample reassessment based on sample size rules

n_{2} (S_{1, OS}^{2})

. We show that for increasing effect sizes, the bias approaches the bias of unblinded sample size reassessment.

Theorem 2

Consider an adaptive two-stage design with a general sample size reassessment rule $n_{2} (S_{1, OS}^{2})$ , such that $0 \leq n_{2 min} \leq n_{2} (S_{1, OS}^{2}) \leq n_{2 max}$ , where $n_{2 max}$ may be infinite. Then,

the expected first stage treatment effect estimate conditional on the variance estimate $S_{1, OS}^{2}$ is given by

\begin{matrix} E ({\bar{Δ}}_{1} | S_{1, OS}^{2}) = \int_{- b}^{b} x ϕ_{δ, \sqrt{2 σ^{2} / n_{1}}} (x) \\ \times χ_{2 n_{1} - 2}^{2} {[(2 n_{1} - 1) S_{1, OS}^{2} - x^{2} n_{1} / 2] / σ^{2}} dx / K \end{matrix}

where ${\bar{Δ}}_{1}$ denotes the unblinded first stage estimate of the mean treatment effect, and $b = \sqrt{2 S_{1, OS}^{2} (2 n_{1} - 1) / n_{1}}, K = \int_{- b}^{b} ϕ_{δ, \sqrt{2 σ^{2} / n_{1}}} (x) χ_{2 n_{1} - 2}^{2} {[(2 n_{1} - 1) S_{1, OS}^{2} - x^{2} n_{1} / 2] / σ^{2}} dx$

the bias of $\bar{Δ}$ is maximized for the sample size rule

\begin{matrix} n_{2} (S_{1, OS}^{2}) = {\begin{matrix} n_{2 min} & if E ({\bar{Δ}}_{1} | S_{1, OS}^{2}) > δ \\ n_{2 max} & otherwise \end{matrix} \end{matrix}

(4)

the maximum bias of $\bar{Δ}$ depends on the true effect size δ and is given by

(\frac{n_{1}}{n_{1} + n_{2 min}} - \frac{n_{1}}{n_{1} + n_{2 max}}) E (E ({\bar{Δ}}_{1} - δ | S_{1, OS}^{2}) 1_{{E ({\bar{Δ}}_{1} | S_{1, OS}^{2}) > δ}})

(5)

For $δ \to \infty$ and σ fixed, the maximum bias of $\bar{Δ}$ caused by blinded sample size reassessment rules based on $S_{1, OS}^{2}$ converges to the maximum bias caused by unblinded sample size reassessment rules given by equation (3).

Note that, by symmetry, the negative bias of $\bar{Δ}$ is maximized by the sample size rule equation (4) and is given by equation (5) with the inequality signs reversed. If $n_{2 min} = 0$ and $n_{2 max} = \infty$ , the first factor in equation (5) reduces to 1.

In Theorem 3, we derive the bias of the variance estimator computed at the end of the trial for general sample size reassessment rules $n_{2} (S_{1, OS}^{2})$ and show that it is symmetric in the true treatment effect δ and converges to 0 for increasing δ under suitable conditions (which are satisfied for $n_{2}^{u}, n_{2}^{a}$ if $n_{2 max} = \infty$ ).

Theorem 3

Consider an adaptive two-stage design with a general sample size reassessment rule $n_{2} (S_{1, OS}^{2})$ . Then,

the bias of the variance estimator S² is given by

\begin{matrix} \int_{0}^{\infty} \int_{- \infty}^{\infty} \frac{1}{(n_{1} - 1 + n_{1} r (q, t + δ))} ((n_{1} - 1) (q - σ^{2}) + \frac{t^{2} n_{1} / 2 - σ^{2}}{2 + 2 r (q, t + δ)}) \\ \times ϕ_{0, \sqrt{2 σ^{2} / n_{1}}} (t) χ_{2 n_{1} - 2}^{2} (q γ) γ d t d q \end{matrix}

where $γ = (2 n_{1} - 2) / σ^{2}$ and r is defined in Theorem 1;

the bias of S² is symmetric in δ around 0;

if ${lim}_{S_{1, OS}^{2} \to \infty} n_{2} (S_{1, OS}^{2}) = \infty$ , the bias of S² converges to 0 for $δ \to \pm \infty$ .

In Theorem 4, we derive a lower bound for the bias of the variance estimator S² (which is negative) in an adaptive two-stage design with sample size reassessment rule $n_{2}^{u}$ .

Theorem 4

Consider a design with blinded sample size reassessment based on $S_{1, OS}^{2}$ , where the second stage sample size is set to $n_{2}^{u} (S_{1, OS}^{2})$ . Under the null hypothesis δ = 0, the bias of the variance estimator S² at the end of the study has the following bounds

- \frac{2 n_{1} - 1}{2 n_{1} - 3} \cdot \frac{1}{v} < {ES}^{2} - σ^{2} < 0

where

v = 2 (z_{1 - α} + z_{1 - β})^{2} / δ_{0}^{2}

. If

n_{2 max} = \infty

(no upper restriction for the sample size), the bias converges to the lower bound for large variances

{ES}^{2} - σ^{2} \to - \frac{2 n_{1} - 1}{2 n_{1} - 3} \cdot \frac{1}{v} for σ^{2} \to + \infty

Note that this lower bound has a similar form as the bound derived for the bias in adaptive two-stage designs with sample size reassessment based on the unblinded variance estimate.⁸ In the latter setting, the bound is given by $- (n_{1} - 1) / [(n_{1} - 2) v]$ . The bound derived in Theorem 4 does not apply to designs where the adjusted rule $n_{2}^{a}$ is applied. The numerical results below suggest that for the adjusted rule the absolute bias is maximized for a finite $σ^{2}$ and not for $σ^{2} \to \infty$ as for the unadjusted rule.

4 Simulation study of the properties of point estimates and confidence intervals

To quantify the bias of mean and variance estimates as well as the actual coverage of corresponding one- and two-sided confidence intervals in adaptive two-stage designs with blinded sample size reassessment, a simulation study was performed. We assumed that a trial is planned for an effect size $δ_{0} = 1$ and a priori variance estimate σ₀ at a one-sided significance level $α = 0.025$ . The preplanned sample size of a two-armed parallel group trial was chosen to provide 80% power (based on the normal approximation) for $σ_{0} \in {1, 1.5, 2}$ . The first stage sample size was set to half of the preplanned total sample size leading to first stage per group sample sizes of $n_{1} \in {8, 18, 32}$ . For each first stage sample size, first stage data were simulated from normal distributions for effect sizes δ ranging from −2 to 2 (in steps of 0.1) and common standard deviations σ, ranging from 0.5 to 2 in steps of 0.5. The second stage sample sizes were then reassessed to $n_{2}^{u}, n_{2}^{a}$ (with $n_{2 min} = 0$ ). Below we refer to the latter reassessment rules as the unadjusted and adjusted sample size reassessment rules. For each scenario $5 \times 10^{7}$ , trials were simulated with R.³⁴ The source code is available in the R-package blindConfidence which can be downloaded and installed from github. Details on the implementation and additional results are presented in the Supplementary Material. Figures 1 and 2 show the mean bias of the final estimates of the mean and variance based on the total sample. If the alternative holds, the estimates may be substantially biased for small first stage sample sizes.

Figure 1.

Bias of the mean under blinded sample size reassessment using the unadjusted (solid line) and adjusted (dashed line) interim variance estimate. The dotted lines show maximum negative and positive bias that can be attained under any blinded sample size reassessment rule according to Theorem 2. The dashed gray line shows the maximum bias that can be attained under any (unblinded) sample size reassessment rule. The treatment effect used for planning is set to $δ_{0} = 1$ . Rows refer to the a priori assumed standard deviations σ₀ determining the first stage sample size n₁. The columns correspond to actual standard deviations. The x-axis in each graph denotes the true treatment effect δ, the y-axis shows the bias.

Figure 2.

Bias of the variance under blinded sample size reassessment using the unadjusted (solid line) and adjusted (dashed line) interim variance estimate. The gray line gives the lower bound from Theorem 4 for the bias under sample size reassessment based on the unadjusted variance estimate. The treatment effect used for planning is set to $δ_{0} = 1$ . Rows refer to the a priori assumed standard deviations σ₀ determining the first stage sample size n₁. The columns correspond to actual standard deviations. The x-axis in each graph denotes the true treatment effect δ, the y-axis shows the bias.

For both sample size reassessment rules, the bias of the mean is zero under the null hypothesis and has the opposite sign as the true effect size, otherwise (in accordance with Theorem 1). For very large positive and negative effect sizes, the bias is close to zero. This is due to the fact that a large effect size results in a large positive bias of the blinded interim variance estimates which in turn results in very large second stage sample sizes. As a consequence, the overall estimates are essentially equal to the (unbiased) second stage estimates, and the bias becomes negligible. In the considered scenarios, the absolute bias is decreasing in the first stage sample size. This also holds for the upper bound of the bias for general sample size reassessment rules. However, while the bias for the adjusted and unadjusted reassessment rules is small for larger sample sizes, the upper bound for general sample size reassessment rules still exceeds 10% in the considered scenarios even in the case where $n_{1} = 32$ subjects per group are recruited in the first stage. In the considered scenarios, we observe that the upper bound for the bias of the mean estimate increases in the true effect size δ and the true standard deviation σ. The numerical results confirm that for increasing $| δ |$ the maximum bias under blinded sample size reassessment approaches to the maximum bias under unblinded sample size reassessment as shown in Theorem 2.

The variance estimate is negatively biased for both considered sample size reassessment rules. For increasing variances, the absolute bias under the null hypothesis δ = 0 for the rule $n_{2}^{u}$ approaches the lower bound derived in Theorem 4. For large positive and negative effect sizes, the bias approaches zero, again, because the overall estimate becomes essentially equal to the (unbiased) second stage estimates. In general, we observe that using the adjusted interim variance estimate for sample size reassessment results in a larger absolute bias of mean and variance estimates.

The difference of the coverage probabilities of the confidence intervals to the nominal coverage probabilities (i.e., 0.975 for the one-sided, 0.95 for the two-sided confidence interval) is shown in Figure 3 for the unadjusted sample size reassessment rule (see Figure 1 of the Supplementary Material for corresponding results for the adjusted sample size reassessment rule). For positive δ, the lower confidence bound is conservative (with coverage probability larger than 0.975) while the upper bound is anti-conservative, and vice-versa for negative δ. In fact, due to symmetry, the coverage probability for the upper confidence bound at a certain true effect size δ is the same as the coverage probability of the lower confidence bound at $- δ$ . If the true $σ^{2}$ and the true $| δ |$ are small, the trial will stop with minimal sample size almost with probability 1. Then, sample size is essentially fixed, no bias occurs and coverage is as desired. This is, e.g., the case for $σ = 1, n_{1} = 32$ and $| δ | < 1.3$ and explains the exact coverage there. Similar cases occur for small $| δ |$ in the three pictures for $σ = 0.5$ as well.

Figure 3.

Difference between actual and nominal coverage probabilities in percentage points under blinded sample size reassessment using the unadjusted interim variance estimate $S_{1, OS}$ : upper confidence bound (dashed line), lower confidence bound (dotted line) and two-sided interval (solid line). Rows refer to the a priori assumed standard deviations σ₀ determining the first stage sample size n₁. The columns correspond to actual standard deviations. The x-axis in each graph denotes the true treatment effect δ; the y-axis shows the difference of actual to nominal coverage probability such that negative values indicate settings where the confidence bound is not valid.

The two-sided coverage probability (which is given by one minus the sum of the non-coverage probabilities of the lower and upper bounds) is not controlled over a large range of δ. However, for δ = 0 (i.e., under the null hypothesis), the inflation of the non-coverage probability, which corresponds to the Type I error rate of the corresponding test, is minor for small sample sizes (0.5 to 0.01 percentage points $n_{1} < 10$ ) and essentially controlled for larger samples sizes. (See Figure 15 of the Supplementary Material for a simulation study that aims to identify the σ where the Type I error rate of the original hypothesis test is maximized.) The good coverage under the null hypothesis is at first sight surprising, given that the variance estimate is negatively biased and the mean estimate is unbiased. We therefore computed the actual variance $σ_{e}^{2}$ of the mean estimate in the simulation study. To compare the actual variance of the mean to the estimated variance, we computed for each simulated trial the variance estimate ${\hat{S}}_{e}^{2} = \frac{2 S^{2}}{n_{1} + n_{2}}$ as well as the actual variance $σ_{f}^{2} = \frac{2 σ^{2}}{n_{1} + n_{2}}$ of a design with fixed sample sizes n₁, n₂ (thus, ignoring that n₂ is dependent on the first stage sample). Both quantities were then averaged over the Monte-Carlo samples. The results show that the true variance of the mean estimate of an adaptive design with blinded sample size reassessment is smaller than the average variance of a fixed sample design with the same sample sizes. The bias of the estimate of the variance of the mean estimate is close to zero around the null hypothesis. This holds both for designs using the adjusted and unadjusted interim variance estimates to reassess the sample size and gives some intuition why there is no relevant inflation of the coverage probability of confidence intervals under the null hypotheses, even though the variance estimate has a considerable negative bias (see Figures 5 and 6 of the Supplementary Material for results that show the variance of the mean estimate for designs that use the unadjusted and adjusted sample size rule).

While Figures 1 to 3 demonstrate the dependence of the bias and coverage probabilities on the true mean and standard deviation, in an actual trial these parameters are typically unknown. Therefore, we computed the maximum absolute bias of the mean, variance and the maximum difference between nominal and actual coverage probabilities for fixed first stage sample sizes over a range of values for the true mean and variance (see Figure 4). The optimization was performed in several steps based on simulations across an increasingly finer grid of true mean differences $δ \in [0, 4]$ and standard deviations $σ \in [. 5, 4]$ and using increasingly larger numbers of up to 10⁸ simulation runs (see Section 2 of the Supplementary Material for computational details of this simulation).

Figure 4.

Maximum absolute mean and variance bias as well as maximum negative difference between actual and nominal coverage probabilities (in percentage points) for given per group first stage sample sizes n₁ between 2 and 50; values of δ between 0 and 4; σ between 0.5 and 4. Left columns show the results for the unadjusted sample size reassessment rule, right columns for the adjusted sample size reassessment rule. The first row shows the value for bias and non-coverage, the second shows the effect size δ and the third shows the standard deviation σ at which the specific value is attained. The gray line denotes a Loess smoothed estimate. For the maximum absolute variance bias, based on the unadjusted rule, the theoretical bound is shown using the slightly thicker gray line.

Overall, the maximum bias of the mean and variance estimates as well as the maximum difference between nominal and actual coverage probabilities using the adjusted sample size reassessment rule exceed that of the unadjusted rule. The maximum absolute bias of the mean bias (Columns 1 and 2 of Figure 4) drops from 0.18 (0.08) for trials with a per group first stage sample size of $n_{1} = 2$ to 0.02 (0.02) for trials with a per group first stage sample size of $n_{1} = 50$ if the adjusted (unadjusted) interim variance estimate is used to reassess the sample size. For increasing values of the first stage sample size, the maximum of the absolute mean bias is found for larger values of the true mean and variance.

For the unadjusted rule, the simulations suggest that the absolute variance bias is maximized for δ = 0 and $σ \to \infty$ (see Figure 2 and the additional simulations in Section 2 of the Supplementary Material). Thus, Column 3 of Figure 4 shows the absolute bias for δ = 0 and σ = 4 (the largest value of σ considered in the parameter grid). For the adjusted rule, Column 4 of Figure 4 shows the maximum absolute bias for δ = 0 and σ maximized over the grid. For the adjusted rule, the maximum bias is attained for a finite σ. The maximum absolute bias of the variance estimate drops from 0.27 (0.19) for trials with a per group first stage sample size of $n_{1} = 2$ to 0.07 (0.06) for trials with a per group first stage sample size of $n_{1} = 50$ if the adjusted (unadjusted) interim variance estimate is used to reassess the sample size. In the simulation, the maxima of the absolute variance bias, using the unadjusted reassessment rule, is practically identical to the theoretical boundary derived in Theorem 4 (see the gray line in the first graph of Column 3 of Figure 4). If the adjusted rule is used, the maximum is attained for values of σ ranging from 1 to 2, increasing with the first stage sample size.

The maximum negative difference between actual and nominal coverage probabilities (“AC-NC,” in Columns 5 and 6 of Figure 4) of the one-sided confidence intervals ranges from about 2.8 (2.4) to 0.4 (0.3) percentage points if the adjusted (unadjusted) variance estimate is used to reassess the sample size. For the two-sided confidence intervals, differences between actual and nominal coverage are slightly lower ranging from 2.4 (2) to 0.2 (0.2) percentage points, respectively. For increasing first stage sample sizes, the maximum is attained for increasing values of both σ and δ where the former ranges between 1 and 3 and the latter between 0.5 and 1.5.

In Section 4 of the Supplementary Material, the corresponding results for restricted sample size rules that limit the maximum second stage sample size to twice the preplanned second stage sample size, are given. We observe that for scenarios where $σ_{0} < σ$ , limiting the second stage sample size results in a reduction of the (absolute) bias and higher coverage probabilities. This, however, comes at the price of limited control in terms of power. When $σ_{0} > σ$ we observe that the (absolute) bias of the mean estimate (in line with the theoretical results in Theorem 2) is reduced by a factor $n_{2 max} / (n_{1} + n_{2 max})$ . The bias of the variance estimate as well as the difference between actual and nominal coverage probabilities are less affected by restricting the second stage sample size. When σ is substantially smaller than σ₀ bias and coverage probabilities under the restricted and unrestricted sample size reassessment rules are nearly identical.

5 Case study

We illustrate the estimation of treatment effects with a randomized placebo controlled trial to demonstrate the efficacy of the kava-kava special extract “WS® 1490” for the treatment of anxiety³⁵ that was used to illustrate procedures for blinded sample size adjustment.¹⁷ The primary endpoint of the study was the change in the Hamilton Anxiety Scale (HAMA) between baseline and end of treatment. Assuming a mean difference between treatment and control of $δ_{0} = 5.5$ and a standard deviation of $σ_{0} = 8$ (i.e., a variance of $σ_{0}^{2} = 64$ ) results in a sample size of 34 patients per group to provide a power of 80%.

As in the previous case study,¹⁷ we assume that sample size reassessment based on blinded data is performed after 15 patients per group, that the sample size reassessment rules were applied with $n_{2 min} = 0$ and $n_{2 max} = \infty$ and consider that the interim estimate of the standard deviation is $S_{1, OS} = 6$ . Consequently, we get $n_{2}^{u} = 4.7$ and $n_{2}^{a} = 0.6$ , such that in the second stage five patients per group would have been recruited based on the unadjusted sample size reassessment rule $n_{2}^{u}$ , and only one patient per group based the adjusted rule $n_{2}^{a}$ .

To estimate the bias of the mean and variance estimate as well as the coverage probabilities of confidence intervals, we performed a simulation study for true effect sizes δ ranging from −11 $(= - 2 δ_{0})$ to 11 $(= 2 δ_{0})$ in steps of 0.05, and σ from 1 to 20 in steps of 1 (see Figure 5).

Figure 5.

Coverage probabilities and bias of mean and variance estimates for the case study. The first row of panels shows actual-nominal coverage probabilities (AC-NC) for the 97.5% upper (dashed line), lower (dotted line) and the 95% two-sided confidence intervals, for the unadjusted reassessment rule $n_{2}^{u}$ . The second row shows the bias of the mean estimate if $n_{2}^{u}$ (solid line) or $n_{2}^{a}$ (dashed line) is used. The dotted line shows upper and lower bounds for the bias for general blinded sample size reassessment rules based on $S_{1, OS}^{2}$ . The dashed gray line shows the bounds for the bias for a general unblinded sample size reassessment rule. The third panel shows the bias of the variance estimate if either $n_{2}^{u}$ (solid line) or $n_{2}^{a}$ (dashed line) is used. The red line shows the theoretical boundary for the bias give in Theorem 4.

5.1 Unadjusted sample size rule

For fixed δ, the absolute bias of the variance estimate is increasing in σ. For σ = 20, the bias of the variance becomes $- 2.06$ which is within simulation error of the theoretical lower bound. The absolute mean bias takes its maximum (minimum) 0.2 (−0.2) for the effect sizes $δ = - 7.98$ $(7.98)$ , respectively, and a standard deviation of $σ = 5$ . The maximum inflation of the non-coverage probabilities of one-sided 97.5% and two-sided 95% confidence intervals is 0.7 and 0.5 percentage points, respectively. The actual coverage probabilities are smallest for large absolute values of the true mean difference and standard deviations of around 5.

5.2 Adjusted sample size rule

If the sample size reassessment is based on the adjusted variance estimate, the absolute bias of the variance and mean estimate will be even larger taking values up to 2.49 for the variance and up to 0.25 for the mean, respectively. The inflation of the non-coverage probabilities goes up to 0.9 percentage points for the one-sided confidence intervals and 0.6 percentage points for the two-sided intervals.

6 Discussion

We investigated the properties of point estimates and confidence intervals in adaptive two-stage clinical trials where the sample size is reassessed based on blinded interim variance estimates. Such adaptive designs are in accordance with current regulatory guidance that proposes blinded sample size reassessment procedures based on aggregated interim data.^9–11 We showed that such blinded sample size reassessment may lead to biased effect size estimates, biased variance estimates and may have an impact on the coverage probability of confidence intervals. The extent of the biases depends on the specific sample size reassessment rule, the first stage sample size, the true effect size and the variance.

We showed that for the unadjusted and the adjusted sample size reassessment rules that aim to control the power at the pre-specified level, the bias of confidence intervals may be large for very small first stage sample sizes but is small otherwise. Under the null hypothesis, even for first stage sample sizes as low as 16, the confidence intervals do not exhibit a relevant inflation of the non-coverage probability. For positive effect sizes, inflations (even though minor) are observed also for somewhat larger sample sizes. This corresponds to previous findings that the type I error rate of superiority as well as non-inferiority tests is hardly affected by blinded sample size reassessment.^17,28 In addition, we show that for positive treatment effects the lower confidence interval (which is often the most relevant because it gives a lower bound of the treatment effect) is strictly conservative in all considered simulation scenarios. The upper bound in contrast shows an inflation of the non-coverage probability, which is however only of relevant size if sample sizes are small.

An approach to obtain conservative confidence intervals even for trials with small first stage sample sizes is to apply methods proposed for flexible designs with unblinded interim analyses^{32,33,36–39} which are based on combination tests or the conditional error rate principle. Such approaches allow to construct confidence intervals which control the coverage probabilities. However, they are based on test statistics that do not equally weigh outcomes from patients recruited in the first and second stage. A hybrid approach that strictly controls the coverage probability could be based on confidence intervals that are the union of the fixed sample confidence interval and the confidence interval obtained from the flexible design methodology. Similar approaches have been proposed for hypothesis testing in adaptive designs.^40,41

We demonstrated that the treatment effect estimates are negatively (positively) biased under the adjusted and unadjusted sample size reassessment rules for all positive (negative) effect sizes. The bias decreases with the first stage sample size but for worst case parameter constellations it remains noticeable also for larger sample sizes. In addition, for the worst case sample size reassessment rule based on blinded variance estimates, the bias may be substantial even for large first stage sample sizes: As the effect size increases the maximum bias of the blinded sample size reassessment rule approaches the bias of unblinded sample size reassessment. As a consequence to maintain the integrity of confirmatory clinical trials with blinded sample size reassessment, binding sample size adaptation rules must be pre-specified and it is important to verify on a case-by-case basis that the reassessment rules used do not have a substantial impact on the properties of estimators should be used.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Martin Posch and Franz Koenig were supported by Austrian Science Fund Grant P23167. Florian Klinglmueller was supported by Austrian Science Fund Grants Number P23167 and J3721.

Supplementary material

Supplementary figures and additional details on the simulation study are presented in an additional document, which is distributed with the paper as Supplementary Material.

The R code used to compute the results of the simulation, and the case study is available as an R-package named blindConfidence which is available on github or on request from the authors.

References

Stein

. A two-sample test for a linear hypothesis whose power is independent of the variance. Ann Math Stat 1945; 16: 243–258.

Wittes

Brittain

. The role of internal pilot studies in increasing the efficiency of clinical trials. Stat Med 1990; 9: 65–72.

Betensky

Tierney

. An examination of methods for sample size recalculation during an experiment. Stat Med 1997; 16: 2587–2598.

Wittes

Schabenberger

Zucker

, et al. Internal pilot studies I: type I error rate of the naive t-test. Stat Med 1999; 18: 3481–3491.

Zucker

Wittes

Schabenberger

, et al. Internal pilot studies II: comparison of various procedures. Stat Med 1999; 18: 3493–3509.

Denne

Jennison

. Estimating the sample size for a t-test using an internal pilot. Stat Med 1999; 18: 1575–1585.

Proschan

Wittes

. An improved double sampling procedure based on the variance. Biometrics 2000; 56: 1183–1187.

Miller

. Variance estimation in clinical studies with interim sample size reestimation. Biometrics 2005; 61: 355–361.

EMEA. Reflection paper on methodological issues in confirmatory clinical trials planned with an adaptive design, London, www.ema.europa.eu/ema/pages/includes/document/open_document.jsp?webContentId=WC500003616 (2007, accessed 8 September 2016).

10.

FDA, CDER. Guidance for industry: adaptive design clinical trials for drugs and biologics (Draft Guidance), Rockville: Food and Drug Administration, 2010.

11.

FDA, CDRH. Draft guidance: adaptive designs for medical device clinical studies, Rockville: Food and Drug Administration, 2015.

12.

Elsäßer

Regnstrom

Vetter

, et al. Adaptive clinical trial designs for European marketing authorization: a survey of scientific advice letters from the European Medicines Agency. Trials 2014; 15: 383–383.

13.

Pritchett

Menon

Marchenko

, et al. Sample size re-estimation designs in confirmatory clinical trialscurrent state, statistical considerations, and practical guidance. Stat Biopharm Res 2015; 7: 309–321.

14.

Gould

Shih

. Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance. Comm Stat Theor Meth 1992; 21: 2833–2853.

15.

Govindarajulu

. Robustness of sample size re-estimation procedure in clinical trials (arbitrary populations). Stat Med 2003; 22: 1819–1828.

16.

Friede

Kieser

. Blinded sample size re-estimation in superiority and noninferiority trials: bias versus variance in variance estimation. Pharmaceut Stat 2013; 12: 141–146.

17.

Kieser

Friede

. Simple procedures for blinded sample size adjustment that do not affect the type I error rate. Stat Med 2003; 22: 3571–3581.

18.

Proschan

. Two-stage sample size re-estimation based on a nuisance parameter: a review. J Biopharm Stat 2005; 15: 559–574.

19.

Proschan

Wittes

Lan

. Statistical monitoring of clinical trials, New York: Springer, 2006.

20.

Posch

Proschan

. Unplanned adaptations before breaking the blind. Stat Med 2012; 31: 4146–4153.

21.

Proschan

Glimm

Posch

. Connections between permutation and t-tests: relevance to adaptive methods. Stat Med 2014; 33: 4734–4742.

22.

Friede

Kieser

. Sample size recalculation for binary data in internal pilot study designs. Pharmaceut Stat 2004; 3: 269–279.

23.

Friede

Schmidli

. Blinded sample size reestimation with count data: methods and applications in multiple sclerosis. Stat Med 2010; 29: 1145–1156.

24.

Schneider

Schmidli

Friede

. Blinded and unblinded internal pilot study designs for clinical trials with count data. Biometrical J 2013; 55: 617–633.

25.

Stucke

Kieser

. Sample size calculations for noninferiority trials with Poisson distributed count data. Biometrical J 2013; 55: 203–216.

26.

Wachtlin

Kieser

. Blinded sample size recalculation in longitudinal clinical trials using generalized estimating equations. Ther Innov Regul Sci 2013; 47: 460–467.

27.

Friede

Miller

. Blinded continuous monitoring of nuisance parameters in clinical trials. J R Stat Soc Series C (Appl Stat) 2012; 61: 601–618.

28.

Friede

Kieser

. Blinded sample size reassessment in non-inferiority and equivalence trials. Stat Med 2003; 22: 995–1007.

29.

. Distribution of the two-sample t-test statistic following blinded sample size re-estimation. Pharm Stat 2016; 15: 208–215.

30.

Żebrowska

Posch

Magirr

. Maximum type I error rate inflation from sample size reassessment when investigators are blind to treatment labels. Stat Med 2015; 35: 1972–1984.

31.

Bretz

Koenig

Brannath

, et al. Adaptive designs for confirmatory clinical trials. Stat Med 2009; 28: 1181–1217.

32.

Brannath

Mehta

Posch

. Exact confidence bounds following adaptive group sequential tests. Biometrics 2009; 65: 539–546.

33.

Brannath

König

Bauer

. Estimation in flexible two stage designs. Stat Med 2006; 25: 3366–3381.

34.

R Core Team. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing, www.R-project.org (2014, accessed 8 September 2016).

35.

Malsch

Kieser

. Efficacy of kava-kava in the treatment of non-psychotic anxiety, following pretreatment with benzodiazepines. Psychopharmacology 2001; 157: 277–283.

36.

Brannath

Posch

Bauer

. Recursive combination tests. J Am Stat Assoc 2002; 97: 236–244.

37.

Lawrence

Hung

. Estimation and confidence intervals after adjusting the maximum information. Biometrical J 2003; 45: 143–152.

38.

Mehta

Bauer

Posch

, et al. Repeated confidence intervals for adaptive group sequential trials. Stat Med 2007; 26: 5422–5433.

39.

Posch

Koenig

Branson

, et al. Testing and estimation in flexible group sequential designs with adaptive treatment selection. Stat Med 2005; 24: 3697–3714.

40.

Posch

Bauer

Brannath

. Issues in designing flexible trials. Stat Med 2003; 22: 953–969.

41.

Burman

Sonesson

. Are flexible designs sound? Biometrics 2006; 62: 664–669.

42.

Behboodian

. Covariance inequality and its application. Int J Math Educ Sci Technol 1994; 25: 643–647.

43.

Johnson

Kotz

Balakrishnan

. Continuous univariate distributions 1994; Vol. 1. Hoboken, NJ, USA: John Wiley & Sons.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.74 MB

Estimation after blinded sample size reassessment

Abstract

Keywords

1 Introduction

2 Blinded sample size reassessment

3 Theoretical results on the bias of the final mean and variance estimators

Theorem 1

Theorem 2

Theorem 3

Theorem 4

4 Simulation study of the properties of point estimates and confidence intervals

5 Case study

5.1 Unadjusted sample size rule

5.2 Adjusted sample size rule

6 Discussion

Footnotes

Declaration of conflicting interests

Funding

Supplementary material

References

Supplementary Material