The population-wise error rate for clinical trials with overlapping populations

Abstract

We introduce a new multiple type I error criterion for clinical trials with multiple, overlapping populations. Such trials are of interest in precision medicine where the goal is to develop treatments that are targeted to specific sub-populations defined by genetic and/or clinical biomarkers. The new criterion is based on the observation that not all type I errors are relevant to all patients in the overall population. If disjoint sub-populations are considered, no multiplicity adjustment appears necessary, since a claim in one sub-population does not affect patients in the other ones. For intersecting sub-populations we suggest to control the average multiple type I error rate, i.e. the probability that a randomly selected patient will be exposed to an inefficient treatment. We call this the population-wise error rate, exemplify it by a number of examples and illustrate how to control it with an adjustment of critical boundaries or adjusted $p$ -values. We furthermore define corresponding simultaneous confidence intervals. We finally illustrate the power gain achieved by passing from family-wise to population-wise error rate control with two simple examples and a recently suggested multiple-testing approach for umbrella trials.

1 Introduction

The aim of precision medicine is to provide each patient with an optimal treatment tailored to his or her genetic and/or clinical profile. One strategy for reaching this goal is to undertake trials where one or several treatments are investigated in multiple sub-populations. Examples of such trials are umbrella and basket trials in oncology. In an umbrella trial patients with the same cancer type but different molecular alterations are enrolled and the treatments are tailored to the specific target sub-populations. In a basket trial, patients with different cancer types but one common molecular alteration are enrolled with the aim to study one specific treatment that is targeted to the common alteration (see e.g., Woodcock and LaVange, ¹ Strzebonska and Waligora ² ). In many cases, the target or sub-populations are disjoint by nature, but when many different biomarkers or cancer types are used, it can also occur that patients belong to more than one sub-population. For example, in the FOCUS4 study, ³ biomarker tests were conducted to define subgroups based on the mutations present in the patients’ tumour DNA. Some patients belonged to more than one subgroup and thus the subgroups were made disjoint by means of a hierarchical ordering structure defined for the different mutations. Another example are subgroup selection and adaptive enrichment designs (e.g. Brannath et al., ⁴ Glimm and Di Scala, ⁵ Wassmer and Brannath, ⁶ Stallard et al. ⁷ ) where a single treatment is tested in an overall patient population and a specific biomarker subgroup of the overall patient population, for which earlier studies indicate that the treatment may be more or even only effective. In this manuscript, we explicitly allow biomarker-defined sub-populations to be overlapping such that patients become eligible for multiple targeted treatment strategies including the case of a single treatment with multiple target populations. This means that, by the trial’s result, future patients of the overlap may be exposed to more than a single test decision on potentially inefficient treatment strategies. Moreover, for such studies suitable allocation procedures have to be defined. Issues of eligibility for multiple target therapies have been addressed e.g. in Malik et al., ⁸ Collignon et al. ⁹ and Kesselmeier et al. ¹⁰

In confirmatory clinical trials with tests of several hypotheses the multiple type I error is usually kept small by controlling the family-wise error rate (FWER). With the growing effort of detecting new and more predictive biomarkers and an increasing focus on rare diseases, it is becoming more and more difficult to undertake clinical trials that are sufficiently powered and also provide sufficient control of type I errors. Since the control of multiple type I errors amplifies this issue, more efficient alternatives to the common approach of FWER control are of strong interest. If a treatment or a treatment strategy is tested in several disjoint populations and each population is affected by only a single hypothesis test, the overall study basically consists of separate trials that merely share the same infrastructure. Therefore, no multiplicity adjustments are needed (e.g. Glimm and Di Scala, ⁵ Collignon et al. ⁹ ). However, if some sub-populations are overlapping, these intersections will contain patients that are possibly exposed to multiple erroneously rejected null hypotheses, implying that one has to adjust for multiplicity (e.g. Collignon et al. ⁹ ). Since only patients in the intersections are concerned with this multiplicity issue, there is no need for adjustments for patients in the complements, who can only be affected by at most one false rejection of a null hypothesis. The FWER would therefore be too conservative also in this case. Especially for small and/or highly stratified populations, as for instance encountered in paediatric oncology, a more efficient approach is desirable (e.g. Fletcher et al. ¹¹ ). The purpose of this manuscript is to propose a new concept of multiple type I error control that is less conservative. With this new error rate, which we name population-wise error rate (PWER), we aim to keep the average multiple type I error rate at a reasonable level. This provides control of the probability that a randomly chosen future patient will be exposed to an inefficient treatment policy.

The paper is outlined as follows. First, the PWER is motivated by means of a simple example, followed by the general mathematical definition. Then, we demonstrate how to control the PWER at a pre-specified level by adjusting critical boundaries or $p$ -values. In Section 4, we present a mathematical result (with a limit) for the strata-wise FWER when adjusting the boundary for PWER control. In the subsequent section, the gain in power by using PWER- instead of FWER-control is illustrated by two examples. In the first example we will investigate the case of two overlapping populations and with (i) two different treatments in each population and (ii) the same treatment in both populations. The second example consists of an application of the PWER to a multiple testing approach for umbrella trials suggested in Sun et al. ¹² In Section 6 we extend the multiple test with PWER-control to simultaneous confidence intervals (SCIs) and discuss their coverage properties. The paper concludes with a discussion in Section 7. All computations and simulations are done in R. The corresponding R-script files are all available at https://github.com/chillner/RCode_Paper_PWER.

2 The population-wise error rate

In this section, the aforementioned PWER is introduced conceptually and formally. Examples for different settings are given to further deepen the understanding.

2.1 General framework and definition

Consider an overall population $\mathcal{P}$ consisting of $m\ge 2$ possibly overlapping sub-populations ${\mathcal{P}}_1,\cdots ,{\mathcal{P}}_m\subseteq \mathcal{P}$ and suppose that we want to investigate for all ${\mathcal{P}}_i$ a treatment $T_i$ in comparison to a control treatment. The population ${\mathcal{P}}_i$ is defined by specific inclusion and exclusion criteria which may include specific biomarker characteristics. In trials on personalized treatments, the appropriate inclusion and exclusion criteria are often a research question itself and then the same treatment is investigated in several different populations simultaneously. This is the case, for instance, in basket and subgroup selection trials. In other trial examples, like umbrella trials, different experimental treatments are investigated in the different sub-populations. In the sequel we call the tuples $({\mathcal{P}}_i,T_i)$ the treatment policies where we allow treatments to be different or equal to each other, so that there can be some $T_i$ and $T_j$ with $i\ne ̸j$ such that $T_i=T_j$ . To each treatment policy ( ${\mathcal{P}}_i,T_i$ ) we assign the null hypothesis $H_i:{\theta }_i\le 0$ , where ${\theta }_i=\theta ({\mathcal{P}}_i,T_i)$ quantifies the efficacy of treatment $T_i$ in comparison to the control treatment in population ${\mathcal{P}}_i$ . Also the control treatments can be the same or can differ between the different populations. The PWER is then given by the risk for a randomly chosen future patient to be assigned to one or more inefficient treatment policies, i.e. to belong to at least one tuple $({\mathcal{P}}_i,T_i)$ where for the true effect ${\theta }_i\le 0$ but $H_i$ has been rejected. This future patient is imagined to be drawn after the study from the same population the study sample was drawn from.

In order to define the PWER mathematically, we partition the overall population into disjoint strata ${\mathcal{P}}_J:=\bigcap _{j\in J}{\mathcal{P}}_j\setminus \bigcup _{k\in I\setminus J}{\mathcal{P}}_k$ for $J\subseteq I:=\{1,\cdots ,m\}$ . In Figure 1 we see an example for such a partition based on three sub-populations ${\mathcal{P}}_i$ , $i=1,2,3$ . Note that ${\mathcal{P}}_{\{1,2,3\}}=\mathrm{\varnothing }$ . For each non-empty stratum ${\mathcal{P}}_J$ , we denote its proportion within the overall patient population $\mathcal{P}$ by ${\pi }_J$ such that $\sum _{J\subseteq I,{\mathcal{P}}_J\ne \mathrm{\varnothing }}{\pi }_J=1$ . In most parts of the paper we will assume that the relative prevalences ${\pi }_J$ are known. In practical cases, they will have to be estimated. The effect of estimation will be discussed in the Section 5.3. For any future patient in ${\mathcal{P}}_J$ , $J\subseteq I$ , we commit a type I error if he/she belongs to at least one $({\mathcal{P}}_i,T_i)$ with $i\in J$ and ${\theta }_i\le 0$ for which $H_i$ has been rejected. The PWER is then defined as

$$\mathrm{PWER}=\sum _{J\subseteq I,{\mathcal{P}}_J\ne \mathrm{\varnothing }}{\pi }_J\mathbb{P}(\mathrm{falsely}\mathrm{reject}\mathrm{any}{H}_i\mathrm{with}i\in J)$$

(1)To determine the PWER, we need to know for each stratum ${\mathcal{P}}_J$ the probability of rejecting at least one true null hypothesis that affects the stratum.

Figure 1.

$m=3$ populations and their disjoint sub-populations.

Compared to the FWER, which controls the maximum risk for future patients to be assigned to an inefficient treatment strategy, the PWER is an average risk. It is more liberal and thereby more powerful, because

$$\begin{array}{rl}\mathrm{PWER}& =\sum _{J\subseteq I,{\mathcal{P}}_J\ne \mathrm{\varnothing }}{\pi }_J\mathbb{P}(\mathrm{falsely}\mathrm{reject}\mathrm{any}{H}_j\mathrm{with}j\in J)\\ & \le \left(\sum _{J\subseteq I,{\mathcal{P}}_J\ne \mathrm{\varnothing }}{\pi }_J\right)\mathbb{P}(\mathrm{falsely}\mathrm{reject}\mathrm{any}{H}_i\mathrm{for}i\in I)=\mathrm{FWER}\end{array}$$

Note that $\mathrm{PWER}=\mathrm{FWER}$ only in the case where $\mathcal{P}={\mathcal{P}}_1=\dots ={\mathcal{P}}_m$ .

2.2 Two intersecting populations

As an example consider a trial with two intersecting sub-populations ${\mathcal{P}}_1$ and ${\mathcal{P}}_2$ and two treatments $T_1$ , $T_2$ to be tested by means of the hypotheses $H_1:\theta ({\mathcal{P}}_1,T_1)\le 0$ and $H_2:\theta ({\mathcal{P}}_2,T_2)\le 0$ . Usually, the two treatments will be compared to the same control, however, the basic idea given in the subsequent sections also applies with treatment-specific controls. As illustrated in the left panel of Figure 2, the overall population can be partitioned into three disjoint sub-populations, ${\mathcal{P}}_{\{1\}}:={\mathcal{P}}_1\setminus {\mathcal{P}}_2$ , ${\mathcal{P}}_{\{2\}}:={\mathcal{P}}_2\setminus {\mathcal{P}}_1$ and ${\mathcal{P}}_{\{1,2\}}:={\mathcal{P}}_1\cap {\mathcal{P}}_2$ . Obviously, we commit a type I error for ${\mathcal{P}}_{\{i\}}$ whenever $H_i$ is falsely rejected, $i=1,2$ , and for ${\mathcal{P}}_{\{1,2\}}$ whenever $H_1$ or $H_2$ are falsely rejected. Hence, if $H_1$ and $H_2$ are both true, then

$$\mathrm{PWER}={\pi }_{\{1\}}\mathbb{P}(\mathrm{reject}{H}_1)+{\pi }_{\{2\}}\mathbb{P}(\mathrm{reject}{H}_2)+{\pi }_{\{1,2\}}\mathbb{P}(\mathrm{reject}{H}_1\mathrm{or}{H}_2)$$

(2)If $H_1$ is true and $H_2$ is false, then $\mathrm{PWER}={\pi }_{\{1\}}\mathbb{P}(\mathrm{reject}{H}_1)+{\pi }_{\{1,2\}}\mathbb{P}(\mathrm{reject}{H}_1)=({\pi }_{\{1\}}+{\pi }_{\{1,2\}})\mathbb{P}(\mathrm{reject}{H}_1)$ . Hence, if only one null hypothesis is true, say $H_i$ , the PWER reduces to the probability of rejecting $H_i$ multiplied by the relative size of the population ${\mathcal{P}}_i$ .

Figure 2.

Left panel: $m=2$ intersecting populations. Right panel: $m=3$ nested populations.

In situations where a single treatment $T=T_1=T_2$ is tested in both sub-populations ${\mathcal{P}}_1$ and ${\mathcal{P}}_2$ , the specific case in which only one hypothesis $H_i$ is rejected deserves some discussion. In clinical trials the final decisions for future patients usually depend on additional analyses that explore efficacy and safety across different populations and strata. Rejection of $H_i$ is then a necessary but not a sufficient condition for the decision to treat future patients from ${\mathcal{P}}_i$ with treatment $T$ . The PWER is then under control in any case, namely even in the worst-case scenario where all future patients of a population will receive treatment $T$ even though $T$ is inefficient for the other, intersecting population.

2.3 Nested populations

In practice, one often faces the problem of nested populations ${\mathcal{P}}_1\supset {\mathcal{P}}_2\supset \cdots \supset {\mathcal{P}}_m$ , as in the right panel of Figure 2. Think of a situation where the optimal eligibility criteria for a single treatment are unknown and thus a sequence of tightening eligibility criteria for testing the efficacy of that treatment is planned which then ultimately leads to a sequence of nested populations. Define the strata ${\mathcal{P}}_{[i]}:={\mathcal{P}}_{\{1,\cdots ,i\}}$ for $i\le m$ , which are ${\mathcal{P}}_{[i]}={\mathcal{P}}_i\setminus {\mathcal{P}}_{i+1}$ for $i<m$ and ${\mathcal{P}}_{[i]}={\mathcal{P}}_i$ for $i=m$ . We commit a type I error for ${\mathcal{P}}_{[i]}$ whenever any true $H_j$ is rejected for $j\le i$ . With relative prevalences ${\pi }_{[i]}:={\pi }_{\{1,\dots ,i\}}$ of ${\mathcal{P}}_{[i]}$ the PWER under the global null hypothesis is given by

$$\mathrm{PWER}=\sum _{i=1}^m{\pi }_{[i]}\mathbb{P}(\mathrm{reject}\mathrm{at}\mathrm{least}\mathrm{one}\mathrm{true}{H}_j\mathrm{for}j\le i)$$

Especially, if ${\mathcal{P}}_i$ is defined by a biomarker $X$ , i.e. ${\mathcal{P}}_i=\{X>t_i\}$ for cut-off points $t_i$ , $i=1,\cdots ,m+1$ (with $t_{m+1}:=\mathrm{\infty }$ ), the PWER under global null hypothesis can be written as

$$\mathrm{PWER}=\sum _{i=1}^m\mathbb{P}(t_i<X\le t_{i+1})\mathbb{P}(\mathrm{reject}\mathrm{at}\mathrm{least}\mathrm{one}\mathrm{true}{H}_j\mathrm{for}j\le i)$$

2.4 Three populations with two intersections

Finally, we give an example where the FWER is strictly conservative even for control of the maximum (instead of the average) type I error rate. Consider three populations ${\mathcal{P}}_1$ , ${\mathcal{P}}_2$ , ${\mathcal{P}}_3$ with ${\mathcal{P}}_1\cap {\mathcal{P}}_2\ne \mathrm{\varnothing }$ , ${\mathcal{P}}_2\cap {\mathcal{P}}_3\ne \mathrm{\varnothing }$ and ${\mathcal{P}}_1\cap {\mathcal{P}}_3=\mathrm{\varnothing }$ , as in Figure 1. Again, hypotheses of the form $H_i:\theta ({\mathcal{P}}_i,T_i)\le 0$ are to be tested in each population, respectively. Under the global null hypothesis, where all null hypotheses $H_i$ are true, the PWER is given by

$$\mathrm{PWER}=\sum _{i=1}^3{\pi }_{\{i\}}\mathbb{P}(\mathrm{reject}{H}_i)+\sum _{i=1}^2{\pi }_{\{i,i+1\}}\mathbb{P}(\mathrm{reject}{H}_i\mathrm{or}{H}_{i+1})$$

The FWER under the global null hypothesis equals $\mathbb{P}(\mathrm{reject}{H}_1\mathrm{or}{H}_2\mathrm{or}{H}_3)$ . Since no patient can belong to ${\mathcal{P}}_1$ and ${\mathcal{P}}_3$ simultaneously, the FWER corrects for a multiplicity no patient is affected by.

3 Control of the PWER

In this section, we demonstrate how to achieve control of the PWER at a pre-specified level $\alpha$ under the general framework of Section 2.1. Suppose that each $H_i$ can be tested with a test statistic $Z_i$ where larger values of $Z_i$ provide evidence against $H_i$ . We assume further that the joint distribution of $\{Z_i{\}}_{i=1}^m$ is known (at least asymptotically). In order to control the PWER at a pre-specified significance level $\alpha \in (0,1)$ , we need to find the smallest critical value $c^{*}\in \mathbb{R}$ such that

$${\mathrm{PWER}}_{{\theta }^{*}}=\sum _{J\subseteq I}{\pi }_J{\mathbb{P}}_{{\theta }^{*}}(\bigcup _{i\in J\cap I({\theta }^{*})}\{Z_i\ge c^{*}\})\le \alpha$$

(3)where ${\theta }^{*}=({\theta }_1^{*},\cdots ,{\theta }_m^{*})$ is the parameter configuration that maximizes the PWER and $I({\theta }^{*})=\{i\in I:{\theta }_i^{*}\in H_i\}$ is the index set of corresponding true null hypotheses. The maximal PWER is usually obtained under the global null hypothesis, i.e. for ${\theta }^{*}=(0,\cdots ,0)$ . This is e.g. the case under the subset pivotality condition (see e.g. Westfall and Young, ¹³ Dickhaus ¹⁴ ) which applies, for instance, to contrast $z$ - or $t$ -statistics for normal data with a variance that is homogeneous across treatment groups and population strata. Since the (asymptotic) correlations between the test statistics usually depend only on the relative prevalences ${\pi }_J$ , $J\subseteq I$ , the PWER-level can be exhausted under ${\theta }^{*}$ . When each $H_i$ is tested by means of a p-value $p_i$ , we can reach $\mathrm{PWER}\le \alpha$ by choice of an adjusted significance level ${\alpha }^{*}$ applied to all $p_i$ . The critical value $c^{*}$ in (3) or adjusted significance level ${\alpha }^{*}$ can be found by applying a univariate root finding method. Because the PWER is always bounded by the FWER, the critical value and adjusted significance level are more liberal than for FWER-control. Therefore the PWER leads to a higher power and a lower sample size to achieve a certain power.

Instead of determining the critical value $c^{*}$ we could report the PWER-adjusted $p$ -values

$$p_j^{PWER}=\sum _{J\subseteq I}{\pi }_J{\mathbb{P}}_{{\theta }^{*}}(\bigcup _{i\in J\cap I({\theta }^{*})}\{Z_i\ge z_j^{\mathrm{obs}}\}),j=1,\dots ,m$$

(4)where $z_j^{\mathrm{obs}}$ is the observed value of $Z_j$ . Obviously, $p_j^{PWER}\le \alpha$ if an only if $z_j^{\mathrm{obs}}\ge c^{*}$ and hence $H_j$ can alternatively be tested with the PWER-adjusted p-value $p_j^{PWER}$ . Furthermore, $p_j^{PWER}$ gives the smallest PWER-level the hypothesis $H_j$ can be rejected with.

Note that we could control the PWER also with population-specific critical values $c_i^{*}$ (or adjusted levels ${\alpha }_i^{*}$ ). Unique solutions for $c_i^{*}$ can be obtained by setting $c_i^{*}=w_i{c}^{*}$ for pre-specified weights $w_i>0$ and searching for the $c^{*}$ that meets the pre-specified PWER-level. Multiplicity adjusted $p$ -values can also be calculated with the weights $w_i$ . The weights may, for instance, be larger for smaller populations ${\mathcal{P}}_i$ in order to increase the chance of finding efficient treatment policies for small sub-populations. However, due to the weighting by ${\pi }_J$ in definition (1) and expression (3), the multiple type I error rate ${\mathbb{P}}_{{\theta }^{*}}(\bigcup _{i\in J\cap I({\theta }^{*})}\{Z_i\ge c^{*}\})$ for ${\mathcal{P}}_J$ will automatically be larger for smaller ${\pi }_J$ . We will therefore only consider equal critical values $c_i^{*}=c^{*}$ in our examples below. This has the additional advantage that, when all test statistics have the same marginal null distribution, the critical value $c^{*}$ cannot fall short of the $(1-\alpha )$ -quantile of the marginal null distribution (see Appendix 1). We further note that due to the natural heterogeneity of the treatment effects within each sub-population, it would be advisable to apply a stratified test to investigate a treatment in each ${\mathcal{P}}_j$ (as done in Section 5).

4 Behaviour of the strata-wise FWER under PWER-control

In order to understand the consequences of PWER-control it is of interest to investigate the behaviour of the FWERs for each stratum ${\mathcal{P}}_J$ , i.e. the risk for future patients in ${\mathcal{P}}_J$ to be exposed to at least one inefficient treatment strategy. Obviously, under PWER-control the strata-wise FWER for each ${\mathcal{P}}_J$ is bounded by $\alpha /{\pi }_J$ . This bound is useful only for sufficiently large ${\pi }_J$ , because there is another bound, which is independent from ${\pi }_J$ : As mentioned in the last section, in the common situation when all test statistics have the same marginal null distribution under ${\theta }^{*}$ , the common critical value $c^{*}$ is bounded from below by the $(1-\alpha )$ -quantile $z_{\alpha }$ of the marginal null distribution. Therefore, the strata-wise FWER for ${\mathcal{P}}_J$ is at most ${\tilde{\alpha }}_J:={\mathbb{P}}_{{\theta }^{*}}(\underset{j\in J}{max}{Z}_j\ge z_{\alpha })$ , see Appendix 1. Note that ${\tilde{\alpha }}_J$ is independent of the prevalences because the defining probability is conditional on the sample sizes. Moreover, ${\tilde{\alpha }}_J$ is at most $\min (|J|\alpha ,1)$ by the Bonferroni inequality. The bound $\alpha /{\pi }_J$ becomes useless latest when it is above the Bonferroni bound which is equivalent to ${\pi }_J>\alpha /{\tilde{\alpha }}_J\ge max(1/|J|,\alpha )$ . The following theorem gives an approximation for small ${\pi }_J$ which is often smaller than ${\tilde{\alpha }}_J$ , but depends on the prevalences ${\tilde{\pi }}_{J^{\prime }}$ of ${\mathcal{P}}_{J^{\prime }}$ for $J^{\prime }\ne ̸J$ relative to the complement $\mathcal{P}\setminus {\mathcal{P}}_J$ . For this theorem we need to define for $J\subseteq I$ the PWER that we obtain when removing ${\mathcal{P}}_J$ from the entire population:

$${cPWER}_J(c)=\sum _{J^{\prime }\subseteq I,J^{\prime }\ne ̸J}{\tilde{\pi }}_{J^{\prime }}{\mathbb{P}}_{{\theta }^{*}}(\underset{j\in J^{\prime }\cap I({\theta }^{*})}{max}{Z}_j\ge c)\mathrm{where}{\tilde{\pi }}_{J^{\prime }}={\pi }_{J^{\prime }}/(1-{\pi }_J)$$

(5)Accordingly, we call ${cPWER}_J(c)$ the complementary PWER of ${\mathcal{P}}_J$ . Note that the sum in (5) is a weighted average of all non- $J$ terms where the weights are the prevalence ${\tilde{\pi }}_{J^{\prime }}$ of ${\mathcal{P}}_{J^{\prime }}$ for $J^{\prime }\ne ̸J$ , relative to $\mathcal{P}\setminus {\mathcal{P}}_J$ .

Theorem. Assume that for all $J\subseteq I$ the strata-wise ${\mathrm{FWER}}_J(c):={\mathbb{P}}_{{\theta }^{*}}(\underset{j\in J\cap I({\theta }^{*})}{max}{Z}_j\ge c)$ is strictly decreasing and differentiable as a function in $c$ . Then ${\mathrm{PWER}}_{{\theta }^{*}}(c)$ and ${\mathrm{cPWER}}_J(c)$ are differentiable and strictly decreasing in $c$ as well and we find $c^{*},d_J^{*}$ such that ${\mathrm{PWER}}_{{\theta }^{*}}(c^{*})={\mathrm{cPWER}}_J(d_J^{*})=\alpha$ . With this we get

$${\mathrm{FWER}}_J(c^{*})={\mathrm{FWER}}_J(d_J^{*})+O({\pi }_J)$$

(6)where $O({\pi }_J)$ converges linearly with ${\pi }_J$ to zero while all ${\tilde{\pi }}_{J^{\prime }}$ , $J^{\prime }\ne ̸J$ , remain constant. Furthermore, when ${\mathrm{FWER}}_J(c)\ge {\mathrm{cPWER}}_J(c)$ for all $c$ , then ${\mathrm{FWER}}_J(c^{*})\le {\mathrm{FWER}}_J(d_J^{*})$ for all ${\pi }_J$ .

Note that the last statement follows for instance when ${\mathrm{FWER}}_J(c)=\underset{J^{\prime }\subseteq I}{max}{\mathrm{FWER}}_{J^{\prime }}(c)$ for all $c$ , in particular, for $J=I$ when ${\pi }_I>0$ . The proof of the theorem can be found in Appendix 2. Its conditions are weak: The strict monotonicity and differentiability of the strata-wise FWER are satisfied when the joint distribution of $Z_1,\dots ,Z_m$ is continuous. This is often the case, at least asymptotically.

We illustrate the approximation (6) with the examples in Figure 2, starting with the left side, i.e. two intersecting populations. In this case ${\mathrm{FWER}}_{\{1,2\}}(c)\ge {\mathrm{cPWER}}_{\{1,2\}}(c)={\tilde{\pi }}_{\{1\}}{\mathbb{P}}_{{\theta }^{*}}(Z_1\ge c)+{\tilde{\pi }}_{\{2\}}{\mathbb{P}}_{{\theta }^{*}}(Z_2\ge c)$ for all $c$ and any ${\tilde{\pi }}_{\{i\}}={\pi }_{\{i\}}/({\pi }_{\{1\}}+{\pi }_{\{2\}})$ and thereby the approximation ${\mathrm{FWER}}_{\{1,2\}}(d_{\{1,2\}}^{*})$ is an upper bound for the maximal strata-wise family wise error rate ${\mathrm{FWER}}_{\{1,2\}}(c^{*})$ . If $Z_1,Z_2$ have the same marginal null distribution, then $d_{\{1,2\}}^{*}$ is the $(1-\alpha )$ -quantile of this distribution, e.g. ${\mathrm{\Phi }}^{-1}(1-\alpha )$ when $Z_1,Z_2$ are standard normal under ${\theta }^{*}$ . The upper bound ${\mathrm{FWER}}_{\{1,2\}}(d_{\{1,2\}}^{*})$ then equals the FWER with two un-adjusted tests that are always below $2\alpha$ and even smaller if we account for the positive correlation between $Z_1$ and $Z_2$ .

For three nested populations (right side of Figure 2) the maximal strata-wise FWER is, according to the theorem, given by ${\mathrm{FWER}}_{\{1,2,3\}}(d^{*})$ where $d^{*}$ solves

$${cPWER}_{\{1,2,3\}}(d^{*})={\tilde{\pi }}_{\{1\}}{\mathbb{P}}_{{\theta }^{*}}(Z_1\ge d^{*})+{\tilde{\pi }}_{\{1,2\}}{\mathbb{P}}_{{\theta }^{*}}(max(Z_1,Z_2)\ge d^{*})=\alpha$$

with ${\tilde{\pi }}_J={\pi }_J/(1-{\pi }_{\{1,2,3\}})$ for $J=\{1\}$ and $J=\{1,2\}$ . The threshold $d^{*}$ depends on the relative prevalences ${\tilde{\pi }}_1$ and ${\tilde{\pi }}_{\{1,2\}}$ and the correlation between $Z_1$ and $Z_2$ , whereby the latter is determined by the sample sizes and the situation whether the same or different treatments are tested in the populations (see Section 5.2 below). For a numerical illustration, we assume the nominal level $\alpha =0.025$ , a single treatment and prevalences ${\pi }_{\{1\}}=0.75$ , ${\pi }_{\{1,2\}}=0.24$ and ${\pi }_{\{1,2,3\}}=0.01$ as well as standard normally distributed endpoints with test statistics $Z_j$ , $j=1,2,3$ . Assuming balanced treatment groups in each stratum and that the sample sizes perfectly match the population prevalences, we obtain the correlations $\mathrm{Corr}(Z_1,Z_2)=0.5$ , $\mathrm{Corr}(Z_1,Z_3)=0.1$ and $\mathrm{Corr}(Z_2,Z_3)=0.2$ . The critical value is then $d^{*}=2.037$ and the upper bound ${\mathrm{cPWER}}_{\{1,2,3\}}(d^{*})$ for the strata-wise FWER for ${\mathcal{P}}_{\{1,2,3\}}$ becomes $0.0572$ . Since this bound results from taking the limit for ${\pi }_{\{1,2,3\}}\downarrow 0$ (while keeping the sample sizes and complementary conditional prevalences fixed), the maximal strata-wise FWER with prevalence ${\pi }_{\{1,2,3\}}=0.01$ is somewhat smaller, namely $0.0566$ .

5 Comparison with FWER-controlling procedures

Due to the PWER being more liberal than the FWER, the next naturally arising question is how much this affects power and sample size. We will at first compare PWER-control with FWER-control for two intersecting sub-populations when (i) the same and (ii) two different treatments are investigated in each sub-population. Secondly, we will apply PWER-control to the multiple testing approach for umbrella trials considered in Sun et al. ¹² and compare it to the originally suggested FWER-control.

5.1 Combination of independent samples

We start with a hypothetical, but statistically simple situation. Assume that a treatment $T$ is investigated for two intersecting populations ${\mathcal{P}}_i$ , $i=1,2$ , that are defined by two different biomarkers. Assume further that a sponsor has decided to test the effect of $T$ for the two biomarker-positive groups in two disjoint samples within a single clinical trial. We consider here the one-sided hypotheses $H_i:{\theta }_i=\theta ({\mathcal{P}}_i,T)\le 0$ for the efficacy of $T$ in ${\mathcal{P}}_i$ against a control, where ${\theta }_i=({\pi }_{\{i\}}{\theta }_{\{i\}}+{\pi }_{\{1,2\}}{\theta }_{\{1,2\}})/{\pi }_i$ (with ${\pi }_i={\pi }_{\{i\}}+{\pi }_{\{1,2\}}$ ) is given as a weighted mean of the unknown effects in the respective strata ${\mathcal{P}}_{\{i\}}$ and ${\mathcal{P}}_{\{1,2\}}$ . Since the analysis of the two samples is done in a single study, regulatory authorities may require multiple testing adjustment. Let us assume that PWER-control is accepted as a compromise between control of the FWER and the unadjusted testing, the latter being the case when submitting two different studies. PWER-control bounds the overall probability for a future patient to be exposed to an inefficient treatment strategy.

Since the two treatment strategies $({\mathcal{P}}_i,T)$ , $i=1,2$ , are investigated in two independent samples, the corresponding test statistics $Z_i$ are stochastically independent. Let us further assume that both $Z_i$ are normally distributed with variance 1. The question is now, what we gain in terms of power by switching from FWER- to PWER-control. We will assume an overlap between the two populations ${\mathcal{P}}_1$ and ${\mathcal{P}}_2$ of probability ${\pi }_{\{1,2\}}$ that will be varied in our investigation.

Let $\mathrm{\Phi }$ and ${\mathrm{\Phi }}^{-1}$ be the standard normal distribution and quantile functions, respectively. By the independence of the test statistics, the $\mathrm{FWER}=1-\mathrm{\Phi }(c_F^{*}{)}^2$ is controlled at $\alpha$ by Šidák's critical value $c_F^{*}={\mathrm{\Phi }}^{-1}(\sqrt{1-\alpha })$ . Following the example in Section 2.2, the PWER is given by

$$\mathrm{PWER}=(1-{\pi }_{\{1,2\}})\{1-\mathrm{\Phi }(c_P^{*})\}+{\pi }_{\{1,2\}}\{1-\mathrm{\Phi }(c_P^{*}{)}^2\}$$

where $c_P^{*}$ is the critical value used for control of the PWER at level $\alpha$ . Note that ${\pi }_{\{1,2\}}$ determines how much multiplicity adjustment is needed for PWER-control. Solving $\mathrm{PWER}=\alpha$ yields

$$c_P^{*}={\mathrm{\Phi }}^{-1}\left(\frac{-(1-{\pi }_{\{1,2\}})+\sqrt{(1-{\pi }_{\{1,2\}}{)}^2+4{\pi }_{\{1,2\}}(1-\alpha )}}{2{\pi }_{\{1,2\}}}\right);$$

(7)see Appendix 3 for the derivation. For ${\pi }_{\{1,2\}}\downarrow 0$ this critical value decreases to ${\mathrm{\Phi }}^{-1}(1-\alpha )$ coinciding with the unadjusted case and for ${\pi }_{\{1,2\}}\uparrow 1$ we have $c_P^{*}\uparrow c_F^{*}$ .

To assess the power gain by using PWER- instead of FWER-control, we consider the factor of sample size increase with PWER- or FWER-control in comparison to the one with no multiplicity correction. Aiming for a marginal power of at least $1-\beta$ , the sample size for each population ${\mathcal{P}}_j$ has to be at least $n_c\ge ({\mathrm{\Phi }}^{-1}(1-\beta )+c{)}^2/{\delta }_j^2$ with critical value $c$ and non-centrality parameter ${\delta }_j$ in ${\mathcal{P}}_j$ . The fractions

$$q_{\alpha }(c):=\frac{n_c}{n_{{\mathrm{\Phi }}^{-1}(1-\alpha )}}={\left(\frac{{\mathrm{\Phi }}^{-1}(1-\beta )+c}{{\mathrm{\Phi }}^{-1}(1-\beta )+{\mathrm{\Phi }}^{-1}(1-\alpha )}\right)}^2\mathrm{for}c\in \{c_P^{*},c_F^{*}\}$$

(8)describe how much more sample size one would need for a marginal power of $1-\beta$ when the multiplicity adjustments are performed.

Figure 3 shows $q_{\alpha }(c)$ for $\alpha =0.025$ and $\beta =0.2$ depending on the size ${\pi }_{\{1,2\}}$ of ${\mathcal{P}}_1\cap {\mathcal{P}}_2$ when both populations are assumed to be of equal size. FWER-control requires an increase in sample size of about 21% while PWER-control requires considerably less depending on ${\pi }_{\{1,2\}}$ . The larger the intersection, the more patients are exposed under the global null hypothesis to two false rejections, therefore the critical value increases and the sample size increases as well. At ${\pi }_{\{1,2\}}=1$ , PWER and FWER coincide and so do the sample sizes. If, for instance, the intersection makes up $40\text{\%}$ of the union of the two populations only around $10\text{\%}$ sample size increase is needed when using PWER-control, less than half of what is necessary with FWER-control.

Figure 3.

Factor of sample size increase compared to the unadjusted case to achieve a marginal power of $1-\beta =80\text{\%}$ with PWER- and FWER-control at $\alpha =0.025$ in a combination of two independent studies with different but overlapping populations. PWER: population-wise error rate; FWER: family-wise error rate.

5.2 Testing population specific effects in one study.

We consider now a single study with two overlapping populations ${\mathcal{P}}_i$ , $i=1,2$ , for each of which a treatment $T_i$ is compared to a common control $C$ . We will investigate two possible scenarios, namely (i) $T_1\ne ̸T_2$ and (ii) $T_1=T_2$ . For simplicity, we assume again that both populations have the same size, i.e. ${\pi }_{\{1\}}={\pi }_{\{2\}}$ . We assume further that the data from each population are normally distributed with mean treatment difference ${\theta }_i$ and a common known variance ${\sigma }^2$ (across treatments and subgroups ${\mathcal{P}}_J$ , $J\subseteq \{1,2\}$ ) and z-tests are used to test $H_i:{\theta }_i\le 0$ . For $J\subseteq \{1,2\}$ , we denote by $n_J=N\cdot {\pi }_J$ the sample size in ${\mathcal{P}}_J$ and by $N=\sum _{J\subseteq \{1,2\}}{n}_J$ the overall total sample size.

(i) Unequal treatments. In scenario (i) we have to think of a way to randomize patients to either treatment or control. In the complements ${\mathcal{P}}_{\{i\}}$ we simply apply 1:1 randomization to treatment $T_i$ or control $C$ . In the intersection ${\mathcal{P}}_{\{1,2\}}$ we apply 1:1:1 randomization to the three groups $T_1,T_2$ and $C$ . By this we can assume that in ${\mathcal{P}}_{\{i\}}$ there are $n_{\{i\}}/2$ patients in the treatment and control group, whereas in the intersection there are $n_{\{1,2\}}/3$ patients in each group.

Obviously, this type of allocation leads to an inconsistency between the sample sizes and prevalences. Say ${\mathcal{P}}_1$ has a prevalence of ${\pi }_1={\pi }_{\{1\}}+{\pi }_{\{1,2\}}=100/170=0.59$ and of 100 patients in ${\mathcal{P}}_1$ , 70 belong to ${\mathcal{P}}_{\{1\}}$ and 30 to ${\mathcal{P}}_{\{1,2\}}$ . However, applying the above allocation rule implies that $35/45\approx 77.7\text{\%}$ of the patients sampled from ${\mathcal{P}}_1$ and assigned to treatment $T_1$ belong to ${\mathcal{P}}_{\{1\}}$ . This means that the proportions of the strata-wise sample sizes within a treatment group do not match their corresponding proportions in the population. Hence, the population-wise means must be estimated by a weighted sum of strata-wise means:

$${\hat{x}}_{i,G_i}=\left(\frac{{\pi }_{\{i\}}}{{\pi }_{\{i\}}+{\pi }_{\{1,2\}}}\right){\overline{x}}_{\{i\},G_i}+\left(\frac{{\pi }_{\{1,2\}}}{{\pi }_{\{i\}}+{\pi }_{\{1,2\}}}\right){\overline{x}}_{\{1,2\},G_i},G_i\in \{T_i,C\}$$

where ${\overline{x}}_{J,G_i}$ is the mean response in strata ${\mathcal{P}}_J$ , $J\subseteq \{1,2\}$ , under treatment $G_i$ . In the above example, we would need to compute ${\hat{x}}_{1,T_1}=0.7\cdot {\overline{x}}_{\{1\},T_1}+0.3\cdot {\overline{x}}_{\{1,2\},T_1}$ for treatment $T_1$ .

The $z$ -test statistic is finally given by $Z_i=({\hat{x}}_{i,T_i}-{\hat{x}}_{i,C})/\sqrt{\mathrm{Var}({\hat{x}}_{i,T_i}-{\hat{x}}_{i,C})}$ . Since in the intersection ${\mathcal{P}}_{\{1,2\}}$ the same control group is used for both test statistics, they are positively correlated. Assuming ${\pi }_{\{1\}}={\pi }_{\{2\}}$ , we obtain $\mathrm{Corr}(Z_1,Z_2)=(3/2){\pi }_{\{1,2\}}/(1+2{\pi }_{\{1,2\}})$ . The calculation of this correlation and an expression for the variance $\mathrm{Var}({\hat{x}}_{i,T_i}-{\hat{x}}_{i,C})$ can be found in Appendix 4.

(ii) Equal treatments. In scenario (ii), we investigate one and the same treatment $T_1=T_2=T$ in both populations and apply the 1:1 randomization to every stratum. By this we can use for $H_i$ the test statistic $Z_i=({\overline{x}}_{T_i}-{\overline{x}}_C)/(2\sigma /\sqrt{n_{\{i\}}+n_{\{1,2\}}})$ . Because we are using the same treatment in both populations, we expect a higher correlation between $Z_1$ and $Z_2$ . Indeed, for ${\pi }_{\{1\}}={\pi }_{\{2\}}$ the correlation is equal to $\mathrm{Corr}(Z_1,Z_2)=2{\pi }_{\{1,2\}}/(1+{\pi }_{\{1,2\}})$ which is greater or equal to the correlation with different treatments for all ${\pi }_{\{1,2\}}\in [0,1]$ ; see Appendix 4.

For both scenarios, we intend to find critical values to control the PWER and FWER, respectively. Following Section 2.2, the PWER under the global null is given by

$$\begin{array}{rl}\mathrm{PWER}& ={\pi }_{\{1\}}\mathbb{P}(\{Z_1\ge c_P^{*}\})+{\pi }_{\{2\}}\mathbb{P}(\{Z_2\ge c_P^{*}\})+{\pi }_{\{1,2\}}\mathbb{P}(\{Z_1\ge c_P^{*}\}\cup \{Z_2\ge c_P^{*}\})\\ & =(1-{\pi }_{\{1,2\}})\{1-\mathrm{\Phi }(c_P^{*})\}+{\pi }_{\{1,2\}}\{1-{\mathrm{\Phi }}_{\rho }(c_P^{*},c_P^{*})\}\end{array}$$

(9)with $c_P^{*}$ being the critical value that is to be found, and ${\mathrm{\Phi }}_{\rho }$ is the cumulative distribution function of the bivariate normal distribution with standard normal marginals and correlation $\rho$ . A univariate root finding algorithm can now be used to solve $\mathrm{PWER}=\alpha$ for $c_P^{*}$ .

As an example, suppose we are in scenario (i) (multiple treatments) with ${\pi }_{\{1\}}={\pi }_{\{2\}}=0.4$ , ${\pi }_{\{1,2\}}=0.2$ , $\beta =0.2$ and $\alpha =0.025$ . Then we have $\rho =\mathrm{Corr}(Z_1,Z_2)\approx 0.01$ . We solve $\mathrm{FWER}=1-{\mathrm{\Phi }}_{\rho }(c_F^{*},c_F^{*})=\alpha$ to obtain $c_F^{*}\approx 2.23$ and $\mathrm{PWER}=\alpha$ to obtain $c_P^{*}\approx 2.03$ . Using (8), this yields a sample size increase of around 20% for the FWER and only an increase of 5% for the PWER.

Figure 4 shows graphs of sample size increases for both scenarios and both types of multiple error control in dependence of ${\pi }_{\{1,2\}}$ . At ${\pi }_{\{1,2\}}=0$ (disjoint populations), for instance, the PWER-approach yields no sample size increase, where the FWER-based method yields an increase of over $20\text{\%}$ . With increasing intersection size the difference between sample sizes for PWER and FWER-control declines until both values fall together at ${\pi }_{\{1,2\}}=1$ where the PWER is equal to the FWER.

Figure 4.

Factor of sample size increase compared to the unadjusted case for FWER- and PWER-control at $\alpha =0.025$ in a single study with two overlapping populations depending on the size of the intersection ${\pi }_{\{1,2\}}$ . The Left panel is for scenario (i) with different experimental treatments and a common control; the right panel is for scenario (ii) with equal experimental treatments. The power is $1-\beta =80\text{\%}$ in both scenarios. PWER: population-wise error rate; FWER: family-wise error rate.

For the PWER, this graphic also illustrates that the correlation of the test statistics and the degree of adjustments needed to correct for multiplicity behave like opposing ’forces’. At ${\pi }_{\{1,2\}}=0$ the test statistics are uncorrelated, but there is no need to adjust for multiplicity with PWER-control. At ${\pi }_{\{1,2\}}=1$ there is only one population, so the correlation is 1 which implies again that no multiplicity adjustment is needed, although we are formally testing two hypotheses for everyone. For ${\pi }_{\{1,2\}}$ between 0 and 1 we obtain the maximum for the PWER and corresponding sample size increase. Mathematically, this can be seen by rewriting the PWER as $1-\mathrm{\Phi }(c^{*})+{\pi }_{\{1,2\}}\{\mathrm{\Phi }(c^{*})-{\mathrm{\Phi }}_{\rho }(c^{*},c^{*})\}$ . For fixed $c^{*}$ , only the second term depends on ${\pi }_{\{1,2\}}$ . It is the product of two non-negative factors, ${\pi }_{\{1,2\}}$ and $\mathrm{\Phi }(c^{*})-{\mathrm{\Phi }}_{\rho }(c^{*},c^{*})$ ), where the first increases from 0 to 1 and the second decreases from 1 to 0.

5.3 Estimation of population prevalences

In clinical practice, the assumption of known prevalences ${\pi }_J$ is rarely justified and it is natural to ask whether the replacement of ${\pi }_J$ by an estimator ${\hat{\pi }}_J$ will significantly inflate the PWER. A suitable choice for ${\hat{\pi }}_J$ is the maximum likelihood estimator (MLE) from the multinomial distribution of $(n_J{)}_{J\subseteq I}$ . Using these estimates instead of the true prevalences, we compute the critical value $c^{*}$ by solving $PWER=\alpha$ . This guarantees asymptotic control of the PWER, since $({\hat{\pi }}_J{)}_{J\subseteq I}$ is consistent and the joint distribution of the test statistics used in the calculation of $c^{*}$ is conditional on $(n_J{)}_{J\subseteq I}$ .

We examine the PWER by means of scenarios (i) and (ii) of Section 5.2. For each constellation of true prevalences, we generate sample size vectors $({\hat{n}}_J{)}_{J\subseteq I}$ from the corresponding multinomial distribution and computed the MLEs $({\hat{\pi }}_J{)}_{J\subseteq I}$ . To see by how much the true PWER is inflated, the probabilities for a type I error for each sub-population ${\mathcal{P}}_J$ are computed by using the ‘estimated’ critical value and the conditional correlation structure of the involved test statistics. By weighting each of these probabilities by their respective true population prevalence ${\pi }_J$ , we obtain the true PWER for the given ‘estimated’ critical value. This procedure was repeated 10.000 times and the mean of each true PWER was taken as approximation of the actual overall PWER. Figure 5 shows contour plots of this approximation of the overall PWER for scenarios (i) and (ii) and $N=50$ and $N=100$ , respectively. The plots indicate that the target PWER of $0.025$ may be missed only slightly, even for $N=50$ . More precisely, the mean true PWER values range from around $2.49\cdot {10}^{-2}$ to at worst $2.51\cdot {10}^{-2}$ where the largest standard error we observed is at around $1.2\cdot {10}^{-5}$ , which all in all displays a fairly negligible deviation from 0.025. To be fair, however, this also implies that the standard deviation of a single true PWER amounts to $1.2\cdot {10}^{-3}$ , meaning that values of around 0.026 are quite possible.

Figure 5.

Contour plots of the actual overall PWER when using ML-estimates ${\hat{\pi }}_J$ for the prevalences ${\pi }_J$ in the determination of the critical value $c_P^{*}$ at level $\alpha =0.025$ . The first row corresponds to scenario (i) and the second row to scenario (ii) from Section 5.2. Because of ${\pi }_{\{1\}}+{\pi }_{\{2\}}\le 1$ , the contour plots are restricted to the lower left rectangle of the squares. PWER: population-wise error rate; ML: maximum likelihood.

In the case of a small relative population size ${\pi }_J$ it might be that, by chance, no patient is recruited in this group. This would mean that we would not account for all the multiplicity of these patients completely. If such an intersection cannot be excluded theoretically from the inclusion and exclusion criteria or due to medical arguments, we could introduce a small minimal number ${\pi }_{min}$ for all ${\pi }_J$ in order to be more conservative. Also, different approaches like shrinkage methods or Bayesian estimation of the ${\pi }_J$ are conceivable options in future research.

Lastly, note that we assumed here that the patient population of the trial is representative of the respective future patient populations. If this cannot be assumed, one could take prevalence estimates from more representative studies. If these estimates are based on sufficiently large sample sizes, the control of the PWER should be achieved similarly or better than with the trial data if the representative data are larger.

5.4 Multiple testing approaches for umbrella trials

We consider now a multiple testing approach for umbrella trials suggested in Sun et al. ¹² and investigate the gain in power by switching from FWER- to PWER-control. Following Sun et al., ¹² we assume $l$ disjoint population strata, which are denoted here by ${\mathcal{S}}_1,\dots ,{\mathcal{S}}_l$ and which are subsets of a larger and more general patient population $\mathcal{P}={\cup }_{i=1}^l{\mathcal{S}}_i$ from which we randomly sample. In each stratum a specific experimental treatment $E_i$ shall be compared to a control treatment $C$ . For simplicity, we assume that each population has the relative population prevalence ${\pi }_i$ which is assumed to equal $n_i/N$ , where $n_i$ is the number of patients in ${\mathcal{S}}_i$ and $N$ is the total number of patients in the sample. This holds in practice at least approximately; see also Section 5.3.

With only small $n_i$ , the establishment of a treatment effect in the individual strata is difficult and impossible to achieve with sufficient power. Therefore, study designs have been suggested that compare $C$ with the global treatment strategy $E$ which assigns treatment $E_i$ to population strata ${\mathcal{S}}_i$ . Application examples for such trial designs where a pooled comparison of the global strategy $E$ to $C$ has been chosen for the primary analysis (with the strata-wise comparisons as secondary analyses only) are Hill et al. ¹⁵ and Owadally et al. ¹⁶ Such an overall comparison of the strategy $E$ with $C$ utilizes the total sample size $N$ and does also not require multiple testing. However, it does not permit a claim for a sub-population when the effect of $E$ is heterogeneous. To improve the approach, Sun et al. ¹² suggest to test all sub-strategies $E^S$ , $S\subseteq \{1,\dots ,l\}$ , that consider only the union ${\mathcal{P}}^S={\cup }_{i\in S}{\mathcal{S}}_i$ with treatment assignments as in $E$ , against the control in ${\mathcal{P}}^S$ . This permits claims also for sub-populations and thereby increases the possibility for efficacy conclusions. Of course, such testing requires an adjustment for multiplicity. Sun et al. ¹² provide a (single-step) procedure that controls the FWER. For the formal description of the procedures, let $\theta =({\theta }_1,\cdots ,{\theta }_l)$ be the vector of unknown treatment effects (mean differences) in the populations, and consider for each $S\subseteq \{1,\dots ,l\}$ the average treatment effect in ${\mathcal{P}}^S$ :

$${\theta }^S=\sum _{i\in S}({\pi }_i/{\pi }^S){\theta }_i$$

with ${\pi }^S=\sum _{i\in S}{\pi }_i$ the relative prevalence of ${\mathcal{P}}^S$ . Sun et al. ¹² assume the linear model

$$Y_{ij}={\mu }_i+{\theta }_i{X}_{ij}+{\epsilon }_{ij}$$

(10)where $X_{ij}$ denotes the treatment indicator for patient $j$ in group $i$ which equals 1 if assigned to the experimental treatment $E_i$ and otherwise $0$ , and ${\theta }_i$ is the treatment effect of $E_i$ in population ${\mathcal{S}}_i$ . The error terms ${\epsilon }_{ij}$ are assumed to be i.i.d. normally distributed with mean $0$ and homogeneous variance ${\sigma }^2$ . As mentioned above, the authors suggest to test

$$H^S:{\theta }^S\le 0\text{vs.}{K}^S:{\theta }^S>0\mathrm{for}\mathrm{all}S\subseteq L=\{1,\dots ,l\}$$

(11)Note that the ${\mathcal{P}}^S$ and $H^S$ , $S\subseteq L$ , correspond to the ${\mathcal{P}}_i$ and $H_i$ , $i\in I$ , Section 2 and 3.

From the least squares estimate of the linear model, we obtain one-sided $t$ -test statistics $T^S$ for testing $H^S$ for each $S\subseteq L$ . In order to control the FWER, Sun et al. ¹² conduct a single-step procedure that compares each $T^S$ with the upper $\alpha$ -quantile $c_F^{*}$ of the distribution of $max\{T^S|S\subseteq L\}$ under the global null hypothesis, i.e. the assumption that none of the treatments $E_i$ is superior to the control. We finally select the subset $S_F^{*}\subseteq L$ for which a positive treatment effect is claimed and that yields the largest value of $T^S$ ,

$$\begin{array}{r}{S}_F^{*}=\{\begin{array}{cc}\arg {\max }_{S\subseteq I}{T}^S,& \mathrm{if}max\{T^S|S\subseteq L\}>c_F^{*}\\ \mathrm{\varnothing },& \mathrm{else}\end{array}\end{array}$$

(12)To achieve PWER-control at the same level $\alpha$ , we determine the critical value $c_{\mathrm{P}}^{*}$ such that $PWER=\alpha$ holds under the global null hypotheses. While ${\mathcal{S}}_1,\cdots ,{\mathcal{S}}_l$ are disjoint, some of their unions ${\mathcal{P}}^S$ overlap. Since not all ${\mathcal{P}}^S$ overlap, the FWER corrects the multiple type I error rate for cases that cannot occur (similar to example 3) and hence may be viewed as overly conservative.

The PWER under the global null hypothesis ( $\theta =\mathbf{0}=(0,\dots ,0)$ ) is given by

$${\mathrm{PWER}}_{\mathbf{0}}=\sum _{i=1}^l{\pi }_i{\mathbb{P}}_{\mathbf{0}}\left(\bigcup _{S\ni i}\{T^S\ge c_P^{*}\}\right)$$

(13)where ‘ $S\ni i$ ’ denotes all $S\subseteq L$ that contain the index $i$ . This is because population ${\mathcal{S}}_i$ is affected by a type I error whenever a hypothesis $H^S$ is erroneously rejected that corresponds to a population ${\mathcal{P}}^S$ for which $i\in S$ (or ${\mathcal{S}}_i\subseteq {\mathcal{P}}^S$ ).

Due to the assumption of a homogeneous residual variance and the $2l$ mean parameter in the linear model (10), $\{T^S{\}}_{S\subseteq L}$ follows a joint $t$ -distribution with $N-2l$ degrees of freedom. In R, the distribution function of the multivariate $t$ -distribution is implemented in the mvtnorm-package (see Genz et al. ¹⁷ ) via the function pmvt which needs the degrees of freedom and the correlation matrix of the test statistics as input (see e.g. Bretz et al. ¹⁸ ). The correlation matrix can be computed using the contrast matrix and the design matrix of the linear model. Probabilities in (13) are then calculated by choosing the appropriate sub-matrices of the correlation matrix. Thus, for known values of ${\pi }_i$ , $i\in L$ , and $l$ , we can numerically determine the critical value $c_P^{*}$ such that $PWER=\alpha$ .

We know that $c_F^{*}>c_P^{*}$ , which implies that whenever the FWER-approach selects a non-empty $S_F^{*}$ , the same set is selected by the PWER-approach, $S_P^{*}=S_F^{*}$ . We may, however, select the empty set with the FWER-approach, $S_F^{*}=\mathrm{\varnothing }$ , while $S_P^{*}\ne ̸\mathrm{\varnothing }$ .

Performance measures. Sun et al. ¹² examined several quality and performance measures to assess how good a selected subset $S^{*}$ is. For example, they considered the average effect in the overall population when applying treatment strategy $E^{S^{*}}$ in ${\mathcal{P}}^{S^{*}}$ and the control in the rest of the population. We will consider the relative quantity $\mathrm{RAE}=100\mathbb{E}(\sum _{i\in S^{*}}{\pi }_i{\theta }_i)/{\theta }_{\mathrm{overall}}$ where $\mathbb{E}$ is the expectation with respect to the sample distribution and ${\theta }_{\mathrm{overall}}$ is the weighted average of the positive treatment effects,

$${\theta }_{\mathrm{overall}}=\sum _{i\in L_{+}}{\pi }_i{\theta }_i/\sum _{i\in L_{+}}{\pi }_i\mathrm{for}{L}_{+}=\{i=1,\dots ,l:{\theta }_i>0\}$$

that describes how efficient the experimental treatment strategy $E$ is for the union of sub-populations that benefit from $E$ . Since the PWER-procedure chooses a non-empty $S^{*}$ more often as the FWER-procedure, this quantity will always be larger for the PWER-approach.

In addition to this measure we will investigate the average size of the ‘correctly’ chosen subgroups within the selected ones, i.e. the average of ${\pi }^{S_{+}^{*}}/{\pi }^{S^{*}}$ where $S_{+}^{*}=\{i\in S^{*}|{\theta }_i>0\}$ and ${\pi }^{S_{+}^{*}}=\sum _{i\in S_{+}^{*}}{\pi }_i$ . This gives the fraction of the patient cohort that benefits from the experimental treatment strategy within the one that is exposed to $E^{S^{*}}$ by the results of the study. Analogously, we are interested in the average of the relative size of the ‘falsely’ chosen subgroups within the chosen ones: ${\pi }^{S_0^{*}}/{\pi }^{S^{*}}$ with $S_0^{*}=\{i\in S^{*}|{\theta }_i\le 0\}$ . Lastly, we consider the probability of rejecting at least one false null hypothesis,

$$\mathrm{Power}=\mathbb{P}(\mathrm{reject}\mathrm{any}{H}^S\mathrm{with}{\theta }^S>0,S\subseteq L),$$

as a way to measure the power of the procedures.

Design of the simulation. To make our results comparable to those of Sun et al., ¹² we conducted simulations with roughly the same parameters. That is, for the cases of $l=2,4,6$ sub-populations and a significance level $\alpha =0.025$ , we chose a total sample size of $N=1056$ and assume that all group-specific intercepts ${\mu }_i$ are equal to 0. Also, for simplicity, each group is assumed to be of equal size, i.e. ${\pi }_1=\cdots ={\pi }_l$ .

As in Sun et al., ¹² we assume non-negative effects ${\theta }_i\ge 0$ and choose $\theta =({\theta }_1,\cdots ,{\theta }_l)$ based on the number of subgroups $l$ and three further characteristics. The first one is the percentage of true null hypothesis: $q=l_0/l$ with $l_0$ the size of $L_0=\{i=1,\dots ,l:{\theta }_i=0\}$ . The second one characterizes the treatment effect heterogeneity and is defined as

$$\tau =({\theta }_{\max }-{\theta }_{\min })/({\theta }_{\max }+{\theta }_{\min })$$

where ${\theta }_{\max }=\underset{i\in L_{+}}{max}{\theta }_i$ and ${\theta }_{\min }={\min }_{i\in L_{+}}{\theta }_i$ . Note that $\tau$ equals the relative half-range of the positive ${\theta }_i$ ’s, i.e. half of their range divided by the average of their extremes. Obviously, a large $\tau$ means a large heterogeneity between the positive ${\theta }_i$ . The third one is the weighted average ${\theta }_{\mathrm{overall}}$ as previously introduced.

Given values for $l$ , $q$ , ${\theta }_{\mathrm{overall}}$ and $\tau$ one finds a gird of $l$ equidistant points such that the three characteristics are met. One easily verifies that this grid is uniquely determined by the four quantities. Following Sun et al. ¹² we chose $q$ such that $q\cdot l$ is always an integer. Note that for $q\ge (l-1)/l$ there is at most one ${\theta }_i\ne 0$ and so $\tau =0$ (no heterogeneity) is the only possible value for $\tau$ . Moreover, we assume ${\theta }_{\mathrm{overall}}=0.1$ in all simulations.

Results. The simulation results for $l=2$ and $4$ are given in Table 1 and for $l=6$ and $8$ in Appendix 5. On can see from the tables that control of the PWER, in comparison to FWER-control, provides a substantially larger power and larger average proportion of ‘correctly’ chosen subgroups and a larger average effect. It also increases the proportion of ‘falsely’ chosen subgroups. This is because a subgroup is selected more frequently with PWER-control.

Table 1.

Simulation results for $l=2$ and $l=4$ , assuming ${\theta }_{\mathrm{overall}}=0.1$ .

		Power	Correct	False	RAE	Power	Correct	False	RAE
	$l=2$	$q=0$
$\tau =0$	PWER	36.4	36.4	0	29
	FWER	31.0	31.0	0	25
$\tau =0.4$	PWER	40.4	40.4	0	33
	FWER	34.6	34.6	0	28
$\tau =0.8$	PWER	51.2	51.2	0	47
	FWER	45.2	45.2	0	42
	$l=2$	$q=1/2$				$q=1$
$\tau =0$	PWER	57.7	52.9	4.8	58	0	0	36	0
	FWER	52.0	47.8	4.2	52	0	0	24	0
	$l=4$	$q=0$			$q=1/4$
$\tau =0$	PWER	36.2	36.2	0	23	42.2	38.8	3.5	30
	FWER	27.4	27.4	0	17	32.7	30.1	2.6	24
$\tau =0.4$	PWER	37.9	37.9	0	25	44.8	41.3	3.6	33
	FWER	29.1	29.1	0	19	35.5	32.7	2.7	26
$\tau =0.8$	PWER	43.0	43.0	0	30	52.7	48.9	3.7	42
	FWER	33.7	33.7	0	24	43.2	40.2	3.0	35
	$l=4$	$q=2/4$
$\tau =0$	PWER	53.2	45.7	7.6	46
	FWER	43.8	37.8	6.1	38
$\tau =0.4$	PWER	58.8	51.1	7.8	50
	FWER	49.5	43.1	6.4	42
$\tau =0.8$	PWER	73.9	65.9	8.0	68
	FWER	65.3	58.5	6.8	60
	$l=4$	$q=3/4$			$q=1$
$\tau =0$	PWER	81.5	70.1	11.5	81	0	0	4.2	0
	FWER	75.1	64.9	10.2	75	0	0	2.4	0

Results for power (%), the percentage of correctly and falsely chosen sub-populations and the RAE for PWER- and FWER-control under parameter configurations $\theta =({\theta }_1,\cdots ,{\theta }_l)$ that depend on the fraction of true null hypotheses $q$ and the relative half-range $\tau$ of the positive ${\theta }_i$ ‘s. RAE: relative average effect; FWER: family-wise error rate; PWER: population-wise error rate.

While the proportion of ‘falsely’ chosen subgroups is increased by at most 2.2% (percentage points) and remains below 5% (one-sided), the proportion of ‘correctly’ chosen subgroups (among the selected ones) and the power are increased by up to 10% and often by more than 5%. The expected effect RAE is always larger with PWER-control.

Under the global null hypothesis ( $P=1$ ) the average proportion of ‘falsely’ selected populations equals by theory the one-sided FWER. With PWER-control at level 2.5% the FWER was found to be between 3.6% and 4.5% for $l=2,4,6,8$ . Note that the average proportion of ‘falsely’ selected populations exceeds the level of 2.5% (sometimes substantially) also with FWER-control when there is an effect in some but not all population strata.

In summary, we see that control of PWER substantially increases the chance for a delivery of efficient treatments while the risk of receiving an inefficient treatment and the percentage of patients that do not benefit from the treatment decisions is increased to a moderate extent and remains comparable to the procedure with FWER-control.

6 Extension to SCIs

We are coming back to the general set-up of Section 2 and 3. Utilizing the duality between (multiple) hypothesis tests and (simultaneous) confidence intervals, the multiple test procedure with control of the PWER, introduced in Section 3, can be extended to confidence intervals for the efficacy parameter ${\theta }_i=\theta ({\mathcal{P}}_i,T_i)$ , $i=1,\dots ,m$ . In this section, we will introduce the dual SCIs and discuss their coverage properties.

To introduce the confidence intervals, let $\delta =({\delta }_1,\dots ,{\delta }_m)$ be a vector of possible values for $\theta =({\theta }_1,\dots ,{\theta }_m)$ and consider the corresponding null hypotheses $H_i^{{\delta }_i}:{\theta }_i={\delta }_i$ , $i=1,\dots ,m$ . Assume further that $T_i^{{\delta }_i},i=1,\dots ,m$ , are (asymptotically) pivotal test statistics for $H_i^{{\delta }_i}$ , i.e., the (asymptotic) joint distribution of $(T_1^{{\delta }_1},\dots ,T_m^{{\delta }_m})$ under $\theta =\delta$ is the same for all $\delta$ . If $T_i^{{\delta }_i}$ decreases in ${\delta }_i$ for the given data, then it makes sense to form the one-sided intervals ${\mathcal{C}}_i=[{\tilde{\theta }}_i,\mathrm{\infty }[$ with the lower bound

$${\tilde{\theta }}_i:=\min \{{\delta }_i:T_i^{{\delta }_i}\le c^{*}\}$$

(14)where $c^{*}$ is the critical value defined in (3) for ${\theta }^{*}=\delta$ . Because $(T_1^{{\delta }_1},\dots ,T_m^{{\delta }_m})$ is pivotal, the critical value $c^{*}$ is independent from $\delta$ . The monotonicity of $T_i^{{\delta }_i}$ applies to most (one-sided) tests and is satisfied e.g. for Wald-type test statistics $T_i^{{\delta }_i}=({\hat{\theta }}_i-{\delta }_i)/S{E}_i$ where ${\hat{\theta }}_i$ is an estimate of ${\theta }_i$ (e.g. the MLE) with an standard error $S{E}_i$ that is independent of the parameter value $\delta$ . In this case, we obtain ${\tilde{\theta }}_i={\hat{\theta }}_i-c^{*}S{E}_i$ .

Upper confidence bounds can be derived by applying the same principle and two-sided confidence intervals are obtained by the intersection of the two one-sided intervals. With Wald-type dual tests we obtain the two-sided intervals ${\mathcal{C}}_i=[{\hat{\theta }}_i-c^{*}S{E}_i,{\hat{\theta }}_i+c^{*}S{E}_i]$ .

We finally discuss the coverage properties of the above-introduced confidence bounds and intervals. We start with the lower confidence bounds ${\tilde{\theta }}_i$ . To this end, consider a patient $P$ that is randomly drawn from $\mathcal{P}$ and let $I_P$ be the set of indices of the sub-populations ${\mathcal{P}}_i$ the patient $P$ belongs to, i.e. $I_P=\{i:P\in {\mathcal{P}}_i\}$ . The set $I_P$ gives all population efficacy parameter ${\theta }_i$ , $i\in I_P$ , that are relevant for patient $P$ . Note that $I_P$ is a random set because $P$ is randomly drawn from $\mathcal{P}$ . If ${\theta }_i$ is the true unknown efficacy parameter, then by the definition (14) we get ${\tilde{\theta }}_i>{\theta }_i$ if and only if $T_i^{{\theta }_i}>c^{*}$ . Since the dual tests for $H_1^{{\theta }_1},\dots ,H_m^{{\theta }_m}$ control the PWER, the (simultaneous) probability that any of the lower confidence bounds ${\tilde{\theta }}_j$ , $j\in I_P$ , fall above the true ${\theta }_j$ is at most $\alpha$ . This gives the coverage property

$${\mathbb{P}}_{\theta }({\tilde{\theta }}_j\le {\theta }_j\text{ for all }j\in I_P)\ge 1-\alpha$$

(15)meaning that with a probability of at most $1-\alpha$ , for a randomly chosen patient $P$ , the lower confidence intervals $[{\tilde{\theta }}_j,\mathrm{\infty }$ $[,j=1,\dots ,m$ , cover all true ${\theta }_j=\theta ({\mathcal{P}}_j,T_j)$ that are relevant to this patient. Because, $I_P=J$ if and only if $P\in {\mathcal{P}}_J={\cap }_{j\in J}{\mathcal{P}}_j\setminus \bigcup _{k\in I\setminus J}{\mathcal{P}}_k$ we can write the coverage probability as

$$\sum _{J\subseteq I}{\pi }_J{\mathbb{P}}_{\theta }({\tilde{\theta }}_j\le {\theta }_j\text{ for all }j\in J)$$

Hence, equation (15) means to control a kind of average simultaneous coverage probability where we focus in each stratum on the relevant confidence statements and average the strata-wise coverage probability over the entire population $\mathcal{P}$ .

The upper confidence bounds and two-sided confidence intervals control the same type of average simultaneous coverage probability. As for the classical confidence intervals, the two-sided interval has a twice as large non-coverage probability as the one-sided intervals.

7 Discussion

This paper introduces a new multiple type I error rate concept for clinical trials with multiple and possibly intersecting populations that permits for more powerful tests than control of the FWER. It relies on the observation that not all patients are affected by all test decisions, since not all hypotheses concern all population strata. By averaging the individually relevant, multiple type I errors over the entire population, it provides control of the probability that a randomly selected future patient will be exposed to an inefficient treatment strategy. This average multiple type I error rate, which we call the population-wise error rate (PWER), is smaller than the maximal FWER a patient strata is exposed to. In Section 4, we have discussed several bounds for the strata-wise FWER that follow from control of the PWER and illustrated them with numerical examples. Hence, control of the PWER guarantees control of the maximal strata-wise FWER on a larger and only indirectly defined level.

Let us recall that we only consider population-wise claims, i.e. claims on treatment strategies that consist of a treatment and a population the treatment is intended for and for which the average treatment effect is the estimand of interest. This is also the case when aiming for FWER-control. No individual efficacy claims are anticipated here. Error control of patient-wise claims is impossible without sacrificing power or making strong assumptions. However, a population-wise claim can be viewed as a proxy or approximation for individual claims in the target population. ²⁶ Test results from more than a single population may be used for a more informed individual decision. Depending on the inconsistency of the efficacy estimates across the individual strata, we may refrain from taking a formal rejection for a claim in the corresponding population. Hence, with PWER-control, we consider the worst-case scenario, where an efficacy claim for a treatment strategy will always lead to an application of the treatment to all patients in the target population. Note that we do not account for a potential off-label use where a treatment is applied to patients outside its target population.

We have presented a simple approach for achieving PWER-control by adjustment of critical values and have illustrated the power gain when passing from FWER to PWER in a number of examples. We have mainly considered the simple situation of multivariate normal distributed test statistics. This situation applies at least asymptotically to a large number of hypothesis tests for which PWER-control is then guaranteed asymptotically. The methods and principle introduced here can also be implemented with other finite sample distributions like e.g. the multivariate $t$ -distribution (as done in Section 5.4) or be improved via resampling methods. Variance heterogeneity across populations is a general issue for trials with multiple populations that applies similarly to procedures with FWER-control (see e.g. Placzek and Friede ²⁴ ). One can say, whenever control of the FWER is possible then control of the PWER is possible as well, since the latter just controls an average of FWERs. We have also extended the suggested multiple test to SCIs and showed that these intervals control, for a randomly chosen patient, the probability of a simultaneously correct statement on the parameters that are relevant for the corresponding stratum the patient belongs to.

One referee asked at which level the PWER should be controlled. We would suggest to control the PWER at the one-sided level of 2.5% which is usually used for FWER-control, because the PWER has an interpretation that is close to the FWER, namely represents the risk to expose a future patient to an inefficient treatment strategy. Of course, this choice of $\alpha$ is as arbitrary as the choice is for FWER-control and there might be reasons for choosing another level, e.g. in phase II studies.

Control of the PWER requires knowledge of the relative prevalences of all disjoint population strata. These may either be obtained from previous studies or may be estimated at the end of the study. This complicates PWER-control. We have illustrated in an example with two populations that the estimation of the prevalences does not strongly harm PWER-control even with moderate sample sizes. However, more examples with more hypotheses are required to fully explore this issue. At least, PWER-control is always guaranteed asymptotically.

Since our procedure simply results in an adjustment of critical values, power calculations and power simulations deviate only minimally from approaches for classical multiple tests, except for the fact that the critical values may depend on the sample via the prevalence estimates. This can be resolved by using a priori estimates of the prevalences based on experience and past studies. The same issues arises from the estimation of the correlation structure of the test statistics used for an efficient PWER and FWER-control. A miss-specification of the prevalences may be corrected in a mid-trial blinded sample size review. ¹⁹

In Section 3, we have suggested a single-step procedure to control the PWER and one might ask whether this procedure can be uniformly improved by a step-down test because this is the case for single-step tests with FWER-control (e.g. Dmitrienko et al. ²⁰ ). For instance, in Example 2.2 with two intersecting hypotheses, we may ask whether we can test $H_2$ with a smaller critical $c_2^{*}<c^{*}$ when $H_1$ has already been rejected with critical value $c^{*}$ . One can quickly see that this is not possible. To this end assume that both hypotheses $H_1$ and $H_2$ are true. Rejection of $H_2$ when $Z_2\ge c^{*}$ or $Z_2\ge c_2^{*}$ with $Z_1\ge c^{*}$ , obviously increases the second and third terms in equation (2) of the PWER. Since we have chosen $c^{*}$ to be the smallest critical value that satisfies (3), which leads to a PWER equal to $\alpha$ with continuously distributed $Z_i$ (a generic and common situation), we do not control the PWER for any $c_2^{*}<c^{*}$ . We may define PWER-controlling step-down tests with an enlarged $c^{*}$ in order to mimic and improve step-down tests with FWER-control. However, such procedures do not uniformly improve the single-step test with PWER-control and are therefore beyond the scope of this paper. The development of step-down tests with PWER-control is a topic for future research.

Single-step procedures have the advantage that they can directly be extended by simple and always informative SCIs. We have illustrated this in Section 6 for single-step tests with PWER-control. An extension to simple and always informative SCIs is impossible for step-down tests: Compatible SCIs often are non-informative in the sense that they do not provide any additional information to the sheer hypothesis tests,^21,22 and sufficiently informative SCIs are compatible only to a modification of the original step-down test. ²³ This justifies the use of single-step tests in practice.

We finally remark that an extension of the presented PWER-approach to multi-stage and adaptive designs is under development by the authors and will be a topic of future contributions. Multi-stage and particularly flexible designs provide the opportunity for adding or dropping populations at interim analyses based on the unblinded interim data (e.g. Brannath et al., ⁴ Wassmer and Brannath, ⁶ Placzek and Friede ²⁴ ). In the example of Section 2.2, we may for instance add and enrich the intersection of the two populations for an investigation in a second stage of the study if the efficacy of the treatment is seen at interim only in one of the two populations. Hence, the development of adaptive and sequential designs with PWER-control is an interesting and valuable research task. It has also the potential to provide a valuable contribution to platform trials for which FWER-control has been discussed (e.g. Stallard et al., ⁷ Collignon et al. ⁹ ) and alternatives (like FDR-control) have been suggested rather recently (e.g. Zehetmayer et al., ²⁵ Robertson et al. ²⁶ ).