Closed testing of each group versus the others combined in a multiple group analysis

Abstract

Background:

Many studies, such as a study of comparative effectiveness, entail a comparison of the beneficial and adverse effects of multiple K $>$ 2 competing therapies. Often, the analysis consists of a comparison of the K groups using an omnibus (T²-like) test for any difference among the groups followed by pairwise comparisons with adjustments for multiple tests.

Methods:

We evaluate the properties of an analysis strategy in which each group is compared to the average of the others in hopes of establishing the overall superiority (or harm) of at least one of the therapies. Testing of one-versus-others can be accomplished for virtually any model using simple tests, and the type I error probability $\alpha$ can be controlled by conducting such tests under the closed testing principle. Testing using linear models, the family of generalized linear models, and Cox proportional hazards models is described with examples.

Results:

Since each tested hypothesis compares one treatment to the average of the others, the K-level null hypothesis in the tree of closed testing is equivalent to any of the (K−1)-level tests, thus reducing the number of tests required. This applies to linear, generalized linear, and Cox proportional hazards models. While the Bonferroni, Holm, and Hommel procedures preserve the desired level $\alpha$ , all are conservative relative to closed one-versus-others testing and closed testing in general provides greater power.

Conclusion:

Testing each of the multiple treatments versus the average of the others is readily and efficiently conducted under the closed testing principle and may be especially useful in the assessment of studies of comparative effectiveness.