Controlling Multiple F Test Errors with an Overall F Test

Abstract

This article presents a theoretical extension of the problem associated with multiple comparisons among means-the increasing rate of false rejection errors-to multiple F tests of effects in multifactor ANOVAs and regression analyses. The authors empirically confirmed the prediction of such errors through an experiment in which 32% of 100 random computer-generated three-factor ANOVAs had one or more false rejections for the usual seven F tests for main and interaction effects (30% were expected), yet only 6% of the overall F ratios-which tested all effects simultaneously-were falsely declared significant (5% were expected). In contrast, the Bonferroni procedure produced false rejections in 11% of the ANOVAs. The authors conclude that researchers can control the predictable rates of false rejection of errors inherent in multifactor analyses by extending the Fisher protection procedure and requiring the overall F to be significant. The article concludes with hypothetical experiments illustrating how researchers can use the procedure.