The Use of an Overall F Test to Control Type I Error Rates in Factorial Analyses of Variance: Limitations and Better Strategies

Abstract

The inflation of Type I error rates caused by the testing of multiple null hypotheses in factorial analyses of variance (ANOVAs) is a problem that is often not recognized in the behavioral sciences. Fletcher, Daw, and Young (1989) described the problem and conducted a limited simulation study to investigate the effectiveness of two strategies to correct the problem: use of an overall F test and use of a Bonferroni adjustment. Unfortunately, two limitations in the design of their simulation led these authors to conclusions about the overall F test that do not hold under all conditions. The present study was designed to overcome these limitations and to provide a more complete evaluation of such strategies. Our results indicated that the overall F test is effective only when all effects in the ANOVA are null. In contrast, the Bonferroni adjustment and recent modifications of the procedure control the Type I error rate regardless of the number of true null hypotheses in the ANOVA.

Get full access to this article

View all access options for this article.

References

Cohen, J. (1988). Statistical power analysis for the behavioral sciences. New York: Academic Press.

Darlington, R. B. (1990). Regression and linear models. New York: McGraw-Hill.

Fletcher, H. J. , Daw, H. , & Young, J. (1989). Controlling multiple F test errors with an overall F test. Journal of Applied Behavioral Science, 25, 101-108.

Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75, 800-802.

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65-70.

Keppel, G. (1991). Design and analysis: A researcher s handbook (3rd ed.). Englewood Cliffs, NJ: Prentice Hall.

Maxwell, S. E. , & Delaney, H. D. (1990). Designing experiments and analyzing data: A model comparison perspective. Belmont, CA: Wadsworth.

Robey, R. R. , & Barcikowski, R. S. (1992). Type I error and the number of iterations in Monte Carlo studies of robustness. British Journal of Mathematical and Statistical Psychology, 45, 283-288.

Ryan, T. A. (1959). Multiple comparisons in psychological research. Psychological Bulletin, 56, 26-47.

10.

Saville, D. J. (1990). Multiple comparison procedures: The practical solution. American Statistician, 44, 174-180.

11.

Stevens, J. (1990). Intermediate statistics: A modern approach. Hillsdale, NJ: Lawrence Erlbaum.

12.

Toothaker, L. E. (1991). Multiple comparisons for researchers. Newbury Park, CA: Sage.