The Effects of Unequal Covariances on the Tukey WSD Test

Abstract

The purpose of this study was to determine the effect of violating the assumption of homogeneity of covariance for the Tukey WSD test on the familywise rate of Type I error, when all other assumptions of the test were met. Determinations of the empirical significance level were made using computerized Monte Carlo simulations varying the number of treatment groups (k = 3 and 5), sample size (n = 5, 15, and 25), and degree of covariance heterogeneity (none, moderate, and high). For each of the resulting 18 combinations, 10,000 separate data sets were generated and analyzed at nominal α = .01 and .05 using the Tukey WSD procedure for all pairwise comparisons of means.

For the conditions investigated in this study, degree of covariance heterogeneity showed a consistent effect on the empirical significance level. As the degree of covariance heterogeneity was increased, the empirical significance levels increased beyond the nominal levels. In general, different combinations of sample size and number of treatments produced no consistent effect on the empirical significance level.

Get full access to this article

View all access options for this article.

References

Box, G. E. P. Some theorems on quadratic forms applied in the study of analysis of variance problems, I. Effect of inequality of variance in the one-way classification . Annals of Mathematical Statistics, 1954, 25, 290-302.

Games, P. A. Inverse relation between the risks of type I and type II errors and suggestions for the unequal n case in multiple comparisons . Psychological Bulletin, 1971, 75 (2), 97-102.

Games, P. A. and Howell, J. F. Pairwise multiple comparison procedures with unequal n's and/or variances: a Monte Carlo study. Journal of Educational Statistics, 1976, 1 (2), 113-125.

Howell, J. F. and Games, P. A. The robustness of the analysis of variance and Tukey WSD test under various patterns of heterogeneous variances. Journal of Experimental Education, 1973, 41 (4), 33-37.

Kennedy, J. J. An introduction to the design and analysis of experiments in education and psychology. Washington, D. C.: University Press of America, Inc., 1978.

Keselman, H. J. , Toothaker, L. E. , and Shooter, M. An evaluation of two unequal nk forms of the Tukey multiple comparison statistic. Journal of the American Statistical Association, 1975, 70, 584-587.

Kirk, R. E. Experimental design: procedures for the behavioral sciences. Belmont, California: Brooks/Cole Publishing Company—A Division of Wadsworth Publishing Company, Inc., 1968.

Myers, J. L. Fundamentals of experimental design (3rd ed.). Boston: Allyn and Bacon, Inc., 1979.

Miller, R. G., Jr. Simultaneous statistical inference. New York: McGraw-Hill Book Company, Inc., 1966.

10.

Ramseyer, G. C. and Tcheng, T. The robustness of the Studentized range statistic to violations of the normality and homogeneity of variance assumptions. American Educational Research Journal, 1973, 10, 235-240.

11.

Rochen, B. L. The effects of unequal covariances on the Tukey WSD test. (Doctoral dissertation, University of Virginia, 1981). Dissertation Abstracts International, in Press.

12.

Rogan, J. C. , Keselman, H. J. , and Breen, L. J. Assumption violations and rates of type I error for the Tukey multiple comparison test: a review and empirical investigation via a coefficient of variance variation. Journal of Experimental Education, 1977, 46 (1), 20-25.

13.

Scheffé, H. The analysis of variance. New York: John Wiley & Sons, Inc., 1959.

14.

Stoloff, P. H. Correcting for heterogeneity of covariance for repeated measures designs of the analysis of variance . EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1970, 30, 909-924.

15.

Tamhane, A. C. A comparison of procedures for multiple comparisons of means with unequal variances . Journal of the American Statistical Association, 1979, 74, 471-480.

16.

Tukey, J. W. The problem of multiple comparisons. Ditto: Princeton University Press, 1953 (copyrighted by J. W. Tukey, 1973).

17.

Winer, B. J. Statistical principles in experimental design (2nd ed.). New York: McGraw-Hill Book Company, 1971.