The authors compare the operating characteristics of the bootstrap-F approach, a direct extension of the work of Berkovits, Hancock, and Nevitt, with Huynh’s improved general approximation (IGA) and the Brown-Forsythe (BF) multivariate approach in a mixed repeated measures design when normality and multisample sphericity assumptions do not hold. The results of the simulation show that the three approaches adequately control Type I error when data are generated from normal or slightly nonnormal distributions. However, when data are generated from distributions with moderate or severe skewness, the approaches tend to produce conservative Type I error rates, except the IGA test of the main effect, which has liberal Type I error rates in some conditions. With regard to power, it was found that the bootstrap-F approach can compete with the IGA approach but not with the BF approach: The power differences favoring the bootstrap-F approach are generally small, whereas those favoring the BF approach are substantial.
Alexander, R. A., & Govern, D. M. (1994). A new and simpler approximation for ANOVA under variance heterogeneity. Journal of Educational Statistics, 19, 91-101.
2.
Algina, J. (1994). Some alternative approximate tests for a split-plot design. Multivariate Behavioral Research, 29, 365-384.
3.
Algina, J., & Keselman, H. J. (1997). Testing repeated measures hypotheses when covariance matrices are heterogeneous: Revisiting the robustness of the Welch-James test. Multivariate Behavioral Research, 32, 255-274.
4.
Algina, J., & Keselman, H. J. (1998). A power comparison of the Welch-James and improved general approximation tests in the split-plot design. Journal of Educational and Behavioral Statistics, 23, 152-169.
5.
Algina, J., & Oshima, T. C. (1994). Type I error rates for Huynh’s general approximation and improved general approximation tests. British Journal of Mathematical and Statistical Psychology, 47, 151-165.
6.
Algina, J., & Oshima, T. C. (1995). An improved general approximation test for the main effect in a split-plot design. British Journal of Mathematical and Statistical Psychology, 48, 149-160.
7.
Berkovits, I., Hancock, G. R., & Nevitt, J. (2000). Bootstrap resampling approaches for repeated measure designs: Relative robustness to sphericity and normality violations. Educational and Psychological Measurement, 60, 877-892.
8.
Box, G. E. P. (1954). Some theorems on quadratic forms applied in the study of analysis of variance problems, I. Effects of inequality of variance in the one-way classification. Annals of Mathematical Statistics, 25, 290-303.
9.
Bradley, J. V. (1978). Robustness?British Journal of Mathematical and Statistical Psychology, 31, 144-152.
10.
Brown, M. B., & Forsythe, A. B. (1974). The small sample behavior of some statistics which test the equality of several means. Technometrics, 16, 129-132.
11.
Coombs, W. T., & Algina, J. (1996). New test statistics for MANOVA/descriptive discriminant analysis. Educational and Psychological Measurement, 58, 382-402.
12.
Efron, B., & Tibshirani, R. (1993). An introduction to the bootstrap. New York: Chapman & Hall.
13.
Fleishman, A. I. (1978). A method for simulating nonnormal distributions. Psychometrika, 43, 521-532.
14.
Hall, P. (1986). On the number of bootstrap simulations required to construct a confidence interval. Annals of Statistics, 14, 1431-1452.
15.
Hughes, J. T., & Saw, J. G. (1972). Approximating the percentage points of Hotelling’s generalized statistic. Biometrika, 59, 224-226.
16.
Huynh, H. (1978). Some approximate tests for repeated measurement designs. Psychometrika, 43, 161-165.
17.
James, G. S. (1951). The comparison of several groups of observations when the ratios of the population variances are unknown. Biometrika, 38, 324-329.
18.
Johansen, S. (1980). The Welch-James approximation of the distribution of the residual sum of squares in weighted linear regression. Biometrika, 67, 85-92.
19.
Johnson, N. J. (1978). Modified t test and confidence intervals for asymmetrical populations. Journal of the American Statistical Association, 73, 536-544.
20.
Kenward, M. G., & Roger, J. H. (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53, 983-997.
21.
Keselman, H. J., Algina, J., Wilcox, R. K., & Kowalchuk, R. K. (2000). Testing repeated measures hypotheses when covariance matrices are heterogeneous: Revisiting the robustness of the Welch-James test again. Educational and Psychological Measurement, 60, 925-938.
22.
Keselman, H. J., Carriere, M. C., & Lix, L. M. (1993). Testing repeated measures hypotheses when covariance matrices are heterogeneous. Journal of Educational Statistics, 18, 305-319.
23.
Keselman, H. J., Huberty, C. J., Lix, L. M., Olejnik, S., Cribbie, R. A., Donohue, B., et al. (1998). Statistical practices of educational researchers: Analyses of their ANOVA, MANOVA, and ANCOVA analyses. Review of Educational Research, 68, 350-386.
24.
Kowalchuk, R. K., Keselman, H. J., Algina, J., & Wolfinger, R. D. (2004). The analysis of repeated measurements with mixed-model adjusted F tests. Educational and Psychological Measurement, 64, 224-242.
25.
Lecoutre, B. (1991). A correction for the approximate test in repeated measures designs with two or more independent groups. Journal of Educational Statistics, 16, 371-372.
26.
Liang, K. R., & Zeger, S. (1986). Longitudinal data analysis using generalized linear models. Biometrika, 73, 13-22.
27.
Lix, L. M., & Keselman, H. J. (1998). To trim or not to trim: Tests of location equality under heteroscedasticity and nonnormality. Educational and Psychological Measurement, 58, 409-429.
28.
Lix, L. M., Keselman, H. J., & Algina, J. (2003). Analyzing multivariate repeated measures designs: A comparison of two approximate degrees of freedom procedures. Multivariate Behavioral Research, 38, 403-431.
29.
Luh, W. M., & Guo, J. H. (1999). A powerful transformation trimmed mean method for one-way fixed effects ANOVA model under nonnormality and inequality of variances. British Journal of Mathematical and Statistical Psychology, 52, 303-320.
30.
Lunneborg, C. E. & Tousignant, J. P. (1985). Efron’s bootstrap with application to the repeated measures design. Multivariate Behavioral Research, 20, 161-178.
31.
Maas, C. J. M., & Snijders, T. A. B. (2003). The multilevel approach to repeated measures for complete and incomplete data. Quality and Quantity, 37, 71-89.
32.
Mehrotra, D. V. (1997). Improving the Brown-Forsythe solution to the generalized Behrens-Fisher problem. Communication in Statistics-Simulation and Computation, 26, 1139-1145.
33.
Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156-166.
34.
Nel, D. G., & van der Merwe, C. A. (1986). A solution to the multivariate Behrens-Fisher problem. Communications in Statistics—Theory and Methods, 15, 3719-3735.
35.
Rao, C. R. (1951). An asymptotic expansion of the distribution of Wilks’s criterion. Bulletin of the International Statistical Institute, 33(2), 177-180.
36.
Rubin, A. S. (1983). The use of weighted contrast in analysis of models with heterogeneity of variance. In Proceedings of the Business and Economic Statistics Section, American Statistical Association (pp. 347-352).
37.
SAS Institute. (2001). SAS/STAT software: Version 8.2 (TS M0). Cary, NC: Author.
38.
Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies?Psychological Bulletin, 105, 309-316.
39.
Vale, C. D., & Maurelli, V. A. (1983). Simulating multivariate nonnormal distributions. Psychometrika, 48, 465-471.
40.
Vallejo, G., Fidalgo, A. M., & Fernández, P. (2001). Effects of covariance heterogeneity on three procedures for analysing multivariate repeated measures designs. Multivariate Behavioral Research, 36, 1-27.
41.
Vallejo, G., & Livacic-Rojas, P. (in press). A comparison of two procedures for analyzing small sets of repeated measures data. Multivariate Behavioral Research.
42.
Welch, B. L. (1951). On the comparison of several mean values: An alternative approach. Biometrika, 38, 330-336.
43.
Westfall, P. H., & Young, S. S. (1993). Resampling-based multiple testing. New York: John Wiley.
44.
Wilcox, R. R. (1994). Some results on the Tukey-McLaughlin and Yuen methods for trimmed means when distributions are skewed. Biometrical Journal, 37, 677-688.
45.
Wilcox, R. R. (2001). Fundamentals of modern statistical methods: Substantially improving power and accuracy. New York: Springer.
