Detecting Unit of Analysis Problems in Nested Designs: Statistical Power and Type I Error Rates of the F Test for Groups-within-Treatments Effects

Abstract

The nested analysis of variance (ANOVA) model is often recommended for analysis of educational data in which students receive treatments within classrooms. The Type I error rate and statistical power of the F test for groups-within-treatments effects associated with such nested ANOVA designs were evaluated in a Monte Carlo study. Data were generated for ANOVA models comprising two and three levels of the treatment variable; two, three, and five groups nested within each treatment; and 3, 10, and 30 observations within each group. The intraclass correlation among scores within groups was controlled at levels ranging from zero (independent observations) to .50, and the F test for groups-within-treatments effects was conducted at nominal alpha levels ranging from .05 to .30. The results indicated that the test for groups within treatments did not indicate sufficient power, even at the most liberal alpha level examined, to detect intraclass correlation in the sample data. The procedure was not effective in supporting decisions to use individual observations rather than group means as the unit of analysis.

Get full access to this article

View all access options for this article.

References

Barcikowski, R. S. (1981). Statistical power with group mean as the unit of analysis. Journal of Educational Statistics, 6, 267-285.

Blair, R. C. , & Higgins, J. J. (1984). Some comments on "The Unit of Analysis: Group Means Versus Individual Observations." Florida Journal of Educational Research, 26, 5-20.

Blair, R. C. , Higgins, J. J. , Topping, M.E.H. , & Mortimer, A. L. (1983). An investigation of the robustness of the t test to unit of analysis violations. Educational and Psychological Measurement, 43, 69-80.

Clinch, J. J. , & Keselman, H. J. (1982). Parametric alternatives to the analysis of variance. Journal of Educational Statistics, 7, 207-214.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences. New York: Academic Press.

Crits-Christoph, P. , & Mintz, J. (1991). Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. Journal of Consulting and Clinical Psychology, 59, 20-26.

Hopkins, K. D. (1982). The unit of analysis: Group means versus individual observations. American Educational Research Journal, 19, 5-18.

Kennedy, J. J. , & Bush, A. J. (1985). An introduction to the design and analysis of experiments in behavioral research. Lanham, MD: University Press of America.

Kirk, R. E. (1982). Experimental design: Procedures for the behavioral sciences. Belmont, CA: Brooks/Cole.

10.

Moran, D. K. , Robison, F. F. , & Hulse-Killacky, D. (1990). Group research and the unit of analysis problem: The use of ANOVA designs with nested factors. Journal for Specialists in Group Work, 15, 10-14.

11.

Scariano, S. M. , & Davenport, J. M. (1987). The effects of violations of independence assumptions in the one-way ANOVA. The American Statistician, 41, 123-129.

12.

Stevens, J. P. (1990). Intermediate statistics: A modern approach. Hillsdale, NJ: Lawrence Erlbaum.

13.

Wilcox, R. R. , Charlin, V. L. , & Thompson, K. L. (1986). New Monte Carlo results on the robustness of the ANOVAF, Wand F* statistics. Communications in Statistics-Simulation and Computation, 15, 933-943.

14.

Winer, B. J. (1971). Statistical principles in experimental design. New York: McGraw-Hill.

15.

Zimmerman, D. W. , & Zumbo, B. D. (1993). Correction for nonindependence of sample observations in ANOVA F tests. Journal of Experimental Education, 60, 367-381.