Sage Journals: Discover world-class research

Abstract

Researchers designing multisite and cluster randomized trials of educational interventions will usually conduct a power analysis in the planning stage of the study. To conduct the power analysis, researchers often use estimates of intracluster correlation coefficients and effect sizes derived from an analysis of survey data. When there is heterogeneity in treatment effects across the clusters in the study, these parameters will need to be adjusted to produce an accurate power analysis for a hierarchical trial design. The relevant adjustment factors are derived and presented in the current article. The adjustment factors depend upon the covariance between treatment effects and cluster-specific average values of the outcome variable, illustrating the need for better information about this parameter. The results in the article also facilitate understanding of the relative power of multisite and cluster randomized studies conducted on the same population by showing how the parameters necessary to compute power in the two types of designs are related. This is accomplished by relating parameters defined by linear mixed model specifications to parameters defined in terms of potential outcomes.

Keywords

power analysis experimental design hierarchical linear modeling

Get full access to this article

View all access options for this article.

References

Angrist

J. D.

(2004). American education research changes tack. Oxford Review of Economic Policy, 20, 198–212.

Angrist

J. D.

Imbens

G. W.

Rubin

D. B.

(1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91, 444–455.

Bloom

H. S.

Hill

Rebeck Black

Lipsey

(2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions. Journal of Research on Educational Effectiveness, 1, 289–328.

Bloom

H. S.

Weiland

(2015). Quantifying variation in head start effects on children’s cognitive and socio-emotional skills using data from the national head start impact study (MDRC Working Paper on Research Methodology). New York, NY: MDRC.

Borenstein

Hedges

L. V.

Rothstein

(2012). CRT power. Teaneck, NJ: Biostat.

Brandon

P. R.

Harrison

G. M.

Lawton

B. E.

(2013). SAS Code for calculating intraclass correlation coefficients and effect size benchmarks for site randomized education experiments. American Journal of Evaluation, 34, 85–90.

Casella

Berger

(2002). Statistical inference (2nd ed.). Pacific Grove, CA: Duxbury.

Chaney

(2016). Reconsidering findings of “no effects” in randomized controlled trials: Modeling differences in treatment impacts. American Journal of Evaluation, 37, 45–62.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

10.

Constas

(2007). Reshaping the methodological identity of education research: Early signs of the impact of federal policy. Evaluation Review, 31, 391–400.

11.

Hedges

L. V.

Hedberg

E. C.

(2007a). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29, 60–87.

12.

Hedges

L. V.

Hedberg

E. C.

(2007b). Intraclass correlations for planning group randomized experiments in rural education. Journal of Research in Rural Education, 22, 1–15.

13.

Hedges

L. V.

Hedberg

E. C.

(2016). Power calculation using the online variance almanac (Web VA): A user’s guide. Retrieved July 2016, from https://arc.uchicago.edu/sites/default/files/VA%20User%20Guide.pdf

14.

Hedges

L. V.

Rhoads

(2009). Statistical power analysis in education research (NCSER 2010-3006). Washington, DC: National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education.

15.

Holland

(1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945–960.

16.

Jacob

Zhu

Bloom

(2010). New empirical evidence for the design of group randomized trials in education. Journal of Research on Educational Effectiveness, 3, 157–198.

17.

Kim

Olson

C. B.

Scarcella

Kramer

Pearson

van Dyk

… Land

(2011). A randomized experiment of a cognitive strategies approach to test-based analytical writing for mainstreamed Latino English language learners in grades 6 to 12. Journal of Research on Educational Effectiveness, 4, 231–263.

18.

Kish

(1965). Survey sampling. New York, NY: Wiley.

19.

Konstantopoulos

(2008). The power of the test for treatment effects in three-level block randomized designs. Journal of Research on Educational Effectiveness, 1, 265–288.

20.

Kosko

Miyazaki

(2012). The effect of student discussion frequency on fifth-grade students’ mathematics achievement in U.S. schools. The Journal of Experimental Education, 80, 173–195.

21.

National Center for Education Research. (2015). FY 2016 request for applications. Retrieved from http://ies.ed.gov/funding/ncer_progs.asp

22.

Raudenbush

S. W.

(1997). Statistical analysis and optimal design for cluster randomization trials. Psychological Methods, 2, 173–185.

23.

Raudenbush

S. W.

Bloom

H. S.

(2015). Learning about and from a distribution of program impacts using multisite trials. American Journal of Evaluation, 36, 475–499.

24.

Raudenbush

S. W.

Liu

(2000). Statistical analysis and optimal design for multisite randomized trials. Psychological Methods, 5, 199–213.

25.

Raudenbush

S. W.

Reardon

Nomi

(2012). Statistical analysis for multisite trials using instrumental variables with random coefficients. Journal of Research on Educational Effectiveness, 5, 303–332.

26.

Raudenbush

S. W.

Spybrook

Congdon

Liu

Martinez

Bloom

Hill

(2011). Optimal design plus empirical evidence (Version 3.0). Retrieved from http://hlmsoft.net/od/od-manual-20111016-v300.pdf

27.

Rhoads

(2011). The implications of contamination for experimental design in education research. Journal of Educational and Behavioral Statistics, 36, 76–104.

28.

Rosenbaum

P. R.

Rubin

D. B.

(1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.

29.

Rubin

D. B.

(1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688–701.

30.

Scammacca

Fall

A. M.

Roberts

(2015). Benchmarks for expected annual academic growth for students in the bottom quartile of the normative distribution. Journal of Research on Educational Effectiveness, 8, 366–379.

31.

Schochet

(2008). Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33, 62–87.

32.

Spybrook

(2014). Introduction to special issue on design parameters for cluster randomized trials in education, Evaluation Review, 37, 435–444.

33.

Spybrook

Cullen

Lininger

(2011). An examination of the impact of changes in federal policy on the landscape of education research. Effective Education, 3, 83–88.

34.

Spybrook

Hedges

Borenstein

(2014). Understanding statistical power in cluster randomized trials: Challenges posted by differences in notation and terminology. Journal of Research on Educational Effectiveness, 7, 384–406.

35.

Weiss

Bloom

H. S.

Brock

(2014). A conceptual framework for studying the sources of variation in program effects. Journal of Policy Analysis and Management, 33, 778–808.

36.

Westine

Spybrook

Taylor

. (2013). An empirical investigation of variance design parameters for planning cluster-randomized trials of science achievement. Evaluation Review, 37, 490–519.

37.

Whitehurst

G. J.

(2004, 4 26). Making education evidence-based: Premises, principles, pragmatics, and politics (IPR Distinguished Public Policy Lecture Series, 2003-2004). Evanston, IL: Northwestern University, Institute for Policy Research.

38.

Nichols

(2010). New estimates of design parameters for clustered randomization studies: Findings from North Carolina and Florida (Calder Working Paper Number 43). Washington, DC: CALDER.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.17 MB

Coherent Power Analysis in Multilevel Studies Using Parameters From Surveys

Abstract

Keywords

Get full access to this article

References

Supplementary Material