Restricted accessResearch articleFirst published online 2020-9
Learning From Cluster Randomized Trials in Education: An Assessment of the Capacity of Studies to Determine What Works,For Whom,and Under What Conditions
Over the past 15 years, we have seen an increase in the use of cluster randomized trials (CRTs) to test the efficacy of educational interventions. These studies are often designed with the goal of determining whether a program works, or answering the what works question. Recently, the goals of these studies expanded to include for whom and under what conditions an intervention is effective. In this study, we examine the capacity of a set of CRTs to provide rigorous evidence about for whom and under what conditions an intervention is effective. The findings suggest that studies are more likely to be designed with the capacity to detect potentially meaningful individual-level moderator effects, for example, gender, than cluster-level moderator effects, for example, school type.
AguinisH.BeatyJ. C.BoikR. J.PierceC. A. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30-year review. Journal of Applied Psychology, 90, 94–107.
2.
BloomH. S. (1995). Minimum detectable effects: A simple way to report the statistical power of experimental designs. Evaluation Review, 19(5), 547–556.
3.
BloomH. S. (2005). Randomizing groups to evaluate place-based programs. In BloomH. S. (Ed.), Learning more from social experiments: Evolving analytic approaches (pp. 115–172). Russell Sage.
4.
BloomH. S.Richburg-HayesL.BlackA. R. (2007). Using covariates to improve precision: Empirical guidance for studies that randomize schools to measure the impacts of educational interventions. Educational Evaluation and Policy Analysis, 29(1), 30–59.
5.
BloomH. S.SpybrookJ. (2017). Assessing the precision of multisite trials for estimating parameters of cross-site program effect distributions. Journal of Research on Educational Effectiveness, 10(4), 877–902.
6.
BrandonP. R.HarrisonG. M.LawtonB. E. (2013). SAS code for calculating intraclass correlation coefficients and effect size benchmarks for site-randomized education experiments. American Journal of Evaluation, 34(1), 85–90.
7.
CookT. D.ForayD. (2007). Building the capacity to experiment in schools: A case study of the Institute of Educational Sciences in the US Department of Education. Economics of Innovation and New Technology, 16(5), 385–402.
8.
DongN.KelceyB.SpybrookJ. (2018). Power analyses for moderator effects in three-level cluster randomized trials. Journal of Experimental Education, 86(3), 489–514.
9.
DongN.KelceyB.SpybrookJ.MaynardR. A. (2016). PowerUp!-Moderator: A tool for calculating statistical power and minimum detectable effect size differences of the moderator effects in cluster randomized trials [Software]. http://www.causalevaluation.org/
10.
DongN.ReinkeW. M.HermanK. C.BradshawC. P.MurrayD. W. (2016). Meaningful effect sizes, intraclass correlations, and proportions of variance explained by covariates for planning two- and three-level cluster randomized trials of social and behavioral outcomes. Evaluation Review, 40(4), 334–377.
11.
HedgesL. V.HedbergE. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87.
12.
HedgesL. V.HedbergE. C. (2013). Intraclass correlations and covariate outcome correlations for planning two- and three-level cluster-randomized experiments in education. Evaluation Review, 37(6), 445–489.
13.
HedgesL. V.RhoadsC. (2009). Statistical power analysis in education research (NCSER 2010-3006). National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education.
14.
HillC.BloomH. S.Rebeck-BlackA.LipseyM. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2(3), 172–177.
15.
Institute of Education Sciences. (2006). Request for applications: Field year 2017. CFDA Number: 84.305A.
16.
Institute of Education Sciences. (2011). Request for applications: Field year 2012. CFDA Number: 84.305A.
17.
Institute of Education Sciences. (2016). Request for applications: Field year 2017. CFDA Number: 84.305A.
18.
JacobR.ZhuP.BloomH. S. (2010). New empirical evidence for the design of group randomized trials in education. Journal of Research on Educational Effectiveness, 3(2), 157–198.
19.
NyeB.KonstantopoulosS.HedgesL. V. (2004). How large are teacher effects?Educational Evaluation and Policy Analysis, 26(3), 237–257.
RaudenbushS. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173–185.
22.
SchochetP. Z. (2008). Strategies for power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33, 62–87.
23.
SpybrookJ.KelceyB.DongN. (2016). Power for detecting treatment by moderator effects in two and three-level cluster randomized trials. Journal of Educational and Behavioral Statistics, 41(6), 605–627.
24.
SpybrookJ.LiningerM.CullenA. (2013). From planning to implementation: An examination of changes in the research design, sample size, and statistical power of group randomized trials launched by the Institute of Education Sciences. Journal of Research on Educational Effectiveness, 64(4), 396–342.
25.
SpybrookJ.RaudenbushS. W. (2009). An examination of the precision and technical accuracy of the first wave of group randomized trials funded by the Institute of Education Sciences. Educational Evaluation and Policy Analysis, 313(3), 298–318.
26.
SpybrookJ.ShiR.KelceyB. (2016). Progress in the past decade: An examination of the precision of cluster randomized trials funded by the U.S. Institute of Education Sciences. International Journal of Research and Method in Education, 39(3), 255–267.
27.
SpybrookJ.WestineC.TaylorC. (2016). Design parameters for impact research in science education: A multistate analysis. AERA Open, 2(1), 1–15.
28.
WestineC.SpybrookJ.TaylorJ. (2013). An empirical investigation of variance design parameters for planning cluster-randomized trials of science achievement. Evaluation Review, 37(6), 490–519.
29.
XuZ. A.NicholsA. (2010). New estimates of design parameters for clustered randomization studies: Findings from North Carolina and Florida (Working Paper No. 43). National Center for Analysis of Longitudinal Data in Education Research.
30.
ZhangQ.SpybrookJ.TiptonE. (2019, March8). The influence of imbalanced subgroup units in statistical power for moderator effects in cluster randomized trials: Empirical evidence from IES-funded studies [Paper presentation]. Annual Meeting for the Society for Research on Educational Effectiveness, Washington, DC.
31.
ZhuP.JacobR.BloomH. S.XuZ. (2012). Designing and analyzing studies that randomize schools to estimate intervention effects on student academic outcomes without classroom-level information. Education Evaluation and Policy Analysis, 34(1), 45–68.
