An Examination of the Precision and Technical Accuracy of the First Wave of Group-Randomized Trials Funded by the Institute of Education Sciences

Abstract

This article examines the power analyses for the first wave of group-randomized trials funded by the Institute of Education Sciences. Specifically, it assesses the precision and technical accuracy of the studies. The authors identified the appropriate experimental design and estimated the minimum detectable standardized effect size (MDES) for each study under plausible assumptions about intra-class correlations, covariate-outcome correlations, and explanatory effects of blocking. The MDESs ranged from .18 to .40 for studies funded by the National Center for Education Evaluation and Regional Assistance (NCEE) and from .20 to 1.0 for studies funded by the National Center for Education Research (NCER). They evaluated the technical accuracy by comparing the estimated MDES to the MDES stated in the proposal. Studies funded by the NCEE tended to be very accurate: estimated MDESs and stated MDESs were similar, whereas studies funded by the NCER were less accurate. However, the technical accuracy of the NCER studies improved over time.