SAS Code for Calculating Intraclass Correlation Coefficients and Effect Size Benchmarks for Site-Randomized Education Experiments

Abstract

When evaluators plan site-randomized experiments, they must conduct the appropriate statistical power analyses. These analyses are most likely to be valid when they are based on data from the jurisdictions in which the studies are to be conducted. In this method note, we provide software code, in the form of a SAS macro, for producing statistical power analyses using such data. The macro is designed to estimate statistical parameters for multiple school subjects and multiple grades in a single computer run. Among other convenient features, the macro calculates intraclass correlation coefficient confidence intervals, which are essential for planning studies conservatively. We also provide a SAS macro for calculating benchmarks–again, using locally-based–data for interpreting the feasibility of achieving the minimum detectable effect sizes that are calculated in statistical power analyses. The benchmarks are potentially useful for interpreting the meaning of the effect sizes that are achieved in completed studies, as well. We describe the macros, describe how we verified their accuracy, show how they can be useful to education evaluators, and give examples of their use with statewide educational assessment data. It is our hope that the macros will help evaluators use local data when conducting group-randomized studies.