An Empirical Study of Design Parameters for Assessing Differential Impacts for Students in Group Randomized Trials

Abstract

Background:

Prior research has investigated design parameters for assessing average program impacts on achievement outcomes with cluster randomized trials (CRTs). Less is known about parameters important for assessing differential impacts.

Objectives:

This article develops a statistical framework for designing CRTs to assess differences in impact among student subgroups and presents initial estimates of critical parameters.

Research design:

Effect sizes and minimum detectable effect sizes for average and differential impacts are calculated before and after conditioning on effects of covariates using results from several CRTs. Relative sensitivities to detect average and differential impacts are also examined.

Subjects:

Student outcomes from six CRTs are analyzed.

Measures:

Achievement in math, science, reading, and writing.

Results:

The ratio of between-cluster variation in the slope of the moderator divided by total variance—the “moderator gap variance ratio”—is important for designing studies to detect differences in impact between student subgroups. This quantity is the analogue of the intraclass correlation coefficient. Typical values were .02 for gender and .04 for socioeconomic status. For studies considered, in many cases estimates of differential impact were larger than of average impact, and after conditioning on effects of covariates, similar power was achieved for detecting average and differential impacts of the same size.

Conclusions:

Measuring differential impacts is important for addressing questions of equity, generalizability, and guiding interpretation of subgroup impact findings. Adequate power for doing this is in some cases reachable with CRTs designed to measure average impacts. Continuing collection of parameters for assessing differential impacts is the next step.

Keywords

education design and evaluation of programs and policies

Get full access to this article

View all access options for this article.

References

Bloom

H. S.

(2005). Randomizing groups to evaluate place-based programs. In Bloom

H. S.

(Ed.), Learning more from social experiments (pp. 115–172). New York, NY: Russell Sage Foundation.

Bloom

H. S.

Michalopoulos

(2010). When is the story in the subgroups? Strategies for interpreting and reporting intervention effects for subgroups. New York, NY: MDRC. Retrieved March 25, 2014, from http://www.mdrc.org/sites/default/files/When%20is%20the%20Story_FULL.pdf

Bloom

H. S.

Richburg-Hayes

Black

A. R.

(2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29, 30–59.

Center for Advancing Research and Communication. (n.d.). Variance Almanac of academic achievement. Retrieved March 25, 2014, from https://arc.uchicago.edu/reese/variance-almanac-academic-achievement

Cook

T. D.

(2002). Randomized experiments in educational policy research: A critical examination of the reasons the education evaluation community has offered for not doing them. Educational Evaluation and Policy Analysis, 24, 175–199.

Cronbach

L. J.

(1975). Beyond the two disciplines of scientific psychology. American Psychologist, 30, 116–127.

Edmunds

J. A.

Bernstein

Unlu

Glennie

Willse

Smith

Arshavsky

(2012). Expanding the start of the college pipeline: Ninth-grade findings from an experimental study of the impact of the early college high school model. Journal of Research on Educational Effectiveness, 5, 136–159.

Gelman

Stern

(2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician, 60, 328–331.

Hedges

Hedberg

E. C.

(2007). Intraclass correlations for planning group randomized experiments in rural education. Journal of Research in Rural Education, 22. Retrieved January 16, 2008, from http://www.umaine.edu/jrre/archive/22-10.pdf

10.

Jaciw

A. P.

Hegseth

Lai

(2012, 11). Assessing impacts of math in focus, a ‘Singapore math” program for American schools: Findings from an RCT (Empirical Education Rep. No. Empirical_MIF121130-FR1-Y1-O.1). Palo Alto, CA: Empirical Education.

11.

Jaciw

A. P.

Toby

Lai

Lin

(2012, 3). Measuring the average impact of an iPad algebra program: Findings from an RCT in four school districts considering one as a special case (Empirical Education Rep. No. Empirical_Fuse120330-FR1-YR1_O.3). Palo Alto, CA: Empirical Education.

12.

Jacob

Zhu

Bloom

(2009). New empirical evidence for the design of group randomized trials in education. MDRC Working Papers on Research Methodology. Retrieved from http://www.mdrc.org/sites/default/files/new_empirical_evidence_fr.pdf

13.

James-Burdumy

Deke

Lugo-Gil

Carey

Hershey

Gersten

… Faddis

(2010). Effectiveness of selected supplemental reading comprehension interventions: Findings from two student cohorts (NCEE 2010-4015). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved March 25, 2014, from http://ies.ed.gov/ncee/pubs/20104015/pdf/20104015.pdf

14.

McClelland

G. H.

Judd

C. M.

(1993). Statistical difficulties of detecting interactions and moderator effects. Psychological Bulletin, 114, 376–390.

15.

Newman

Finney

P. B.

Bell

Turner

Jaciw

A. P.

Zacamy

J. L.

Feagans Gould

(2012). Evaluation of the effectiveness of the Alabama math, science, and technology initiative (AMSTI) (NCEE 2012–4008). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

16.

Northwestern University Institute for Policy Research. (n.d.). Intraclass correlation database. Retrieved from http://stateva.ci.northwestern.edu/

17.

Puma

M. J.

Olsen

R. B.

Bell

S. H.

Price

(2009). What to do when data are missing in group randomized controlled trials (NCEE 2009-0049). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

18.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models (2nd ed.). Thousand Oaks, CA: Sage.

19.

Raudenbush

S. W.

Liu

(2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5, 199–213.

20.

SAS Institute. (2008). SAS/STAT software: Changes and enhancements (Through release 9.2). Cary, NC: Author.

21.

Schochet

P. Z.

(2008a). Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33, 62–87.

22.

Schochet

P. Z.

(2008b). Technical methods report: Guidelines for multiple testing in impact evaluations (NCEE 2008-4018). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

23.

Shadish

Cook

Campbell

(2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

24.

Singer

J. D.

(1998). Using SAS PROC MIXED to fit multilevel models, hierarchical models, and individual growth models. Journal of Educational and Behavioral Statistics, 24, 323–355.

25.

Somers

M.-A.

Corrin

Sepanik

Salinger

Levin

Zmach

(2010). The enhanced reading opportunities study final report: The impact of supplemental literacy courses for struggling ninth-grade readers (NCEE 2010-4021). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved March 25, 2014, from http://ies.ed.gov/ncee/pubs/20104021/pdf/20104021.pdf

26.

Spybrook

(2013). Detecting intervention effects across context: An examination of the precision of cluster randomized trials. The Journal of Experimental Education, 82, 1–24.

27.

Spybrook

Bloom

Congdon

Hill

Martinez

Raudenbush

S. W.

(2011). Optimal design for longitudinal and multilevel research: Documentation for the “Optimal Design” software (through version 3.0). Retrieved from http://hlmsoft.net/od/od-manual-20111016-v300.pdf

28.

Westine

C. D.

Spybrook

Taylor

J. A.

(2014). An empirical investigation of variance design parameters for planning cluster-randomized trials of science achievement. Evaluation Review, 1–30. doi:10.1177/0193841X14531584

29.

Whisman

M. A.

McClelland

G. H.

(2005). Designing, testing, and interpreting interactions and moderator effects in family research. Journal of Family Psychology, 19, 111–120.

30.

Nichols

(2010). New estimates of design parameters for clustered randomization studies: Findings from North Carolina and Florida (Working Paper no. 43). Washington, DC: National Center for Analysis of Longitudinal Data in Education Research (CALDER).

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.20 MB