Sage Journals: Discover world-class research

Abstract

Design-based methods have recently been developed as a way to analyze randomized controlled trial (RCT) data for designs with a single treatment and control group. This article builds on this framework to develop design-based estimators for evaluations with multiple research groups. Results are provided for a wide range of designs used in education research, including clustered and blocked designs. Because analysis in the multi-armed setting involves pairwise contrasts across the research groups, the key methodological question addressed is: How do the estimators for the two-group design need to be adjusted for multi-armed trials? The critical insight is that in multi-armed trials where the goal is to identify the most effective treatments, the samples for each pairwise contrast are representative of the full set of randomized units, not just of themselves. The implication is that variance terms need to be adjusted slightly under the finite-population framework that can reduce precision, and blocks need to be weighted to reflect the full randomized sample in the block or biases can result. An empirical example using data from a multi-armed education RCT demonstrates the issues.

Keywords

multi-armed RCTs average treatment effects design-based estimators impact and variance estimation

Get full access to this article

View all access options for this article.

References

Agodini

Harris

Atkins-Burnett

Heaviside

Novak

Murphy

(2009). Achievement effects of four early elementary school math curricula: Findings from first graders in 39 schools. Washington, DC: U.S. Department of Education, Institute of Education Sciences.

Aronow

P. M.

Green

D. P.

Lee

D. K. K.

(2014). Sharp bounds on the variance in randomized experiments. Annals of Statistics, 42, 850–871.

Bell

McCaffrey

(2002). Bias reduction in standard errors for linear regression with multi-stage samples. Survey Methodology, 28, 169–181.

Box

G. E.

Hunter

J. S.

Hunter

W. G.

(2005). Statistics for experiments: Design, innovation, and discovery (2nd ed.). New York, NY: Wiley.

Castleman

B. L.

Page

L. C.

(2015). Summer nudging: Can personalized text messages and peer mentor outreach increase college going among low-income high school graduates? Journal of Economic Behavior & Organization, 115, 144–160.

Dasgupta

Pillai

N. S.

Rubin

(2015). Casual inference from 2 ^k factorial designs by using potential outcomes. Journal of the Royal Statistical Society, B, 77, 727–753.

Freedman

(2008). On regression adjustments to experimental data. Advances in Applied Mathematics, 40, 180–193.

Hansen

C. B.

(2007). Asymptotic properties of a robust variance matrix estimator for panel data when T is large. Journal of Econometrics, 141, 597–620.

Holland

P. W.

(1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945–960.

10.

Hsu

J. C.

(1996). Multiple comparisons: theory and methods. London, England: Chapman & Hall.

11.

Imbens

Rubin

(2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge, England: Cambridge University Press.

12.

James-Burdumy

Mansfield

Deke

Carey

Lugo-Gil

Hershey

… Faddis

(2009). Effectiveness of selected supplemental reading comprehension interventions. Washington, DC: U.S. Department of Education, Institute of Education Sciences.

13.

Kraft

M. A.

Rogers

(2014). The underutilized potential of teacher-to-parent communication: Evidence from a field experiment (Working Paper RWP14-049). Cambridge, MA: Harvard Kennedy School.

14.

Ding

(2017). General forms of finite population central limit theorems with applications to causal inference. Journal of the American Statistical Association, 112, 520, 1759–1769. doi:10.1080/01621459.2017.1295865

15.

Lin

(2013). Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique. Annals of Applied Statistics, 7, 295–318.

16.

Neyman

(1990). On the application of probability theory to agricultural experiments: Essay on principles. Section 9. Statistical Science, 5, 465–472. (Original work published 1923)

17.

Pashley

Miratrix

(2017). Insights on variance estimation for blocked and matched pair designs (Working Paper). Harvard University Statistics Department.

18.

Rubin

D. B.

(1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Education Psychology, 66, 688–701.

19.

Rubin

D. B.

(1977). Assignment to treatment group on the basis of a covariate. Journal of Education Statistics, 2, 1–26.

20.

Rubin

D. B.

(1986). Which ifs have causal answers? Discussion of Holland’s “Statistics and causal inference.” Journal of the American Statistical Association, 81, 961–962.

21.

Scher

Cole

(2017). Evidence review standards considerations when using RCT-YES. Washington, DC: Analytic Technical Assistance and Development, National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from http://www.rct-yes.com

22.

Schochet

P. Z.

(2009). An approach for addressing the multiple testing problem in social policy impact evaluations. Evaluation Review, 33, 539–567.

23.

Schochet

P. Z.

(2010). Is regression adjustment supported by the Neyman model for causal inference? Journal of Statistical Planning and Inference, 140, 246–259.

24.

Schochet

P. Z.

(2013). Estimators for clustered education RCTs using the Neyman model for causal inference. Journal of Educational and Behavioral Statistics, 38, 219–238.

25.

Schochet

P. Z.

(2016). Statistical theory for the RCT-YES software: Design-based causal inference for RCTs (NCEE 2015–4011; 2nd ed.). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from https://ies.id.gov/ncee/pubs/20154011/pdf/20154011.pdf (Original work published 2015, 1st ed.)

26.

Schochet

P. Z.

(2017). Multi-armed RCTs: A design-based framework (NCEE 2017–4027). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from https://ies.ed.gov/ncee/pubs/20174027/pdf/20174027.pdf

27.

Schochet

P. Z.

Kautz

(2018). Design-based estimators for clustered RCTs and how they compare to robust estimators (Working Paper No. 1). Princeton, NJ: Mathematica Policy Research.

28.

Toledo

Humpage-Liuzzi

Murray

Glazerman

(2015). Data-driven instruction in Honduras: An impact evaluation of the educAccion promising reading intervention evaluation plan. Washington, DC: Mathematica Policy Research, June 2016. Retrieved from https://www.socialscienceregistry.org/trials/780

29.

Westfall

P. H.

Tobias

Rom

Wolfinger

Hochberg

(1999). Multiple comparisons and multiple tests using SAS. Cary, NC: SAS Institute.

30.

C. F. J.

Hamada

M. S.

(2009). Experiments: Planning, analysis and parameter design optimization (2nd ed.). Hoboken, NJ: John Wiley.

31.

Yang

Tsiatis

(2001). Efficiency study of estimators for a treatment effect in a pretest-posttest trial. American Statistician, 55, 314–321.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.17 MB

Design-Based Estimators for Average Treatment Effects for Multi-Armed RCTs

Abstract

Keywords

Get full access to this article

References

Supplementary Material