Sage Journals: Discover world-class research

Abstract

Background:

Random design experiments are a powerful device for estimating average treatment effects, but evaluators sometimes seek to estimate the distribution of treatment effects. For example, an evaluator might seek to learn the proportion of treated units who benefit from treatment, the proportion who receive no benefit, and the proportion who are harmed by treatment.

Method:

Imbens and Rubin (I&R) recommend a Bayesian approach to drawing inferences about the distribution of treatment effects. Drawing on the I&R recommendations, this article explains the approach; provides computing algorithms for continuous, binary, ordered and countable outcomes; and offers simulated and real-world illustrations.

Results:

This article shows how the I&R approach leads to bounded uncertainty intervals for summary measures of the distribution of treatment effects. It clarifies the nature of those bounds and shows that they are typically informative.

Conclusions:

Despite identification issues, bounded solutions provide useful insight into the distribution of treatment effects. We recommend that evaluators incorporate analyses of the distribution of treatment effects into new studies and that evaluators revisit completed studies to estimate the distribution of treatment effects.

Keywords

treatment effect heterogeneity Bayesian inference random design experiments

Get full access to this article

View all access options for this article.

References

Bench

Allen

. (2003). Investigating the stigma or prison classification: An experimental design. The Prison Journal, 83(4), 367–382.

Berk

(2005). Randomized experiments as the bronze standard. Journal of Experimental Criminology, 1(4), 417–433.

Berk

Ladd

Graziano

Baek

(2003). Randomized experiment testing inmate classification systems. Criminology and Public Policy, 2(2), 215–242.

Bertsekas

Tsitsdklis

(2008). Introduction to probability (2nd ed.). Athena Scientific Books.

Bloom

(1984). Accounting for no-shows in experimental evaluation designs. Evaluation Review, 8(2), 225–246.

Bloom

(2005). Learning more from social experiments: Evolving analytical approaches. Russell Sage Foundation.

Burtless

(1995). The case for randomized field trials in econometrics and policy research. Journal of Economic Perspectives, 9(2), 63–84.

Chen

Shapiro

(2007). Do harsher prison conditions reduce recidivism? A discontinuity based approach. American Law and Economics Review, Advanced Access 9(1), 1–29.

Cronbach

Snow

(1977). Aptitudes and instructional methods. Irvington.

10.

Deaton

Cartwright

(2016, September). Understanding and misunderstanding randomized controlled trials. [NBER Working Paper 22595]. http://www.nber.org/papers/w22595.pdf

11.

Donner

Klar

(2004). Pitfalls and controversies in cluster randomization trials. American Journal of Public Health, 93(3), 416–422.

12.

Enders

(2010). Applied missing data analysis. Guilford Press.

13.

Frangakis

Rubin

(2002). Principal stratification in causal inference. Biometrics, 58, 21–29.

14.

Freedman

(2008). On regression adjustments to experimental data. Advances in Applied Mathematics, 30(6), 180–193.

15.

Gaes

Camp

(2009). Unintended consequences: Experimental evidence from the criminogenic effect of prison security level placement on post-release recidivism. Journal of Experimental Criminology, 5(2), 139–162.

16.

Garner

Fagan

Maxwell

(1995). Published findings from the spouse assault replication program: A critical review. Journal of Quantitative Criminology, 11(1), 3–27.

17.

Gelman

(2002). Prior Distribution. In El-Shaarawi

A. H.

Piegorsch

W. W.

(Eds.), Encyclopedia of Environomics, 3, 1634–1637.

18.

Guo

Fraser

(2010). Propensity score analysis: Statistical methods and applications. Sage Publications.

19.

Hayes

Moulton

. (2009). Cluster randomization designs. Chapman & Hall/CRC Interdisciplinary Series.

20.

Heckman

(1979). Sample selection bias as a specification error. Econometrica, 47(1), 153–161.

21.

Heckman

Vytlacil

(2007a). Econometric evaluation of social programs, Part I: Causal models, structural models and econometric policy evaluations. In Heckman

Leamer

(Eds.), Handbook of econometrics (Vol. 6b, pp. 740) North Holland Press.

22.

Hoff

(2009). A first course in Bayesian statistical methods. Springer.

23.

Imbens

Angrist

(1994). Identification and estimation of local average treatment effects. Econometrica, 62(2), 467–475.

24.

Imbens

Rubin

(2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge University Press.

25.

Johnson

Neyman

(1936). Tests of certain linear hypotheses and their applications to some educational problems. Statistical Research Memoirs, 1, 57–93.

26.

Kruschke

(2015). Doing Bayesian analysis: A tutorial with R, Jags and Stan. Elsevier.

27.

Lancaster

(2004). An introduction to modern Bayesian econometrics. Blackwell Publishing.

28.

Lin

. (2012). Agnostic notes on regression adjustments in experimental data: Reexamining freedman’s critique. Annals of Applied Statistics, 7(1), 295–318.

29.

Little

Rubin

(2002). Statistical analysis with missing data (2nd ed.). Wiley.

30.

Maddala

(1983). Limited-dependent and qualitative variables in econometrics. Cambridge University Press.

31.

Manski

(2007). Identification for prediction and decision. Harvard University Press.

32.

Morgan

Winship

(2015). Counterfactuals and causal inference: Methods and principals for social research (2nd ed.). Cambridge University Press.

33.

Page

(2012). Principal stratification as a framework for investigating mediational processes in experimental sessions. Journal of Research on Educational Effectiveness, 5, 215–244.

34.

Page

Feller

Grindal

Miratrix

Somers

(2015). Principal stratification: A tool for understanding variation in program effects across endogenous subgroups. American Journal of Evaluation, 36(4), 514–531.

35.

Peck

(2003). Subgroup analysis in social experiments: Measuring program impacts based on post-treatment choice. American Journal of Evaluation, 24, 157–187.

36.

Raudenbush

Bloom

. (2015a, May 1). It’s no longer all about the mean. William T. Grant Foundation. http://wtgrantfoundation.org/resource/its-no-longer-all-about-the-mean-using-multi-site-trials-to-learn-about-and-from-impact-variation

37.

Raudenbush

Bloom

(2015b). Learning about and from variation in program impacts using multi-site trials. http://wtgrantfoundation.org/library/uploads/2015/10/Learning-About-and-From-Variation-in-Program-Impacts-Using-Multi-site-Trials1.pdf

38.

Rhodes

(2014). Pairwise cluster randomization: An exposition. Evaluation Review, 38(3), 217–250.

39.

Rubin

(1978). Bayesian inferences for causal effects: The role of randomization. Annals of Statistics, 6, 34–58.

40.

Sampson

(2010). Gold standard myths: Observations on the experimental turn in quantitative criminology. Journal of Quantitative Criminology, 26(4), 489–500.

41.

Schaefer

(1997). Analysis of incomplete missing data. Chapman & HALL/CRC.

42.

Sherman

Berk

(1984a). The Minneapolis domestic violence experiment. Police Foundation.

43.

Sherman

Berk

(1984b). The specific deterrent effects of arrest for domestic assault. American Sociological Review, 49, 261–272.

44.

Thompson

(2014). Bayesian analysis with Stata. Stata Press.

45.

Xie

Brand

Jann

(2012). Estimating heterogeneous treatment effects with observational data. Sociological Methodology, 42, 314–347.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.05 MB

Estimating the Distribution of Treatment Effects From Random Design Experiments

Abstract

Background:

Method:

Results:

Conclusions:

Keywords

Get full access to this article

References

Supplementary Material