In experimental evaluations of policy interventions, the so-called Bloom adjustment is commonly used to estimate the impact of the treatment on the treated. It does so by rescaling the estimated impact of the intention to treat—that is, the overall treatment-control group difference in outcomes for the entire experimental sample—by the percentage of cases that took up the treatment offer. The practice of also rescaling the variance, as is common in the literature, imposes simplifying assumptions that may lead to biased variance estimates. We compare variances using the Bloom adjustment to variances that capture all the estimation error. While the difference between these variances is negligible in three experimental evaluations, we highlight three conditions that could result in larger relative bias: large impacts, large variability in compliance, and/or substantial endogeneity bias. The presence of these conditions could potentially result in a different conclusion for policy or practice.
BloomH. S. (1984). Accounting for no-shows in experimental evaluation designs. Evaluation Review, 8, 225–246.
4.
BloomH. S. (2006). The core analytics of randomized experiments for social research(Working paper). New York, NY: MDRC.
5.
BloomH. S.OrrL. L.CaveG.BellS. H.DoolittleF. (1993). The national JPTA study: Title II-A impacts on earnings and employment at 18 Months. Bethesda, MD: Abt Associates.
6.
BoundJ.JaegerD. A.BakerR. M. (1995). Problems with instrumental variables estimation when the correlation between the instruments and the endogenous explanatory variable is weak. Journal of the American Statistical Association, 90, 443–450.
7.
CollinsA. M.BriefelR.KlermanJ. A.WolfA.RoweG.LoganC.…LyskawaJ. (2016). Summer Electronic Benefit Transfer for Children (SEBTC) demonstration: Summary report. Washington, DC: U.S. Department of Agriculture Food and Nutrition Service. Retrieved fromhttps://www.fns.usda.gov/sites/default/files/ops/sebtcfinalreport.pdf
8.
CopsonE.MartinsonK.BensonV.DiDomenicoM.WilliamsJ.NeedelsK.MastriA. (2016). The green jobs and health care impact evaluation: Findings from the implementation study of four training programs for unemployed and disadvantaged workers. Washington, DC: U.S. Department of Labor Employment and Training Administration. Retrieved fromhttps://wdr.doleta.gov/research/FullText_Documents/ETAOP-2017-08%20Green%20Jobs%20-%20Implementation.pdf
9.
DynarskiM.RuiN.WebberA.GutmannB. (2017). Evaluation of the DC Opportunity Scholarship Program: Impacts after one year (NCEE 2017-4022). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
10.
FrangakisC. E.RubinD. B. (2002). Principal stratification in causal inference. Biometrics, 58, 21–29.
11.
HeckmanJ. J.HohmannN.SmithJ.KhooM. (2000). Substitution and dropout bias in social experiments: A study of an influential social experiment. Quarterly Journal of Economics, 115, 651–694.
12.
HollandP. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945–960.
13.
ImbensG. W.AngristJ. D. (1994). Identification and estimation of local average treatment effects. Econometrica, 62, 467–475.
14.
ImbensG. W.RubinD. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge, England: Cambridge University Press.
15.
JohnsonM.DemersA.JohnsonC. J.GentileC. (2016). Ewing Marion Kauffman School evaluation impact report year 4. Princeton, NJ: Mathematica Policy Research.
16.
KangH.PeckL. R.KeeleL. (2016). A comparison of inferential techniques for instrumental variables methods(Unpublished working paper).
17.
KlermanJ. A.WolfA.CollinsA.BellS.BriefelR. (2017). The effects the Summer Electronic Benefits for Children demonstration has on children's food supply. Manuscript submitted for publication.
18.
LoveJ. M.KiskerE. E.RossC.RaikesH.ConstantineJ.BollerK.…FuligniA. S. (2005). The effectiveness of early head start for 3-year-old children and their parents: Lessons for policy and programs. Developmental Psychology, 41, 885–901.
19.
LudwigJ.DuncanG.GennetianL.KatzL.KesslerR.KlingJ.SanbonmatsuL. (2013, October4). Moving to opportunity: Final impacts evaluation science article data, 2008-2010 (ICPSR34860). Ann Arbor, MI: Inter-university Consortium of Political and Social Research. doi:10.3886/ICPSR34860.v2
20.
MartinsonK.WilliamsJ.NeedelsK.PeckL.MoultonS.PaxtonN.…Brown-LyonsM. (2016). The green jobs and health care impact evaluation: Findings from the impact study of four training programs for unemployed and disadvantaged workers. Washington, DC: U.S. Department of Labor Employment and Training Administration. Retrieved fromhttps://wdr.doleta.gov/research/keyword.cfm?fuseaction=dsp_resultDetails&pub_id=2604&mp=y
21.
MillsG.McKernanS.RatcliffeC.EdelsteinS.PergamitM.BragaB.…ElkinS. (2016). Building savings for success: Early impacts from the assets for independence program randomized evaluation (OPRE Report #2016-59 for the US Department of Health and Human Services). Washington, DC: The Urban Institute.
22.
PageL. C.FellerA.GrindalT.MiratrixL.SomersM. (2015). Principal stratification: A tool for understanding variation in program effects across endogenous subgroups. American Journal of Evaluation, 36, 514–531.
23.
PeckL. R.LitwokD.KeeleL. (2016). The Green Jobs and Health Care impact evaluation: Special topic paper on standard error estimation in evaluations with no-shows. Washington, DC: U.S. Department of Labor, Employment and Training Administration.
24.
RubinD. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688.
25.
SanbonmatsuL.LudwigJ.KatzL. F.GennetianL. A.DuncanG. J.KesslerR. C…LindauS. T. (2011). Moving to Opportunity for Fair Housing Demonstration program—Final impacts evaluation. Washington, DC: U.S. Department of Housing and Urban Development, Office of Policy Development and Research.
26.
SchochetP. Z. (2008). Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33, 62–87.
27.
SchochetP. Z.ChiangH. (2011). Estimation and identification of the complier average causal effect parameter in education RCTs. Journal of Educational and Behavioral Statistics, 36, 307–345.
28.
TheodosB.PergamitM. R.DerianA.EdelsteinS.StolteA. (2016). Solutions for youth: An evaluation of the Latin American Youth Center’s Promotor Pathway Program. Washington, DC: The Urban Institute.
29.
TrenholmC.DevaneyB.FortsonK.QuayK.WheelerJ.ClarkM. (2007). Impacts of four title V, section 510 abstinence education programs: Final report. Princeton, NJ: Mathematica Policy Research, Inc.
30.
WooldridgeJ. M. (2015). Introductory econometrics: A modern approach. Mason, OH: Cengage Learning, Nelson Education.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
