Background: In randomized controlled trials, attrition rates often differ by treatment status, jeopardizing causal inference. Inverse probability weighting methods and estimation of treatment effect bounds have been used to adjust for this bias. Objectives: We compare the performance of various methods within two samples, both generated through lottery-based randomization: one with considerable differential attrition and an augmented dataset with less problematic attrition. Research Design: We assess the performance of various correction methods within the dataset with problematic attrition. In addition, we conduct simulation analyses. Results: Within the more problematic dataset, we find the correction methods often performed poorly. Simulation analyses indicate that deviations from the underlying assumptions for bounding approaches damage the performance of estimated bounds. Conclusions: We recommend the verification of the underlying assumptions in attrition correction methods whenever possible and, when verification is not possible, using these methods with caution.
AngristJ. (1990). Lifetime earnings and the Vietnam era draft lottery: Evidence from social security administrative records. The American Economic Review, 80(3), 313-336.
4.
AngristJ.BettingerE.BloomE.KremerMKingE. (2002). The effects of school vouchers on students: Evidence from Colombia. American Economic Review, 92(5), 1535-1558.
5.
AngristJ.BettingerE.KremerM. (2006). Long-term educational consequences of secondary school vouchers: Evidence from administrative records in Colombia. American Economic Review,96(3), 847-862.
6.
Aron-DineA.EinavL.FinkelsteinA. (2013). The RAND health insurance experiment, three decades later. Journal of Economic Perspectives, 27(1), 197-222.
7.
AustinP. C.StuartE. A. (2015). Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Statistics in Medicine, 34(28), 3661-3679.
8.
BaileyM. A.HopkinsD.J.RogersT. (2016). Unresponsive, unpersuaded: The unintended consequences of voter persuasion efforts. Political Behavior, 38(3), 713-746.
9.
BarnowB. S.GreenbergD. (2015). Do estimated impacts on earnings depend on the source of the data used to measure them? Evidence from previous social experiments. Evaluation Review, 39(2), 179-228.
10.
BarrowL.Richburg-HayesL.RouseC. E.BrockT. (2014). Paying for performance: The education impacts of a community college scholarship program for low-income adults. Journal of Labor Economics, 32(2), 563-599.
11.
BifulcoR. (2012). Can nonexperimental estimates replicate estimates based on random assignment in evaluations of school choice? A within-study comparison. Journal of Policy Analysis and Management, 31, 729-751.
BooF. L.PalloniG.UrzuaS. (2014). Cost-benefit analysis of a micronutrient supplementation and early childhood stimulation program in Nicaragua. Annals of the New York Academy of Sciences, 1308, 139-148.
14.
BussoM.DiNardoJ.McCraryJ. (2014). New evidence on the finite sample properties of propensity score reweighting and matching estimators. The Review of Economics and Statistics, 96(5), 885-897.
15.
ClarkM.RothsteinJ.SchanzenbachD. W. (2009). Selection bias in college admissions test scores. Economics of Education Review, 28, 295-307.
16.
CookT. D. (2008). “Waiting for Life to Arrive”: A history of the regression-discontinuity design in Psychology, Statistics and Economics. Journal of Econometrics, 142, 636-654.
17.
CookT. D.SteinerP. M.PohlS. (2009). How bias reduction is affected by covariate choice, unreliability, and mode of data analysis: Results from two types of within-study comparisons. Multivariate Behavioral Research, 44(6), 828-847.
18.
CullenJ. B.JacobB. A.LevittS. (2006). The effect of school choice on participants: Evidence from randomized lotteries. Econometrica, 74(5), 1191-1230.
19.
DehejiaR. H.WahbaS. (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association, 94(448), 1053-1062.
20.
DemingD. J.HastingsJ. S.KaneT. J.StaigerD. O. (2014). School choice, school quality, and postsecondary attainment. The American Economic Review, 104(3), 991-1023.
21.
DiNardoJ.McCraryJ.SanbonmatsuL. (2006). Constructive proposals for dealing with attrition: An empirical example. Retrieved February 28, 2018, fromhttp://eml.berkeley.edu/∼jmccrary/DMS_v9.pdf
22.
DobbieW.FryerR. G. (2009). Are high quality schools enough to close the achievement gap? Evidence from a social experiment in Harlem. No. w15473. National Bureau of Economic Research. Retrieved February 28, 2018, fromhttp://www.nber.org/papers/w15473.pdf
23.
EngbergJ.EppleD.ImbrognoJ.SiegH.ZimmerR. (2014). Evaluating education programs that have lotteried admission and selective attrition. Journal of Labor Economics, 32(1), 27-63.
24.
FrölichM.HuberM. (2014). Treatment evaluation with multiple outcome periods under endogeneity and attrition. Journal of the American Statistical Association, 109(508), 1697-1711.
GoldbergerA. S. (1972). Selection Bias in Evaluating Treatment Effects: Some Formal Illustrations. (Discussion Paper No. 123). Institute for Research on Poverty, University of Wisconsin – Madison. Retrieved February 28, 2018, fromhttp://www.irp.wisc.edu/publications/dps/pdfs/dp12372.pdf
28.
GreenbergD.BarnowB. S. (2014). Flaws in evaluations of social programs: Illustrations from randomized controlled trials. Evaluation review, 38(5), 359-387.
29.
GrilliL.MealliF. (2008). Nonparametric bounds on the causal effect of university studies on job opportunities using principal stratification. Journal of Educational and Behavioral Statistics, 33, 111-130.
30.
HastingsJ. S.NeilsonC. A.ZimmermanS. D. (2012). The effect of school choice on intrinsic motivation and academic outcomes. NBER Working Paper 18324. Retrieved February 28, 2018, fromhttp://www.nber.org/papers/w18324
31.
HeckmanJ. (1979). Sample selection bias as a specification error. Econometrica,47(1), 153-162.
HiranoK.ImbensG. W.RidderG. (2003). Efficient estimation of average treatment effects using the estimated propensity score. Econometrica, 71(4), 1161-1189.
34.
HoganJ. W.LancasterT. (2004). Instrumental variables and inverse probability weighting for causal inference from longitudinal observational studies. Statistical Methods in Medical Research, 13(1), 17-48.
35.
HolmA.JaegerM. M. (2009). Selection bias in educational transition models: Theory and empirical evidence. University of Copenhagen Department of Economics Working Paper No. 2009-05. Retrieved February 28, 2018, fromhttps://core.ac.uk/download/pdf/6476769.pdf
36.
HorowitzJ. L.ManskiC. F. (1998). Censoring of outcomes and regressors due to survey nonresponse: Identification and estimation using weights and imputations. Journal of Econometrics, 84, 37-58.
37.
HorowitzJ. L.ManskiC. F. (2000). Nonparametric analysis of randomized experiments with missing covariate and outcome data. Journal of the American Statistical Association, 95(449), 77-84.
ImbensG. W.ManskiC. F. (2004) Confidence intervals for partially identified parameters. Econometrica, 72, 1845-1857.
41.
ImbensG. W.WooldridgeJ. M. (2009). Recent developments in the econometrics of program evaluation. Journal of Economic Literature, 47(1), 5-86.
42.
KarlanD.FairlieR. W.ZinmanJ. (2012). Behind the GATE experiment: Evidence on effects of and rationales for subsidized entrepreneurship training. Yale University Department of Economics Working Paper No. 95. Retrieved February 28, 2018, fromhttps://www.dartmouth.edu/∼jzinman/Papers/GATE_2012_11.pdf
43.
KremerM.MiguelE.ThorntonR. (2009). Incentives to learn. The Review of Economics and Statistics, 91(3), 437-456.
LaLondeR. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. The American Economic Review, 76(4), 604-620.
46.
LechnerM.MellyB. (2010). Partial identification of wage effects of training programs. Brown University Department of Economics Working Paper. Retrieved February 28, 2018, fromhttps://www.alexandria.unisg.ch/38673/4/Bounds.pdf
47.
LeeD.S. (2009). Training, wages, and sample selection: Estimating sharp bounds on treatment effects. Review of Economic Studies,76, 1071-1102.
48.
ManskiC. F. (1990). Nonparametric bounds on treatment effects. American Economic Review Papers and Proceedings, 80, 319-323.
49.
ManskiC. F. (1995). Identification problems in the social sciences. Harvard University Press.
50.
MelenbergB.van SoestA. (1996). Parametric and semi-parametric modelling of vacation expenditures. Journal of Applied Econometrics, 11(1), 59-76.
MrozT. A. (1987). The sensitivity of an empirical model of married women’s hours of work to economic and statistical assumptions. Econometrica, 55(4), 75-799.
53.
MuralidharanK.SundararamanV. (2014). The aggregate effect of school choice: Evidence from a two-stage experiment in India. NBER Working Paper 19441. Retrieved February 28, 2018, fromhttp://www.nber.org/papers/w19441
54.
NeweyW. K.PowellJ. L.WalkerJ. R. (1990). Semiparametric estimation of selection models: Some empirical results. The American Economic Review, 80(2), 324-328.
55.
PumaM. J.OlsenR. B.BellS. H.PriceC. (2009). What to do when data are missing in group randomized controlled trials (NCEE 2009-0049). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
56.
ReynoldsA. J.TempleJ. A.OuSArteagaI. A.WhiteB. A. B. (2011). School-based early childhood education and age-28 well-being: Effects by timing, dosage, and subgroups. Science, 333, 360-364.
57.
RobinsP. K.WestR. W. (1986). Sample attrition and labor supply response in experimental panel data: A study of alternative correction procedures. Journal of Business & Economic Statistics, 4(3), 329-338.
58.
RouseC. E. (1998). Private school vouchers and student achievement: An evaluation of the Milwaukee parental choice program. Quarterly Journal of Economics, 113(2), 553-602.
59.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin Company.
60.
SmithJ. A.ToddP. E. (2005). Does matching overcome LaLonde’s critique of nonexperimental estimators?Journal of Econometrics, 125, 305-353.
61.
SteeleJ. L.SlaterR. O.ZamarroG.MillerT.LiJ.BurkhauserS.BaconM. (2017). Effects of dual-language immersion programs on student achievement: Evidence from lottery data. American Educational Research Journal, 54(1_suppl), 282S-306S.
62.
SteinerP. M.CookT. D.ShadishW. R.ClarkM. H. (2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods, 15(3), 250-267.
63.
SteinerP. M.WongV. C. (2018). Assessing correspondence between experimental and nonexperimental estimates in within-study comparisons. Evaluation Review, 42(2), 214-247.
TauchmannH. (2014). Lee’s treatment effect bounds for non-random sample selection – an implementation in Stata. The Stata Journal, 14(4), 884-894.
66.
TryonW. W. (2001). Evaluation statistical difference, equivalence, and indeterminacy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis tests. Psychological Methods, 6, 371-386.
67.
What Works Clearinghouse. (2013). Assessing Attrition Bias V 2.1. What Works Clearinghouse, Institute of Education Sciences, U.S. Department of Education. Retrieved January 16, 2020, fromhttps://ies.ed.gov/ncee/wwc/Document/243
WooldridgeJ. M. (2007). Inverse probability weighted estimation for general missing data problems. Journal of econometrics, 141(2), 1281-1301.
70.
ZhangJ. L.RubinD. B. (2003). Estimation of causal effects via principal stratification when some outcomes are truncated by death. Journal of Educational and Behavioral Statistics, 28(4), 353-368.
71.
ZhangJ. L.RubinD. B.MealliF. (2008). Evaluating the effects of job training programs on wages through principal stratification. In FombyT.HillC. R.MillimetD. LSmithJ.VytlacilE. J. (Eds.), Advances in econometrics: Modelling and evaluating treatment effects in econometrics (pp. 117-145). Emerald Group Publishing Limited.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
