This paper considers recent methodological developments in the treatment effects literature, describes their value for applied evaluation work, and suggests next steps. It pays particular attention to documenting the presence of treatment effect heterogeneity, to the quest to attach treatment effect heterogeneity to particular subgroups and other moderators, and to the recent application of machine learning methods in this domain.
BaldA.ChynE.HastingsJ.MachelettM. (2021). The causal impact of removing children from abusive and neglectful homes. Journal of Political Economy, Forthcoming. https://doi.org/10.1086/719856
2.
BehrmanJ. R.ChengY.ToddP. E. (2004). Evaluating preschool programs when length of exposure to the program varies: A nonparametric approach. Review of Economics and Statistics, 86(1), 108–132. https://doi.org/10.1162/003465304323023714
3.
BellS. H.OrrL. L. (2002). Screening (and creaming?) applicants to job training programs: The AFDC homemaker-home health aide demonstrations. Labour Economics, 9(2), 279–301. https://doi.org/10.1016/s0927-5371(02)00006-4
4.
BitlerM. P.GelbachJ. B.HoynesH. W. (2006). What mean impacts miss: Distributional effects of welfare reform experiments. American Economic Review, 96(4), 988-1012. https://doi.org/10.1257/aer.96.4.988.
5.
BitlerM. P.GelbachJ. B.HoynesH. W. (2017). Can variation in subgroups’ average treatment effects explain treatment effect heterogeneity? Evidence from a social experiment. Review of Economics and Statistics, 99(4), 683-697. https://doi.org/10.1162/rest_a_00662.
6.
BlackDanJooJoonwhiLaLondeRobertSmithJeffreyTaylorEvan (2020). Simple tests for selection: Learning more from instrumental variables. HCEO Working Paper No. 2020-048.
7.
BlackD. A.SmithJ. A.BergerM. C.NoelB. J. (2003). Is the threat of reemployment services more effective than the services themselves? Evidence from random assignment in the UI system. American Economic Review, 93(4), 1313-1327. https://doi.org/10.1257/000282803769206313.
BloomH.OrrL.GeorgeC.BellS.DoolittleF. (1993). The national JTPA study: Title II-A impacts on earnings and employment at 18 months. Abt Associates.
10.
Brudevold-NewmanA.HonoratiM.JakielaP.OwenO. (2017). A firm of one’s own: Experimental evidence on credit constraints and occupational choice. World Bank Policy Research Working. Paper No. 7977.
11.
Buhl-WiggersJ.KerwinJ.Muñoz-MoralesJ.SmithJ.ThorntonR. (2022a). Some children left behind: Variation in the effects of an educational intervention. Journal of Econometrics, Forthcoming. https://doi.org/10.1016/j.jeconom.2021.12.010.
12.
Buhl-WiggersJ.KerwinJ.SmithJ.ThorntonR. (2022b). Learning more about teachers: Value-added and treatment effects on value-added in Northern Uganda. University of Wisconsin-Madison. Unpublished manuscript.
13.
BushwayS.SmithJ. (2007). Sentencing using statistical treatment rules: What we don’t know can hurt us. Journal of Quantitative Criminology, 23(4), 377–387. https://doi.org/10.1007/s10940-007-9035-1
14.
BykerT.SmithJ. (2021). Chapter 6: Evidence from Connecticut jobs first. In SmithJ.WhalleyA.WilcoxN. (Eds.), Are participants good evaluators? (pp. 145-196). Upjohn Institute for Employment Research.
15.
CalónicoS.SmithJ. (2021). Chapter 5: Evidence from the national supported work demonstration. In SmithJ.WhalleyA.WilcoxN. (Eds.), Are participants good evaluators? (pp. 101-144). Upjohn Institute for Employment Research.
16.
ChernozhukovV.DemirerM.DufloE.Fernández-ValI.2018. Generic machine learning inference on heterogeneous treatment effects in randomized experiments, with an application to Immunization in India. NBER Working Paper No. 24678.
17.
ChettyR.FriedmanJ. N.RockoffJ. E. (2014). Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates. American Economic Review, 104(9), 2593–2632. https://doi.org/10.1257/aer.104.9.2593
18.
ChungE.OlivaresM. (2021). Permutation test for heterogeneous treatment effects with a nuisance parameter. Journal of Econometrics, 225(2), 148-174. https://doi.org/10.1016/j.jeconom.2020.09.015.
19.
CookT.CampbellD. (1979). Quasi-experimentation: Design & analysis issues for field settings. Houghton-Mifflin.
20.
CréponB.DufloE.GurgandM.RathelotR.ZamoraP. (2013). Do labor market policies have displacement effects? Evidence from a clustered randomized experiment. Quarterly Journal of Economics, 128(2), 531-580. https://doi.org/10.1093/qje/qjt001.
21.
CunhaF.HeckmanJ. J. (2007). Identifying and estimating the distributions of ex post and ex ante returns to schooling. Labour Economics, 14(6), 870–893. https://doi.org/10.1016/j.labeco.2007.06.002
22.
DavisJ. M. V.HellerS. B. (2017). Using causal forests to predict treatment heterogeneity: An application to summer jobs. American Economic Review, 107(5), 546–550. https://doi.org/10.1257/aer.p20171000
23.
DeatonA. (2010). Understanding the mechanisms of economic development. Journal of Economic Perspectives, 24(3), 3–16. https://doi.org/10.1257/jep.24.3.3
DillonE. W.SmithJ. A. (2020). The consequences of academic match between students and colleges. Journal of Human Resources, 55(3), 767–808. https://doi.org/10.3368/jhr.55.3.0818-9702r1
26.
DingP.FellerA.MiratrixL. (2019). Decomposing treatment effect variation. Journal of the American Statistical Association, 114(525), 304–317. https://doi.org/10.1080/01621459.2017.1407322
FarrellM. H. (2015). Robust inference on average treatment effects with possibly more covariates than observations. Journal of Econometrics, 189(1), 1–23. https://doi.org/10.1016/j.jeconom.2015.06.017
29.
FerberR.HirschW. (1981). Social experimentation and economic policy. Cambridge University Press.
30.
FisherR. (1935). The design of experiments. Oliver and Boyd.
31.
FrostR. (1920). The road not taken. In FrostR. (Ed.), Mountain interval. Henry Holt.
HarcourtB. (2006). Against prediction: Profiling, policing, and punishing in an actuarial age. University of Chicago Press.
34.
HeckmanJ.HohmannN.SmithJ.KhooM. (2000). Substitution and drop out bias in social experiments: A study of an influential social experiment. Quarterly Journal of Economics, 115(2), 651–694. https://doi.org/10.1162/003355300554764
HeckmanJ. J.RobbR. (1985). Alternative methods for evaluating the impact of interventions. In HeckmanJ.SingerB. (Eds.). Longitudinal analysis of labor market data (pp. 156–246). Cambridge University Press for Econometric Society Monograph Series
37.
HeckmanJ.SmithJ. (1998). Evaluating the Welfare State. In StromS. (Ed.), Econometrics and economic theory in the 20th century: The Ragnar Frisch centennial (pp. 241-318). Cambridge University Press for Econometric Society Monograph Series.
38.
HeckmanJ. J.SmithJ.ClementsN. (1997).Making the most out of programme evaluations and social experiments: Accounting for heterogeneity in programme impacts. Review of Economic Studies, 64(4), 487–535. https://doi.org/10.2307/2971729
39.
HollisterR.KemperP.MaynardR.(1984). The national supported work demonstration. University of Wisconsin Press.
40.
HorváthH. (2015). Classroom assignment policies and implications for teacher value-added estimation. Institute of Education, University College London. Unpublished manuscript.
41.
HotzV. J.GuidoI.MortimerJ. (2005). Predicting the efficacy of future training programs using past experiences at other locations. Journal of Econometrics, 125(1–2), 241-270. https://doi.org/10.1016/j.jeconom.2004.04.009.
42.
ImbensG. W.AngristJ. D. (1994). Identification and estimation of local average treatment effects. Econometrica, 62(2), 467-475. https://doi.org/10.2307/2951620.
43.
JacobB. A.LefgrenL. (2008). Can principals identify effective teachers? Evidence on subjective performance evaluation in education. Journal of Labor Economics, 26(1), 101–136. https://doi.org/10.1086/522974
44.
KlermanJ. A. (2017). “Editor in chief’s comment: External validity in systematic reviews.” Evaluation Review41(5), 391–402. https://doi.org/10.1177/0193841X17746740
45.
KlermanJ.SaundersC.DastrupE.EpsteinZ.DouglasW.AdamT.BarnowB. (2019). Evaluation of impacts of the reemployment and eligibility assessment (REA) program: Final report. Abt Associates.
46.
KlineP.TartariM. (2016). Bounding the labor supply responses to a randomized welfare experiment: A revealed preference approach. American Economic Review, 106(4), 972–1014. https://doi.org/10.1257/aer.20130824
47.
KnausM. C. (2021). A double machine learning approach to estimate the effects of musical practice on student's skills. Journal of the Royal Statistical Society: Series A, 184(1), 282–300. https://doi.org/10.1111/rssa.12623
48.
KnausM. C.LechnerM.StrittmatterA. (2022). Heterogeneous employment effects of job search programmes: A machine learning approach. Journal of Human Resources, 57(2), 597–636. https://doi.org/10.3368/jhr.57.2.0718-9615r1
49.
LaLondeR. (2003). Employment and training programs. In MoffittR. (Ed.). Means- tested transfer programs in the United States (pp. 517–585). University of Chicago Press.
LechnerM.WiehlerS. (2011). Kids or courses? Gender differences in the effects of active labor market policies. Journal of Population Economics, 24(3), 783-812. https://doi.org/10.1007/s00148-009-0267-2.
52.
LeiH.Nahum-ShaniI.LynchK.OslinD.MurphyS. A. (2012). A “SMART” design for building individualized treatment sequences. Annual Review of Clinical Psychology, 8(1), 14-28. https://doi.org/10.1146/annurev-clinpsy-032511-143152.
LinhartH.ZucchiniW. (1986). Model selection. Wiley.
55.
LiseJ.ShannonS.SmithJ. (2004). Equilibrium policy experiments and the evaluation of social programs. NBER WorkingPaper No. 10283.
56.
McCallB.SmithJ.WunschC. (2016). Government-sponsored vocational training. In HanushekE.ManchinS.WoessmanL. (Eds.), Handbook of the Economics of Education (Volume 5, pp. 479-652). North-Holland.
57.
McKenzieD. (2018). Can business owners form accurate counterfactuals? Eliciting treatment and control beliefs about their outcomes in the alternative treatment status. Journal of Business & Economic Statistics, 36(4), 714-722.
58.
MichalopoulosC.CardD.GennetianL.HarknettK.RobinsP. (2000). The self-sufficiency project at 36 months: Effects of a financial work incentive on employment and income. Social Research and Demonstration Corporation.
59.
MillJ. S. (1843). A system of logic. John W. Parker.
60.
MullerS. M. (2015). Causal interaction and external validity: Obstacles to the policy relevance of randomized evaluations. World Bank Economic Review, 29(Suppl 1), S217-S225. https://doi.org/10.1093/wber/lhv027.
61.
NeymanJ. (1935). Statistical problems in agricultural experiments. Journal of the Royal Statistical Society, 2(2), 107-180.
62.
OrrL.BloomH.BellS.DoolittleF.LinW. (1996). Does training for the disadvantaged work? Evidence from the national JTPA study. Urban Institute Press.
63.
PhilipsonT.HedgesL. V. (1998). Subject evaluation in social experiments. Econometrica, 66(2), 381–408. https://doi.org/10.2307/2998563
64.
PittM. M.RosenzweigM. R.HassanM. N. (2012). Human capital investment and the gender division of labor in a brawn-based economy. American Economic Review, 102(7), 3531–3560. https://doi.org/10.1257/aer.102.7.3531
RothschildM.WhiteL. J. (1995). The analytics of the pricing of higher education and other services in which the customers are inputs. Journal of Political Economy, 103(3), 573–586. https://doi.org/10.1086/261995
RubinD. B. (1974). Estimating causal effects of treatments in randomized and non-randomized studies. Journal of Educational Psychology, 66(5), 688–701. https://doi.org/10.1037/h0037350
69.
SandnerM.CornelissenT.JungmannT.HerrmannP. (2018). Evaluating the effects of a targeted home visiting program on maternal and child health outcomes. Journal of Health Economics, 58(1), 269-283. https://doi.org/10.1016/j.jhealeco.2018.02.008.
70.
SmithJ.WhalleyA.WilcoxN. (2020). Are program participants good evaluators?IZA Working Paper No. 13584.
71.
VanderWeeleT. J.HernanM. A. (2013). Causal inference under multiple versions of treatment. Journal of Causal Inference, 1(1), 1–20. https://doi.org/10.1515/jci-2012-0002
72.
WagerS.AtheyS. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523), 1228–1242. https://doi.org/10.1080/01621459.2017.1319839
73.
WaiteL.GallagherM. (2000). The case for marriage: Why married people are happier, healthier and better off financially. Doubleday.
74.
Watkins-HayesC. (2009). The new welfare bureaucrats: Entanglements of race, class, and policy reform. University of Chicago Press.
75.
WeissM. J.BloomH. S.BrockT. (2014). A conceptual framework for studying the sources of variation in program effects. Journal of Policy Analysis and Management, 33(3), 778–808. https://doi.org/10.1002/pam.21760
WoodR. G.MooreQ.ClarkwestA.KillewaldA. (2014). The long-term effects of building strong families: A program for unmarried parents. Journal of Marriage and Family, 76(2), 446-463. https://doi.org/10.1111/jomf.12094.
