Estimating the Impact of Emergency Assistance on Educational Progress for Low-Income Adults: Experimental and Nonexperimental Evidence

Abstract

Methods for estimating causal impact aim to either remove or reduce bias. This study estimates the degree of bias reduction obtained from regression adjustment and propensity score methods when only a weak set of predictors are available. The study uses an experimental test of providing emergency financial assistance to participants in a job training program to estimate an experimental benchmark and compares it to nonexperimental estimates of the impact of receiving assistance. When estimating the impact of receiving assistance, those who received it constitute the treatment group. The study explores two different comparison groups: those who could have (because they were assigned to the experimental treatment group) but did not receive emergency assistance; and those who could not receive emergency assistance because they were randomly assigned to the experimental control group. It uses these groups to estimate impacts by applying three estimation strategies: unadjusted mean comparison, regression adjustment, and inverse propensity weighting. It then compares these estimates to the experimental benchmark using statistical tests recommended by the within-study comparison literature. The nonexperimental approaches to addressing selection bias suggest large positive impacts. These are statistically different from the experimental benchmark, which shows that receipt of emergency assistance does not improve educational progress. Further, over 90% of the bias from a simple comparison of means remains. Unless a stronger set of predictors are available, future evaluations of such interventions should be wary of relying on these methods for either unbiased estimation of impacts or bias reduction.

Keywords

nonexperimental evaluation experimental evaluation treatment on the treated within-study comparison emergency assistance

Get full access to this article

View all access options for this article.

References

Andrews

Armstrong

T. B.

(2017). Unbiased instrumental variables estimation under known first‐stage sign. Quantitative Economics, 8(2), 479–503. https://doi.org/10.3982/QE700.

Andrews

Armstrong

T. B

Nichols

(2019). aaniv: Stata module for unbiased IV regression.

Angrist

J. D.

Imbens

G. W.

Rubin

D. B.

(1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434), 444–455. DOI: 10.1080/01621459.1996.10476902.

Angrist

J. D.

Pischke

J. S.

(2008). Mostly harmless econometrics: An empiricist’s companion. Princeton University Press.

Austin

P. C.

(2009). Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity‐score matched samples. Statistics in Medicine, 28(25), 3083–3107. https://doi.org/10.1002/sim.3697.

Bang

Robins

J. M.

(2005). Doubly robust estimation in missing data and causal inference models. Biometrics, 61(4), 962–973. https://doi.org/10.1111/j.1541-0420.2005.00377.x.

Barnow

B. S.

Smith

(2015). Employment and training programs Economics of means tested transfer programs in the United States (pp. 127–234). University of Chicago Press.

Baum

Schwartz

(2015). Student aid, student behavior, and educational attainment. In Castleman

B. L.

Schwartz

Baum

(Eds), Decision making for student success: Behavioral insights to improve college access and persistence (pp. 38–62). Routledge.

Bell

S. H.

Orr

L. L.

Blomquist

J. D.

Cain

G. G.

(1995). Program applicants as a comparison group in evaluating training programs: Theory and a test. Kalamazoo, MI: W. E. Upjohn Institute for Employment Research.

10.

Black

D. A.

Joo

LaLonde

Smith

Taylor

E. J.

(2017). Simple tests for selection: Learning more from instrumental variables (No. 6392). CESifo Working Paper.

11.

Bloom

(1984). Accounting for no-shows in experimental evaluation designs. Evaluation Review, 8(2), 225–246. https://doi.org/10.1177/0193841X8400800205.

12.

Busso

DiNardo

McCrary

(2014). New evidence on the finite sample properties of propensity score reweighting and matching estimators. Review of Economics and Statistics, 96(5), 885–897. https://doi.org/10.1162/REST_a_00431.

13.

Calónico

Smith

(2017). The women of the national supported work demonstration. Journal of Labor Economics, 35(S1), S65–S97. https://doi.org/10.1086/692397.

14.

Cook

T. D.

Shadish

W. R.

Wong

V. C.

(2008). Three conditions under which experiments and observational studies often produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27(4), 724–750. https://doi.org/10.1002/pam.20375.

15.

Deaton

Cartwright

(2018). Understanding and misunderstanding randomized controlled trials. Social Science & Medicine, 210, 2–21. https://doi.org/10.1016/j.socscimed.2017.12.005.

16.

Dehejia

R. H.

Wahba

(1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association, 94(448), 1053–1062. DOI: 10.1080/01621459.1999.10473858.

17.

Duke-Benfield

A. E.

(2015). Bolstering non-traditional student success: A comprehensive student aid system using financial aid, public benefits, and refundable tax credits. Available at: https://www.clasp.org/sites/default/files/public/resources-and-publications/publication-1/Bolstering-NonTraditional-Student-Success.pdf

18.

Fraker

Maynard

(1987). The adequacy of comparison group designs for evaluations of employment-related programs. Journal of Human Resources, 22(2), 194–227. https://doi.org/10.2307/145902.

19.

Geckeler

(2008). Helping community college students cope with financial emergencies: Lessons from the dreamkeepers and angel fund emergency financial aid programs. MDRC.

20.

Glazerman

Levy

D. M.

Myers

(2003). Nonexperimental versus experimental estimates of earnings impacts. The Annals of the American Academy of Political and Social Science, 589(1), 63–93. https://doi.org/10.1177/0002716203254879.

21.

Harvill

Litwok

Moulton

Rulf Fountain

Peck

L. R.

(2018). Technical supplement to the Health profession opportunity grants (HPOG) impact study interim report: Report appendices. Washington, D.C.: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, OPRE report 2018-16b.

22.

Heckman

Hohmann

Smith

Khoo

(2000). Substitution and dropout bias in social experiments: A study of an influential social experiment. The Quarterly Journal of Economics, 115(2), 651–694. https://doi.org/10.1162/003355300554764.

23.

Heckman

Smith

Taber

(1998). Accounting for dropouts in evaluations of social programs. Review of Economics and Statistics, 80(1), 1–14. https://doi.org/10.1162/003465398557203.

24.

Heckman

J. J.

Hotz

V. J.

(1989). Choosing among alternative nonexperimental methods for estimating the impact of social programs: The case of manpower training. Journal of the American Statistical Association, 84(408), 862–874. DOI: 10.1080/01621459.1989.10478848.

25.

Heckman

J. J.

Ichimura

Todd

P. E.

(1997). Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. The Review of Economic Studies, 64(4), 605–654. https://doi.org/10.1111/1467-937X.00044.

26.

Heckman

J. J.

Smith

J. A.

(1995). Assessing the case for social experiments. Journal of Economic Perspectives, 9(2), 85–110. DOI: 10.1257/jep.9.2.85.

27.

Imbens

G. W.

Angrist

J. D.

(1994). Identification and estimation of local average treatment effects. Econometrica, 62(2), 467–475. https://doi.org/10.2307/2951620.

28.

King

Nielsen

(2019). Why propensity scores should not be used for matching. Political Analysis, 27(4), 435–454. https://doi.org/10.1017/pan.2019.11.

29.

Kowalski

A. E.

(2016). Doing more when you're running late: Applying marginal treatment effect methods to examine treatment effect heterogeneity in experiments (No. w22363). National Bureau of Economic Research.

30.

LaLonde

R. J

(1986). Evaluating the econometric evaluations of training programs with experimental data. The American Economic Review, 76(4), 604–620. https://www.jstor.org/stable/1806062.

31.

Litwok

Walton

Harmon

Peck

L. R.

(2022). Health profession opportunity grants (HPOG) impact study: Six-year impacts report appendix . Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, OPRE Report 2022–45.

32.

Litwok

Walton

Peck

L. R.

Harvill

(2019). Health profession opportunity grants (HPOG) impact study’s three-year follow-up analysis plan . Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. OPRE Report 2018–124.

33.

Michalopoulos

Bloom

H. S.

Hill

C. J.

(2004). Can propensity-score methods match the findings from a random assignment evaluation of mandatory welfare-to-work programs? The Review of Economics and Statistics, 86(1), 156–179. https://doi.org/10.1162/003465304323023732.

34.

Montiel-Olea

J. L.

Pflueger

(2013). A robust test for weak instruments. Journal of Business & Economic Statistics, 31(3), 358–369. https://doi.org/10.1080/00401706.2013.806694.

35.

Orozco

Mayo

(2011). Keeping students enrolled: How community colleges are boosting financial resources for their students. Available at: https://www.demos.org/sites/default/files/publications/KeepingStudentsEnrolled_Demos.pdf

36.

Peck

L. R.

Litwok

Walton

(2022). Health profession opportunity grants (HPOG 1.0) impact study: Six-year impacts report. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, OPRE Report 2022-45.

37.

Peck

L. R.

Litwok

Walton

Harvill

Werner

(2019). Health profession opportunity grants (HPOG 1.0) impact study: Three-year impacts report. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, OPRE Report 2019-114.

38.

Peck

L. R.

Werner

Harvill

Litwok

Moulton

Rulf Fountain

Locke

(2018). Health profession opportunity grants (HPOG 1.0) impact study interim report: Program implementation and short-term Impacts. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, OPRE Report 2018-16a.

39.

Rosenbaum

P. R.

Rubin

D. B.

(1983). The central role of the propensity score inobservational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41.

40.

Smith

J. A.

Todd

P. E.

(2005). Does matching overcome LaLonde’s critique of nonexperimental estimators? Journal of Econometrics, 125(1–2), 305–353. https://doi.org/10.1016/j.jeconom.2004.04

41.

StataCorp (2019a). User manual: Kdensity. Stata Statistical Software: Release.

42.

StataCorp (2019b). User manual: Teffects aipw. Stata Statistical Software: Release.

43.

Steiner

P. M.

Wong

V. C.

(2018). Assessing correspondence between experimental and nonexperimental estimates in within-study comparisons. Evaluation Review, 42(2), 214-247. Assessing Correspondence Between Experimental and Nonexperimental Estimates in Within-Study Comparisons Peter M. Steiner, Vivian C. WongFirst Published May 17, 2018 Research Article Find in PubMed. https://doi.org/10.1177/0193841X18773807.

44.

Werner

Loprest

Schwartz

Koralek

Sick

(2018). National implementation evaluation of the first round Health Profession Opportunity Grants (HPOG 1.0). Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, OPRE Report 2018-09.

45.

Wong

V. C.

Steiner

P. M.

(2018). Designs of empirical evaluations of nonexperimental methods in field settings. Evaluation Review, 42(2), 176–213. https://doi.org/10.1177/0193841X18778918.

46.

Wong

V. C.

Steiner

P. M.

Anglin

K. L.

(2018). What can be learned from empirical evaluations of nonexperimental methods? Evaluation Review, 42(2), 147–175. https://doi.org/10.1177/0193841X18776870.

47.

Wooldridge

J. M.

(2010). Econometric analysis of cross section and panel data. MIT press.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.58 MB