This article uses propensity scores to identify subgroups of individuals most likely to experience a reduction in cash benefits because of sanctions in some of the programs that make up the National Evaluation of Welfare-to-Work Strategies. It extends program evaluation methodology by using propensity scoring to identify the subgroups of sanctioned and nonsanctioned welfare recipients. Specifically, the propensity score is used to identify the sample subset most likely to experience program sanction. In this application, the propensity score helps deal with an omitted variable problem, that of not knowing what the sanction status is in the control group (because they were not subject to the policies being tested). Findings reveal that being high sanction risk induces greater work levels and therefore higher earnings, but it also results in receiving less cash assistance so that sanctioned recipients have roughly the same net incomes as nonsanctioned ones.
Agodini, R., & Dynarski, M. (2001). Are experiments the only option? A look at dropout prevention programs. Unpublished report.
2.
Dehejia, R.H., & Wahba, S. (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association,94(448), 1053—1062.
3.
Dehejia, R.H., & Wahba, S. (2002). Propensity score-matching methods for nonexperimental causal studies. Review of Economics and Statistics, 84(1), 151—161.
4.
Fein, D.J., Beecroft, E., Hamilton, W., Lee, W.S., Holcomb, P.A., Thompson, T.S., et al. (1998). The Indiana welfare reform evaluation: Program implementation and economic impacts after two years. Bethesda, MD: Abt.
5.
Gibson, C.M. (2003). Privileging the participant: The importance of subgroup analysis in social welfare evaluations. American Journal of Evaluation, 24(4), 443—469.
6.
Hamilton, G., Brock, T., & Farkas, J. (1994). The JOBS evaluation: Early lessons from seven sites. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Office of the Assistant Secretary for Planning Evaluation; U.S. Department of Education, Office of the Under Secretary, Office of Vocational and Adult Education.
7.
Hamilton, G., Brock, T., Farrell, M., Friedlander, D., & Harknett, K. (with Hunter-Manns, J., Walter, J., & Weissman , J.). (1997). Evaluating two welfare-to-work program approaches: Two-year findings on the labor force attachment and human capital development programs in three sites . Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Office of the Assistant Secretary for Planning and Evaluation; U.S. Department of Education, Office of the Under Secretary, Office of Vocational and Adult Education.
8.
Harknett, K. (2006). Does receiving an earnings supplement affect union formation? Estimating effects for program participants using propensity score matching. Evaluation Review, 30(6), 741—778.
9.
Hill, J., Waldfogel, J., & Brooks-Gunn, J. (2002). Differential effects of high-quality child care. Journal of Policy Analysis and Management , 21(4), 601—627.
10.
Hollister, R.G., & Metcalf, C.E. (1977). Family labor-supply response in the New Jersey experiment. In H. Watts & A. Rees (Eds.), The New Jersey income-maintenance experiment. Volume 2: Labor supply responses. New York : Academic Press.
11.
Jenkins-Smith, H.C. (1990). Democratic politics and policy analysis. New York: Harcourt Brace.
12.
Kemple, J.J., & Snipes, J.C. (2000). Career academies: Impacts on students' engagement and performance in high school. New York: Manpower Demonstration Research Corporation.
13.
Lens, V. (2006). Examining the administration of work sanctions on the frontlines of the welfare system. Social Science Quarterly , 87(3), 573—590.
14.
Leow, C., Marcus, S., Zanutto, E., & Boruch, R. (2004). Effects of advanced course-taking on math and science achievement: Addressing selection bias using propensity scores. American Journal of Evaluation, 25(4), 461—478.
15.
Luellen, J.K., Shadish, W.R., & Clark, M.H. (2005). Propensity scores: An introduction and experimental test. Evaluation Review, 29(6), 530—558.
16.
Morris, P., Bloom, D., Kemple, J., & Hendra, R. (2003). The effects of a time-limited welfare program on children: The moderating role of parents' risk of welfare dependency. Child Development, 74(3), 851—874.
17.
Pavetti, L., Derr, M.K., & Hesket, H. (2003). Review of sanction policies and research studies: Final literature review. Washington, DC: Mathematica Policy Research, Inc.
18.
Peck, L.R. (2002). Subgroup analysis in social experiments. (Doctoral dissertation, Robert F. Wagner Graduate School of Public Service, New York University, 2002). Dissertation Abstracts International, 63, 359.
19.
Peck, L.R. (2003). Subgroup analysis in social experiments: Measuring program impacts based on posttreatment choice. American Journal of Evaluation, 24(2), 157—187.
20.
Rosenbaum, P.R., & Rubin, D. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41—55.
Schochet, P.Z., & Burghardt, J. (2007). Using propensity scoring to estimate program-related subgroup impacts in experimental program evaluations. Evaluation Review, 31(2), 95—120.
23.
Shadish, W.R., Luellen, J.K., & Clark, M.H. (2006). Propensity scores and quasi-experiments: A testimony to the practical side of Lee Sechrest. In R. R. Bootzin & P. E. McKnight (Eds.), Strengthening research methodology: Psychological measurement and evaluation (pp. 143—157). Washington, DC: American Psychological Association.
24.
U.S. Department of Health and Human Services, Office of Planning Research and Evaluation. (2001). Temporary Assistance for Needy Families program (TANF): Fourth annual report to Congress . Retrieved October 15, 2002, from http://www.acf.hhs.gov/programs/opre/ar2001/indexar.htm
25.
Wilde, E.T., & Hollister, R. (2002). How close is close enough? Testing nonexperimental estimates of impact against experimental estimates of impact with education test scores as outcomes (Discussion Paper No. 1242—02). Madison, WI: Institute for Research on Poverty.
26.
Yoshikawa, H., Magnuson, K.A., Bos, J.M., & Hsueh. J. (2003). Effects of earnings-supplement policies on adult economic and middle-childhood outcomes differ for the “hardest to employ.”Child Development, 74(5), 1500—1521.
