A fundamental question within the field of program evaluation is “Do social programs work?” Although experiments allow us to answer this question with certainty, they have some limitations. Experiments generate mean program impacts and even mean impacts by subgroup, but they often leave unexplored the impacts on subgroups determined by treatment use. This work proposes a methodology for analyzing the impacts of social programs on previously unexamined subgroups. Rather than using a single trait to define subgroups—which is currently the dominant method of subgroup analysis—the proposed approach estimates the impact of programs on subgroups identified by a post-treatment choice while still maintaining the integrity of the experimental research design. Analysis of data from the experimental evaluation of New York State’s Child Assistance Program (CAP) provides an application of the proposed technique.
Get full access to this article
View all access options for this article.
References
1.
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91, 444-455.
2.
Angrist, J. D., & Krueger, A. B. (1994). Split sample instrumental variables. Cambridge, MA: National Bureau of Economic Research, Technical Working Paper #150.
3.
Beecroft, E., & Lee, W. S. (2000). Looking beyond mean impacts to see who gains and who loses with time-limited welfare: Evidence from the Indiana welfare reform evaluation. Paper presented at the Annual Meeting of the National Association for Welfare Research and Statistics, Scottsdale, AZ.
4.
Bloom, H. S. (1984). Accounting for no-shows in experimental evaluation designs. Evaluation Review, 8, 225-246.
5.
Bloom, H. S., Hill, C. J., & Riccio, J. (2001). Modeling the performance of welfare-to-work programs: The effects of program management and services, economic environment, and client characteristics. New York, NY: Manpower Demonstration Research Corporation Methodology Working Paper.
6.
Bos, J. M. (1995). The labor market value of remedial education: Evidence from time series data on an experimental program for school dropouts (Doctoral dissertation, Robert F. Wagner Graduate School of Public Service, New York University, 1995). Dissertation Abstracts International, 56,4139.
7.
Boudett, K. P., & Friedlander, D. (1997). Does mandatory basic education improve achievement test scores of AFDC recipients? A reanalysis of data from California’s GAIN Program. Evaluation Review, 21, 568-588.
8.
Dehejia, R. H., & Wahba, S. (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association, 94, 1053-1062.
9.
Eberwein, C., Ham, J. C., & LaLonde, R. J. (1997). The impact of being offered and receiving classroom training on the employment histories of disadvantaged women: Evidence from experimental data. Review of Economic Studies, 64, 655-682.
10.
Fein, D. J., Beecroft, E., Hamilton, W., Lee, W. S., Holcomb, P. A., Thompson, T. S., & Ratcliffe, C. E. (1998). The Indiana welfare reform evaluation: Program implementation and economic impacts after two years. Cambridge, MA: Abt Associates Inc.
11.
Frangakis, C. E., & Rubin, D. B. (2000). The defining role of “principal causal effects” in comparing treatments using general post-treatment variables. In Proceedings of the Epidemiology Section, American Statistical Association (pp. 23-32).
12.
Friedlander, D. (1993). Subgroup impacts of large-scale welfare employment programs. Review of Economics and Statistics, 75, 138-143.
13.
Friedlander, D., & Robins, P. K. (1997). The distributional impacts of social programs. Evaluation Review, 21, 531-553.
14.
Hamilton, W. L., Burstein, N. R., Baker, A. J., Earle, A., Gluckman, S., Peck, L., & White, A. (1996). The New York State Child Assistance Program: Five year impacts, costs, and benefits. Cambridge, MA: Abt Associates Inc.
15.
Hargreaves, M. (1992). The New York Child Assistance Program: Interim report on implementation. Cambridge, MA: Abt Associates Inc.
16.
Heckman, J. J. (1999). Accounting for heterogeneity, diversity and general equilibrium in evaluating social programs. Cambridge, MA: National Bureau of Economic Research, Working Paper #7230.
17.
Heckman, J. J., & Robb, R., Jr. (1985a). Alternative methods for evaluating the impact of interventions. In J. Heckman & B. Singer (Eds.), Longitudinal analysis of labor market data (pp. 145-245). New York, NY: Cambridge University Press.
18.
Heckman, J. J., & Robb, R., Jr. (1985b). Alternative methods for evaluating the impact of interventions: An overview. Journal of Econometrics, 30, 239-367.
19.
Heckman, J., Smith, J., & Clements, N. (1997). Making the most out of program evaluations and social experiments: Accounting for heterogeneity in program impacts. Review of Economic Studies, 64, 487-535.
20.
Heckman, J., Smith, J., & Taber, C. (1998). Accounting for dropouts in evaluations of social programs. Review of Economics and Statistics, 130, 1-14.
21.
Hirano, K., Imbens, G. W., Rubin, D., & Zhou, X. (1999). Assessing the effect on an influenza vaccine in an encouragement design. Unpublished manuscript.
22.
Hollister, R. G., & Metcalf, C. E. (1977). Family labor-supply response in the New Jersey experiment. In H. Watts & A. Rees (Eds.), The New Jersey income-maintenance experiment: Volume II: Labor supply responses (pp. 185-220). New York, NY: Academic Press.
23.
Imbens, G. W., & Rubin, D. B. (1997). Estimating outcome distributions for compliers in instrumental variable models. Review of Economic Studies, 64, 555-574.
24.
Kemple, J. J., & Snipes, J. C. (2000). Career academies: Impacts on students’ engagement and performance in high school. New York, NY: Manpower Demonstration Research Corporation.
25.
Manski, C. F. (1995). Identification problems in the social sciences. Cambridge, MA: Harvard University Press.
26.
Manski, C. F. (1996). Learning about treatment effects from experiments with random assignment of treatments. Journal of Human Resources, 31, 709-733.
27.
Manski, C. F. (1997). The mixing problem in program evaluation. Review of Economic Studies, 64, 537-553.
28.
Michalopolous, C., & Schwartz, C. (2000). What works best for whom: Impacts of 20 welfare-to-work programs by subgroup. Washington, DC: U.S. Department of Health and Human Services, Office of Assistant Secretary for Planning and Evaluation and Administration for Children and Families; and U.S. Department of Education.
29.
Nagin, D. S., & Land, K. C. (1993). Age, criminal careers, and population heterogeneity: Specification and estimation of a nonparametric mixed poisson model. Criminology, 31, 327-362.
30.
Peck, L. (1999). Do social programs affect nonparticipants? Evidence from the child assistance program. Paper presented at the New York University Robert F. Wagner Graduate School of Public Service Doctoral Colloquium, New York, NY.
31.
Peck, L. R. (2002). Subgroup analysis in social experiments. (Doctoral dissertation, Robert F. Wagner Graduate School of Public Service, New York University, 2002). Dissertation Abstracts International, 63, 359.
32.
Yoshikawa, H., Rosman, E. A., & Hsueh, J. (2001). Variation in teenage mothers’ experiences of child care and other components of welfare reform: Selection processes and developmental consequences. Child Development, 72, 299-317.
