Do Family Caps on Welfare Affect Births Among Welfare Recipients? Reconciling Efficacy and Effectiveness Estimates of Impact through a Blended Design Strategy

Abstract

This paper proposes a blended experimental and quasi-experimental research strategy which we believe will help improve the external validity of evaluations of welfare reform and other social policies. We draw upon data from New Jersey’s imposition of a family cap, where welfare benefits instead of increasing remained the same for women who conceive and bear a child while on public assistance. We directly compare estimates of the policy’s effectiveness on the state’s welfare population using (a) a point-in-time sample randomly assigned to experimental and control group conditions; (b) an initial cohort sample similarly assigned; and (c) a before and after policy implementation of the entire welfare caseload. We find that these estimates can differ dramatically, reflecting sensitivity to a time-induced heterogeneity we have labeled the Heraclitus effect. We discuss how multiple estimates of effectiveness can be used to bound expectations of a welfare reform policy impact when population dynamics data are available.

Get full access to this article

View all access options for this article.

References

Blank, R. M. (1989). Analyzing the length of welfare spells. Journal of Public Economics, 39, 45-273.

Burtless, G. (1995). The case for randomized field trials in economic and policy research. Journal of Economic Perspectives, 9, 63-84.

Burtless, G. , & Orr, L. L. (1986). Are classical experiments needed for manpower policy? Journal of Human Resources, 21, 606-639.

Cain, G. G. (1997). Controlled experiments in evaluating the new welfare programs. Focus, 18, 25-28.

Camasso, M. J. (in press). Isolating the family cap effect on fertility behavior: Evidence from New Jersey’s Family Development Program Experiment. Journal of Contemporary Economic Policy.

Camasso, M. J. , Jagannathan, R. , Harvey, C. , & Killingsworth, M. (2003). The use of client surveys to gauge the threat of contamination in welfare reform experiments. Journal of Policy Analysis and Management, 22, 207-223.

Camasso, M. J. , Jagannathan, R. , Killingsworth, M. , & Harvey, C. (2003). New Jersey’s family cap and family size decisions: Findings from a five-year evaluation. Research in Labor Economics, 22, 71-112.

Campbell, D. T. , & Stanley, J. C. (1966). Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally.

Cook, T. D. , & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago, IL: Rand McNally.

10.

Ellwood, D. T. , & Bane, M. J. (1994). Understanding welfare dynamics. In M. J. Bane & D. T. Ellwood (Eds.), Welfare realities: From rhetoric to reform (pp. 28-66). Cambridge, MA: Harvard University Press.

11.

Friedlander, D. , & Robins, P. K. (1995). Evaluating program evaluations: New evidence on commonly used nonexperimental methods. American Economic Review, 85, 923-937.

12.

Garfinkel, I. , Manski, C. F. , & Michalopoulos, C. (1992). Micro experiments and macro effects. In C. F. Manski & I. Garfinkel (Eds.), Evaluating welfare and training programs (pp. 253-273) Cambridge, MA: Harvard University Press.

13.

Greenberg, D. , & Shroder, M. (1997). The digest of social experiments (2nd ed.). Washington, DC: The Urban Institute Press.

14.

Grissmer, D. , Flanagan, A. , Kawata, J. , & Williamson, S. (2000). Improving students achievement. Santa Monica, CA: Rand.

15.

Gueron, J. M. , & Pauly, E. (1991). From welfare to work. New York: Russell Sage.

16.

Heckman, J. J. (1992). Randomization and social policy evaluation. In C. F. Manski & I. Garfinkel (Eds.), Evaluating welfare and training programs (pp. 201-230). Cambridge, MA: Harvard University Press.

17.

Heckman, J. J. , & Hotz, V. J. (1989). Choosing among alternative nonexperimental methods for estimating the impact of social programs: The case of manpower training. Journal of the American Statistical Association, 84, 862-874.

18.

Heckman, J. J. , & Smith, J. A. (1995). Assessing the case for social experiments. Journal of Economic Perspectives, 9, 85-110.

19.

Heckman, J. J. , Smith, J. A. , & Clements, N. (1997). Making the most out of programme evaluations and social experiments: Accounting for heterogeneity in programme impacts. Review of Economic Studies, 64, 487-535.

20.

Horvath-Rose, A. , & Peters, H. E. (2001). Welfare waivers and nonmarital childbearing. In G. Duncan & P. L. Chase-Lansdale (Eds.), For better and for worse: Welfare reform and the wellbeing of children and families. New York, NY: Russell Sage.

21.

Hotz, V. J. , Imbens, G. W. , & Mortimer, J. H. (1999). Predicting the efficacy of future training programs using past experiences (NBER-238). Cambridge, MA: National Bureau of Economic Research.

22.

Hoynes, H. W. (1997). Work, welfare, and family structure: What have we learned? In A. J. Auerbach (Ed.), Fiscal policy: Lessons from economic research (pp. 101-146). London: MIT Press.

23.

Jagannathan, R. , Camasso, M. J. , & Killingsworth, M. (2004). New Jersey’s Family Cap experiment: Do fertility impacts differ by racial density? Journal of Labor Economics, 22, 431-460.

24.

Manski, C. F. (1996). Learning about treatment effects from experiments with random assignment of treatments. Journal of Human Resources, 31, 709-733.

25.

Manski, C. F. (1997). The mixing problem in programme evaluation. Review of Economic Studies, 64, 537-553.

26.

Meyer, B. (1995). Natural and quasi-experiments in economics. Journal of Business and Economic Statistics, 13, 151-161.

27.

Moffitt, R. A. (1992). Evaluating methods for program entry effects. In C. F. Manski & I. Garfinkel (Eds.), Evaluating welfare and training programs (pp. 231-252). Cambridge, MA: Harvard University Press.

28.

Moffitt, R. A. (1998). The effect of welfare on marriage and fertility. In R. A. Moffitt (Ed.), Welfare, the family, and reproductive behavior (pp. 50-97). Washington, DC: National Academy Press.

29.

Moffitt, R. A. , & Ver Ploeg, M. (Eds.). (1999). Evaluating welfare reform: A framework and review of current work. Washington, DC: National Academy Press.

30.

Moffitt, R. A. , Ver Ploeg, M. (Eds.). (2001). Evaluating welfare reform in an era of transition. Washington, DC: National Academy Press.

31.

Mohr, L. B. (1995). Impact analysis for program evaluation. Thousand Oaks, CA: Sage.

32.

Nathan, P. E. (2004). The clinical utility of therapy research: Bridging the gap between the present and the future. In A. R. Roberts & K. R. Yeager (Eds.), Evidence-based practice manual (pp. 949-960). New York, NY: Oxford University Press.

33.

Nathan. P. E. , & Gorman, J. M. (2002). Efficacy, effectiveness and clinical utility of psychotherapy research. In P. E. Nathan & J. M. Gorman (Eds.), A guide to treatments that work (pp. 643-654). New York, NY: Oxford University Press.

34.

O’Neill, D. M. , & O’Neill, J. F. (1997). Lessons for welfare reform: An analysis of the AFDC caseload and past welfare-to-work programs. Kalamazoo, MI: WE Upjohn Institute.

35.

O’Neill, J. A. , Bassi, L. J. , & Wolf, D. A. (1987). The duration of welfare spells. Review of Economics and Statistics, 69, 241-249.

36.

Orr, L. L. (1999). Social experiments: Evaluating public programs with experimental methods. Thousand Oaks, CA: Sage.

37.

Pepper, J. V. (2003). Using experiments to evaluate performance standards: What do welfare-to-work demonstrations reveal to welfare reformers? Journal of Human Resources, 38, 860-880.

38.

Plotnick, R. (1983). Turnover in the AFDC population: An event history analysis. Journal of Human Resources, 18, 65-81.

39.

Rank, M. R. (1985). Exiting from welfare: A life-table analysis. Social Service Review, 59, 358-376.

40.

Robinson, T. M. (1987). Fragments: A text and translation/Heraclitus. Toronto: University of Toronto Press.

41.

Rosenbaum, P. R. (2002). Observational studies (2nd ed.). New York, NY: Springer.

42.

Rossi, P. H. , Freeman, H. E. , & Lipsey, M. W. (1999). Evaluation: A systematic approach. Thousand Oaks, CA: Sage.

43.

Shadish, W. R. , Cook, T. D. , & Campbell. D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

44.

Sommer, A. , & Zeger, S. L. (1991). On estimating efficacy from clinical trials. Statistics in Medicine, 10, 45-52.

45.

Stevens, D. (2000). Welfare, employment, and earnings. Memorandum proposal for the panel on data and methods for measuring the effects of changes in social welfare programs. Committee on National Statistics. Baltimore, MD: University of Baltimore.

46.

Stock, J. H. , & Watson, M. W. (2003). Introduction to econometrics. Boston, MA: Addison Wesley.

47.

Toklas, A. B. (1963). What is remembered. New York: Holt, Rinehart, and Winston.

48.

Ver Ploeg, M. (2002). Preexit benefit receipt and employment histories and postexit outcomes of welfare leavers. In M. Ver Ploeg , R. A. Moffitt , & C. F. Citro (Eds.), Studies of welfare populations: Data collection and research issues (pp. 415-472). Washington, DC: National Academy Press.