Integrating a Comparison Group Design into a Theory of Change Evaluation: The Case of the Urban Health Initiative

Abstract

This paper describes how we strengthened the theory of change approach to evaluating a complex social initiative by integrating it with a quasi-experimental, comparison group design. We also demonstrate the plausibility of selecting a credible comparison group through the use of cluster analysis, and describe our work in validating that analysis with additional measures. The integrated evaluation design relies on two points of comparison: (1) program theory to program experience; and (2) program cities to comparison cities. We describe how we are using this integrated design to evaluate the Robert Wood Johnson Foundation’s Urban Health Initiative, an effort that aims to improve health and safety outcomes for children and youth in five distressed urban areas through a process of citywide, multi-sector planning and changed public and private systems. We also discuss how the use of two research frameworks and multiple methods can enrich our ability to test underlying assumptions and evaluate overall program effects. Using this integrated approach has provided evidence that the earliest phases of this initiative are unfolding as the theory would predict, and that the comparison cities are not undergoing a similar experience to those in UHI. Despite many remaining limitations, this integrated evaluation can provide greater confidence in assessing whether future changes in health and safety outcomes may have resulted from the Urban Health Initiative (UHI).

Get full access to this article

View all access options for this article.

References

The Brookings Institution . (1998). Learning what works: Evaluating complex social interventions. Washington, DC: The Brookings Institute.

Chen, H. (1990). Theory-driven evaluations. London: Sage Publications.

Cook, T. D. (2000). The false choice between theory-based evaluation and experimentation. Program theory evaluation: Challenges and opportunities (No. 87, pp. 27-34). San Francisco, CA: Jossey-Bass.

Granger, R. C. (1998). Establishing causality in evaluations of Comprehensive Community Initiatives. In K. Fulbright-Anderson , A. C. Kubisch , & J. P. Connell (Eds.), New approaches to evaluating Community Initiatives: Vol. 2. Theory, measurement, and analysis (pp. 221-246). Washington, DC: The Aspen Institute.

Gruenewald, P. J. , Treno, A. J. , Taff, G. , & Klitzner, M. (1997). Measuring community indicators: A systems approach to drug and alcohol problems. Thousand Oaks, CA: Sage.

Hacsi, T. A. (2000). Using program theory to replicate successful programs. Program theory evaluation: Challenges and opportunities (No. 87, pp. 71-78). San Francisco, CA: Jossey-Bass.

Hatry, H. , Winnie, R. E. , & Fisk, D. (1981). Program analysis for state and local governments (2nd ed.). Washington, DC: The Urban Institute.

Hauser, R. M. , Brown, B. V. , & Prosser, W. R. (Eds.). (1997). Indicators of children’s well-being. New York: Russell Sage.

Hollister, R. G. , & Hill, J. (1995). Problems in the evaluation of community-wide initiatives. In J. Connell , A. Kubisch , L. Schorr , & C. Weiss (Eds.), New approaches to evaluating Community Initiatives: Concepts, methods, and contexts (pp. 127-171). Washington, DC: The Aspen Institute.

10.

Kubisch, A. C. , Fullbright-Anderson, K. , & Connell, J. P. (1998). Evaluating Community Initiatives: A progress report. In A. Kubisch , K. Fulbright-Anderson , & J. Connell (Eds.), New approaches to evaluating Community Initiatives: Theory, measurement, and analysis. Washington, DC: The Aspen Institute.

11.

Kubisch, A. , Weiss, C. , Schorr, L. , & Connell, J. (1995). Introduction. In J. Connell , A. Kubisch , L. Schorr , & C. Weiss (Eds.), New approaches to evaluating community initiatives: Concepts, methods, and contexts (pp. 1-23). Washington, DC: The Aspen Institute.

12.

Lippman, L. , Burns, S. , & McArthur, E. (1996). Urban schools: The challenge of location and poverty. Washington, DC: National Center for Education Statistics.

13.

National Evaluation of the Urban Health Initiative . (2000, November). State of the cities of the Urban Health Initiative: A baseline report. New York: New York University. http://www.nyu.edu/wagner/chpsr.

14.

Rogers, P. J. (2000). Casual models in program theory evaluation. In Program theory evaluation: Challenges and opportunities (No. 87, pp. 47-56). San Francisco, CA: Jossey-Bass.

15.

Rogers, P. , Petrosino, A. , Huebner, T. , & Hacsi T. (2000). Program theory evaluation: Practice, promise, and problems. In Program theory evaluation: Challenges and opportunities (No. 87, pp. 5-13). San Francisco, CA: Jossey-Bass.

16.

Rusk, D. (1996). Baltimore unbound: A strategy for regional renewal. Baltimore, MD: The Abell Foundation.

17.

Savitch, H. V. , & Vogel, R. K. (Eds.). (1996). Regional politics: America in a post-city age. In Urban affairs annual reviews (Vol. 45). Thousand Oaks, CA: Sage.

18.

Silver, D. , Weitzman, B. W. , & Brecher, C. (2002, in press). Setting an agenda for local action: The limits of expert opinion and community voice. Policy Studies Journal.

19.

Stufflebeam, D. (2001). Evaluation models. New Directions for Evaluation (No. 89).

20.

Swanstrom, T. (2001). What we argue about when we argue about regionalism. Journal of Urban Affairs, 33, 479-496.

21.

U.S. Department of Housing and Urban Development . (2000). State of the cities 2000. Washington, DC: HUD.

22.

Weiss, C. (1997). How can theory-based evaluation make greater headway? Evaluation Review, 21, 501-524.