Sage Journals: Discover world-class research

Abstract

This article uses the evaluation of the Robert Wood Johnson Foundation’s (RWJF) Urban Health Initiative (UHI), a 10-year effort to improve health and safety outcomes in distressed cities, to demonstrate the strength of an evaluation design that integrates theory of change and quasi-experimental approaches, including the use of comparison cities. This paper focuses on the later stages of implementation and, especially, our methods for estimating program impacts. While the theory of change was used to make preliminary identification of intended outcomes, we used the sites’ plans and early implementation to refine this list and revisit our strategy for estimating impacts. Using our integrated design, differences between program and comparison cities are considered impacts only if they were predicted by program theory, local plans for action, and early implementation. We find small, measurable changes in areas of greatest programmatic effort. We discuss the importance of the integrated design in identifying impacts.

Keywords

theory of change comprehensive community initiatives quasi-experimental designs child health urban health

Get full access to this article

View all access options for this article.

References

Bloom, H.S. , Riccio, J.A. , & Verma, N. ( 2005). Promoting work in public housing: The effectiveness of Jobs-Plus. New York: MDRC.

Brecher, C. , Searcy, C. , Silver, D. , & Weitzman, B.C. (2004). What does government spend on children? Evidence from five cities. Washington, DC : The Brookings Institution.

Burton, P. , Goodlad, R. , & Croft, J. ( 2006). How would we know what works? Context and complexity in the evaluation of community involvement. Evaluation, 12, 294-312.

Chen, H. ( 1990). Theory-driven evaluations. London : Sage.

Connell, J.P. , & Kubisch, A.C. ( 1998). Applying a theory of change approach to the evaluation of comprehensive community initiatives: Progress, prospects and problems. In K. Fulbright-Anderson, A. Kubisch, & J. Connell (Eds.), New approaches to evaluating community initiatives, vol 2: Theory, measurement and analysis. Washington, DC: The Aspen Institute.

Cook, T.D. ( 2000). The false choice between theory-based evaluation and experimentation. New Directions in Evaluation, 87, 27-34.

Davidson, E.J. ( 2000). Ascertaining causality in theory-based evaluation. Program Theory in Evaluation: Challenges and Opportunities, 87, 5-13.

Emshoff, J.G. , Darnell, A.J. , Darnell, D.A. , Erickson, S.W. , Schneider, S. , & Hudgins, R. ( 2007). Systems change as an outcome and a process in the work of community collaboratives for health. American Journal of Community Psychology, 39, 255-267.

Hollister, R.G. , & Hill, J. ( 1995). Problems in the evaluation of community-wide initiatives . In J. Connell , A. Kubisch , L. Schorr , & C. Weiss (Eds.), New approaches to evaluating community initiatives: Concepts, methods, and contexts. Washington, DC: The Aspen Institute.

10.

Jolliffe, I.T. ( 2002). Principal component analysis. New York: Springer-Verlag.

11.

Judge, K. , & Bauld, L. ( 2001). Strong theory, flexible methods: Evaluating complex community-based initiatives. Critical Public Health, 11, 19-38.

12.

Kling, J.R. , Liebman, J.B. , & Katz, L.F. ( 2007). Experimental analysis of neighborhood effects. Econometrica, 75, 83-119.

13.

Kubisch, A.C. , Fulbright-Anderson, K. , & Connell, J.P. ( 1998). Evaluating community initiatives: A progress report. In K. Fulbright-Anderson, A. Kubisch, & J. Connell (Eds.), New approaches to evaluating community initiatives, vol 2: Theory, measurement and analysis. Washington, DC: The Aspen Institute.

14.

Rindskopf, D. , & Saxe, L. ( 1998). Zero effects in substance abuse programs: Avoiding false positives and false negatives in the evaluation of community based programs . Evaluation Review, 22, 78-94.

15.

Rogers, P.J. , Petrosino, A. , Huebner, T.A. , & Hacsi, T.A. ( 2000). Program theory evaluation: Practice, promise, and problems . Program Theory in Evaluation: Challenges and Opportunities , 87, 5-13.

16.

Silver, D. , & Weitzman, B.C. (2009). The pros and cons of comprehensive community initiatives at the city level: The case of the Urban Health Initiative . The Foundation Review, 1, 85-95.

17.

Silver, D. , Weitzman, B.C. , & Brecher, C. ( 2002). Setting an agenda for local action: The limits of expert opinion and community voice. Policy Studies Journal, 30, 363-378.

18.

Swanson, C.B. ( 2003). Who Graduates? Who Doesn’t? A Statistical Portrait of Public High School Graduation, Class of 2001. Washington, D.C.: The Urban Institute.

19.

Weitzman, B.C. , Silver, D. , & Brazill, C. ( 2006). Efforts to improve public policy and programs through data practice: Experiences in 15 distressed cities. Public Administration Review, 66, 386-399.

20.

Weitzman, B.C. , Silver, D. , & Dillman, K.N. ( 2002). Integrating a comparison group design into a theory of change evaluation: The case of the Urban Health Initiative. American Journal of Evaluation, 23, 371-385.

Finding the Impact in a Messy Intervention: Using an Integrated Design to Evaluate a Comprehensive Citywide Health Initiative

Abstract

Keywords

Get full access to this article

References