When is a Program Ready for Rigorous Impact Evaluation? The Role of a Falsifiable Logic Model

Abstract

Background:

Recent reviews suggest that many plausible programs are found to have at best small impacts not commensurate with their cost, and often have no detectable positive impacts at all. Even programs with initial rigorous impact evaluation (RIE) that show them to be effective often fail a second test with an expanded population or at multiple sites.

Objective:

This article argues that more rapid movement to RIE is a partial cause of the low success rate of RIE and proposes a constructive response: process evaluations that compare program intermediate outcomes—in the treatment group, during the operation of the program—against a more falsifiable extension of the conventional logic model.

Conclusion:

Our examples suggest that such process evaluations would allow funders to deem many programs unlikely to show impacts and therefore not ready for random assignment evaluation—without the high cost and long time lines of an RIE. The article then develops the broader implications of such a process analysis step for broader evaluation strategy.

Keywords

design and evaluation of programs and policies outcome evaluation program design and development

Get full access to this article

View all access options for this article.

References

Anderson

2010. “Proven Programs are the Exception, not the Rule [Web log post].” Accessed January 3, 2013, http://blog.givewell.org/2008/12/18/guest-post-proven-programs-are-the-exception-not-the-rule/

Azurdia

Barnes

. 2008. The Employment Retention and Advancement Projects: Impacts for Portland’s Career Builders Program. New York: Manpower Demonstration Research Corporation.

Berk

(2011). Evidence-Based Versus Junk-Based Evaluation Research: Some Lessons from 35 Years of the Evaluation Review. Evaluation Review, 35(3), 191–203.

Bernstein

Dun Rappaport

Olsho

Hunt

Levin

. 2009. “Impact Evaluation of the U.S. Department of Education’s Student Mentoring Program” NCEE 2009-4047, National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education, Washington, DC.

Bos

J. M.

Scrivener

Snipes

Hamilton

. 2002. Improving Basic Skills: The Effects of Adult Education in Welfare-to-Work Programs. New York: Manpower Demonstration Research Corporation.

Brown

Klerman

J. A.

. 2012. “Insights from Public Accounting for Contracting for Independent Evaluation”. Evaluation Review 36:4(June):185–2197. doi 10.1177/0193841X12450163

Campbell

D. T.

1969. “Reforms as Experiments.” American Psychologist 24:409–29.

Campbell

D. T.

1984. “Can we be Scientific in Applied Social Science?” Evaluation Studies Review Annual 9:26–48.

Coalition for Evidence Based Policy. 2009. “The Coalition is Pleased with GAO’s Confirmation of the Top Tier Initiative’s Adherence to Rigorous Standards and Overall Transparency.” Accessed January 3, 2013, http://coalition4evidence.org/wordpress/?page_id=410.

10.

Connell

J. P.

Kubisch

A. C.

. 1998. “Applying a Theory of Change Approach to the Evaluation of Comprehensive Community Initiatives: Progress, Prospects, and Problems.” The Aspen Institute. Accessed January 3, 2013, https://communities.usaidallnet.gov/fa/system/files/Applying+Theory+of+Change+Approach.pdf

11.

Conrad

K. J.

Randolph

Kirby

Jr Bebout

R. R.

1999. “Creating and using Logic Models—Four Perspectives.” Alcoholism Treatment Quarterly 17:17–31.

12.

GiveWell. (n.d.). “Social Programs that Just don’t Work.” Accessed January 3, 2013, http://www.givewell.org/giving101/Social-Programs-That-Just-Dont-Work

13.

Ioannidis

J. P. A.

Haidich

A. B.

Pappa

Pantazis

Kokori

S. I.

Tektonidou

M. G.

Contopoulos-Ionnidis

D. G.

Lau

. 2001. “Comparison of Evidence of Treatment Effects in Randomized and Nonrandomized Studies.” Journal of the American Medical Association 286:821–30.

14.

Jalbert

Rhodes

. 2010. Reduced Caseloads Improve Probation Outcomes in Oklahoma City. Cambridge, MA: Abt Associates.

15.

James-Burdumy

Deke

Lugo-Gil

Crey

Hershey

Gersten

Newman-Gonchar

Dimino

Haymond

Faddis

. 2010. “Effectiveness of Selected Supplemental Reading Comprehension Interventions: Findings from Two Student Cohorts.” NCEE 2010-4015, National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education, Washington, DC.

16.

Judkins

St.Pierre

R. B.

Goodson

Gutmann

von Glatz

Hamilton

Webber

Troppe

Rimdzius

2008. “A Study of Classroom Literacy Interventions and Outcomes in Even Start.” NCEE 20084028, National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education, Washington, DC.

17.

Klerman

J. A.

2010. “Contracting for Independent Evaluation: Approaches to an Inherent Tension.” Evaluation Review 34(4, August):299–333.

18.

Leviton

L. C.

Gutman

M. A.

. 2010. “Overview and Rationale for the Systematic Screening and Assessment Method.” In Leviton

L. C.

Kettel Khan

Dawkins

(Eds.), The Systematic Screening and Assessment Method: Finding Innovations Worth Evaluating. New Directions for Evaluation 125:7–31.

19.

McDonald

S.-K.

2009. “Scale-up as a Framework for Intervention, Program, and Policy Evaluation Research.” In Handbook of Education Policy Research (pp. 191–208), edited by Sykes

Schneider

Plank

D. N.

. New York: Routledge.

20.

McDonald

S.-K.

2010. “Developmental Stages for Evaluating Scale.” The Evaluation Exchange 15. Accessed January 3, 2013, http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/current-issue-scaling-impact/developmental-stages-for-evaluating-scale

21.

McLaughlin

J. A.

Jordan

G. B.

. 1999. “Logic Models: A Tool for Telling your Program’s Performance Story.” Evaluation and Program Planning 22:65–72.

22.

Meckstroth

Burwick

Ponza

Marsh

Novak

Phillips

Diaz-Tena

. 2006. “Paths to Work in Rural Places: Key Findings and Lessons from the Impact Evaluation of the Future Steps Rural Welfare-to-Work Program.” Final report MPR No. 8762-192, 202. Mathematica Policy Research, Princeton, NJ.

23.

Miller

Martin

Hamilton

. 2008. The Employment Retention and Advancement Projects: Findings for the Cleveland Achieve Model: Implementation and Early Impacts of an Employer-Based Approach to Encourage Employment Retention among Low-Wage Workers. New York, NY: Manpower Demonstration Research Corporation.

24.

Molina

Cheng

W.-L.

Hendra

. 2008. The Employment Retention and Advancement Project: Results from the Valuing Individual Success and Increasing Opportunities Now (VISION) Program in Salem, Oregon. New York: Manpower Demonstration Research Corporation.

25.

Obama

2009. “Remarks of President Obama on Community Solutions Agenda.” Accessed January 3, 2013, http://www.nationalservice.gov/about/newsroom/statements_detail.asp?tbl_pr_id=1828

26.

Orszag

P. R.

2009a

“Building Rigorous Evidence to Drive Policy [Web log post].”

Accessed January 3, 2013, http://www.whitehouse.gov/omb/blog/09/06/08/BuildingRigorousEvidencetoDrivePolicy

27.

Orszag

P. R.

2009b. “Increased Emphasis on Program Evaluation [Memorandum].” Accessed January 3, 2013, http://www.whitehouse.gov/omb/assets/memoranda_2010/m10-01.pdf

28.

Pawson

Tilley

. 1997. Realistic Evaluation. London, England: Sage.

29.

Ravallion

2008. “Should the Randomistas Rule?” The Economists’ Voice 6: article 6.

30.

Rogers

Petrosino

Huebner

Hacsi

. 2000. “Program Theory Evaluation: Practice, Promise, and Problems.” New Directions in Evaluation 87:5–13.

31.

Rogers

P. J.

2005. “Logic Models.” In Encyclopedia of Evaluation, edited by Mathison

, 232. Beverly Hills, CA: Sage.

32.

Sawhill

I. V.

Baron

. 2010. “We Need a New Start for Head Start.” Education Week 29:22–23.

33.

Scrivener

Azurdia

Page

. 2005. The Employment Retention and Advancement Projects: Results from the South Carolina ERA site. New York: Manpower Demonstration Research Corporation.

34.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

. 2001. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston, MA: Houghton Mifflin.

35.

Shah, Shivam

Mallick

Jolin

Michele

2012. “Social Sector Innovation Funds: Lessons Learned and Recommendations.” Accessed January 3, 2013, http://www.americanprogress.org/wp-content/uploads/2012/11/JolinInnovationFunds-2.pdf. America Achieves and the Center for American Progress.

36.

Society for Prevention Research. 2004. Standards of Evidence: Criteria for Efficacy, Effectiveness and Dissemination. Accessed January 3, 2013, http://www.preventionresearch.org/sofetext.php

37.

United States Agency for International Development. 2011. USAID Evaluation Policy. Bureau for Policy, Planning, and Learning. Accessed January 3, 2013, http://www.usaid.gov/evaluation/

38.

Valley of the Sun United Way. 2008. Logic Model Handbook 2008. Accessed January 3, 2013, http://vsuw.org/file/logic_model_handbook_updated_2008.pdf

39.

Weiss

1997. Evaluation, 2nd ed. New York: Prentice Hall.

40.

Wholey

1994. “Assessing the Feasibility and Likely Usefulness of Evaluation.” In Handbook of Practical Program Evaluation, edited by Hatry

H. P.

Wholey

J. S.

Newcomer

K. E.

. San Francisco, CA: Jossey-Bass.

41.

Wood

R. G.

McConnell

Moore

Clarkwest

Hseuh

. 2010. “The Building Strong Families Project: Strengthening Unmarried Parents’ Relationships: The Early Impacts of Building Strong Families: Executive Summary.” MPR No. 08935.155. Mathematica Policy Research, Princeton, NJ.

42.

W. K. Kellogg Foundation. 2004. Using Logic Models to Bring Together Planning, Evaluation and Action: Logic Model Development Guide. Battle Creek, MI: W.W. Kellogg Foundation. Accessed January 3, 2013, http://www.ncga.state.nc.us/PED/Resources/documents/LogicModelGuide.pdf