What Doesn’t Work? Three Failures,Many Answers

Abstract

Current debates on impact evaluation have addressed the question ‘what works and what doesn’t?’ mainly focussing on methodology failures when providing evidence of impact. In order to answer that question, this article contrasts different approaches to evaluation in terms of the way they address different kinds of possible failures. First, there is more to be debated than simply methodological failures: there are also programme theory failures and implementations failures. Moreover, not all methodological failures are a simple matter of selection bias. Second, the article reviews issues that have recently been raised within different approaches relative to each failure. For programme theory failure, it is a matter of complexity and providing rival explanations; for implementation failure: how to use guidelines, and how to take context into account; and for methodology failure: how to move from internal to external validity, and to syntheses, within the framework of ‘situational responsiveness’. All these issues disclose a terrain for potential exchange between the protagonists of different approaches to impact evaluation.