Building the house of evaluation: Reconsidering the blueprints and bricks for knowledge accumulation in evaluation

Abstract

The push for evidence-based practice is persistent in the public sector—what counts is what works. One central premise for evidence-based practice is the existence of an evidence base; that is, an accumulated and generalizable body of knowledge. Informed by a recent systematic review, we examine the promises and pitfalls of meta-analysis (the statistical workhorse of systematic reviews) as the primary blueprint for cumulative knowledge building in evaluation. This analysis suggests that the statistical assumptions underlying the meta-analytic framework raise issues that, at least in regards to producing generalizable knowledge, may cut even deeper than is suggested by common criticisms. Advancing beyond meta-analysis, we consider alternative approaches for knowledge building and reflect on the implications of these for individual evaluations.