‘What if’: The counterfactual in program evaluation

Abstract

How do evaluators provide the best evidence of program outcomes to stakeholders? The current focus on impact evaluation and the adjoining debate over the suitability of various designs for rigorously measuring program outcomes begs the question of what are the most suitable alternative outcomes with which to compare program outcomes. In essence this is asking what is the counterfactual, how should it be measured and what is an appropriate method of comparing the outcomes of a program and the counterfactual. Although measurement of outcomes is a major issue in evaluation, the counterfactual does not appear often in the evaluation literature, whereas it is widely used in the fields of history, political science, philosophy and psychology. This article attempts to commence the process of filling in this gap by focusing on the role and measurement of the counterfactual in program evaluation. The underlying logic of the counterfactual is explored and some approaches to its measurement are discussed. The approach is described in detail using an example from a recent evaluation study of a crime prevention program in Western Australia. The potential benefits of measuring the counterfactual are explored, particularly in relation to providing influential information to key stakeholders.