One concern about impact evaluation is whether a program has made a difference (Owen 2006). Essentially, in trying to understand this, an evaluation seeks to establish causality—the causal link between the social program and the outcomes (Mohr 1999; Rossi & Freeman 1993). However, attributing causality to social programs is difficult because of the inherent complexity and the many and varied factors at play (House 2001; Mayne 2001; Pawson & Tilley 1998; White 2010).
Evaluators can choose from a number of theories and methods to help address causality (Davidson 2000). Measuring the counterfactual—the difference between actual outcomes and what would have occurred without the intervention—has been at the heart of traditional impact evaluations. While this has traditionally been measured using experimental and quasi-experimental methods, a counterfactual does not always need a comparison group (White 2010) and can be constructed qualitatively (Cummings 2006).
With these in mind, this article explores the usefulness of the concept of additionality, a mixed-methods framework developed by Buisseret et al. (cited in Georghiou 2002) as a means of evaluative comparison of the counterfactual.
Get full access to this article
View all access options for this article.
References
1.
BambergerM.2006, Conducting quality impact evaluations under budget, time and data constraints, The World Bank, Washington, DC.
2.
BambergerM., RughJ., & MabryL.2006, Real world evaluation: working under budget, time, date and political constraints, Sage, Thousand Oaks, California.
3.
BlameyA., & MackenzieM.2007, ‘Theories of change and realistic evaluation: peas in a pod or apples and oranges?’, Evaluation, vol. 13, no. 4, pp. 439–455.
4.
CampbellD., & RussoM.J.1999, Social experimentation, Sage, Thousand Oaks, California.
5.
CardenF., BambergerM., & RughJ.2009, Alternatives to the conventional counterfactual: follow-up note on Session 713 Think Tank, American Evaluation Association annual conference, November, Orlando, viewed 26 April 2010, <http://www.realworldevaluation.org/Additional_RWE_materials.html>.
6.
CarrS., DancerS., RussellG., TylerP., RobsonB., LawlessP., & HaighN.2004, Additionality guide: a standard approach to assessing the additional impact of projects, 2nd edn, English Partnerships, London.
7.
CookT.D., ScrivenM., CorynC.L.S., & EvergreenS.D.H.2009, ‘Contemporary thinking about causation in evaluation: a dialogue with Tom Cook and Michael Scriven’, American Journal of Evaluation, vol. 31, no. 1, pp. 105–117.
8.
CummingsR.2006, ‘What if?: the counterfactual in program evaluation’, Evaluation Journal of Australasia, vol. 6(new series), no. 2, pp. 6–15.
9.
DavidsonE.J.2000, ‘Ascertaining causality in theory-based evaluation’, in PetrosinoA (eds), Program theory in evaluation—opportunities: New directions for evaluation, 87, Jossey-Bass, San Francisco.
10.
DavidsonE.J.2005, Evaluation methodology basics: the nuts and bolts of sound evaluation, Sage, Thousand Oaks, California.
11.
GarbarinoS., & HollandJ.2009, Quantitative and qualitative methods in impact evaluation and measuring results: issues paper, Governance and Social Development Resource Centre, Department for International Development, London.
12.
GarganiJ.2010, ‘A welcome change from debate to dialogue about causality’, American Journal of Evaluation, vol. 31, no. 1, pp. 131–132.
13.
GeorghiouL.2002, ‘Impact and additionality of innovation policy’, paper presented to Six Countries Programme on Innovation, Spring Conference, Brussels, 28 February – 1 March.
14.
GeorghiouL., & ClarysseB.2006, ‘Introduction and synthesis’, in OECD (ed.), Government R&D funding and company behaviour: measuring behavioural additionality, OECD Publishing, Paris.
15.
GreeneJ.C., LiseyM.W., SchwandtT.A., SmithN.L., & TharpR.G.2007, ‘Method choice: five discussant commentaries’, in JulnesGJ, & RogDJ (eds), Informing federal policies on evaluation methodology—building the evidence base for method choice in government sponsored evaluation: New directions for evaluation, 113, Jossey-Bass, San Francisco.
16.
HouseE.2001, ‘Unfinished business: causes and values’, American Journal of Evaluation, vol. 22, no. 3, pp. 309–315.
17.
International Finance Corporation2008, Evaluating private sector development in developing countries: assessing the additionality of multilateral development banks—IFC's experience, IFC, The World Bank Group, Washington, DC.
JulnesG., & MarkM.M.1998, ‘Evaluation as sensemaking: knowledge construction in a realist world’, in HenryGT, JulnesG, & MarkMM (eds), Realist evaluation: an emerging theory in support of practice: New directions for evaluation, 78, Jossey-Bass, San Francisco.
20.
KirkhartK.E.2000, ‘Reconceptualizing evaluation use: an integrated theory of influence’, in CaracelliVJ, & PreskillH (eds), The expanding scope of evaluation use: New directions for evaluation, 88, Jossey-Bass, San Francisco.
21.
KotvojsF., & ShrimptonB.2007, ‘Contribution analysis’, Evaluation Journal of Australasia, vol. 7, no. 1, pp. 27–35.
22.
LevitonL.C., & GutmanM.A.2010, ‘Overview and rationale for the systematic screening and assessment method’, in LevitonLC, KhanLK, & DawkinsN (eds), The systematic screening and assessment method: finding innovations worth evaluating: New directions for evaluation, 125, Jossey-Bass, San Francisco.
23.
LipseyM.W.2001, ‘Unsolved problems and unfinished business’, American Journal of Evaluation, vol. 22, no. 3, pp. 325–328.
MackieJ.L.2002, The cement of the universe: a study of causation, Oxford Press, Oxford.
26.
MarkM.M., & HenryG.T.1998, ‘Social programming and policy-making: a realist perspective’, in HenryGT, JulnesG, & MarkMM (eds), Realist evaluation: an emerging theory in support of practice: New directions for evaluation, 78, Jossey-Bass, San Francisco.
27.
MarkM.M., HenryG.T., & JulnesG.1998, ‘A realist theory of evaluation practice’, in HenryGT, JulnesG, & MarkMM (eds), Realist evaluation: an emerging theory in support of practice: New directions for evaluation, 78, Jossey-Bass, San Francisco.
28.
MayneJ.2001, ‘Addressing attribution through contribution analysis using performance measures sensibly’, The Canadian Journal of Program Evaluation, vol. 16, no. 1, pp. 1–24.
29.
MohrL.1999, ‘The qualitative method of impact analysis’, American Journal of Evaluation, vol. 20, no. 1, pp. 69–84.
30.
NärfeltK-H, & WildbergerA.2006, ‘Additionality and funding agencies: opening the black box’, paper presented at the New Frontiers in Evaluation Conference, Vienna, 24-25 April.
31.
OECD2006, ‘Executive summary’, in OECD (ed.), Government R&D funding and company behaviour: measuring behavioural additionality, OECD Publishing, Paris.
32.
OwenJ.2006, Program evaluation: forms and approaches, 3rd edn, Allen & Unwin, Crows Nest, New South Wales.
33.
PattonM.Q.2002, Qualitative research and education methods, 3rd edn,Sage, Thousand Oaks, California.
RogersP.J.2000, ‘Causal models in program theory evaluation’, in PetrosinoA (eds), Program theory in evaluation: challenges and opportunities: New directions for evaluation, 87, Jossey-Bass, San Francisco.
37.
RogersP., PetrosinoA., HuebnerT.A., & HacsiT.A.2000, ‘Program theory evaluation: practice, promise, and problems’, in PetrosinoA (eds), Program theory in evaluation: challenges and opportunities: New directions for evaluation, 87, Jossey-Bass, San Francisco.
38.
RoldánJ.2002, ‘Designing a practical instrument for measuring developmental additionality: the IIC approach’, paper presented at inter-agency roundtable on additionality of private sector development programs and operations supported by the international financial institutions, Inter-American Investment Corporation, Washington, DC, 23 May.
39.
RossiP.H., & FreemanH.E.1993, Evaluation: a systematic approach, 5th edn, Sage, Newbury Park, California.
SchweigertF.2006, ‘The meaning of effectiveness in assessing community initiatives’, American Journal of Evaluation, vol. 27, no. 4, pp. 416–436.
42.
ScrivenM.1991, Evaluation thesaurus, 4th edn, Sage, Newbury Park, California.
43.
ScrivenM.2008, ‘A summative evaluation of RCT methodology: an alternative approach to causal research’, Journal of MultiDisciplinary Evaluation, vol. 5, no. 9, pp. 11–24.
44.
ShadishW.R., CookT.D., & CampbellD.T.2002, Experimental and quasi-experimental designs for generalized causal inference, Houghton Mifflin, Boston.
45.
WeissC.H.2000, ‘Which links in which theories shall we evaluate?’, in PetrosinoA (eds), Program theory in evaluation: challenges and opportunities: New directions for evaluation, 87, Jossey-Bass, San Francisco.
46.
WhiteH.2010, ‘A contribution to current debates in impact evaluation’, Evaluation, vol. 16, no. 2, pp. 153–164.
