Evaluators are often called upon to assess the effects of programs. To assess a program effect, evaluators need a clear understanding of how a program effect is defined. Arguably, the most widely used definition of a program effect is the counterfactual one. According to the counterfactual definition, a program effect is the difference between what happened after the program was implemented and what would have happened if the program had not been implemented, but everything else had been the same. Such a definition is often said to be linked to the use of quantitative methods. But the definition can be used just as effectively with qualitative methods. To demonstrate its broad applicability in both qualitative and quantitative research, I show how the counterfactual definition undergirds seven common approaches to assessing effects. It is not clear how any alternative to the counterfactual definition is as generally applicable as the counterfactual definition.
AbellP. (2004). Narrative explanation: An alternative to variable-centered explanation?Annual Review of Sociology, 30, 287–310.
2.
AngristJ. D.PischkeJ.-S. (2009). Mostly harmless econometrics: An empiricist’s companion. Princeton University Press.
3.
AstburyB.LeeuwF. L. (2010). Unpacking the black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation, 31(3), 363–381.
4.
BeachD. (2016). What are we actually tracing? Process tracing and the benefits of conceptualizing causal mechanisms as systems. Qualitative and Multi-Method Research, 14(1/2), 15–22.
5.
BeachD.PedersenR. B. (2019). Process-tracing methods: Foundations and guidelines (2nd ed.). The University of Michigan Press.
6.
BefaniB.MayneJ. (2014). Process tracing and contribution analysis: A combined approach to generative causal inference for impact evaluation. IDS Bulletin, 45(6), 17–36.
7.
BennettA. (2010). Process tracing and causal inference. In BradyH. E.CollierD. (Eds.), Rethinking social inquiry: Diverse tools, shared standards (pp. 179–189). Rowman & Littlefield.
BradyH. E. (2008). Causation and explanation in social science. In Box-SteffensmeierJ. M.BradyH. E.CollierD. (Eds.), The Oxford handbook of political methodology (pp. 217–270). Oxford University Press.
10.
BrinkerhoffR. O. (2005). Success case method. In MathisonS. (Ed.), Encyclopedia of evaluation (pp. 402–403). Sage.
11.
CampbellD. T.StanleyJ. C. (1966). Experimental and quasi-experimental designs for research. Skokie, IL: Rand McNally.
12.
ChenH. T. (1990). Theory-driven evaluations. Sage.
13.
CookT. D. (2000). The false choice between theory-based evaluation and experimentation. New Directions for Evaluation, 87, 27–34.
14.
CookT. D.CampbellD. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Rand McNally.
15.
CopestakeJ. (2014). Credible impact evaluation in complex contexts: Confirmatory and exploratory approaches. Evaluation, 20(4), 412–427.
16.
CopestakeJ. (2019). Comparing the QuIP with other approaches to developmental impact evaluation. In CopestakeJ.MorsinkM.RemnantF. (Eds.), Attributing development impact: The qualitative impact protocol (QuIP) case book (pp. 29–57). Practical Action Publishing.
17.
CopestakeJ.RemnantF.MorsinkM. (2019). Introducing the causal attribution challenge and the QuIP. In CopestakeJ.MorsinkM.RemnantF. (Eds.), Attributing development impact: The qualitative impact protocol (QuIP) case book (pp. 1–27). Practical Action Publishing.
18.
CorynC. L. S.NoakesL. A.WestineC. D.SchröterD. C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199–226.
19.
DelahaisT.ToulemondeJ. (2012). Applying contribution analysis: Lessons from five years of practice. Evaluation, 18(3), 281–293.
20.
FearonJ. D. (1991). Counterfactuals and hypothesis testing in political science. World Politics, 43(2), 169–195.
21.
GatesE.DysonL. (2017). Implications of the changing conversation about causality for evaluators. American Journal of Evaluation, 38(1), 29–46.
22.
GeorgeA. L.BennettA. (2004). Case studies and theory development in the social sciences. The MIT Press.
GoertzG.MahoneyJ. (2012). A tale of two cultures: Qualitative and quantitative research in the social sciences. Princeton University Press.
25.
HillL. G.BetzD. L. (2005). Revisiting the retrospective pretest. American Journal of Evaluation, 26(4), 501–517.
26.
HollandP. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–970.
27.
HumeD. (1748). An enquiry concerning human understanding. Bebbington.
28.
JohnsonR. B.RussoF.SchoonenboomJ. (2019). Causation in mixed methods research: The meeting of philosophy, science, and practice. Journal of Mixed Methods Research, 13(2), 143–162.
29.
LamT. C. M.ValenciaE. (2019). Retrospective pretest and counterfactual self-report: Different or same?Journal of MultiDisciplinary Evaluation, 15(33), 37–53.
30.
LeeuwF. L. (2012). Linking theory-based evaluation and contribution analysis: Three problems and a few solutions. Evaluation, 18(3), 348–363.
31.
LeeuwF. L.VaessenJ. (2009). Impact evaluation and development. NONIE and World Bank.
32.
MackieJ. L. (1965). Causes and conditions. American Philosophical Quarterly, 2(4), 245–264.
33.
MahoneyJ. (2008). Toward a unified theory of causality. Comparative Political Studies, 41(4/5), 412–436.
34.
MahoneyJ.GoertzG. (2006). A tale of two cultures: Contrasting quantitative and qualitative research. Political Analysis, 14(3), 227–249.
35.
MahoneyJ.KimballE.KoivuK. L. (2009). The logic of historical explanation in the social sciences. Comparative Political Studies, 42(1), 114–146.
36.
MarkM. M.HenryG. T.JulnesG. (1998). A realist theory of evaluation practice. New Directions for Evaluation, 78, 3–32.
37.
MayneJ. (2008). Contribution analysis: An approach to exploring cause and effect (ILAC Brief No. 16). The Institutional Learning and Change Initiative.
38.
MayneJ. (2012). Contribution analysis: Coming of age?Evaluation, 18(3), 270–280.
39.
MohrL. B. (1995). Impact analysis for program evaluation. Sage.
40.
MohrL. B. (1999). The qualitative method of impact analysis. American Journal of Evaluation, 20(1), 69–84.
41.
MorganS. L.WinshipC. (2015). Counterfactuals and causal inference: Methods and principles for social research (2nd ed.). Cambridge University Press.
42.
MuellerC. E.GausH.RechJ. (2014). The counterfactual self-estimation of program participants: Impact assessment without control groups or pretests. American Journal of Evaluation, 35(1), 8–25.
43.
PattonM. Q. (2015). Qualitative research and evaluation methods (4th ed.). Sage.
44.
PattynV.MolenveldA.BefaniB. (2019). Qualitative comparative analysis as an evaluation tool: Lessons from an application in development cooperation. American Journal of Evaluation, 40(1), 55–74.
PrattC. C.McGuiganW. M.KatzevA. R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3), 341–349.
47.
RathbunB. C. (2008). Interviewing and qualitative field methods: Pragmatism and practicalities. In Box-SteffensmeierJ. M.BradyH. E.CollierD. (Eds.), The Oxford handbook of political methodology (pp. 685–701). Oxford University Press.
48.
ReichardtC. S. (2006). The principle of parallelism in the design of studies to estimate treatment effects. Psychological Methods, 11(1), 1–18.
49.
ReichardtC. S. (2011). Evaluating methods for estimating program effects. American Journal of Evaluation, 32(2), 246–272.
50.
ReichardtC. S. (2019). Quasi-experimentation: A guide to design and analysis. Guilford Press.
51.
RicksJ.LiuA. H. (2018). Process-tracing research designs: A practical guide. PS: Political Science and Politics, 51(4), 842–846.
52.
RohlfingI.SchneiderC. Q. (2018). A unifying framework for causal analysis in set-theoretic multimethod research. Sociological Methods and Research, 47(1), 37–63.
53.
RoughleyA.DartJ. (2009). Developing a performance story report: Users guide. Commonwealth of Australia.
54.
RubinD. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701.
55.
RubinD. B. (2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469), 322–331.
56.
ScrivenM. (1976). Maximizing the power of causal investigations: The modus operandi method. In GlassG. V. (Ed.), Evaluation studies review annual (Vol. 1, pp. 101–118). Sage.
57.
ScrivenM. (2005a). Can we infer causation from cross-sectional data? [Conference session]. A Symposium on the Use of School-Level Data for Evaluating Federal Education Programs, Board on Testing and Assessment, The National Academies Press, Washington, DC, United States.
58.
ScrivenM. (2005b). Causation. In MathisonS. (Ed.), Encyclopedia of evaluation (pp. 44–48). Sage.
59.
ScrivenM. (2008). A summative evaluation of RCT methodology: An alternative approach to causal research. Journal of MultiDisciplinary Evaluation, 5(9), 11–24.
60.
ScrivenM. (2009). Demythologizing causation and evidence. In DonaldsonS. I.ChristieC. A.MarkM. M. (Eds.), What counts as credible evidence in applied research and evaluation practice? (pp. 134–152). Sage.
61.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin Harcourt.
62.
SternE.StameN.MayneJ.ForssK.DaviesR.BefaniB. (2012). Broadening the range of designs and methods for impact evaluation: Report of a study commissioned by the Department of International Development (Working paper 38). Department of International Development.
63.
TacqJ. (2011). Causality in qualitative and quantitative research. Quality and Quantity, 45(2), 263–291.
WeissC. H. (1997). Theory-based evaluation: Past, present, and future. New Directions for Evaluation, 76, 41–55.
66.
WestS. G.ThoemmesF. (2010). Campbell’s and Rubin’s perspectives on causal inference. Psychological Methods, 15(1), 18–37.
67.
WhiteH. (2011). Theory-based impact evaluation: Principles and practice (3ie Working paper 3). International Initiative for Impact Evaluation.
68.
WhiteH.PhillipsD. (2012). Addressing attribution of cause and effect in small n impact evaluations: Towards an integrative framework (3ie Working paper 15). International Initiative for Impact Evaluation.
69.
Wilson-GrauR.BrittH. (2013). Outcome harvesting. Ford Foundation Middle East and North Africa Office.
