Theory-based impact evaluations have been put forward increasingly as an alternative for counterfactual impact evaluations. However, this raises questions regarding the foundations of drawing causal inference on the basis of such approaches. Case study methods such as QCA (Quantitative Comparative Analysis), process tracing and congruence analysis are emerging as a way to match the methodological rigor of counterfactuals. While QCA relies on multiple cases, process tracing and congruence analysis are methods that claim to be able to draw causal inference within a single case. In this article, a completed theory-based impact evaluation of a European Social Fund intervention is used as a foundation to demonstrate and discuss the differences between process tracing and congruence analysis and their relative (dis)advantages.
AstburyBLeeuwF (2010) Unpacking black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation31(3): 363–81.
2.
BamanyakiPAHolvoetN (2016) Integrating theory-based evaluation and process tracing in the evaluation of civil society gender budget initiatives. Evaluation22(1): 72–90.
3.
BeachDPedersenRB (2013) Process-Tracing Methods. Ann Arbor, MI: University of Michigan Press.
4.
BeachDPedersenRB (2016a) Causal Case Study Methods. Ann Arbor, MI: University of Michigan Press.
5.
BeachDPedersenRB (2016b) Selecting appropriate cases when tracing causal mechanisms. Sociological Methods and Research. Epub ahead of print 13 January 2016. DOI: 10.1177/0049124115622510.
6.
BefaniBMayneJ (2014) Process tracing and contribution analysis: A combined approach to generative causal inference and impact evaluation. IDS Bulletin45(6): 17–36.
7.
BefaniBD’ErricoSBookerFet al. (2016) Clearing the fog: New tools for improving the credibility of claims. IIED briefingApril2016.
8.
BennettA (2014) Appendix. In: BennettACheckelJT (eds) Process Tracing: From Metaphor to Analytical Tool. Cambridge: Cambridge University Press.
9.
BlatterJHaverlandM (2012) Designing Case Studies. Houndmills: Palgrave Macmillan.
10.
CartwrightNHardieJ (2012) Evidence-based Policy: Doing It Better: A Practical Guide to Predicting if a Policy Will Work for You. Oxford: Oxford University Press.
11.
ChenH (2005) Practical Program Evaluation. London: SAGE.
12.
ChenH (2006) A theory-driven evaluation perspective on mixed methods research. Research in the Schools13(1): 74–82.
13.
CorynCLSNoakesLAWestineCDet al. (2011) A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation32(2): 199–226.
14.
De RickKDriegheLJacobsLet al. (2014) Het POP: een effectief instrument in het arbeidsmarktbeleid? Een kwalitatieve analyse van de uitvoeringspraktijk en effecten van POPs in drie cases. Leuven: HIVA – KU Leuven.
15.
European Commission (2013) Evalsed Sourcebook: Methods and Techniques. Brussels: European Commission.
16.
FunnelSCRogersPJ (2011) Purposeful Program Theory. San Francisco, CA: Jossey-Bass.
17.
HedströmP (2005) Dissecting The Social. Cambridge: Cambridge University Press.
18.
KhandkerSRKoolwalGBSamadHA (2010) Handbook on Impact Evaluation: Quantitative Methods and Practices. Washington, DC: World Bank.
19.
LeeuwFL (2012) Linking theory-based evaluation and contribution analysis: Three problems and a few solutions. Evaluation18(3): 348–63.
20.
MilesMBHubermanAMSaldanaJ (2014) Qualitative Data Analysis. London: SAGE.
21.
PawsonR (2003) Nothing as practical as a good theory. Evaluation9(4): 471–90.
PuntonMWelleK (2015) Straws-in-the-wind, hoops and smoking guns: What can process tracing offer to impact evaluation?Centre for Development impact practice paper number 10: April2015.
24.
RyanRMDeciEL (2000) Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist55: 68–78.
25.
SchmittJBeachD (2015) The contribution of process tracing to theory-based evaluations. Evaluation21(4): 429–47.
26.
SternE (2015) Impact Evaluation: A Guide for Commissioners and Managers. London: Bond.
27.
SternEStameNMayneJet al. (2012) Broadening the Range of Designs and Methods for Impact Evaluation. London: Department for International Development.
28.
Van EveraS (1997) Guide to Methods for Students of Political Science. Ithaca, NY: Cornell University Press.
29.
YinRK (2003) Case Study Research Design and Methods. Thousand Oaks, CA: SAGE.
