Debate continues over how best international development agencies can evaluate the impact of actions intended to reduce poverty, insecurity and vulnerability in diverse and complex contexts. There are strong ethical grounds for simply asking those intended to benefit what happened to them, but it is not obvious how to do so in a way that is sufficiently free from bias in favour of confirming what is expected. This article considers scope for addressing this problem by minimizing the prior knowledge participants have of what is being evaluated. The tensions between more confirmatory and exploratory methodological approaches are reviewed in the light of experience of designing and piloting a qualitative impact assessment protocol for evaluating NGO interventions in complex rural livelihood transformations. The article concludes that resolving these tensions entails using mixed methodologies, and that the importance attached to exploratory (nested within confirmatory) approaches depends on contextual complexity, the type of evidence sought and the level of trust between stakeholders.
AbbottJ (2000) Chaos of the Disciplines. Chicago and London: University of Chicago Press.
2.
AndersonMBBrownDJeanI (2012) Time to Listen: Hearing People on the Receiving End of International Aid. Cambridge, MA: CDA Collaborative Learning Projects.
3.
BevanP (2013) Researching Social Change and Continuity: A Complexity-Informed Study of Twenty Rural Community Cases in Ethiopia, 1994-2015. Oxford: Mokoro Ltd.
4.
CartwrightN (2011) A philosopher’s view of the long road from RCTs to effectiveness. The Lancet377(9775): 1400–1.
5.
DeatonA (2010) Instruments, randomization, and learning about development. Journal of Economic Literature48(2): 424–55.
6.
DietzT (2012) Participatory assessment of development in Africa. In: PouwNBaudI (eds) Local Governance and Poverty in Developing Nations. New York: Routledge, 215–39.
7.
DietzTBymoltRBelemvireAvan der GeestKde GrootDMillarD et al. (2013) PADev Guidebook: Participatory Assessment of Development. Leiden: University of Amsterdam, Tamale University for Development Studies, Expertise pour le Developpement du Sahel, ICCO Alliance, Prisma, Woord en Daad.
8.
DuvendackMPalmer-JonesRCopestakeJHooperLLokeYRaoN (2011) What is the Evidence of the Impact of Microfinance on the Well-being of Poor People?London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London.
9.
EasterlyW (2006) The White Man’s Burden: Why the West’s Efforts to Aid the Rest Have Done So Much Ill and So Little Good. New York: Penguin Press.
10.
EybenR (2013) Uncovering the Politics of ‘Evidence’ and ‘Results’. A Framing Paper for Development Practitioners. Brighton: Institute of Development Studies. URL: http://www.bigpushforward.net (accessed 10 November 2013).
11.
GreenDRocheCEybenRDerconSWittyC (2013) The political implications of evidence-based approaches to development. URL: http://www.oxfamblogs.org/fp2p/?p=13344 (accessed 10 November 2013).
12.
GulrajaniN (2010) New vistas for development management: examining radical-reformist possibilities and potential. Public Administration and Development30(2): 136–48.
13.
HaidtJ (2012) The Righteous Mind: Why Good People are Divided by Politics and Religion. London: Penguin.
14.
HammersleyM (2013) The Myth of Research-based Policy and Practice. Thousand Oaks, CA: SAGE.
15.
HeckmanJJSmithJA (1995) Assessing the case for social experiments. The Journal of Economic Perspectives9: 85–110.
KahnemanD (2011) Thinking, Fast and Slow. London: Allen Lane.
18.
LevinsR (1966) The strategy of model building in population biology. American Scientist54(4): 11.
19.
LewisJRitchieJ (2003) Generalising from qualitative research: a guide for social science students and researchers. In: RitchieJLewisJ (eds) Qualitative Research Practice. London: SAGE, 263–86.
20.
MayneJ (2012) Contribution analysis: coming of age?Evaluation18(3): 270–80.
21.
McGilchristI (2010) The Master and his Emissary: The Divided Brain and the Making of the Western World. New Haven, NJ: Yale University Press.
22.
PawsonRManzano-SantaellaA (2012) A realist diagnostic workshop. Evaluation18(2): 176–91.
23.
PawsonRTilleyN (1994) What works in evaluation research?British Journal of Criminology34(3): 291–306.
24.
PicciottoR (2012) Experimentalism and development Evaluation: will the bubble burst?Evaluation18(2): 213–29.
25.
RavallionM (2009) Should the randomistas rule?The Economists’ Voice6: 6.
26.
RowsonJMcGilchristI (2013) Divided brain, divided world: why the best part of us struggles to be heard. London: RSA Action and Research Centre.
27.
ScheiferU (2008) Integrated Evaluation of Change. Lisbon: Lisbon University Institute.
28.
ShafferP (2013) Ten years of ‘Q-squared’: implications for understanding and explaining poverty. World Development45: 269–85.
TonG (2012) The mixing of methods: a three-step process for improving the rigour in impact evaluations. Evaluation18(1): 5–25.
31.
WhiteH (2010) A contribution to current debates in impact evaluation. Evaluation16(2): 153–64.
32.
WhiteHPhillipsD (2012) Addressing Attribution of Cause and Effect in ‘Small n’ Impact Evaluations: Towards an Integrated Framework. London: International Initiative for Impact Evaluation.
