How Mixed-Methods Research Can Improve the Policy Relevance of Impact Evaluations

Abstract

This paper describes how mixed methods can improve the value and policy relevance of impact evaluations, paying particular attention to how mixed methods can be used to address external validity and generalization issues. We briefly review the literature on the rationales for using mixed methods; provide documentation of the extent to which mixed methods have been used in impact evaluations in recent years; describe how we developed a list of recent impact evaluations using mixed methods and the process used to conduct full-text reviews of these articles; summarize the findings from our analysis of the articles; discuss three exemplars of using mixed methods in impact evaluations; and discuss how mixed methods have been used for studying and improving external validity and potential improvements that could be made in this area. We find that mixed methods are rarely used in impact evaluations, and we believe that increased use of mixed methods would be useful because they can reinforce findings from the quantitative analysis (triangulation), and they can also help us understand the mechanism by which programs have their impacts and the reasons why programs fail.

Keywords

mixed methods program evaluation impact evaluation qualitative evaluation methodological development external validity transferability

Get full access to this article

View all access options for this article.

References

Angrist

Pischke

J.-S.

(2010). The Credibility Revolution in empirical economics: How better research design is taking the con out of Econometrics. The Journal of Economic Perspectives, 26(2), 3–30. https://doi.org/10.3386/w15794

Bamberger

Vaessen

Raimondo

(2016, online 2019). Dealing with complexity in development evaluation: A practical approach. Sage Publications. https://methods.sagepub.com/book/dealing-with-complexity-in-development-evaluation

Barnow

B. S.

(2010). Setting up social experiments: The good, the bad, and the ugly. Journal for Labour Market Research, 43(2), 91–105. https://doi.org/10.1007/s12651-010-0042-6

Barnow

B. S.

Greenberg

D. H.

(2020). Conducting evaluations using multiple trials. American Journal of Evaluation, 41(4), 564–580.

Bonilla

Zarzur

R. C.

Handa

Nowlin

Peterman

Ring

Seidenfeld

(2017). Cash for women’s empowerment? A mixed-methods evaluation of the government of Zambia’s child grant program. World Development, 95, 58. https://doi.org/10.1016/j.worlddev.2017.02.017

Bryman

(2006). Integrating quantitative and qualitative research: How is it done? Qualitative Research, 6(1), 107. https://doi.org/10.1177/1468794106058877

Cook

T. D.

(2018). Twenty-Six assumptions that have to be met if single random assignment experiments are to warrant “gold Standrd” status: A commentary on Deaton and Cartwright. Social Science & Medicine, 210, 37–40.

Cronbach

L. J.

(1982). Designing evaluations of educational and social programs. Jossey-Bass.

Deaton

Cartwright

(2018). Understanding and misunderstanding randomized controlled trials. Social Science and Medicine, 210(2), 2–21. https://doi.org/10.3386/w22595

10.

de Milliano

Barrington

Angeles

Gbedemah

(2021). Crowding-out or crowding-in? Effects of LEAP 1000 unconditional cash transfer program on household and community support among women in rural Ghana. World Development, 143(486), 105466. https://doi.org/10.1016/j.worlddev.2021.105466

11.

Fetters

M. D.

(2020. Online 2023). The mixed methods research workbook: Activities for designing, implementing, and publishing projects. Sage Publications. https://methods-sagepub-com.proxygw.wrlc.org/book/the-mixed-methods-research-workbook

12.

Fetters

M. D.

Molina-Azorin

J. F.

(2020). Utilizing a mixed methods approach for conducting interventional evaluations. Journal of Mixed Methods Research, 14(2), 131–144.

13.

Gertler

Wilde

M. S.

Premand

Rawlings

Vermeersch

(2016). Impact evaluation in practice. World Bank Group.

14.

Green

Roberts

Petticrew

Steinbach

Goodman

Jones

Edwards

(2015). Integrating quasi-experimental and inductive designs in evaluation: A case study of the impact of free bus travel on public health. Evaluation, 21(4), 391–406.

15.

Greenberg

Ashworth

Cebulla Walker

(2005). When welfare-to-work programs seem to work well: Explaining why Riverside and Portland shine so brightly. Industrial and Labor Relations Review, 59(1), 34–50.

16.

Greene

J. C.

Caracelli

V. J.

Graham

W. F.

(1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 259. https://doi.org/10.3102/01623737011003255

17.

Hendren

Luo

Q. E.

Pandey

S. K.

(2018). The state of mixed methods research in public administration and public policy. Public Administration Review, 78(6), 904–916. https://doi.org/10.1111/puar.12981

18.

Hughes

Morgan

Baylis

Oduol

Smith-Dumont

Vågen

T.-G.

Kegode

(2020). Assessing the downstream socioeconomic impacts of agroforestry in Kenya. World Development, 128(1), 104835. https://doi.org/10.5716/wp18033.pdf

19.

Imbens

G. W.

(2018). Comments on understanding and misunderstanding randomized controlled trials: A commentary on Cartwright and Deaton. Social Science and Medicine, 210, 50–52.

20.

Leamer

E. E.

(2010). Tantalus on the road to asymptopia. The Journal of Economic Perspectives, 24(2), 31–46.

21.

Littell

J. H.

(2024). The logic of generalization from systematic reviews and meta-analyses of impact evaluations. Evaluation Review.

22.

Maynard

(2024). Improving the usefulness and use of meta-analysis to inform policy and practice. Evaluation Review.

23.

Nichols-Barrer

Protik

Berman

Sloan

(2018). Understanding the impacts of a civic participation initiative in Rwanda: Experimental findings. RIDIE Datasets, Vol. 154. https://doi.org/10.23846/ridie028

24.

Pawson

Tilley

(1997). An introduction to scientific realist evaluation. In Evaluation for the 21st century: A handbook. Sage. https://doi.org/10.4135/9781483348896.n29

25.

Rao

Ananthpur

Malik

(2017). The anatomy of failure: An ethnography of a randomized trial to deepen democracy in rural India. World Development, 99, 481–497. https://doi.org/10.1016/j.worlddev.2017.05.037

26.

Richwine

Luo

Q. E.

Thorkildsen

Chong

N. J.

Morris

Barnow

B. S.

Pandey

S. K.

(2022). Defining and assessing the value of canonical mixed methods research designs in public policy and Public Administration. Journal of Policy Analysis and Management, 41(3), 891–920. https://doi.org/10.1002/pam.22392

27.

Roelen

Saha

(2021). Pathways to stronger futures? The role of social protection in reducing psychological risk factors for child development in Haiti. World Development, 142(Part 2), 105423. https://doi.org/10.1016/j.worlddev.2021.105423

28.

Starr

(2014). Qualitative and mixed-methods research in economics: Surprising growth, promising future. Journal of Economic Surveys, 28(2), 242. https://doi.org/10.1111/joes.12004

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB