Sage Journals: Discover world-class research

Abstract

This introduction to the second special issue of Evaluation review on external validity and generalizability opens a dialogue between different ways to think about generalizability in program evaluation. It argues that generalizability in impact evaluation fundamentally is about inferring some form of causality at a level broader than the specific circumstances of the initial study or studies from which these inferences are drawn. The question, then, is about how one apprehends causality: in other words, what is being generalized and how? The first special issue mainly relied on a counterfactual conception of causality, embodied by experimental and quasi-experimental methods, that aimed at impact measurement. The articles in this volume, drawing on mixed methods, also mobilize generative and configurational causal inferences to provide further levers of generalizability, focusing on how the impact is produced. The introduction insists on the specific input of qualitative methods in this respect, as theorized by grounded theory.

Keywords

generalization external validity generalizability transferability grounded theory qualitative methods mixed methods causality

Get full access to this article

View all access options for this article.

References

Alamos-Concha

Pattyn

Cambré

Rihoux

(2025). Increasing internal and external validity through a multimethod evaluation with qualitative comparative analysis (QCA) and process tracing: The case of training transfer effectiveness in Flemish SMEs. Evaluation Review, 193841X2, Article 51352020. https://doi.org/10.1177/0193841x251352020

Alkin

M. C.

King

J. A.

(2017). Definitions of evaluation use and misuse, evaluation influence, and factors affecting use. American Journal of Evaluation, 38(3), 434–450. https://doi.org/10.1177/1098214017717015

Barnow

B. S.

Pandey

S. K.

Luo

(2024). How mixed-methods research can improve the policy relevance of impact evaluations. Evaluation Review, 48(3), 495–514. https://doi.org/10.1177/0193841X241227480

Beach

Pedersen

R. B.

(2016). Causal case study methods: Foundations and guidelines for comparing, matching, and tracing. University of Michigan Press.

Beach

Pedersen

R. B.

(2019). Process-tracing methods: Foundations and guidelines (2nd ed). University of Michigan Press.

Becker

H. S.

(1998). Tricks of the trade: How to think about your research while you’re doing it. University of Chicago Press.

Besharov

D. J.

(2024). Program evaluation’s path to greater policy relevance: Learning from Rossi’s iron laws. Evaluation Review, 48(3), 403–409. https://doi.org/10.1177/0193841x241238031

Bodenstein

Kemmerling

(2025). Taking Stock of Qualitative Methods of Evaluation: A Study of Practices and Quality Criteria. Evaluation review. https://doi.org/10.1177/0193841X251370426

Bryman

(1988). Quantity and quality in social research. Routledge.

10.

Bryman

(2007). Barriers to integrating quantitative and qualitative research. Journal of Mixed Methods Research, 1(1), 8–22. https://doi.org/10.1177/2345678906290531

11.

Burawoy

(1998). The extended case method. Sociological Theory, 16(1), 4–33. https://doi.org/10.1111/0735-2751.00040

12.

Carminati

(2018). Generalizability in qualitative research: A tale of two traditions. Qualitative Health Research, 28(13), 2094–2101. https://doi.org/10.1177/1049732318788379

13.

Corbin

J. M.

Strauss

(1990). Grounded theory research: Procedures, canons, and evaluative criteria. Qualitative Sociology, 13(1), 3–21. https://doi.org/10.1007/BF00988593

14.

Corrigan

Onwuegbuzie

(2020). Toward a meta-framework for conducting mixed methods representation analyses to optimize meta-inferences. The Qualitative Report, 25(3), 785–812. https://doi.org/10.46743/2160-3715/2020.3579

15.

Cresswell

J. W.

Plano Clark

V. L.

(2017). Designing and conducting mixed methods research. Sage.

16.

Delahais

(2025). External validity for complex interventions: Towards pragmatic strategies for evaluators. Evaluation Review, 193841X2, Article 51343130. https://doi.org/10.1177/0193841x251343130

17.

Devaux-Spatarakis

(2023). Theory-based evaluation. LIEPP Methods Brief / Fiches Méthodologiques Du LIEPP, 28, 1–5. https://sciencespo.hal.science/hal-04159080v1

18.

Donmoyer

(2009). Generalizability and the single-case study. In Gomm

Hammersley

Foster

(Eds.), Case study method (pp. 45–68). Sage.

19.

Findley

M. G.

Kikuta

Denly

(2021). External validity. Annual Review of Political Science, 24(1), 365–393. https://doi.org/10.1146/annurev-polisci-041719-102556

20.

Ganesh

N. P.

(2020). Teaching the scope and limits of generalizability in qualitative research. New Trends in Qualitative Research, 1(1), 186–192. https://doi.org/10.36367/ntqr.1.2020.186-192

21.

Glaser

B. G.

Strauss

A. L.

(1967). The discovery of grounded theory: Strategies for qualitative research (pp. 271). Aldine Pub. Co.

22.

Greenhalgh

Manzano

(2022). Understanding ‘context’ in realist evaluation and synthesis. International Journal of Social Research Methodology, 25(5), 583–595. https://doi.org/10.1080/13645579.2021.1918484

23.

Guba

E. G.

Lincoln

Y. S.

(1981). Effective evaluation. Jossey-Bass Publishers.

24.

Handel

D. V.

Hanushek

E. A.

(2024). Contexts of convenience: Generalizing from published evaluations of school finance policies. Evaluation Review, 48(3), 461–494. https://doi.org/10.1177/0193841x241228335

25.

Hays

D. G.

McKibben

W. B.

(2021). Promoting rigorous research: Generalizability and qualitative research. Journal of Counseling & Development, 99(2), 178–188. https://doi.org/10.1002/jcad.12365

26.

Hong

Q. N.

(2023). Mixed methods systematic reviews. LIEPP Methods Brief/Fiches Méthodologiques Du LIEPP, 32, 1–5. https://sciencespo.hal.science/hal-04159206

27.

Hong

Q. N.

Pluye

(2019). A conceptual framework for critical appraisal in systematic mixed studies reviews. Journal of Mixed Methods Research, 13(4), 446–460. https://doi.org/10.1177/1558689818770058

28.

Lincoln

Guba

(2009). The only generalization is: There is no generalization. In Gomm

Hammersley

Foster

(Eds.), Case study method (pp. 27–47). Sage. https://doi.org/10.4135/9780857024367

29.

Ling

(2024). Transferability of lessons from program evaluations: Iron laws, hiding hands and the evidence ecosystem. Evaluation Review, 48(3), 410–426. https://doi.org/10.1177/0193841x241228332

30.

Littell

J. H.

(2024). The logic of generalization from systematic reviews and meta-analyses of impact evaluations. Evaluation Review, 48(3), 427–460. https://doi.org/10.1177/0193841X241227481

31.

Maynard

(2024). Improving the usefulness and use of meta-analysis to inform policy and practice. Evaluation Review, 48(3), 515–543. https://doi.org/10.1177/0193841x241229885

32.

Pattyn

(2023). Qualitative comparative analysis (QCA). LIEPP Methods Brief / Fiches Méthodologiques du LIEPP, 39, 1–5. https://hal.science/LIEPP-METHOD/hal-04160570v1

33.

Pattyn

Molenveld

Befani

(2019). Qualitative comparative analysis as an evaluation tool: Lessons from an application in development cooperation. American Journal of Evaluation, 40(1), 55–74. https://doi.org/10.1177/1098214017710502

34.

Pawson

(2010). Middle range theory and program theory evaluation: From provenance to practice. In Vaessen

Leeuw

F. L.

(Eds.), Mind the gap perspectives on policy evaluation and the social sciences (pp. 171–202). Routledge.

35.

Pawson

Tilley

(1997). Realistic evaluation. Sage.

36.

Pluye

Hong

Q. N.

(2014). Combining the power of stories and the power of numbers: Mixed methods research and mixed studies reviews. Annual Review of Public Health, 35(1), 29–45. https://doi.org/10.1146/annurev-publhealth-032013-182440

37.

Raimondo

(2020). Getting practical with causal mechanisms: The application of process-tracing under real-world evaluation constraints. New Directions for Evaluation, 2020(167), 45–58. https://doi.org/10.1002/ev.20430

38.

Raimondo

(2023). Process tracing. LIEPP Methods Brief / Fiches méthodologiques du LIEPP, 41, 1–5. https://sciencespo.hal.science/hal-04160661

39.

Rangarajan

(Ed.), (2023) The Oxford handbook of program design and implementation evaluation. Oxford University Press. https://doi.org/10.1093/oxfordhb/9780190059668.001.0001

40.

Revillard

(2023). Policy evaluation: Methods and approaches. ESBC. https://scienceetbiencommun.pressbooks.pub/pubpolevaluation/

41.

Revillard

(2024). How people experience policies: The study of policy reception. In Sager

Mavrot

Keiser

(Eds.), Handbook of public policy implementation (pp. 493–504). Edward Elgar Publishing.

42.

Revillard

Lucas

(2025). Identifier et expliquer les effets de l’animation de la vie sociale. CNAF, Dossier d’étude 236.

43.

Roth

E. M.

Eggleston

R. G.

(2010). Forging new evaluation paradigms: Beyond statistical generalization (1st ed., pp. 203–220). CRC Press. https://doi.org/10.1201/9781315593173-16

44.

Shadish

W. R.

(1995). The logic of generalization: Five principles common to experiments and ethnographies. American Journal of Community Psychology, 23(3), 419–428. https://doi.org/10.1007/BF02506951

45.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Wadsworth Publishing.

46.

Small

M. L.

(2009). “How many cases do I need?” on science and the logic of case selection in field-based research. Ethnography, 10(1), 5–38. https://doi.org/10.1177/1466138108099586

47.

Stake

R. E.

Trumbull

D. J.

(1982). Naturalistic generalizations. Review Journal of Philosophy and Social Science, 7(1), 1–12.

48.

Suchman

E. A.

(1967). Evaluative research. Principles and practice in public service and social action programs. Russell Sage Foundation.

49.

Tashakkori

Johnson

Teddlie

(2021). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences (2nd ed). Sage Publications, Inc.

50.

Timmermans

Tavory

(2012). Theory construction in qualitative research:from grounded theory to abductive analysis. Sociological Theory, 30(3), 167–186.

51.

Van Belle

Abejirinde

I.-O.

Ssennyonjo

Srinivas

P. N.

Hebbar

Marchal

(2023). How to develop a realist programme theory using margaret Archer’s structure–agency–culture framework: The case of adolescent accountability for sexual and reproductive health in urban resource-constrained settings. Evaluation, 29(3), 259–275. https://doi.org/10.1177/13563890231185167

52.

Weiss

C. H.

(1998). Evaluation: Methods for studying programs and policies. Prentice-Hall.

53.

Woolcock

(2013). Using case studies to explore the external validity of ‘complex’ development interventions. Evaluation, 19(3), 229–248. https://doi.org/10.1177/1356389013495210

54.

Woolcock

(2022). Will it work here? Using case studies to generate ‘Key Facts’ about complex development programs. In Widner

J. A.

Woolcock

Ortega Nieto

(Eds.), The case for case studies (1st ed., pp. 87–116). Cambridge University Press. https://doi.org/10.1017/9781108688253.006

55.

Yin

(2010). Analytic generalization. In Mills

Durepos

Wiebe

(Eds.), Encyclopedia of case study research (pp. 20–22). Thousand Oaks: SAGE. https://doi.org/10.4135/9781412957397.n8

External Validity and Generalizability in Program Evaluation: Embracing Complexity

Abstract

Keywords

Get full access to this article

References