As a theory-driven evaluation design grounded in the critical realism philosophy, realist evaluation relies on eliciting the assumptions of stakeholders about how programmes are expected to produce outcomes. Little is known about key principles for involving stakeholders in realist evaluations. In this scoping review of realist evaluation studies, we explored how stakeholders were involved in the conduct of realist evaluation. A total of 26 ‘participatory’ realist evaluation studies have been included. We analysed the approach to the involvement of stakeholders, using an analytical framework rooted in existing theories and models of participatory research and evaluations, using a ‘Best Fit Framework Synthesis’ methodology. We found a large range of approaches to stakeholder involvement, related to the types of stakeholders involved, their depth of involvement and means by which researchers/evaluators fostered their involvement. We discuss key lessons learnt and contribute to foster transparent and meaningful involvement of stakeholders in Realist Evaluation.
AbmaTAWiddershovenGAM (2008) Evaluation and/as social relation. Evaluation14(2): 209–25.
2.
AbmaTABanksSCookT, et al. (2019) Participatory Research for Health and Social Well-Being. Cham: Springer.
3.
AbramsRParkSWongG, et al. (2021) Lost in reviews: Looking for the involvement of stakeholders, patients, public and other non-researcher contributors in realist reviews. Research Synthesis Methods12(2): 239–47.
AlongeORodriguezDCBrandesN, et al. (2019) How is implementation research applied to advance health in low-income and middle-income countries?BMJ Global Health4(2): e001257.
6.
ArcherMS (1995) Realist Social Theory: The Morphogenetic Approach. Cambridge: Cambridge University Press.
7.
ArkseyHO’MalleyL (2005) Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology8(1): 19–32.
8.
AstburyBLeeuwFL (2010) Unpacking black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation31(3): 363–81.
9.
BaumFMacDougallCSmithD (2006) Participatory action research. Journal of Epidemiology and Community Health60(10): 854–7.
10.
BeckettKFarrMKothariA, et al. (2018) Embracing complexity and uncertainty to create impact: Exploring the processes and transformative potential of co-produced research through development of a social impact model. Health Research Policy and Systems16(1): 118.
11.
BergeronDATalbotLRGabouryI (2020) Intersectoral oral health promotion interventions for schoolchildren living in remote rural Andean communities: A realist evaluation. Global Health Promotion27(3): 103–12.
12.
BhaskarR (2008) A Realist Theory of Science. London: Verso books.
13.
BoothACarrollC (2015) How to build up the actionable knowledge base: The role of ‘best fit’ framework synthesis for studies of improvement in healthcare. BMJ Quality & Safety24(11): 700.
14.
BrandonPRFukunagaLL (2014) The state of the empirical research literature on stakeholder involvement in program evaluation. American Journal of Evaluation35(1): 26–44.
15.
BrisolaraS (1998) The history of participatory evaluation and current debates in the field. New Directions for Evaluation1998(80): 25–41.
16.
BrysonJMPattonMQBowmanRA (2011) Working with evaluation stakeholders: A rationale, step-wise approach and toolkit. Evaluation and Program Planning34(1): 1–12.
17.
BulleyCBaerGMahalD, et al. (2021) Supporting families and carers of people living with multiple sclerosis: A rapid realist review and realist evaluation. International Journal of Care and Caring5(2): 263–82.
18.
BungeM (1997) Mechanism and explanation. Philosophy of the Social Sciences27(4): 410–65.
19.
ButlerCWilsonPAbrahamsonV, et al. (2022) Optimum models of hospice at home services for end-of-life care in England: A realist-informed mixed-methods evaluation. Health and Social Care Delivery Research10(24): 1–304.
20.
CardiffSSandersKWebsterJ, et al. (2020) Guiding lights for effective workplace cultures that are also good places to work. International Practice Development Journal10(2): 2.
21.
CarrollCBoothACooperK (2011) A worked example of ‘best fit’ framework synthesis: A systematic review of views concerning the taking of some potential chemopreventive agents. BMC Medical Research Methodology11(1): 29.
22.
CarrollCBoothALeavissJ, et al. (2013) ‘Best fit’ framework synthesis: Refining the method. BMC Medical Research Methodology13(1): 37.
23.
ChevalierJMBucklesDJ (2019) Participatory Action Research. Theory and Methods for Engaged Inquiry. London: Routledge.
24.
ChouinardJACousinsJB (2021) Developing an ethical rationale for collaborative approaches to evaluation. Evaluation27(3): 364–81.
25.
ChouinardJAMilleyP (2016) Mapping the spatial dimensions of participatory practice: A discussion of context in evaluation. Evaluation and Program Planning54: 1–10.
26.
CopestakeJ (2021) Case selection for robust generalisation: Lessons from QuIP impact evaluation studies. Development in Practice31(2): 150–60.
27.
CornwallAJewkesR (1995) What is participatory research?Social Science & Medicine41(12): 1667–76.
28.
CousinsJBWhitmoreE (1998) Framing participatory evaluation. New Directions for Evaluation1998(80): 5–23.
29.
DanielsJEBellBHorrocksC (2018) Capturing the realities of sports programmes: Systematic ‘messiness’?International Journal of Sport Policy and Politics10(4): 779–94.
30.
DarlingtonEJViolonNJourdanD (2018) Implementation of health promotion programmes in schools: An approach to understand the influence of contextual factors on the process?BMC Public Health18(1): 163.
31.
DaviesSBurtonCRWilliamsL, et al. (2019) Brief smoking cessation in acute Welsh hospitals: A realist approach. Health Promotion International35(2): 244–54.
32.
De WegerEBaanCBosC, et al. (2022a) ‘They need to ask me first’. Community engagement with low-income citizens. A realist qualitative case-study. Health Expectations25(2): 684–96.
33.
De WegerEDrewesHWVan VoorenNJE, et al. (2022b) Engaging citizens in local health policymaking. A realist explorative case-study. PLoS ONE17(3): e0265404.
34.
De WegerEVan VoorenNJEDrewesHW, et al. (2020) Searching for new community engagement approaches in the Netherlands: A realist qualitative study. BMC Public Health20(1): 508.
35.
DewingJ (2007) Participatory research: A method for process consent with persons who have dementia. Dementia6(1): 11.
36.
EmmelNGreenhalghJManzanoA, et al. (2018) Doing Realist Research. Los Angeles, CA; London; New Delhi, India; Singapore; Washington, DC; Melbourne, VIC, Australia: Sage.
37.
Figueiredo NascimentoSCuccillatoESchadeS, et al. (2016) Citizen engagement in science and policy-making. EUR 28328 EN. Luxembourg. Publications Office of the European Union. JRC105004.
38.
FettermanDM (2001) Foundations of Empowerment Evaluation. London: Sage.
39.
FettermanDMRodríguez-CamposLWandersmanA, et al. (2014) Collaborative, participatory, and empowerment evaluation: Building a strong conceptual foundation for stakeholder involvement approaches to evaluation (a response to Cousins, Whitmore, and Shulha, 2013). American Journal of Evaluation35(1): 144–8.
40.
GibbsG (1988) Learning by Doing: A Guide to Teaching and Learning Methods. Oxford: Further Education Unit Oxford Polytechnic.
41.
GreenhalghTJacksonCShawS, et al. (2016) Achieving research impact through co-creation in community-based health services: Literature review and case study. Milbank Quarterly94(2): 392–429.
HarréR (1986) Varieties of Realism: A Rationale for the Natural Sciences. Oxford: Blackwell.
44.
HeatonJDayJBrittenN (2015) Inside the ‘black box’ of a knowledge translation program in applied health research. Qualitative Health Research25(11): 1477–91.
45.
HerzogLLepeniesR (2022) Citizen science in deliberative systems: Participation, epistemic injustice, and civic empowerment. Minerva60(4): 489–508.
46.
HigginsJPThomasJChandlerJ, et al. (2019) Cochrane Handbook for Systematic Reviews of Interventions. Chichester: John Wiley & Sons.
47.
HinrichsenCNelausenMKNielsenL, et al. (2022) The workings of an action learning program for building mental health promotion capacity: A realist evaluation. Evaluation and Program Planning92: 102080.
48.
HouseERHoweKR (2003) Deliberative democratic evaluation. In: KellaghanTStufflebeamDL (eds) International Handbook of Educational Evaluation. Dordrecht: Springer, 79–100.
49.
JacksonCMcBrideTManleyK, et al. (2021) Strengthening nursing, midwifery and allied health professional leadership in the UK: A realist evaluation. Leadership in Health Services34(4): 392–453.
50.
JagoshJMacaulayACPluyeP, et al. (2012) Uncovering the benefits of participatory research: Implications of a realist review for health research and practice. Milbank Quarterly90(2): 311–46.
51.
JanssenMMVosWLuijkxKG (2019) Development of an evaluation tool for geriatric rehabilitation care. BMC Geriatrics19(1): 206.
52.
LaskerRDWeissESMillerR (2001) Partnership synergy: A practical framework for studying and strengthening the collaborative advantage. The Milbank Quarterly79(2): 179–205.
53.
LevacDColquhounHO’BrienKK (2010) Scoping studies: Advancing the methodology. Implementation Science5(1): 69.
54.
LhussierMBangashSDykesF, et al. (2012) Development and implementation of a nutrition intervention programme in North West Pakistan: A realist framework. Health Promotion International27(4): 453–62.
55.
MacaulayACCommandaLEFreemanWL, et al. (1999) Participatory research maximises community and lay involvement. North American Primary Care Research Group. BMJ319(7212): 774–8.
56.
MaitlisSChristiansonM (2014) Sensemaking in organizations: Taking stock and moving forward. The Academy of Management Annals8(1): 57–125.
57.
MalengreauxSDoumontDScheenB, et al. (2022) Realist evaluation of health promotion interventions: A scoping review. Health Promotion International37(5): daac136.
58.
ManzanoA (2016) The craft of interviewing in realist evaluation. Evaluation22(3): 342–60.
59.
ManzanoA (2022) Conducting focus groups in realist evaluation. Evaluation28(4): 406–25.
60.
ManzanoA (2024) User and stakeholder involvement in realist evaluation. Sciences Po. LIEPP working paper n°158, hal-04410009.
61.
MarchalBAbejirindeIOSulaberidzeL, et al. (2021) How do participatory methods shape policy? Applying a realist approach to the formulation of a new tuberculosis policy in Georgia. BMJ Open11(6): e047948.
62.
MarchalBKegelsGBelleSV (2018) Theory and realist methods. In: EmmelNGreenhalghJManzanoA, et al. (eds) Doing Realist Research. Los Angeles, CA; London; New Delhi, India: Sage, 79–90.
63.
MarchalBVan BelleSVan OlmenJ, et al. (2012) Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research. Evaluation18(2): 192–212.
64.
McConnellTBestPSturmT, et al. (2020) A translational case study of empowerment into practice: A realist evaluation of a member-led dementia empowerment service. Dementia (London)19(6): 1974–96.
65.
McInnesRJAitken-ArbuckleALakeS, et al. (2020) Implementing continuity of midwife carer – Just a friendly face? A realist evaluation. BMC Health Services Research20(1): 304.
66.
MertensDM (2008) Transformative Research and Evaluation. New York: Guilford Publications.
67.
MukumbangFCvan WykB (2020) Leveraging the photovoice methodology for critical realist theorizing. International Journal of Qualitative Methods19: 1609406920958981.
68.
MunnZPetersMDJSternC, et al. (2018) Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology18(1): 143.
69.
NelsonGOchockaJGriffinK, et al. (1998) ‘Nothing about me, without me’: Participatory action research with self-help/mutual aid organizations for psychiatric consumer/survivors. American Journal of Community Psychology26(6): 881–912.
70.
OetzelJGZhouCDuranB, et al. (2015) Establishing the psychometric properties of constructs in a community-based participatory research conceptual model. American Journal of Health Promotion29(5): e188–202.
71.
OuzzaniMHammadyHFedorowiczZ, et al. (2016) Rayyan – A web and mobile app for systematic reviews. Systematic Reviews5: 1–10.
72.
PalmRFahsoldARoesM, et al. (2021) Context, mechanisms and outcomes of dementia special care units: An initial programme theory based on realist methodology. PLoS ONE16(11): e0259496.
73.
PawsonR (2013) The Science of Evaluation: A Realist Manifesto. London: Sage.
74.
PawsonRManzano-SantaellaA (2012) A realist diagnostic workshop. Evaluation18(2): 176–91.
PeekSTMLuijkxKGVrijhoefHJM, et al. (2017) Origins and consequences of technology acquirement by independent-living seniors: Towards an integrative model. BMC Geriatrics17(1): 189.
PolaninJRPigottTDEspelageDL, et al. (2019) Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Research Synthesis Methods10(3): 330–42.
79.
RenmansDCastellano PleguezueloV (2023) Methods in realist evaluation: A mapping review. Evaluation and Program Planning97: 102209.
80.
RenmansDSarkarNVan BelleS, et al. (2022) Realist evaluation in times of decolonising global health. The International Journal of Health Planning and Management37(Suppl. 1): 37–44.
81.
Rycroft-MaloneJBurtonCRWilkinsonJ, et al. (2015a) Collective action for implementation: A realist evaluation of organisational collaboration in healthcare. Implementation Science11: 17.
82.
Rycroft-MaloneJBurtonCRWilkinsonJ, et al. (2015b) Collective Action for Knowledge Mobilisation: A Realist Evaluation of the Collaborations for Leadership in Applied Health Research and Care. Health Services and Delivery Research 3.
83.
SayerA (1992) Method in Social Science: A Realist Approach. London; New York: Routledge.
84.
SayerA (2000) Realism and Social Science. London: Sage.
85.
ScherBDScott-BarrettJHickmanM, et al. (2023) Participatory research emergent recommendations for researchers and academic institutions: A rapid scoping review. Journal of Participatory Research Methods4(2): 1–34.
86.
SchwandtT (2015) Cultivating a Life of the Mind for Practice. Redwood City, CA: Stanford University Press.
87.
SewardNHanlonCHinrichs-KraplesS, et al. (2021) A guide to systems-level, participatory, theory-informed implementation research in global health. BMJ Global Health6(12): e005365.
88.
ShulhaLMWhitmoreECousinsJB, et al. (2016) Introducing evidence-based principles to guide collaborative approaches to evaluation: Results of an empirical process. American Journal of Evaluation37(2): 193–215.
89.
TaylorJJonesRMO’ReillyP, et al. (2010) The Station Community Mental Health Centre Inc: Nurturing and empowering. Rural and Remote Health10(3): 1411.
90.
TolsonDSchofieldI (2012) Football reminiscence for men with dementia: Lessons from a realistic evaluation. Nursing Inquiry19(1): 63–70.
91.
TriccoACLillieEZarinW, et al. (2018) PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Annals of Internal Medicine169(7): 467–73.
92.
TylerIPaulyBWangJ, et al. (2019) Evidence use in equity focused health impact assessment: A realist evaluation. BMC Public Health19(1): 230.
93.
Van BelleSB (2022) At the interface between the formal and informal, the actual and the real: A realist study protocol for governance and accountability practices in urban settings focusing on adolescent sexual and reproductive health and rights. International Journal for Equity in Health21(1): 40.
94.
Van BelleSBMarchalBDubourgD, et al. (2010) How to develop a theory-driven evaluation design? Lessons learned from an adolescent sexual and reproductive health programme in West Africa. BMC Public Health10(1): 741.
95.
Vanden BosscheDWillemsSDecatP (2022) Understanding trustful relationships between community health workers and vulnerable citizens during the COVID-19 pandemic: A realist evaluation. International Journal of Environmental Research and Public Health19(5): 2496.
96.
Van der VekenKLauwerierEWillemsSJ (2020) How community sport programs may improve the health of vulnerable population groups: A program theory. International Journal for Equity in Health19(1): 74.
97.
VerkooijenKTSuperSMulderijLS, et al. (2020) Using realist interviews to improve theory on the mechanisms and outcomes of sport for development programmes. Social Inclusion8(3): 152–61.
98.
WallersteinNOetzelJDuranB, et al. (2008) What predicts outcomes in CBPR? In: MinklerMWallersteinN (eds) Community Based Participatory Research for Health: Process to Outcomes. San Francisco, CA: Jossey-Bass, 371–92.
99.
WesthorpG (2014) Realist Impact Evaluation: An Introduction. London: Overseas Development Institute.
WesthorpGStevensKRogersPJ (2016) Using realist action research for service redesign. Evaluation22(3): 361–79.
102.
WongGWesthorpGManzanoA, et al. (2016) RAMESES II reporting standards for realist evaluations. BMC Medicine14(1): 96.
103.
WoodEOhlsenSFentonS-J, et al. (2021) Social prescribing for people with complex needs: A realist evaluation. BMC Family Practice22(1): 53.
104.
WrightMTRocheBvon UngerH, et al. (2010) A call for an international collaboration on participatory research for health. Health Promotion International25(1): 115–22.
105.
YardleySWilliamsHBowieP, et al. (2022) Which human factors design issues are influencing system performance in out-of-hours community palliative care? Integration of realist approaches with an established systems analysis framework to develop mid-range programme theory. BMJ Open12(1): e048045.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
