This overview aims to stimulate conceptual and practical discussions to help unlock the full potential of realist evaluation in health systems research. Based on a structured literature search, this review maps how the concepts of realist evaluation are applied in health systems research and which methodological problems are encountered. We found a great diversity in the depth of application of the philosophical concepts, use of terminology and scope of application in the research process. Terms of theory-driven evaluation, theories of change and realist evaluation are often used interchangeably. Diverging views exist regarding the nature of ‘mechanism’ and the difference between mechanism and essential context condition. A lack of methodological guidance was highlighted by many authors. Realist evaluation is slowly gaining traction in health systems research, but more clarity is needed concerning the definitions of mechanisms and context and how the configuration of context, mechanism and intervention can be described and assessed.
AstburyBLeeuwF (2010) Unpacking black boxes: mechanisms and theory building in evaluation. American Journal of Evaluation31: 363–81.
2.
BambergerP (2008) Beyond contextualisation: using context theories to narrow the micro-macro gap in management research. Academy Of Management Journal51: 839–46.
3.
BarnesMMatkaESullivanH (2003) Evidence, understanding and complexity: evaluation in non-linear systems. Evaluation9: 265–84.
4.
BerwickDM (2008) The science of improvement. JAMA299: 1182–4.
5.
BhaskarR (1998) Philosophy and scientific realism. In: ArcherMBhaskarRCollierALawsonTNorrieA (eds) Critical Realism. Essential Readings. London & New York: Routledge.
6.
BlaisePKegelsG (2004) A realistic approach to the evaluation of the quality management movement in health care systems: a comparison between European and African contexts based on Mintzberg’s organizational models. International Journal of Health Planning and Management19: 337–64.
7.
BlameyAMackenzieM (2007) Theories of change and realistic evaluation: peas in a pod or apples and oranges?Evaluation13(4): 439–55.
8.
ByngRNormanIRedfernS (2005) Using realistic evaluation to evaluate a practice-level intervention to improve primary healthcare for patients with long-term mental illness. Evaluation11: 69–93.
9.
ByngRNormanIRedfernSJonesR (2008) Exposing the key functions of a complex intervention for shared care in mental health: case study of a process evaluation. BMC Health Services Research8: 274.
10.
CalnanMFerlieE (2003) Analysing process in healthcare: the methodological and theoretical challenges. Policy and Politics31: 185–93.
11.
CampbellNCMurrayEDarbyshireJEmeryJFarmerAGriffithsFGuthrieBLesterHWilsonPKinmonthAL (2007) Designing and evaluating complex interventions to improve health care. British Medical Journal334: 455–9.
12.
ChecklandP (1999) Systems Thinking, Systems Practice. Chichester: Wiley.
13.
ChenH-T (1989) The conceptual framework of the theory-driven perspective. Evaluation and Program Planning12: 391–6.
14.
ChenH-T (1990) Theory-Driven Evaluations. Newbury Park, CA: SAGE.
15.
ChenH-T (2005) Practical Program Evaluation. Thousand Oaks, CA: SAGE.
16.
ChenH-TRossiP (1980) Multi-goal theory-driven approach: a model linking basic and applied science. Social Forces59(1): 106–22.
17.
ChenH-TRossiP (1983) Evaluating with Sense – The Theory Driven Approach. Evaluation Review7(3): 283–302.
18.
ChenH-TRossiP (1987) The theory-driven approach to validity. Evaluation and Program Planning10: 95–103.
19.
ChenH-TRossiP (1989) Issues in the theory-driven perspective. Evaluation and Program Planning12: 299–306.
20.
ClarkAMWhelanHKBarbourRMacintyrePD (2005) A realist study of the mechanisms of cardiac rehabilitation. Journal of Advanced Nursing52: 362–71.
21.
ColeG (1999) Advancing the development and application of theory-based evaluation in the practice of public health. American Journal of Evaluation20: 453–70.
22.
ConnellJPKubischASchorrLWeissC (1995) New Approaches To Evaluating Community Initiatives. Concepts, Methods, and Contexts. Washington, DC: The Aspen Institute.
23.
ConnellyJ (2000) A realistic theory of health sector management: the case for critical realism. Journal of Management in Medicine14: 262–71.
24.
ConnellyJ (2004) Realism in evidence based medicine: interpreting the randomised controlled trial. Journal of Health Organization and Management18: 70–5.
25.
ConnellyJ (2007) Evaluating complex public health interventions: theory, methods and scope of realist enquiry. Journal of Evaluation in Clinical Practice13(6): 935–41.
26.
CorynCNoakesLWestineCSchröterD (2011) A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation32: 199–226.
27.
CostnerH (1989) The validity of conclusions in evaluation research: a further development of Chen and Rossi’s theory-driven approach. Evaluation and Program Planning12: 345–53.
28.
DickinsonH (2006) The evaluation of health and social care partnerships: an analysis of approaches and synthesis for the future. Health and Social Care in the Community14: 375–83.
29.
DouglasFCGrayDAVan TeijlingenER (2010) Using a realist approach to evaluate smoking cessation interventions targeting pregnant women and young people. BMC Health Services Research10: 49.
30.
EcclesMGrimshawJCampbellMRamsayC (2003) Research designs for studies evaluating the effectiveness of change and improvement strategies. Quality and Safety in Health Care12: 47–52.
31.
EvansDKilloranA (2000) Tackling health inequalities through partnership working: lessons from a realistic evaluation. Critical Public Health10: 125–40.
32.
FissP (2007) A set-theoretic approach to organizational configurations. Academy of Management Review32: 1180–98.
33.
Fulbright-AndersonAKubischAConnellJ (1998) New Approaches To Evaluating Community Initiatives. Theory, Measurement, and Analysis. Washington, DC: Aspen Institute.
34.
FulopNAllenPClarkeABlackN (2001) Studying the Organisation and Delivery of Health Services: Research Methods. London: Routledge.
35.
GreenhalghTHumphreyCHughesJMacfarlaneFButlerCPawsonR (2009) How do you modernize a health service? A realist evaluation of whole-scale transformation in London. Milbank Q87: 391–416.
36.
GrimshawJWilsonBCampbellMEcclesMRamsayC (2001) Epidemiological methods. In: FulopNAllenPClarkeABlackN (eds) Studying the Organisation and Delivery of Health Services: Research Methods. London: Routledge.
37.
GrocottPCowleySRichardsonA (2002) Solving methodological challenges using a theory-driven evaluation in the study of complex patient care. Evaluation8: 306–21.
JulnesGMarkMHenryG (1998) Promoting realism in evaluation. realistic evaluation and the broader context. Evaluation4: 483–504.
40.
KanekoM (1999) A methodological inquiry into the evaluation of smoking cessation programmes. Health Education Research14(3): 433–41.
41.
KaziM (2003a) Realist evaluation for practice. British Journal of Social Work33: 803–18.
42.
KaziM (2003b) Realist Evaluation in Practice. Health and Social Work. London: SAGE.
43.
KernickD (2006) Wanted–new methodologies for health service research. is complexity theory the answer?Family Practice23: 385–90.
44.
KernickDMannionR (2005) Developing an evidence base for intermediate care delivered by GPs with a special interest. British Journal of Medical Practice55(521): 908–10.
45.
LeeuwF (2003) Reconstructing program theories: methods available and problems to be solved. American Journal of Evaluation24: 5–20.
46.
LeoneL (2008) Realistic evaluation of an illicit drug deterrence programme. Evaluation14: 19–28.
47.
MackenzieMBlameyA (2005) The practice and theory: lessons from the application of a theories of change approach. Evaluation11(2): 151–68.
48.
MackenzieMKoshyPLeslieWLeanMHankeyC (2009) Getting beyond outcomes: a realist approach to help understand the impact of a nutritional intervention during smoke cessation. European Journal of Clinical Nutrition63: 1136–42.
49.
MalukaSKamuzoraPSan SebastianMByskovJNdawiBOlsenOHurtigAK (2011) Implementing accountability for reasonableness framework at district level in Tanzania: a realist evaluation. Implementation Science6(1): 11.
50.
Manzano-SantaellaA (2011) A realistic evaluation of fines for hospital discharges: incorporating the history of programme evaluations in the analysis. Evaluation17: 21–36.
51.
MarchalBDedzoMKegelsG (2010a) A realist evaluation of the management of a well-performing regional hospital in Ghana. BMC Health Services Research10: 24.
52.
MarchalBDedzoMKegelsG (2010b) Turning around an ailing district hospital: a realist evaluation of strategic changes at Ho Municipal Hospital (Ghana). BMC Public Health10: 787.
53.
MasonPBarnesM (2007) Constructing theories of change: methods and sources. Evaluation13: 151–70.
54.
McEvoyPRichardsD (2003) Critical realism: a way forward for evaluation research in nursing?Journal of Advanced Nursing43: 411–20.
55.
MercierCPiatMPeladeauNDagenaisC (2000) An application of theory-driven evaluation to a drop-in youth center. Evaluation Review1: 73–91.
56.
MertonRK (1968) Social Theory and Social Structure. New York: The Free Press.
57.
MillsAGilsonLHansonKPalmerNLargardeM (2008) What do we mean by rigorous health-systems research?Lancet372: 1527–9.
58.
MingersJ (2000) The contribution of critical realism as an underpinning philosophy for Or/Ms and systems. Journal of the Operational Research Society51: 1256–70.
59.
MintzbergH (1989) Mintzberg on Management. Inside Our Strange World of Organizations. New York: The Free Press.
60.
OgrincGBataldenP (2009) Realist evaluation as a framework for the assessment of teaching about the improvement of care. Journal of Nursing Education48: 661–7.
61.
PawsonR (2002) Evidence-based policy: the promise of ‘realist synthesis’. Evaluation8: 340–58.
62.
PawsonR (2006) Evidence-Based Policy. A Realist Perspective. London: SAGE.
PawsonRGreenhalghTHarveyGWalsheK (2005) Realist review–a new method of systematic review designed for complex policy interventions. Journal of Health Services Research and Policy10(Suppl. 1): 21–34.
65.
PedersenLRieperO (2008) Is realist evaluation a realistic approach for complex reforms?Evaluation14: 271–93.
66.
PommierJGuévelM-RJourdanD (2010) Evaluation of health promotion in schools: a realistic evaluation approach using mixed methods. BMC Public Health10: 43.
67.
RedfernSChristianSNormanI (2003) Evaluating change in health care practice: lessons from three studies. Journal of Evaluation in Clinical Practice9: 239–49.
68.
RiddeVGuichardA (Forthcoming) Perception de quelques mécanismes favorables a la reduction des inégalités sociales de santé en France. Global Health Promotion.
69.
RodriguezEMeadJ (1997) Evaluating a community oriented primary care program: lessons learned through a theory-oriented approach. Evaluation and Program Planning20: 217–24.
70.
RogersP (2007) Theory-based evaluation: reflections ten years on. New Directions for Evaluation114: 63–84.
71.
Rycroft-MaloneJFontenlaMBickDSeersK (2010) A realistic evaluation: the case of protocol-based care. Implementation Science5(1): 38.
72.
SidaniSDoranDMMitchellPH (2004) A theory-driven approach to evaluating quality of nursing care. Journal of Nursing Scholarship36(1): 60–5.
73.
SmithR (2000) Quality improvement reports: a new kind of article. British Medical Journal321: 1428.
74.
StameN (2004) Theory-based evaluation and types of complexity. Evaluation10: 58–76.
75.
StecklerAMcleroyKR (2008) The importance of external validity. American Journal of Public Health98: 9–10.
76.
SturmbergJMartinC (2009) Complexity and health – yesterday’s traditions, tomorrow’s future. Journal of Evaluation in Clinical Practice15: 543–8.
77.
SullivanHBarnesMMatkaE (2002) Building collaborative capacity through ‘theories of change’: early lessons from the evaluation of health action zones in England. Evaluation8: 205–26.
78.
TolsonDMcintoshJLoftusLCormieP (2007) Developing a managed clinical network in palliative care: a realistic evaluation. International Journal of Nursing Studies44: 183–95.
79.
TrochimW (1989) Pattern matching and program theory. Evaluation and Program Planning12: 355–66.
80.
Van BelleSMarchalBDubourgDKegelsG (2010) How to develop a theory-driven evaluation design – lessons learned from an adolescent sexual and reproductive health programme in West-Africa. BMC Public Health10: 741.
81.
VictoraCGHabichtJPBryceJ (2004) Evidence-based public health: moving beyond randomized trials. American Journal of Public Health94: 400–5.
82.
WalsheK (2007) Understanding what works–and why–in quality improvement: the need for theory-driven evaluation. International Journal for Quality in Health Care19: 57–9.
83.
WandTWhiteKPatchingJ (2010) Applying a realist(ic) framework to the evaluation of a new model of emergency department based mental health nursing practice. Nursing Inquiry17: 231–9.
84.
WeissC (1995) Nothing as practical as good theory: exploring theory-based evaluation for comprehensive community initiatives. In: ConnellJKubischASchorrLWeissC (eds) New Approaches to Evaluating Comprehensive Community Initiatives: Concepts, Methods and Contexts. Washington, DC: The Aspen Institute.
85.
WeissC (1997) Theory-based evaluation: past, present, and future. New Directions for Program Evaluation76: 41–55.
86.
Ying HoS (1999) Evaluating urban regeneration programmes in Britain. exploring the potential of the realist approach. Evaluation5: 422–38.
