The concept of generative mechanisms is central to the realistic evaluation approach. They are seen as the drivers of change. Qualitative evidence is especially well suited to unearthing how mechanisms work. However, when evaluating a large-N programme, a realist evaluation might benefit from quantitative tests of the mechanisms hypothesized in the programme theory. Despite this, quantitative tests of mechanisms are rarely applied in realistic evaluation. The purpose of this article is therefore to illustrate how widely used quantitative social science methods can be utilized to test mechanisms in realistic evaluation when evaluating large-N programmes. The proposed methods focus on intra-programme comparison based on the strength of a quantitatively measured mechanism. The article illustrates how simple statistical methods in the form of descriptive statistics and logistic regression can be used to test the influence of mechanisms in generating outcomes.
AgrestiAFinlayB (2009) Statistical Methods for the Social Sciences. Upper Saddle River, NJ: Pearson Education.
2.
ArendtJNHøjbjergR (2017) Litteraturreview Om Progression for Ikke-arbejdsmarkedsparate Ledige. København: VIVE -Det Nationale Forsknings- og Analysecenter for Velfærd.
3.
BefaniBStedman-BryceG (2017) Process tracing and Bayesian updating for impact evaluation. Evaluation23(1): 42–60.
BordinES (1979) The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: Theory, Research and Practice16: 252–60.
6.
BordinES (1983) A working alliance based model of supervision. The Counseling Psychologist11(1), 35–42.
7.
BordinES (1994) Theory and research on the therapeutic working alliance new directions. In: HorvathAOGreenbergLS (eds) The Working Alliance. New York: Wiley, 13–37.
8.
DalkinSMGreenhalghJJonesDet al. (2015) What’s in a mechanism? Development of a key concept in realist evaluation. Implementation Science10(1): 49.
9.
de VausD (2014) Surveys in Social Research, vol. 6. New York: Routledge.
10.
Doi JepsonLRHardieS (2017) Realist evaluation of an enhanced health visiting programme. PLoS ONE12(7): e0180569.
11.
Donker-CoolsBHPMWindHFrings-DresenMHW (2016) Prognostic factors of return to work after traumatic or non-traumatic acquired brain injury. Disability and Rehabilitation38: 733–41.
12.
Ekspertgruppen om udredning af den aktive beskæftigelses indsats (2015) Nye veje mod job – for borgere i udkanten af arbejdsmarkedet. The Danish Ministry of Employment. Available at: https://bm.dk/media/6783/carsten_koch_2-pdf.pdf
13.
FordJAJonesAWongGet al. (2018) Access to primary care for socio- economically disadvantaged older people in rural areas: Exploring realist theory using structural equation modelling in a linked dataset. BMC Medical Research Methodology18: 57.
14.
FunnelSCRogersPJ (eds) (2011) Processes to identify or develop a program theory. In: Purposeful Program Theory: Effective Use of Theories of Change and Logic Models, vol. 1. San Francisco, CA: John Wiley & Sons, 95–148.
15.
GiffoniFSaliniSSirtoriE (2018) Evaluating business support measures: The Bayesian Network approach. Evaluation24(2): 133–52.
16.
GlaeserELLaibsonDIScheinkmanJAet al. (2000) Measuring trust. The Quarterly Journal of Economics115: 811–46.
17.
GreenhalghTHumphreyCHughesJet al. (2009) How do you modernize a health service? A realist evaluation of whole-scale transformation in London. The Milbank Quarterly87(2): 391–416.
18.
HawkinsAJ (2016) Realist evaluation and randomised controlled trials for testing program theory in complex social systems. Evaluation22(3): 270–85.
HorvathAO (1994) Empirical validation of Bordin’s pantheoretical model of the Alliance: The Working Alliance Inventory perspective. In: The Working Alliance. New York, 109–30.
21.
JagoshJBushPLSalsbergJet al. (2015) A realist evaluation of community-based participatory research: Partnership synergy, trust building and related ripple effects. BMC Public Health15: 725. doi: 10.1186/s12889-015-1949-1
22.
JensenB (2014) Hvad ved vi om modtagerne af kontanthjælp?vol. 1. København: Syddansk Universitetsforlag.
23.
LammertsLSchaafsmaFGEikelenboomMet al. (2016) Longitudinal associations between biopsychosocial factors and sustainable return to work of sick-listed workers with a depressive or anxiety disorder. Journal of Occupational Rehabilitation26(1): 70–9
24.
MahoneyJGoertzG (2006) A tale of two cultures: Contrasting quantitative and qualitative research. Political Analysis14(3): 227–49.
25.
NaefMSchuppJ (2009) Measuring trust: Experiments and surveys in contrast and combination. IZA Discussion Paper, 4087. Available at: http://ftp.iza.org/dp4087.pdf
26.
ØyeflatenIHysingMEriksenHR (2008) Prognostic factors associated with return to work following multidisciplinary vocational rehabilitation. Journal of Rehabilitation Medicine40(7): 548–54.
27.
PawsonR (ed.) (2013a) A Complexity Checklist. In: The Science of Evaluation – A Realist Manifesto. London: SAGE, 33–45.
28.
PawsonR (ed.) (2013b) First principles: A realist diagnostic workshop. In: The Science of Evaluation – A Realist Manifesto. London: SAGE, 13–31.
29.
PawsonR (2013c) The Science of Evaluation – A Realist Manifesto. London: SAGE.
30.
PawsonRTilleyN (eds) (1997a) How to construct realistic data: Utilizing stakeholder’ knowledge, In:Realistic Evaluation. London: SAGE, 153–82.
31.
PawsonRTilleyN (eds) (1997b) No smoking without firing mechanisms: A ‘realistic’ consultation. In: Realistic Evaluation. London: SAGE, 183–99.
SelanderJMarnetoftS-UAsellM (2007) Predictors for successful vocational rehabilitation for clients with back pain problems. Disability and Rehabilitation29(3): 215–20.
37.
Van BelleSWongGWesthorpGet al. (2016) Can ‘realist’ randomised controlled trials be genuinely realist?Trials17(1): 313.
WilliamsM (2018) Making up mechanisms in realist research. In: EmmelNGreenhalghJManzanoAet al. (eds) Doing Realist Research. London: SAGE, 25–40.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
