What can be said about effect of an intervention without a control group? The lack of evaluative evidence is a long-standing problem for regulatory policies against work-related health and safety risks. The European Union Occupational Safety and Health Framework has been in operation for three decades and covers more than 200 million workers, but the most recent evaluation was inconclusive about the benefits generated by this framework. A theory-based evaluation focusing on mechanisms in combination with a design capturing within-intervention variations offers a way forward. The idea is to measure the prevalence of most likely mechanisms and their correlation with outcomes. This approach is illustrated in a large-N evaluation of the use of workplace assessments in the public sector in Denmark. The strengths and weakness of the workplace assessment legislation are assessed. It is shown how findings based on the presented approach contribute to the public debate about workplace assessments.
AstburyBLeeuwF (2010) Unpacking Black Boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation31(3): 363–81.
2.
CartwrightN (2007) Hunting Causes and Using them. Cambridge: Cambridge University Press.
3.
CheungKLten KloosterPMSmitC, et al. (2017) The impact of non-response bias due to sampling in public health studies: A comparison of voluntary versus mandatory recruitment in a Dutch national survey on adolescent health. BMC Public Health17: 276.
4.
ColdwellM (2019) Reconsidering context: Six underlying features of context to improve learning from evaluation. Evaluation25(1): 99–117.
5.
CousinsJBWhitmoreE (2004) Framing participatory evaluation. New Directions for Evaluation1998(80): 5–23.
6.
CoxTKaranikaMGriffithsA, et al. (2007) Evaluating organizational level work stress interventions: Beyond traditional methods. Work & Stress21(4): 348–62.
7.
Da SilvaSLCAmaralFG (2019) Critical factors of success and barriers to the implementation of occupational health and safety management systems: A systematic review of the literature. Safety Science117: 123–32.
European Commission (2017) Ex-Post Evaluation of the European Union Occupational Safety and Health Directives. Brussels: European Commission.
14.
FontaineG (2020) The contribution of policy design to realist evaluation. Evaluation26: 296–314.
15.
FordJAJonesAWongG, et al. (2018) Access to primary care for socioeconomically disadvantaged older people in rural areas: Exploring realist theory using structural equation modelling in a linked dataset. BMC Medical Research Methodology18: 57.
16.
JagoshJBushPLSalsbergJ, et al. (2015) A realist evaluation of community-based participatory research: Partnership synergy, trust building and related ripple effects. BMC Public Health15(1): 725.
17.
JamalFFletcherAShackletonN, et al. (2015) The three stages of building and testing mid-level theories in a realist RCT: A theoretical and methodological case-example. Trials16(1): 466.
GriffithsA (1999) Organizational interventions: Facing the limits of the natural science paradigm. Scandinavian Journal of Work, Environment & Health25(6): 589–96.
20.
HarrisKHendersonSWinkB (2019) Mobilising Q methodology within a realist evaluation: Lessons from an empirical study. Evaluation25(4): 430–48.
21.
HaslePZwetslootG (2011) Editorial: Occupational health and safety management systems: Issues and challenges. Safety Science49(7): 961–63.
22.
HaslePLimborgHNielsenK (2014) Working environment interventions – Bridging the gap between policy instruments and practice. Safety Science68: 73–80.
23.
HawkinsAJ (2016) Realist evaluation and randomised controlled trials for testing program theory in complex social systems. Evaluation22(3): 270–85.
24.
HelboAJHohnenPHasleP (2016) Internal audits of psychosocial risks at workplaces with certified OHS management systems. Safety Science84: 201–9.
25.
HohnenPHasleP (2011) Making work environment auditable: A ‘critical case’ study of certified occupational health and safety management systems in Denmark. Safety Science49(7): 1022–9.
26.
KrollA (2015) Drivers of performance information use: Systematic literature review and directions for future research. Public Performance & Management Review38(3): 459–86.
LascoumesPLe GalèsP (2007) Introduction: Understanding public policy through its instruments: From the nature of instruments to the sociology of public policy instrumentation. Governance20(1): 1–21.
LedermannS (2012) Exploring the necessary conditions for evaluation use in program change. American Journal of Evaluation33(2): 159–78.
31.
MarchalBWesthorpGWongG, et al. (2013) Realist RCTs of complex interventions – An oxymoron. Social Science & Medicine94: 124–128.
32.
NielsenK (2000) Organization theories implicit in various approaches to OHS management. In: FrickKJensenPLQuinlanM, et al. (eds) Systematic Occupational Health and Safety Management Perspectives on an International Development. Oxford: Pergamon Press, 99–124.
33.
NielsenKRandallR (2013) Opening the black box: Presenting a model for evaluating organizational-level interventions. European Journal of Work and Organizational Psychology22(5): 601–17.
34.
ØvretveitJ (2011) Understanding the conditions for improvement: Research to discover which context influences affect improvement success. BMJ Quality & Safety 20(Suppl. 1): i18–i23.
35.
PattynV (2019) Towards appropriate impact evaluation methods. The European Journal of Development Research31: 174–9.
36.
PawsonR (2013) The Science of Evaluation: A Realist Manifesto. London: SAGE.
PedersenLMNielsenKJKinesP (2012) Realistic evaluation as a new way to design and evaluate occupational safety interventions. Safety Science50: 48–54.
39.
PodsakoffPMMacKenzieSBLeeJ-Y (2003) Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology88(5): 879–903.
40.
PorterS (2015) The uncritical realism of realist evaluation. Evaluation21(1): 65–82.
41.
RadaelliCMMeuweseA (2010) Hard questions, hard solutions: Proceduralisation through impact assessment in the EU. West European Politics33(1): 136–53.
42.
RavnR (2019) Testing mechanisms in large-N realistic evaluations. Evaluation25(7): 171–88.
43.
RobsonLSClarkeJACullenKL, et al. (2007) The effectiveness of occupational health and safety management system interventions: A systematic review. Safety Science45(3): 329–53.
44.
SchmidtV (2012) Democracy and legitimacy in the European Union revisited: Output, input and throughput. Political Studies13(61): 2–22.
45.
SchwandtT (2015) Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice. Stanford: Stanford University Press.
46.
SmismansS (2003) Towards a new community strategy on health and safety at work? Caught in the institutional web of soft procedures. The International Journal of Comparative Labour Law and Industrial Relations19(1): 55–84.
47.
StameN (2010) What doesn’t work? Three failures, many answers. Evaluation16(4): 371–87.
48.
SternEStameNMayneJ, et al. (2012) Broadening the range of designs and methods for impact evaluations. Report of a Study Commissioned by the Department for International Development, Working Paper 38. Department for International Development. Available at: https://www.oecd.org/derec/50399683.pdf (accessed 18 January 2019).
49.
ValovirtaV (2002) Evaluation utilization as argumentation. Evaluation8(1): 60–80.
50.
Van BelleSWongGWesthorpG, et al. (2016) Can “realist” randomised controlled trials be genuinely realist?Trials17(1): 313.
51.
WautersBBeachD (2018) Process tracing and congruence analysis to support theory-based impact evaluation. Evaluation24(3): 284–305.
52.
WeissCH (1997) Theory-based evaluation: Past, present, and future. New Directions for Evaluation76: 41–55.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
