Sage Journals: Discover world-class research

Abstract

This article argues that evaluation experts stand to benefit significantly from engaging with concepts and methodologies derived from both realist evaluation and implementation science. First, realist evaluation can facilitate the development of nuanced understandings of the intricate interplay among contextual factors, mechanisms of implementation strategies, and implementation outcomes. This enables evaluators to explore how, why, and under what circumstances implementation strategies work. Second, viewing the underlying workings of implementation strategies as ripple effects provides insight into their links with the implementation object, that is, the intervention itself. This provides insights into how implementation strategies can influence the workings of an implementation object. Third, realist evaluation allows for an exploration of unintended and unexpected outcomes, which are often overlooked in traditional evaluation frameworks. This is crucial as unforeseen effects may shape the generation of both implementation and client outcomes. These arguments are illustrated through theoretical and empirical reflections drawn from realist evaluation studies.

Keywords

implementation research and evaluation implementation strategies mechanisms realist evaluation ripple effect unintended outcomes

Introduction

Evaluation requires defining an analytical object and selecting which aspects of the evaluand—whether a health program, technology, or policy—to investigate due to practical constraints and complexity (Funnell and Rogers, 2011; Pawson, 2024). Evaluators must strategically focus on specific elements to ensure feasibility and relevance of the evaluation. However, this inevitably leaves some aspects unexplored, often turning them into “black boxes,” which limits the ability to understand their structure, interactions, and influence within the evaluand. In both implementation and intervention evaluation, there is a persistent risk of reducing either the implementation strategy (e.g. staff training and education, audit and feedback, or employment of implementation champions) or the intervention itself (e.g. mental health promotion (MHP) campaigns, smoking cessation programs, or new IT solutions) to a black box. For example, this “objectification” is reflected in program theories outlining the expected workings of health promotion interventions, where implementation processes are depicted as a single, obscure black box (see, for example, Andersen et al., 2015; Bonnesen et al., 2020; Pisinger et al., 2020). While this allows for in-depth investigations of the intervention itself, such simplifications entail the risk of hiding processes and factors that drive implementation success or failure. Conversely, focusing solely on implementation without thoroughly investigating the intervention can obscure critical aspects of how it produces its effects.

The implications of these simplifications are increasingly recognized. Albers et al. (2020) document a notable rise in implementation-focused publications over the past three decades, reflecting growing interest in how innovations are enacted in real-world settings. The broader field of implementation research encompasses diverse studies examining how innovations, programs, or policies are enacted in practice. A cornerstone of implementation research is the recognition that an intervention or treatment is unlikely to be effective without proper implementation (Albers et al., 2020). Building on this premise, and influenced by the evidence-based medicine movement, implementation science has emerged as a distinct sub-discipline of implementation research (Albers et al., 2020). A central objective is to establish shared taxonomies and standardized methodologies for assessing the effectiveness of implementation strategies (Albers et al., 2020).

To support this effort, studies within the implementation science framework often build on the distinction between implementation strategies and implementation objects (Albers et al., 2020; Powell et al., 2015). Implementation strategies refer to “methods or techniques used to enhance the adoption, implementation, and sustainability of a clinical program or practice” (Powell et al., 2015), whereas the implementation object describes the intervention, policy, or technology being implemented (Nilsen, 2020). Proctor et al. (2011) operationalize this distinction by outlining how implementation outcomes serve as essential prerequisites for achieving desired improvements in client or service outcomes. This entails the distinction between implementation outcomes (e.g. acceptability, adoption, feasibility, and fidelity), systems outcomes (e.g. efficiency, safety, effectiveness, and equity), and client outcomes (e.g. health and social outcomes and satisfaction), which has gained increasing attention since Proctor et al. (2011) first introduced their taxonomy on implementation outcomes in 2011 (Proctor et al., 2023).

Despite these advances, critiques of implementation science persist. For instance, describing work as “implementation science”—whether intentionally or not—can imply the existence of a coherent science, a notion that Fixsen et al. (2024) critically question. While implementation science provides a strong foundation for studying “what works,” that is, the effectiveness of implementation strategies (Albers et al., 2020), it has traditionally placed less emphasis on explanatory and mechanistic investigations of implementation strategies (Lewis et al., 2024). A 2020 systematic review highlights that, although implementation studies have examined implementation mechanisms, the applied terminology and methodologies remain inconsistent and require further development (Lewis et al., 2020). In a research agenda co-authored by leading scholars within the field of implementation science, a main conclusion is to expand the research focus “from what works to how and why certain strategies work, for whom, when, and in which contexts” (Lewis et al., 2024). This explanatory focus aligns closely with the aims of realist evaluation (RE), a theory-driven methodology, aiming to unpack black boxes by elucidating and theorizing about the interplay between contextual factors, mechanisms, and outcomes (Pawson, 2024). In other words, RE pursuits answering how, for whom, and under what circumstances programs and policies generate change. Scholars within implementation research have recognized RE as a relevant approach to further develop implementation research methodology and to foster a context sensitive and mechanistic thinking (Brownson et al., 2022; Harvey et al., 2023; Kim et al., 2023).

We argue that evaluation experts stand to benefit from engaging with concepts and methodologies derived from both the RE approach and implementation science. This enables a more granular understanding of implementation processes and mechanisms while maintaining a focus on context-specific explanatory insights. In addition, it allows evaluators to disentangle the dynamic interplay between the interventions being implemented and the strategies designed to facilitate their uptake. Specifically, distinguishing analytically between implementation strategies and implementation objects as well as distinguishing between implementation mechanisms and intervention mechanisms, and consequently implementation outcomes and client outcomes, provides a robust framework for understanding complex change processes. These distinctions foster greater clarity and insight into the mechanisms through which outcomes are generated, thereby improving the practical relevance and theoretical rigor of evaluation findings.

In this article, we propose three arguments elaborating and substantiating the above propositions. First, we argue that RE can facilitate the development of nuanced understandings of the intricate interplay among contextual factors, mechanisms of implementation strategies, and implementation outcomes. Essentially, this means opening the “black box” of implementation strategies. Second, we propose that by viewing the underlying workings of implementation strategies as ripple effects¹, we gain a nuanced comprehension of the links between implementation strategies and the implementation object. Finally, we argue that investigating implementation strategies from a RE perspective strengthens our ability to capture and investigate unintended and unexpected outcomes of such strategies. This is crucial, as unforeseen effects may shape the generation of both implementation and client outcomes. To support and illustrate our arguments, we provide theoretical and methodological reflections and present empirical examples based on three RE studies that were led by the authors of this article. The details and results of these three studies are described elsewhere (Christoffersen et al., 2023; Hinrichsen et al., 2022; Kirk et al., 2024). With this article, we wish to foster interdisciplinary dialogue and contribute to the discussions on how to strengthen the understanding of and research on the inner workings of implementation strategies and the interactions between implementation strategies and the implementation object.

Background—Realist Evaluation

With its philosophical roots in critical realism, RE has challenged the common understanding of causation and methods for evaluating and studying interventions within disciplines such as health, policy, and educational sciences (Mukumbang et al., 2023; Pawson, 2013). In that respect, an essential attribute of RE is that it is based on an understanding of causation as generative. The assumption of generative causation calls for an analytical focus on how program mechanisms and contextual factors interact to produce outcomes or generate change (Pawson, 2024). The programs referred to are interventions, practices, or methods intended to generate change (Pawson, 2024), which we argue also includes implementation strategies. From a RE perspective, programs are based on a program theory, that is, theoretical assumptions about how and why they are expected to work (Pawson and Tilley, 2004). These can be either explicit or implicit. Similarly, implementation strategies build on assumptions about how they generate change, that is, program theories. Therefore, we argue that RE is a suitable approach to develop and refine theory for implementation strategies by shedding light on the inner workings of such implementation strategies.

Integrating RE with the concepts from implementation research prompts a rethinking of the definitions of the implementation mechanism and implementation outcome found within implementation research (Brownson et al., 2022; Lewis et al., 2020). While several definitions of implementation mechanisms are applied within implementation research, a prevalent definition is: “Processes or events through which an implementation strategy operates to affect desired implementation outcomes” (Brownson et al., 2022; Lewis et al., 2020). This definition reflects an epistemological stance common in implementation science, often based on quantitative methods. RE, however, encourages multimethod studies and an analytical openness toward undesired and unexpected outcomes. This highlights that, within RE, outcomes are understood more broadly, that is, as context-dependent changes that may include unintended or emergent effects (Pawson, 2024), rather than being limited to a pre-specified set of outcomes such as those outlined in Proctor et al.’s taxonomy. We will further elaborate on the definitions of mechanisms and outcomes below.

Three arguments for applying a RE approach to investigate implementation strategies

In the following sections, we elaborate and exemplify how the RE logic of inquiry can be applied to the study of implementation strategies. This provides us with a foundation for linking implementation strategies and objects as well as understanding and investigating unintended and unexpected outcomes of implementation strategies.

Opening the black box of implementation strategies

First, in this section, we argue that RE can facilitate unearthing of the underlying workings of implementation strategies, that is, how specific resources introduced as a part of implementation strategies in conjunction with specific contextual factors generate implementation outcomes. This entails our explication of how concepts from RE, including ripple effects (Jagosh et al., 2015), can be applied to investigate implementation strategies. Table 1 presents an overview of central concepts and their operationalizations as we propose they can be used for specifically RE studies of implementation strategies.

Table 1.

Concepts and their operationalization for studying implementation strategies.

Concept	Operationalization
Concepts adapted from implementation research
Implementation strategy	“Methods or techniques used to enhance the adoption, implementation, and sustainability of research informed programs” (Albers et al., 2020).
Implementation object	The thing being implemented such as “research, guidelines, interventions, innovations and evidence (i.e. research-based knowledge in a broad sense)” (Nilsen, 2020).
Concepts adapted from realist evaluation literature
Context	Features of the conditions in which implementation strategies are introduced that are relevant to the operation of the implementation mechanisms (adapted from Pawson and Tilley, 1997). These conditions may change over time and changes may be induced by implementation strategies (adapted from Jagosh et al., 2015).
Implementation mechanism	Implementation mechanisms describe what it is about implementation strategies that generate outcomes and are composed of (1) the resources (knowledge, materials, opportunities, etc.) provided or introduced by an implementation strategy and (2) how individuals interpret and respond to these resources (adapted from Astbury and Leeuw, 2010).
Outcome	Intended and unintended consequences of implementation strategies resulting from the activation of implementation mechanisms (adapted from (Pawson and Tilley, 1997).
CMO-configuration	Causal explanations about how outcomes (O) are interrelated with combinations of mechanisms (M) and contextual factors (C), often represented as C + M = O (Jagosh et al., 2015).
Ripple effect	CMO-configurations configured in a series in which the outcome of one CMO-configuration constitutes the context of another CMO-configuration (Jagosh et al., 2015).

CMO-configurations and implementation strategies

The central argument is that we can develop CMO-configurations and program theories for implementation strategies, the same way it is done for, for example, MHP and preventive programs (Funnell and Rogers, 2011), through the application of the RE approach. These program theories articulate the underlying assumptions about how the implementation strategy is expected to generate specific implementation outcomes under given circumstances. By systematically developing and testing program theories for implementation strategies, researchers can refine their understanding of the intricate interplay between contextual factors, implementation mechanisms, and outcomes of an implementation strategy. The cumulative construction of program theories for various implementation strategies sets the stage for developing middle-range theories (Pawson, 2013) for implementation strategies, as has been called for in the implementation science literature (Kislov et al., 2019; Meza et al., 2023).

When investigating implementation strategies, the central outcomes of interest are implementation outcomes as opposed to client outcomes such as improved health and satisfaction with a treatment. Proctor et al. (2011) define “implementation outcomes as the effects of deliberate and purposive actions to implement new treatments, practices and services.” Moreover, the authors present a heuristic taxonomy of implementation outcomes that we propose to provide direction for conceptualizing and identifying implementation outcomes. The taxonomy includes eight types of implementation outcomes: acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability (Proctor et al., 2011). Importantly, the RE approach encourages researchers to stay open toward the generation of unintended and unexpected outcomes, and in general to all consequences resulting from the activation of implementation mechanisms. Thus, one should not be limited by Proctor et al.’s (2011) taxonomy as the only outcomes of interest. Our argument is not intended as a critique of the work of Proctor et al. (2011); rather, we suggest that an openness to unexpected and unintended outcomes may contribute to expanding their suggested list of implementation outcomes, a development the authors themselves have advocated.

Applying the term mechanism from RE (Astbury and Leeuw, 2010) to the investigation of implementation strategies, we arrive at the following operationalization: implementation mechanisms describe what it is about implementation strategies that generate outcomes and are composed of (1) the resources (knowledge, materials, opportunities, etc.) provided or introduced by an implementation strategy and (2) how individuals interpret and respond to these resources. The individuals mentioned here can include both implementers and clients (i.e. the target group of the implementation object) and can be part of groups, teams, or organizations. Moreover, the term context describes features of the conditions in which implementation strategies are introduced that are relevant to the operation of the implementation mechanisms (adapted from Pawson and Tilley (1997)). In line with systems theory (Kim et al., 2023), these conditions may change over time, and changes may be induced by implementation strategies (Jagosh et al., 2015).

While studies employing RE methodology in implementation studies are limited, researchers support the argument for applying RE in implementation research (Dossou et al., 2021; Oladimeji and Fatusi, 2022). For example, in the context of policy implementation, Dossou et al. (2021) conclude that “realist evaluation made it possible to formulate plausible explanatory theories on what drives change in policy implementation outputs over time” (p. 14). In addition, Kirk et al. (2024) applied a RE approach to open the black box of an implementation strategy to explain why and how it operates. The authors explored the implementation mechanisms of “oilcloth sessions” used as an implementation strategy in the implementation of a new emergency department in a Danish hospital. While exploring participants’ reasoning in response to the resources provided by the strategy, the authors elicited CMO-configurations such as: being heard, listened to, and involved (M) by the hospitals’ executive board in the oilcloth sessions (C) increased the acceptability of the implementation process among the department managers (O), thereby highlighting how mechanisms generate implementation outcomes. Another CMO-configuration (see Table 2) showed that the time spent at the oilcloth sessions was an important contextual factor. The time away from busy routines (C) created an opportunity for the participants to engage in reflexive thinking without being disturbed (M), which influenced managers and key employees to engage in the implementation of the new emergency department (O). The finding that the implementation strategy created time for reflexive and creative thinking provides important insights into the participants’ motivation to engage in the implementation process, contributing to the proposed program theory of oilcloth sessions. Exploring contextual factors that activate the implementation mechanisms of the oilcloth sessions thus fostered important knowledge about how the strategy was understood, interpreted, and thus acted upon. Opening the ‘black box’ of oilcloth sessions with the use of CMO-configurations provided nuances to the initial assumptions about how oilcloth sessions work, which is conducive to future successful operation of the implementation strategy.

Table 2.

Illustration of a CMO-configuration from (Kirk et al., 2024).

Context	Mechanism (resource and reasoning)	Outcome
Time free from disturbances	Reflexive and creative thinking (resource) → Managers and key employees motivated to engage in the implementation of the new emergency department (reasoning)	Increase in new and better ideas and increased acceptability and quality of the implementation process

Another example where the RE approach proved suitable and valuable for studying implementation strategies is a study led by the first author that explored the CMO-configurations of an implementation strategy (Hinrichsen et al., 2022). The study explored the underlying workings of an action learning program (ALP), applied as an implementation strategy, aimed at improving existing and implementing new MHP practices at an organizational level. The study drew on qualitative methods to uncover and describe the nuances of the CMO-configurations at play. Intervention research studies in the field of public health usually investigate the health-related outcomes of such programs (i.e. client outcomes). In the Hinrichsen et al. study, however, the analytical focus was on the generation of organizational capacity and how the ALP facilitated the adoption and adaption of MHP practices, thus linking the implementation strategy with the MHP practices (Hinrichsen et al., 2022). The study produced nuanced CMO-configurations depicting how conducting an ALP resulted in changes in organizational practices, individual learning, and strengthened inter- and intraorganizational collaboration. For example, a central mechanism identified in the ALP was the iterative reflective processes triggered by e.g. facilitated discussions and action plans which participants continuously developed throughout the course of the intervention. In a configured account, the iterative reflections on MHP and existing practices (M) generated an increase in participants’ awareness of and knowledge about MHP (O). A central contextual factor was the motivation of participants to engage in MHP efforts (C).

Ripple effects within implemenation strategies

Inspired by Jagosh et al. (2015), we propose that viewing the underlying workings of implementation strategies as ripple effects (i.e. in a sequential manner) allows for a more nuanced comprehension of the interaction between various implementation mechanisms—providing more depth to the analysis of the underlying workings of implementation strategies. Implementation strategies, regardless of whether they are single-component or multifaceted strategies (Proctor et al., 2013), often combine several events and/or actions that together support the implementation of the implementation object (Powell et al., 2012). By examining multiple CMO-configurations embedded in a specific implementation strategy, we gain insights into the intricate relationships and potential dependencies that shape an implementation process. This level of analysis is crucial for unraveling the complexity inherent in the interactions between various implementation mechanisms within a given implementation strategy and how it potentially creates change across multiple levels (e.g. individual, organizational, and system-level).

Jagosh et al. (2015) suggest using the concept of the ripple effect to conceptualize the interconnectedness or embeddedness of CMO-configurations. They propose that one CMO-configuration can be embedded in another or configured in a series of CMO-configurations. According to Jagosh et al. (2015), the ripple effect describes a causal chain, providing a comprehensive understanding of how programs unfold under specific circumstances. Applying this to the study of implementation strategies, for example, the outcome of one CMO-configuration can be a relevant contextual factor for the next CMO-configuration in a series of steps inherent in an implementation strategy.

Returning to the Hinrichsen et al. study as an example, we demonstrate how ripple effects can elucidate connections between several implementation mechanisms within an implementation strategy. The ALP introduced several resources, such as action plans and social interactions among participants from across organizations, intended to trigger several implementation mechanisms (Hinrichsen et al., 2022). Figure 1 shows an interpretation of the study results as a “chain” of CMO-configurations, illustrating the inner workings of the implementation strategy. Here, we focus on how the results shed light on the implementation processes that precede the generation of MHP effects. The first CMO-configuration describes the generation of participants’ change in motivation to engage in MHP practices (O₁) through the use of appealing facilitation styles and peer discussions (M₁). The increased motivation to work with MHP (C₂) allowed for iterative reflection processes on MHP and existing practices (M₂), which in turn generated an increase in awareness and knowledge about MHP among participants (O₂). Finally, in the context of increased awareness and knowledge about MHP (C₃), time and effort were allocated to work with MHP (M₃), leading to MHP initiatives being developed and adopted (O₃), which included, for example, a peer-to-peer initiative aimed at promoting mental health.

Figure 1.

Using the ripple effect to illustrate the interconnection between multiple CMO-configurations of an implementation strategy.

Linking implementation strategies and objects

Expanding our argument, we assert that ripple effects can shed light on how implementation strategies link to the inner workings of an implementation object. This is done by illuminating how the CMO-configurations of an implementation strategy relate to the CMO-configurations of the implementation object, for example, how staff training relates to activities in an MHP intervention. From a RE perspective, a contextual factor can enable, trigger, support as well as hinder, block, or act as a barrier to an intervention (Greenhalgh and Manzano, 2022), that is, the implementation object. Building on this logic, the outcomes of an implementation strategy can constitute critical contextual factors for intervention activities or introduce resources that are part of the intervention mechanisms, and hereby influencing change generated through an intervention. For example, conducting training sessions to increase knowledge about mental health among staff (implementation strategy) can be understood as altering contextual factors, allowing the mechanisms of an MHP intervention (implementation object) to work effectively. This analytical approach fosters a comprehensive understanding of how the implementation strategy sets the stage for subsequent CMO-configurations of the implementation object. Our proposed approach is equally relevant for both discrete and multifaceted implementation strategies. Even discrete strategies, such as training sessions, embody a degree of complexity that necessitates nuanced examination through RE, revealing how mechanisms interact with contextual factors to produce outcomes. Furthermore, applying this perspective enables researchers or evaluators to understand and “manage” contextual aspects as dynamic and changing over time, which we consider a strength.

Returning to the Hinrichsen et al. study, we now exemplify this argument by illustrating how the CMO-configurations link to the working of the implemented MHP initiative. The process depicted in Figure 1 is from a specific case investigated in the study where the participants from an educational institution developed and implemented a peer-to-peer initiative (the MHP initiative). O₃ represents the development and adoption of the peer-to-peer initiative. We argue that the outcomes of increased employee motivation to engage in MHP practices (O₁) and employee awareness and knowledge about MHP (O₂) are very likely to play crucial roles in how the peer-to-peer initiative unfolds and works in practice. These outcomes can be seen as capacity-building outcomes that constitute central elements of the CMO-configurations of the peer-to-peer initiative (the implementation object). This is illustrated in Figure 2 (the figure is based on the results presented by Hinrichsen et al. (2022)). The links between the CMO-configurations of the ALP and the succeeding peer-to-peer initiative are depicted through the dotted arrows, which indicate the links between the changes in motivation and knowledge generated by the implementation strategy and intervention processes. The workings of the peer-to-peer initiative can also be described as a series or a set of CMO-configurations; however, this was not within the scope of the Hinrichsen et al. study and, therefore, is not detailed in Figure 2.

Figure 2.

Using the ripple effect to illustrate the interconnectedness of CMO-configurations of the implementation strategy and the MHP initiative.

Employing a RE approach for investigating both implementation strategies and implementation objects provides a common theoretical foundation and logic of inquiry in evaluation and research efforts. This approach not only ensures conceptual coherence but also enhances the compatibility and interpretability of results. Moreover, this alignment strengthens applicability of implementation research, facilitating a more effective translation of findings into practice. The goal is not to construct a single, comprehensive theory but to clarify the relationship between two theoretical domains: the implementation strategy (e.g. an ALP) and the implementation object (e.g. a peer-to-peer initiative). As Pawson (2024) argues, fragmented and incremental understanding is central to enlightenment, making it essential to clearly define the scope of research or evaluation as a foundational step in any inquiry. We believe that this closely aligns with the aim of implementation research in bridging the gap between research findings and real-world applications (Albers et al., 2020; Na, 2024).

Unintended and unexpected outcomes

By now, we have made the argument that RE and applying the concept of the ripple effect can help us understand the intricate details of how implementation strategies work to generate implementation outcomes and link to the inner workings of implementation objects. Finally, in this section, we assert that investigating implementation strategies from a RE perspective strengthens our ability to capture and explore not only intended but also unintended and unexpected outcomes of these strategies. These might in turn affect the effectiveness of the implementation object.

Designing activities of change in complex systems, such as implementation strategies, can be characterized as “informed guesswork” that is underpinned with theory, results from empirical research, and practice knowledge (Pawson, 2013). When these strategies meet reality, unintended and unexpected influences on involved subjects’ reasoning and behavior (corresponding to a mechanism in RE, cf. Table 1) may occur, which, in turn, can lead to unintended and unexpected outcomes (Kim et al., 2023; Pullmann et al., 2022). These can be both desirable and undesirable. Attention to unintended outcomes of implementation strategies may broaden our understanding of how, for whom, and under what circumstances different implementation strategies work, which in turn can help researchers and practitioners anticipate or avoid unwanted consequences of executing implementation strategies (Bonell et al., 2015; Kim et al., 2023; Pullmann et al., 2022).

The methodological and conceptual “choices” made when designing and conducting implementation studies greatly affect the ability to investigate unintended and unexpected outcomes of implementation strategies. Within the RE approach, outcomes are defined broadly, encompassing not only the intended but also the unintended and unexpected consequences of programs (Pawson, 2013). The emphasis on unintended outcomes is rooted in the recognition that social programs, including implementation strategies, operate within complex and dynamic contexts, where unforeseen interactions and consequences will probably arise (Pawson, 2013). With this theoretical underpinning in mind, evaluators and researchers should be open to and explore the full spectrum of outcomes.

Turning to the implementation science literature, the definition of implementation mechanism suggested by Lewis et al. (2020) (i.e. “a process or event through which an implementation strategy operates to affect desired implementation outcomes”) omits the unintended and unexpected outcomes of implementation strategies. This might lead investigators to overlook, or minimize their focus on, the generation of unintended and unexpected outcomes. The same issue applies to studies that focus narrowly on quantifiable implementation mechanisms and intended outcomes. These approaches are often underpinned by epistemological assumptions that favor generalizability (Albers et al., 2020) and the application of predefined taxonomies and variables, entailing the risk of underestimating the dynamic and adaptive nature of real-world practice. They might fail to capture how actors interpret, negotiate, and adapt interventions in context, thereby missing emergent phenomena and unintended consequences (Pawson, 2013). Context, in such models, tends to be treated as a confounder rather than as a generative part of the causal mechanism, as conceptualized in RE. We fully recognize the strengths of developing and applying quantifiable measures for investigating implementation mechanisms (Kim et al., 2023; Lewis et al., 2022; Ravn, 2019). However, it is equally important to consider the limitations of investigating the implementation mechanisms and causal processes solely through quantitative methodologies and predefined quantitative measures. Integrative strategies that combine structured tools from implementation science (e.g. ERIC; Powell et al., 2015) with theory-informed, context-sensitive methods from RE may offer a more nuanced understanding of how implementation efforts unfold—and why they succeed or fail—in specific settings.

To illustrate our argument, we turn to the realist synthesis of Christoffersen et al. (2023). The study aimed to explore why, how, and for whom preventive health checks work toward adverse health behavior in high-risk groups. Regarding implementation outcomes, the synthesis showed that collaboration between those who delivered preventive activities generated an unintended outcome in the form of capacity building Christoffersen et al. (2023). In particular, professional relations were strengthened, which might influence future collaboration about preventive services in a local context, for example, the working relationship between a municipality and general practice. This means that implementing preventive health checks appears to increase local health organizations’ capacity to draw on each other’s resources and competences—an unexpected positive result of health check interventions—which, in turn, might facilitate future implementation processes.

The RE approach to Oilcloth sessions additionally revealed unexpected findings (Kirk et al., 2024). The authors found that the facilitator’s use of militaristic metaphors led several participants to perceive the sessions as battlefields where struggles for involvement and influence took place. This, in turn, fostered mistrust among participants regarding each other’s intentions and professionalism. The managers’ mistrust, identified as a factor contributing to reduced fidelity in the implementation process, may serve as a crucial contextual element influencing how employees’ reason in future implementation efforts. Thus, this finding provides valuable insights into the delivery of Oilcloth sessions in similar contexts (specifically, the avoidance of war metaphors) and enhances the generalizability of the implementation strategy.

By acknowledging and exploring a broader spectrum of outcomes, implementation (and intervention) researchers and evaluators can provide a more holistic and multilevel assessment of the impact of implementation strategies in diverse contexts. Thus, the RE approach provides an opportunity to explore how implementation strategies lead to a compilation of unintended consequences, which permits to learn from negative studies and improve implementation strategies (Kirk et al., 2024).

Discussion

In this article, we present three arguments and examples that highlight the potential benefits of applying the RE approach to investigate not only interventions themselves, that is, the implementation objects, but also implementation strategies. First, we argue that RE can help understand the intricate details of how implementation strategies work to generate implementation outcomes. Second, we argue that viewing the underlying workings of implementation strategies as ripple effects helps us gain a nuanced comprehension of the interactions between implementation strategies and the implementation object. Third, we assert that investigating implementation strategies from a RE perspective strengthens our ability to capture and explore the intended as well as unintended and unexpected outcomes of these strategies.

Applying RE to investigate implementation strategies, aligns with the call for methodological approaches to generate knowledge on implementation mechanism (Lewis et al., 2024) to establish a thorough understanding of why, how, and under what circumstances implementation strategies work (Lewis et al., 2021). Implementation scholars suggest investigating the mechanisms of change in implementation strategies to broaden our understanding of why these strategies succeed or fail (Geng et al., 2022; Kislov et al., 2019; Lewis et al., 2020; Lewis et al., 2018). This entails generating knowledge on how implementation mechanisms work in different and dynamic contexts (Kim et al., 2023). In other words, the study of implementation mechanisms moves evaluation of implementation strategies from what it is and does it work to what it does (Kislov et al., 2019). These insights, that can be gained through RE, are crucial for understanding which implementation barriers and facilitators the implementation strategy addresses. Furthermore, a thorough understanding of implementation mechanisms helps researchers, policy makers, and practitioners make informed decisions on when, how, and in what contexts to use certain implementation strategies and, importantly, when not to use them (Geng et al., 2022; Lewis et al., 2022).

The cumulative construction of program theories for various implementation strategies sets the stage for the development of middle-range theories (Pawson and Tilley, 1997). These theories offer a level of abstraction that transcends specific implementation strategies and contexts and provide a more generalizable understanding of how implementation strategies operate in diverse settings. While middle-range theories aim to transcend specific settings, the CMO-configurations that underpin them are often context-dependent, reflecting the unique interplay of factors within a given setting. These configurations, however, offer critical insights into how contextual factors, mechanisms, and outcomes interact, providing a foundation for adapting implementation strategies to different contexts. For example, going back to the study of oil cloth sessions, the use of war metaphors should be considered in relation to the implementation context. By bridging context-specific findings with broader theoretical principles, RE supports both the refinement of program theories and their practical application across diverse settings. The realist review and realist synthesis (Pawson, 2013) are study designs that facilitate this process by systematically evaluating and synthesizing evidence.

While the aim of this article is not to further develop the terminology of implementation outcomes, we believe that the RE approach to investigating implementation strategies additionally holds significant potential in this area. Proctor et al. define implementation outcomes as “the effects of deliberate and purposive actions to implement new treatments, practices, and services” (Proctor et al., 2011) and present a heuristic taxonomy of implementation outcomes. We propose that future research can combine RE with existing implementation science frameworks, models, or theories (Nilsen, 2015), aiming to provide detailed insights into how these outcomes are generated and explore the possibility of expanding Proctor et al.’s list of implementation outcomes.

Our argument for operationalizing the inner workings of an implementation strategy as ripple effects, that is, a sequence of processes operating at various levels, has similarities to the approaches proposed by Williams (2016). They argue that the study of implementation strategies (particularly evidence-based treatment implementation in mental health) “can be most effectively advanced through the integration of theories and research on implementation antecedents and the testing of these antecedents as multilevel mechanisms” (Williams, 2016: 15). Attention to series of events in implementation research and evaluation, for example, by linking CMO-configurations as ripple effects, can help model the complexity of implementation strategies. This approach can guide the tailoring of implementation strategies to specific circumstances (e.g. health organizations or systems) and provide valuable information for adapting implementation strategies—knowledge which is asked for within implementation research (Lewis et al., 2021). We are aware that a sequential conceptualization of CMO-configurations does not capture potential feedback loops or circular processes (Kim et al., 2023). However, the linear depiction of the links between the CMO-configurations in Figure 1 should be seen as a simplification aiming to provide an overview of the relationships between processes, not an exact processual map.

Scholars highlight that a stronger link and mutual understanding between implementation and intervention research will benefit both fields (Beidas et al., 2022; Brownson et al., 2022). Partnering the two fields allows for knowledge exchange and both fields to grow synergistically (Brownson et al., 2022). Moreover, Beidas et al. (2022) highlight the need for more evidence on the relationship between implementation outcomes and health outcomes and prioritizing both types of outcomes. We suggest applying the ripple effect to understand and investigate the interconnectedness of CMO-configurations, as this allows us to distinguish between implementation mechanisms and intervention mechanisms (and thus between the generation of implementation outcomes and systems/client outcomes). If implementation mechanisms and intervention mechanisms are not separated, we risk conflating them and, for example, not being able to decide whether the lack of anticipated client outcomes can be ascribed to failure of the implementation strategy or the implementation object. Therefore, we need to be conscious of and carefully consider both implementation mechanisms and intervention mechanisms when planning, implementing, and evaluating clinical and public health interventions and implementation strategies.

Limitations

Some limitations deserve mentioning. First, we underpin our arguments with examples from mainly three studies. Thus, our arguments should be tested and developed through further research, for example, in different sectors and including researchers from various disciplines. However, the studies that we draw on vary widely in their study designs (realist syntheses, mixed qualitative methods, and ethnographic methods), fields of interest (the hospital sector, educational sector, and primary/GP sector), and implementation objects (MHP initiatives, health checks, and a new emergency department), resulting in a diverse spectrum of research.

Second, in this article, we briefly address some aspects of methodological considerations (i.e. strengths/weaknesses of qualitative and quantitative approaches), but we do not delve into the study designs that may be applicable for RE studies of implementation strategies. Our aim was not to write a methodology paper, but to highlight that the same methodological approach used in program evaluation should apply for evaluation of implementation strategies as well. For more details on RE study designs, see publications on RE lead by Ray Pawson (Pawson and Tilley, 1997; Pawson and Tilley, 2004; Pawson, 2013) and the RAMSES guidelines. Future research should explore and test the feasibility of (1) applying RE study designs in implementation research and (2) combining a RE approach with theories, frameworks, and models from implementation research. However, the emerging implementation research drawing on a RE approach seems promising (Dossou et al., 2021; Kirk et al., 2024; Oladimeji and Fatusi, 2022).

Third, we want to direct attention to some shortcomings of RE studies that are often highlighted in the RE literature. Among others, it is usually resource- and labor-intensive to conduct a RE study, RE is recommended for experienced researchers because there are no formula books or methodological prescriptions, and assessing and entangling contextual factors, mechanisms, and outcomes can be an intellectually challenging task (Nielsen et al., 2022; Pawson and Tilley, 2004; Salter and Kothari, 2014).

Conclusion

By combining RE with implementation science concepts, evaluation experts gain a comprehensive toolkit to assess and explore the inner workings of both implementation strategies and interventions themselves. This includes the assessment of diverse outcomes, including implementation, health-related, and unintended outcomes. While RE is becoming established in intervention research, it represents a novel methodological avenue for studying implementation strategies. In this article, we outline how the RE approach helps researchers and evaluators answer the questions of “how, for whom, and under what circumstances implementation strategies work” rather than (merely) “which implementation strategies are most effective.” In addition, we argue that applying the same theoretical lens to explore implementation strategies and implementation objects provides a compatible knowledge base, enabling us to theorize and investigate the links between them. By iteratively refining CMO-configurations, developing program theories, and constructing middle-range theories, researchers and evaluators advance the field of implementation and foster a deeper comprehension of the mechanisms that drive successful implementation strategies in various contexts. Such knowledge can facilitate the selection of implementation strategies that cater to specific contextual circumstances, leading to more successful interventions, such as MHP and preventive efforts. Although RE is a relatively new approach in implementation research and evaluation, its emerging application suggests a promising avenue for studying implementation strategies. Future studies should explore the methodological applications of RE and potential synergies with existing implementation frameworks.

Footnotes

Acknowledgements

The authors sincerely thank the Mental Health Research Group at the National Institute of Public Health, University of Southern Denmark, for their constructive feedback on the work presented in this paper.

Author contributions

Concept and Design: CH and MBJ. Initial drafting: CH and MBJ. Critically revising the manuscript for important intellectual content: CH, JWK, and MBJ. All the authors read and approved the final manuscript.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by departmental resources/independently financed by the authors.

Ethical considerations

Not applicable. This work did not involve any humans as research participants.

Consent to participate

Not applicable. This paper is based on the results of other publications. Aspects related to consent for publication are presented in these publications.

Consent for publication

Not applicable.

ORCID iDs

Carsten Hinrichsen

Marie Broholm-Holst

Data availability

Data sharing is not applicable to this article, as no datasets were generated or analyzed during the current study.

Notes

Carsten Hinrichsen holds a position as Assistant Professor at the University of Southern Denmark. His research interest lies within mental health promotion, implementation and intervention research, and realist evaluation.

Jeanette Wassar Kirk holds positions as Senior Research and Associate Professor in implementation research.

Marie Broholm-Holst holds a position as Associate Professor at the University of Southern Denmark. Her research focuses on implementation in general practice and evaluation methodology including realist evaluation.

References

Albers

Shlonsky

Mildon

(2020) En Route to Implementation Science 3.0. In: Albers

Shlonsky

Mildon

(eds) Implementation Science 3.0. Cham: Springer, 1–38.

Andersen

Tolstrup

Rod

, et al. (2015) Shaping the social: Design of a settings-based intervention study to improve well-being and reduce smoking and dropout in Danish vocational schools. BMC Public Health 15(1): 568.

Astbury

Leeuw

(2010) Unpacking black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation 31(3): 363–81.

Beidas

Dorsey

Lewis

, et al. (2022) Promises and pitfalls in implementation science from the perspective of US-based researchers: Learning from a pre-mortem. Implementation Science 17(1): 55.

Bonell

Jamal

Melendez-Torres

, et al. (2015) “Dark logic”: Theorising the harmful consequences of public health interventions. Journal of Epidemiology and Community Health 69(1): 95.

Bonnesen

Toftager

Madsen

, et al. (2020) Study protocol of the healthy high school study: A school-based intervention to improve well-being among high school students in Denmark. BMC Public Health 20(1): 95.

Brownson

Shelton

Geng

, et al. (2022) Revisiting concepts of evidence in implementation science. Implementation Science 17(1): 26.

Christoffersen

Nilou

Thilsing

, et al. (2023) Exploring targeted preventive health check interventions: A realist synthesis. BMC Public Health 23(1): 1928. https://doi.org/10.1186/s12889-023-16861-8

Dossou

J-P

Van Belle

Marchal

(2021) Applying the realist evaluation approach to the complex process of policy implementation—The case of the user fee exemption policy for cesarean section in Benin. Frontiers in Public Health 9: 553980.

10.

Fixsen

Van Dyke

Blase

(2024) Is implementation science a science? Not yet. Frontiers in Public Health Volume 12: 512.

11.

Funnell

Rogers

(2011) Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. San Francisco, CA: Jossey-Bass.

12.

Geng

Baumann

Powell

(2022) Mechanism mapping to advance research on implementation strategies. PLOS Medicine 19(2): e1003918.

13.

Greenhalgh

Manzano

(2022) Understanding “context” in realist evaluation and synthesis. International Journal of Social Research Methodology 25(5): 583–95.

14.

Harvey

Rycroft-Malone

Seers

, et al. (2023) Connecting the science and practice of implementation—Applying the lens of context to inform study design in implementation research. Frontiers in Health Services 3: 1162762.

15.

Hinrichsen

Nelausen

Nielsen

, et al. (2022) The workings of an action learning program for building mental health promotion capacity: A realist evaluation. Evaluation and Program Planning 102080. https://doi.org/https://doi.org/10.1016/j.evalprogplan.2022.102080

16.

Jagosh

Bush

Salsberg

, et al. (2015) A realist evaluation of community-based participatory research: Partnership synergy, trust building and related ripple effects. BMC Public Health 15: 725.

17.

Kim

Cruden

Crable

, et al. (2023) A structured approach to applying systems analysis methods for examining implementation mechanisms. Implementation Science Communications 4(1): 127.

18.

Kirk

Stefansdottir

Andersen

, et al. (2024) How do oilcloth sessions work? A realist evaluation approach to exploring ripple effects in an implementation strategy. Journal of Health Organization and Management 38(9): 195–215.

19.

Kislov

Pope

Martin

, et al. (2019) Harnessing the power of theorising in implementation science. Implementation Science 14(1): 103.

20.

Lewis

Boyd

Walsh-Bailey

, et al. (2020) A systematic review of empirical studies examining mechanisms of implementation in health. Implementation Science 15(1): 21.

21.

Lewis

Frank

Cruden

, et al. (2024) A research agenda to advance the study of implementation mechanisms. Implementation Science Communications 5(1): 98.

22.

Lewis

Klasnja

Lyon

, et al. (2022) The mechanics of implementation strategies and measures: Advancing the study of implementation mechanisms. Implement Sci Commun 3(1): 114.

23.

Lewis

Klasnja

Powell

, et al. (2018) From classification to causality: Advancing understanding of mechanisms of change in implementation science. Frontiers in Public Health 6: 00136.

24.

Lewis

Powell

Brewer

, et al. (2021) Advancing mechanisms of implementation to accelerate sustainable evidence-based practice integration: Protocol for generating a research agenda. BMJ Open 11(10): e053474.

25.

Meza

Moreland

Pullmann

, et al. (2023) Theorizing is for everybody: Advancing the process of theorizing in implementation science. Frontiers in Health Services 3: 1134931.

26.

Mukumbang

De Souza

Eastwood

(2023) The contributions of scientific realism and critical realism to realist evaluation. Journal of Critical Realism 22(3): 504–24.

27.

Na (2024) Implementation Science—Aims and Scope. Available at: https://implementationscience.biomedcentral.com/about (accessed 12/12)

28.

Nielsen

Lemire

Tangsig

(2022) Unpacking context in realist evaluations: Findings from a comprehensive review. Evaluation 28(1): 91–112.

29.

Nilsen

(2015) Making sense of implementation theories, models and frameworks. Implementation Science 10: 53.

30.

Nilsen

(2020) Making sense of implementation theories, models, and frameworks. In: Albers

Shlonsky

Mildon

(eds) Implementation Science 3.0. Cham: Springer International Publishing, 53–79.

31.

Oladimeji

Fatusi

(2022) Realist Evaluation of the “Abiye” Safe Motherhood Initiative in Nigeria: Unveiling the Black-Box of Program Implementation and Health System Strengthening. Frontiers in Health Services 2: 779130.

32.

Pawson

(2013) The science of evaluation: A realist manifesto (ed. Metzler

). London: Sage.

33.

Pawson

(2024) How to Think Like a Realist: A Methodology for Social Science. Cheltenham: Edward Elgar.

34.

Pawson

Tilley

(1997) Realistic Evaluation. Thousand Oaks, CA: Sage.

35.

Pawson

Tilley

(2004) Realist Evaluation. London: Sage.

36.

Pisinger

VSC

Hoffmann

Rosing

, et al. (2020) Study protocol for a cluster randomised controlled trial testing the effectiveness of the “High schools High on life” intervention on reducing excessive drinking in Danish high schools. BMJ Open 10(8): e038857.

37.

Powell

McMillen

Proctor

, et al. (2012) A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev 69(2): 123–57.

38.

Powell

Waltz

Chinman

, et al. (2015) A refined compilation of implementation strategies: Results from the expert recommendations for implementing change (ERIC) project. Implementation Science 10(1): 21.

39.

Proctor

Silmere

Raghavan

, et al. (2011) Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research 38(2): 65–76.

40.

Proctor

Bunger

Lengnick-Hall

, et al. (2023) Ten years of implementation outcomes research: A scoping review. Implementation Science 18(1): 31.

41.

Proctor

Powell

McMillen

(2013) Implementation strategies: Recommendations for specifying and reporting. Implementation Science 8(1): 139.

42.

Pullmann

Dorsey

Duong

, et al. (2022) Expect the unexpected: A qualitative study of the ripple effects of children’s mental health services implementation efforts. Implementation Research and Practice. Epub ahead of print 28 August. DOI: 10.1177/26334895221120797.

43.

Ravn

(2019) Testing mechanisms in large-N realistic evaluations. Evaluation 25(2): 171–88.

44.

Salter

Kothari

(2014) Using realist evaluation to open the black box of knowledge translation: A state-of-the-art review. Implementation Science 9(1): 115.

45.

Williams

(2016) Multilevel mechanisms of implementation strategies in mental health: Integrating theory, research, and practice. Administration and Policy in Mental Health 43(5): 783–98.

A realist lens on implementation strategies: An opportunity for linking implementation and intervention processes and exploring unintended outcomes

Abstract

Keywords

Introduction

Background—Realist Evaluation

Three arguments for applying a RE approach to investigate implementation strategies

Opening the black box of implementation strategies

CMO-configurations and implementation strategies

Ripple effects within implemenation strategies

Linking implementation strategies and objects

Unintended and unexpected outcomes

Discussion

Limitations

Conclusion

Footnotes

Acknowledgements

Author contributions

Declaration of conflicting interests

Funding

Ethical considerations

Consent to participate

Consent for publication

ORCID iDs

Data availability

Notes

References