Leveraging Systemic Contributors and Adaptations to Sharpen Interventions: The IMPActS Workshop

Abstract

A common pattern among intervention implementations is that, in many cases, initial adoption and early signals of success are followed by dwindling participation, signals that the intervention is not achieving its goals, and abandonment of the intervention as it was originally conceived. The IMPActS Framework facilitates anticipation of misalignment among stakeholders during the design, pitch, implementation, and sustainability of a candidate intervention’s implementation. We adapted the IMPActS Framework to design an IMPActS Workshop to occur within the context of an ongoing research with a partner organization to develop their proactive performance monitoring and sustained adaptability management program. The IMPActS Workshop prospectively tests the implementation sustainability of candidate interventions and sharpened intervention design to improve the likelihood of sustainment across the organization. We report the motivation, methods, results, and lessons learned from this effort.

Introduction

Organizations seek to make improvements in areas such as safety, efficiency, and cost savings by introducing changes and innovations to their existing systems. However, initial dissemination and adoption of new ideas and processes into a system does not guarantee implementation locally or across the system (Nilsen & Bernhardsson, 2019; Squires et al., 2023; Tabak et al., 2012). A general pattern emerges among unsuccessful implementations: early signals of success are followed by reduced interest, dwindling participation, and signals that the intervention is not achieving its goals, leading to the abandonment of the intervention as it was originally conceived. A variety of factors contribute to the reimagination or abandonment of interventions, and the burgeoning field of implementation science seeks to reveal contributors and offer guidance to improve the sustained implementation of interventions (Rapport et al., 2022).

Numerous frameworks have been created to evaluate the success and sustainment of interventions and implementation systems. These frameworks focus on behavioral contributors, characteristics of the intervention and target system, contributory organizational factors, realized value of the intervention, and characteristics of the implementation process (Nilsen, 2020). Success may be claimed when the intervention is implemented in the target environment and is determined to have achieved its goals (Rycroft-Malone, 2004). Sustainment of the intervention, monitored over time since its initial implementation, reflects continued operationalization in the target environment (Glasgow et al., 1999). Rabin et al. (2008) describe three facets of operationalization: maintenance, or cultivation of the system’s ability to continuously actualize initial outcomes; institutionalization, or the integration into local system culture; and capacity building, or the continued apportionment of system resources required by the intervention. Sustainment, therefore, is closely tied to the organization’s active mitigation of changing and competing system pressures.

Advances in the implementation science and resilience engineering domains have revealed several factors that are critical for intervention implementation success. Research into the dynamics of system pressures indicates that action to mitigate conflicting system goals, enacted by individuals in various organizational levels, has the potential to preserve or undermine long term viability of the organization (Walker, 2021; Woods & Alderson, 2021). These system pressures and local adaptations to resolve goal conflicts contribute to all work, irrespective of events or outcomes (Hollnagel et al., 2013). Initial adoptions of intervention implementations are subject to these same pressures. Implementation sustainability depends on these same local work system adaptations that mitigate goal conflicts inevitably introduced by the new intervention. According to Girth et al. (2022), goal alignment across all layers of the organization in promoting the intervention’s sustained implementation and procuring the system’s capacity to support that intervention is a cornerstone of success and sustainment. Organizational support is also a critical determinant of implementation success and is included in virtually all published implementation science frameworks (Nilsen & Bernhardsson, 2019). While such frameworks describe what contributes to or might influence organizational support, they do not describe what actions to take to garner organizational support. Indeed, Fitzgerald (2019) finds that nearly all descriptive implementation frameworks lack guidance for the practical application of their findings.

The IMPActS Framework bridges this gap (Fitzgerald, 2019). IMPActS is an acronym that stands for the Ideas, Mental Model Alignment, Pragmatics, Actors, and Sustainment efforts required for the durable success of a scientifically grounded intervention. IMPActS is a goal-directed guide to garnering organizational support through the preemptive mitigation of barriers to implementation sustainability during the intervention design phase. Specifically, IMPActS facilitates recognition of misalignment among stakeholders regarding the design and during the pitch, adoption, and implementation of a candidate intervention. Whereas implementation concerns are often considered separately from intervention design (Tabak et al., 2012), the IMPActS Framework facilitates the identification and integration of implementation system requirements within the intervention design itself. The framework leverages research on engineering resilient systems (Hollnagel, 2008; Woods, 2018) and systems thinking for design (Roesler et al., 2005; Sanders & Stappers, 2012) to centralize the interdependencies of the multiple design and implementation processes being managed within a larger system. IMPActS allows teams to move beyond descriptions of context and phenomena to direct the engineering focus of intervention design and target system implementation.

This paper reports our first attempt to operationalize the IMPActS Framework to proactively guide design-phase discussions of candidate interventions and generate novel insights about their anticipated implementations to sharpen the intervention and increase the likelihood of sustained implementation in the target system.

The IMPActS Framework

Ideas

The ideas underpinning an intervention must be scientifically founded. While frameworks including the Consolidated Framework for Implementation Research (CFIR; Damschroder et al., 2022) and Promoting Action on Research Implementation in Health Services (PARIHS; Rycroft-Malone, 2004) assert the requirement for high-quality evidence underpinning intervention designs, IMPActS explains that, although evidence-based support is stronger than other forms of support (such as anecdotal support), it is not as strong as mechanism-based support for explanations of the observed phenomenon. Importantly, stakeholder alignment on mechanisms including deterministic patterns, interactions, pressures, and structure relating to the intervention indicates that common ground (Klein et al., 2005) has been achieved among all parties regarding how the larger system responds to conflicts generated by changing pressures. When an intervention’s design accounts for the underlying mechanisms that shape system performance, it is more likely to succeed in resolving local conflicts to changing pressures, rather than contributing to or exacerbating them.

Mental Model Alignment

All stakeholders generate and maintain mental models of how their system operates. For an intervention to be adopted for implementation, decision makers must first be aligned on their mental models of system problems (Mitroff & Featheringham, 1974) and the system mechanisms by which those problems manifest; only then can solutions be proposed to fit an identified problem. This approach requires the inclusion of frontline operators whose work contributes to and resolves problematic systems as well as managers and leaders who bring their own informed understanding of contributors to system problems. Observability across roles is generated during this work to align mental models, which assists in overcoming local cognitive dissonance (Fitzgerald, 2019), distancing through differencing (Cook & Woods, 2006), and folk models (Dekker & Hollnagel, 2004) during dissemination and implementation.

Pragmatics

The perceived demands of any intervention must be equal or less than what that system can and will bear (Fitzgerald, 2019). Familiar implementation science constructs such as usability, appropriateness, feasibility, and return on investment can oversimplify and even obscure the particular costs, risks, energy expenditures, and uncertainties associated with the implementation of an intervention. Stakeholders must consider what the intervention and its implementation will require of the system, such as new and adjusted resource allocations, changes to policy and functional infrastructure, and different utilization of system capabilities in the context of work as it is done, which may be substantially different from the way work is imagined or described (Shorrock, 2016; Woods et al., 1996). To “fit through the door” (Fitzgerald, 2019, p. 20), the design of an intervention must be intentionally pragmatic.

Actors

The IMPActS Framework considers system resources and capabilities, alongside human and machine partners, to be contributing actors in the implementation of an intervention. All actors require provisions for coordination, communication, and goal alignment to preempt any potential resistance to the new initiative. While implementation frameworks that address behavioral contributors offer best practices for promoting awareness, encouraging engagement, and managing through change, the reduction in friction and resistance across actors must be engineered to deploy, mobilize, and generate the resources (Woods et al., 2013) sustaining the implementation at the targeted scale.

Sustainment

Continuous monitoring capabilities that inform replanning and revising of the intervention and its implementation are essential to the sustainability of the intervention in context (Woods & Hollnagel, 2006). Beyond modifications to the intervention itself, system changes such as role restructuring and building resource capacity must evolve to accommodate the implementation over time (Rabin et al., 2008; Stirman et al., 2019). Continuous monitoring and sensemaking of these emergent changes facilitate coordination and generate foresight about the changing shape of risk across the larger system (Provan et al., 2020), enabling course correction ahead of increasing system stress and strain.

Methods

We adapted the IMPActS Framework to design an interactive workshop to be attended by expert practitioners within the partner organization who would be involved with and/or affected by the implementation of one or more candidate interventions. The IMPActS Workshop is designed to achieve the four goals via the following plan.

• (1) Guide design-phase discussions. The workshop opens with didactic and interactive training on systems thinking and the IMPActS Framework. Researchers coach participants in using each IMPActS factor as a lens to apply systems thinking and generate foresight about implementing the candidate intervention in their work environment. Resulting insights inform co-construction of refinements to the intervention’s design.

• (2) Sharpen the intervention. Participants contribute their institutional knowledge and lived experience of implementing interventions to the discussion of the candidate intervention. At the same time, researchers leverage extensive findings on adaptive systems to ask incisive questions about how the intervention design itself could mitigate the added complexity it would introduce and what features may be integrated to act as countermeasures to known mechanisms of complex system failure (Branlat & Woods, 2010; Patterson et al., 2010; Woods & Rayo, 2022).

• (3) Generate novel insights about the intervention’s anticipated implementation. The workshop recruits participants of varying roles, responsibilities, and experience to maximize the opportunity to converge multiple diverse perspectives (Woods, 2018) about the system’s dynamics, its problems, and the candidate interventions. As suggested by Roesler et al. (2005), recruitment considerations include participants’ variety of work experiences, specialty knowledge, breadth of interactions across the organization, and the mutual informativeness and new observability across roles that could be generated among these demographics during the workshop.

• (4) Increase the likelihood of sustained implementation. Discussions driving refinements of the candidate intervention continue until workshop participants are satisfied that they have imagined and mitigated all potential implementation barriers and facilitators through intervention design changes or deferred them as out of their reach of influence.

Representatives from the partner organization who have insight into the qualifications and mix of specific participants should drive recruitment. It may be challenging to coordinate a workshop time in which all desired participants are available, given that they are experts actively working in operational environments and whose technical skills are in high demand. Greater than the desired number of participants should be initially recruited, flexibility in selecting participants may become necessary, and additional outreach may be required to ensure the workshop represents an accurate cross-section of the organization.

Experts in how frontline work is done in the target system should identify candidate interventions. Our research team was steeped in recent findings from a systems analysis of the partner organization using the Systemic Contributors and Adaptations Diagramming technique (SCAD; Jefferies et al., 2022). Stories of everyday work revealed complex landscapes of system pressures, goal conflicts, and adaptations that create successful work and incentivize innovation in this organization (Girth et al., 2022). From the SCAD stories, researchers and an organization liaison selected three adaptations that had the greatest potential for application across the organization and reframed them as candidate interventions for the workshop. The researcher who led the SCAD data analysis also participated in the workshop to “speak for” the amalgamated data, that is, as the summative voice of adaptive, innovative behavior within the organization as gleaned during the SCAD interviews. This researcher generated precise, work system-informed probes during the intervention sharpening discussion.

Workshop materials are minimal. Paper handouts may be used in person. Remote participants may access web-based whiteboard software such as Miro (Khusid & Shardin, 2011). The main handout provides rating scales for each of the five IMPActS factors and a space to provide explanations for their ratings. Qualitative rating scales are utilized to encourage nuanced discussion and facilitate schema development in context (Bingham & Kahl, 2013). In addition to low and high endpoint labels, each scale includes descriptive language to specify the meaning of low, high, and halfway points with respect to each IMPActS factor. For example, the Pragmatic factor scale aligns these markers with “costs far outweigh yield,” “yield far outweighs cost,” and “break even,” respectively.

Independently from each other, participants rate each candidate intervention across the five IMPActS Framework factors. The group then convenes to discuss initial ratings and rationales. Robust discussion among participants aims to reveal and tailor the intervention design to address the mechanisms by which local system pressures and goals contribute to anticipated barriers and facilitators to implementation of the candidate intervention. The group’s collaborative redesign of the candidate intervention helps participants think more broadly about potential issues and barriers to successful implementation. After the intervention has been sharpened, participants independently rate the new intervention design. The workshop concludes when participants are confident that a robust proposal for system-wide adoption of the sharpened candidate intervention would be well-received by diverse stakeholders and senior leadership within the organization.

Results and Discussion

Our findings suggest that the IMPActS workshop facilitated our participants’ ability to generate novel and useful insights in tailoring the candidate intervention for successful, sustained implementation. The workshop was conducted with one mid-level administrative leader, one mid-level operational leader, and one immediate supervisor of frontline personnel. Implications stemming from the number and type of participants were considered when conducting this analysis.

• (1) Guide design-phase discussions. Participants quickly adopted a systems view and engaged in its application throughout the workshop’s activities. They stated that, having achieved competence by working through the first IMPActS assessment, subsequent assessments would likely take 1 hr or less, and developing an action plan to address the necessary mitigations would likely take at least another hour per candidate intervention.

• (2) Sharpen the intervention. Participants unanimously reported that the workshop facilitated efficient assessment of the candidate intervention even though the discussion required them to adopt a new mindset towards the system and how it operates. They also reported that the five IMPActS factors identified and focused new opportunities to leverage existing system attributes and pressures to refine the intervention’s design. Participants recognized that this effort would not only increase the likelihood of successful implementation but would also increase the intervention’s projected operational value to the organization.

• (3) Generate novel insights about the intervention’s anticipated implementation. Researchers’ questions stimulated thoughtful discussion among participants, who reported that the workshop improved both the efficiency of design decisions and comprehensive, systematic consideration of factors that might deliver value to disparate yet interdependent areas of the organization’s operations. Converging multiple perspectives about the design of candidate interventions facilitated emergent insights that were noted as unique to this workshop’s structure and facilitation method, compared to other intervention design efforts that participants had experienced in their careers. Additionally, participants reflected on the “intrinsic value” of the IMPActS Framework to drive decision making around operational resource apportionment and stakeholder involvement that may otherwise go underappreciated.

• (4) Increase the likelihood of sustained implementation. Following structured discussion to strengthen the candidate intervention’s IMPActS factors, group members reported strong confidence in the intervention itself and a sense of agency and ownership over the co-redesign process. Through this experience, participants realized that the resource profiles and mechanisms by which organizations initiate change and sustain initiatives over time are different and require intentional strategies to effect. Of particular concern to the group was what strategies would be required to maintain mental model alignment among stakeholders amidst the changing tempo of work and system pressures that disincentivize innovation in their industry (see also Girth et al., 2022). Future workshops will elaborate on the additional steps needed to mitigate the risks and overcome the barriers raised during these discussions.

Design-phase discussions were constrained primarily by time and availability of participants. One participant co-located with researchers and the other two participants joined virtually from separate work sites. The workshop lasted 2 hr with roughly 40 min of instructional time focused on systems thinking and the IMPActS framework and the remainder of the time focused on assessing the interventions. As discussions continued, the workshop became constrained by time, so participants and researchers selected one of three prepared candidate interventions to co-construct improvements relative to IMPActS factors on which it scored weakly. Future workshops will be scheduled for 8 hr to provide for additional instructional time and 2 hr to workshop each candidate intervention.

Although the IMPActS Workshop was not designed to assess the validity of the SCAD system analysis technique, some insight was unexpectedly gleaned in this area. We noted striking similarities between the SCAD data researcher’s IMPActS factor projections and concerning system pressures and those of the frontline supervisor and mid-level operational leader, respectively. These findings suggest that the SCAD technique of eliciting stories of frontline work is effective not only in revealing current systems dynamics that affect frontline work but also in anticipating future system dynamics in relation to a planned interjection of change, such as the implementation of an intervention.

Limitations

This pilot study had a small sample size with three participants and four researchers attending one workshop. While the lived experiences of smaller samples within the same organization are noted to provide insights into influential institutional memory (March et al., 1991) and contribute to foresight generation about emergent organizational behavior (Woods, 2005), future IMPActS workshop recruitments will take steps to mitigate the contributors to this pilot’s low turnout. Exposition into the validity and reliability of specific findings from this IMPActS Workshop, while crucial to the effective sharpening of candidate interventions, is outside the scope of this methodology assessment paper.

Conclusion

Introducing change and innovation into a dynamic and fast-paced work setting can be challenging. Scarce resources, heavy workload, competing demands for attention, and other factors can derail well-intentioned efforts. Many organizations approach this challenge by funneling extensive resources into the implementation effort or by instituting a senior leadership mandate to force the change. While potentially sufficient in some cases, those approaches do not scale to address the myriad of changes an organization needs to effect to remain competitive. By contrast, the IMPActS approach uses a lightweight method that draws from extensive research findings and institutional and domain expertise to give each intervention the greatest chance of sustainment in an operational environment.

The methods we used to design, pilot, and refine the IMPActS Workshop facilitate our ability to offer insights to other research-industry partnerships seeking to implement sustainable system interventions. We found that the design of interventions and implementation systems benefits from collaboration among both internal and external experts. Convening multiple, diverse perspectives reveals a nuanced landscape of systemic pressures and adaptations that can be leveraged to favor the intervention and its implementation. While the IMPActS Framework systematically assesses key contributing factors to successful implementation, it also broadens the range of considerations when designing any new initiative. Organizations may therefore leverage these insights to increase the likelihood of sustaining any implementation.

Most importantly, perhaps, were the takeaways noted by the participants themselves: the structure of the IMPActS Workshop was lightweight and high value, enabling it to be conducted even in high tempo, production-pressured environments to improve intervention and implementation system design.