Abstract
The rapidly advancing field of implementation science is steadily moving beyond seeking to understand whether implementation efforts were successful and past what factors might have led to unsuccessful efforts, toward issues of causality, or exploring why and how implementation efforts work. Indeed, in 2017, the 4th biennial conference of the Society for Implementation Research Collaboration (SIRC) centered its theme on implementation mechanisms titled, “What Makes Implementation Work and Why?” Implementation mechanisms are defined as processes or events through which implementation strategies operate to affect one or more implementation outcomes (Lewis et al., 2020); or how and why strategies work. For example, an electronic health record reminder prompt works through the mechanism of increasing awareness to a forgotten clinical practice. A recent systematic review identified 46 studies that explored implementation mechanisms (Lewis et al., 2020); however, none empirically established causal links between strategies, mechanisms, and outcomes consistent with criteria for this type of research (e.g., strong association, specificity, consistency, experimental manipulation) (Hill, 1965; Kazdin, 2007). Understanding the mechanisms through which strategies work is critical to informing strategy matching, tailoring, and optimization (Lewis et al., 2021a); without this knowledge, we run the risk of deploying costly and complex multifaceted strategies that may not offer an enhanced impact.
To support the study of implementation mechanisms, the Agency for Healthcare Research and Quality (HS025632) recently funded a 3-year R13 (HS025632) initiative with two aims: (1) establish priorities to guide a research agenda on implementation mechanisms and (2) disseminate the agenda to research, policy, and practice audiences. The protocol for this mechanistic implementation science initiative is detailed elsewhere (Lewis et al., 2021b). The research agenda will synthesize and build on unique challenges or gaps surfaced from six diverse sources, including expert input, evidence syntheses, and working meetings with global attendees.
This brief report presents findings from data extraction of one of the sources for the R13 implementation mechanisms research agenda, notably abstracts accepted for presentation at the 5th biennial SIRC conference themed, “Where the Rubber Meets the Road: The Intersection of Research, Policy, and Practice.” Specifically, we explored the extent to which empirical investigations of mechanisms were present in SIRC's 2019 programming in an effort to identify gaps that would directly inform the research agenda, which would complement the 2020 systematic review that did not include studies after 2018. Although the 2019 SIRC theme did not squarely focus on implementation mechanisms, those leading in this area suggest that implementation mechanisms may be the most efficient path toward generating practical implementation strategies that policymakers may fund. That is, if mechanisms were pervasive in the field, we would expect to see them mentioned in the SIRC conference abstracts. Reported herein is a detailed account of the components of implementation causal pathways (Lewis et al., 2018) that were present in published empirical abstracts, with a specific focus on capturing the extent to which studies articulated implementation strategies, mechanisms, determinants, and outcomes (see Figure 1 for a parsimonious, linear causal pathway diagram).
Implementation causal pathway.
Method
SIRC conference abstracts
This study examined the content of abstracts included in the program for the 2019 SIRC conference. SIRC is one of at least three implementation science focused conferences. These abstracts provide a snapshot of the state of the science, with respect to the study of implementation mechanisms, deemed worthy of featuring in one of the field's leading conferences. As per SIRC, abstracts were limited to 350 words, required three to five learning objectives, and a description of the current stage of data collection. Of the 205 abstracts accepted for conference presentation, 81 were 15–20 min oral presentations, five were 30–60 min panel presentations, and 119 were posters.
Data extraction
Trained coders reviewed all 205 accepted abstracts (across categories of oral presentations and posters) and determined first whether they were empirical or theoretical. Theoretical articles were omitted from further analyses due to their inability to offer data related to implementation mechanisms (N = 54). We extracted information across five main areas to describe the studies and characterize the presence of data related to causal pathways. (1) We extracted information about the sample to align with our funding agency's (i.e., Agency for Healthcare Research and Quality) list of priority populations: children, chronic care, elderly, end-of-life care, inner city, low income, rural, minority, and women. (2) We extracted the study setting to align with categories applied in a recent and relevant systematic review of implementation mechanisms to maintain consistency and create common terminology across projects (Table 1) (Lewis et al., 2020). (3) We extracted whether the study was qualitative, quantitative, or mixed methods. (4) If an abstract reported a guiding theory or framework, we recorded this verbatim to align with the extant literature.
Results of abstract data extraction.
Note. Percentages were calculated using only empirical studies (N = 151) as the denominator.
End-of-life care population not assessed; “Other” populations included elderly, inner city, low income, and minority.
Settings not assessed included dental offices, public library systems, and university nursing programs; “Other” settings assessed included emergency departments, hospitals, juvenile courts, and nursing homes.
(5) Consistent with the primary objectives of this report, we extracted information regarding implementation strategy causal pathways, specifically the following: discrete implementation strategies, determinants of implementation, implementation mechanisms, service outcomes, and implementation outcomes, which are often lumped together in implementation studies (Lewis et al., 2020). To improve precision reporting, aligned with the call to action in this recent systematic review, we attempted to code for the factors implicated in an implementation strategy pathway from strategy to mechanism to determinant, to outcome(s). We applied terms from the extant literature wherever possible to capture each of the causal pathway elements. For implementation strategies, we used labels from a recently published compilation (Powell et al., 2015), for determinants of implementation we used the Consolidated Framework for Implementation Research (Damschroder et al., 2009), for service outcomes, we used those articulated by the Institute of Medicine (Institute of Medicine, 2006), and for implementation outcomes, we drew on the seminal work of Proctor and colleagues (Proctor et al., 2011). Implementation mechanisms were coded only if the authors reported studying “mechanisms” or “mediators.” Unfortunately, since no list of putative mechanisms yet exists, we used exact terms from authors or summarized lengthier descriptions into precise terms as needed.
Coder training and procedures
Coders were two research specialists and the lead author. A coding manual was developed by the lead author to include the recently published systematic review of implementation mechanisms (Lewis et al., 2020) and an article articulating steps for developing implementation causal pathways (Lewis et al., 2018) to establish foundational knowledge to guide extraction. The manual also included the articles from which the standardized terms came as well as the terms themselves and associated definitions. The manual was reviewed by the senior author and coders were trained to become savvy with relevant implementation science terminology. During training, coders met weekly with the senior author to review coding questions. Once the coders reached a consensus on two consecutive abstracts, they were given a weekly list of abstract assignments for formal coding. Formal abstract coding data was entered into REDCap and 5% of abstracts were coded by all three coders to establish interrater reliability. These abstracts were chosen from primary coders’ weekly assignments to assess and correct for drift over time. Overall percent agreement was 95.81% across 10 abstracts.
Following completion of coding, data were exported, and frequencies run to explore the need to collapse across categories for meaningful summarization. The senior author assessed the abstracts the coders flagged as discussing implementation mechanisms, dropping any that solely examined determinants, and categorized the remaining implementation mechanisms to reflect a simple and consistent use of terms.
Results
The majority of the 205 abstracts were empirical investigations (73.66%, N = 151), employing a variety of qualitative (N = 46), quantitative (N = 44), and mixed methods (N = 39); there were 22 abstracts without sufficient information regarding study design. The minority of abstracts embraced a theory or framework to guide their work (14.47%, N = 22); the Consolidated Framework for Implementation Research was the most common among those invoking any theory/framework. The most prevalent priority population studied was children, followed by those with chronic conditions. Consistent with the origins of SIRC (i.e., National Institute of Mental Health-funded conference series; MH086159), the most prevalent setting was behavioral health facilities. See Table 1 for details.
Most abstracts reported using at least one implementation strategy (59.6%, N = 90) and 45 different strategies were reported across all abstracts (see Tables 2 and 3). “Assess for readiness and identify determinants” was the most commonly reported implementation strategy (21.19%, N = 32). The majority of abstracts (65.56%, N = 99) did not report determinants of implementation. The most commonly reported determinant of implementation was an inner setting determinant, “available resources” (N = 16). For intervention characteristics, “complexity” was the most frequently reported determinant (N = 7). For the outer setting, the most frequently reported determinant was “external policy and incentives” (N = 13). Within characteristics of individuals, “knowledge and beliefs about the intervention” were reported with the highest frequency (N = 12). For process, “engaging” was the most frequently reported determinant (N = 9).
Results of implementation strategy causal pathways factor extraction.
Note. Percentages were calculated using only empirical studies as the denominator.
Implementation strategies reported.
Note. Percentages were calculated using only empirical studies as the denominator; Strategies not reported were not included in this table; Nine strategies were reported that did not fit into this framework, examples include: hiring intermediaries, deploying web-based tools, conducting rapid evidence reviews.
Just 11 (7.28%) of the 151 empirical abstracts reported studying mechanisms. Mechanisms were examined in projects utilizing 14 different implementation strategies (Table 4). “Assess for readiness and identify determinants” was linked to the greatest number of abstracts studying mechanisms, though this reflects a mere 12.5% (N = 4) of abstracts reporting the implementation strategy. When mechanisms were explored, they were most often measured using qualitative methods (36.36%, N = 4) or self-report (36.36%, N = 4), followed by mixed methods design (27.27%, N = 3). No studies reported using administrative data to study mechanisms. Accountability was the most commonly reported implementation mechanism (N = 4).
Implementation causal pathways reported.
Note. This table reports the mechanisms studied by implementation strategy reported in the abstract and links the implementation and service outcomes as well as any barriers reported.
*Same abstract.
+Same abstract.
Service outcomes were reported for almost half of empirical investigations (47.02%, N = 71). Quality was the most commonly reported outcome (42.38%, N = 64) and was reported in 54.55% (N = 6) of abstracts examining mechanisms. Just over half (63.64%, N = 7) of abstracts studying implementation mechanisms reported an implementation outcome. Fidelity was the most commonly reported implementation outcome overall (22.52%, N = 34), but the most commonly reported implementation outcome in abstracts including the study of mechanisms was adoption (36.36%, N = 4).
We were able to construct implementation causal pathways for just two implementation strategies, “assess for readiness and identify determinants,” representing information pulled from four different abstracts (Adams et al., 2019; Laukvik et al., 2019; Moon et al., 2019; Stanick et al., 2019), and “create a learning collaborative,” with data pulled from just one abstract (Are et al., 2019). The remaining 12 implementation strategies linked to implementation mechanisms had missing information (Table 4) suggesting most studies were more exploratory perhaps contributing to hypothesis generation rather than hypothesis testing.
Discussion
The purpose of this project was to identify the extent to which implementation mechanisms were empirically investigated in studies presented at the 2019 SIRC biennial conference. We explored this objective through a qualitative analysis of the accepted abstracts. Although the majority of presentations were empirical studies, only a small minority included the investigation of mechanisms. Overall, the abstracts that included implementation mechanisms offered a preliminary view into mechanisms as opposed to establishing or confirming mechanism activation via quantitative methods. The majority of abstracts with mechanism information reported adoption as the primary implementation outcome illustrating that mechanistic evaluations are most often occurring in the earliest phases of the implementation process. These findings indicate that, at least based on SIRC conference abstracts, the empirical investigation of implementation mechanisms remains both rare and limited in scope.
Our team reviewed the two constructed causal pathways through the lens of “plausibility” defined as “coherence of an explanation of how a mechanism operates” (Kazdin, 2007), which is one of seven criteria for establishing mechanisms. Although acknowledged as subjective, this process entailed asking ourselves questions such as, “What processes may be activated when deploying X strategy?” and “Are the steps that unfold from strategy X likely to resolve Y determinant?” In doing so, we found coherence in three of four mechanistic pathways for “assess for readiness and identify determinants,” but found it questionable as to whether this strategy alone could activate the mechanism of “reforming state policy” as one abstract suggested. In our view, this particular pathway would have been made plausible if this strategy was supplemented with another system-level strategy. The learning community causal pathway came only from one abstract, and compared to the other causal pathway, it appears quite plausible and coherent (see Figure 2). Specifically, this study articulated that a learning collaborative could address determinants of networks and communication as well as access to knowledge and information through the mechanism of “building community capacity” to improve outcomes of appropriateness, quality, and fidelity. These types of pathways are critical to testing and establishing implementation mechanisms, but are rare still in the implementation literature. Although there is rich literature on needs assessments and learning collaboratives, there is little mention, to our knowledge, about their mechanisms of action or about theory or causality more broadly.
Learning collaborative implementation causal pathway.
The study of how implementation strategies influence implementation outcomes must be conducted through a theoretical lens, though this has been complicated by the plethora of implementation science related models and frameworks (Tabak et al., 2012) for which their distinction from theory has only recently been made (Nilsen, 2020). It is the case that there are very few implementation theories that can inform causal relations, prediction, and testing (Lewis et al., 2020). It is noteworthy that very few of the empirical abstracts reported any guiding theory, model, or framework. Without a theoretical foundation through which to interpret the progress of an implementation strategy, it is likely that the causal links between strategies, mechanisms, and outcomes will remain understudied, and instead knowledge about whether a strategy works, but not how, will remain the focus. As evidence of this, 90 abstracts mentioned a specific implementation strategy while just 11 examined implementation mechanisms, suggesting an emphasis on understanding implementation strategy impact without much insight into how implementation strategies are operating to influence implementation and service outcomes.
An important part of the implementation causal pathway is the study of determinants. In abstracts that included mechanisms, the most commonly missing piece of the implementation causal pathway was the study of determinants. In our team's related systematic review of implementation mechanisms, we often saw conflation of determinants and mechanisms, which remains an unresolved issue for the field. In the current study, we saw conflation of strategies and mechanisms, such as “providing feedback,” arguably a strategy, was instead characterized as a mechanism. We hope the field will harmonize around the differentiation of factors in a causal pathway, acknowledging that determinants are factors that enable or hinder strategy impact (Krause et al., 2014) whereas mechanisms are processes through which strategies operate to affect their desired outcome (Lewis et al., 2018). Strategies, activities, or methods deployed to achieve implementation outcomes should target determinants and address them by activating conceptually related mechanisms. Clearly, more work is needed conceptually to advance the study of implementation mechanisms.
It is important to consider this brief report within the context of several noteworthy limitations. First, the theme of the 2019 SIRC conference was “Where the Rubber Meets the Road” and thus the more practical and policy-related theme might have de-emphasized the inclusion of mechanistic investigations, in general, or in abstract language, in specific. Second, the results reported herein come from the qualitative coding of conference abstracts and not full study reports so these findings might reflect how people were aligning their presentations to the conference theme and its review criteria, and this summary may therefore not be a true reflection of the study of implementation mechanisms. Third, this review was a snapshot of a single implementation conference at one time point. It could be interesting to see if the study of implementation mechanisms shows up more over time within SIRC or across conferences or the field more broadly.
These findings highlight the need for focused research on the study of implementation mechanisms. Although this focus is clearly relevant to advancing the field of implementation science, there is great potential for practical impact as isolating mechanisms can help with strategy selection, matching, tailoring, and optimizing. This paradigm shift for the field will require greater attention to the theoretical underpinnings of why and how implementation strategies work. As a result, the field will uncover more cost-effective and impactful strategies by virtue of establishing causal links between strategies, mechanisms, and outcomes (implementation, service, and clinical).
Footnotes
Declaration of Conflicting Interests
The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Dr. Cara C. Lewis is an author of this manuscript and editor of the journal, Implementation Research and Practice. Due to this conflict, Dr. Lewis was not involved in the editorial or review process for this manuscript.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Agency for Healthcare Research and Quality awarded to Dr. Cara C. Lewis as principal investigator [HS025632] and by a National Institute of Child Health and Human Development funded National Research Service Award awarded to Sarah F. Vejnoska [F31HD10215502].
