Adapting the stages of implementation completion to an evidence-based implementation strategy: The development of the NIATx stages of implementation completion

Abstract

Background

Dissemination and implementation frameworks provide the scaffolding to explore the effectiveness of evidence-based practices (EBPs) targeting process of care and organizational outcomes. Few instruments, like the stages of implementation completion (SIC) examine implementation fidelity to EBP adoption and how organizations differ in their approach to implementation. Instruments to measure organizational competency in the utilization of implementation strategies are lacking.

Method

An iterative process was utilized to adapt the SIC to the NIATx implementation strategies. The new instrument, NIATx-SIC, was applied in a randomized controlled trial involving 53 addiction treatment agencies in Washington state to improve agency co-occurring capacity. NIATx-SIC data were reported by state staff and external facilitators and through participating agency documentation. Proportion and duration scores for each stage and phase of the NIATx-SIC were calculated for each agency. Competency was assessed using the NIATx fidelity tool. Comparisons of proportion, duration, and NIATx activities completed were determined using independent sample t-tests by agency competency level.

Results

The NIATx-SIC distinguished between agencies achieving competency (n = 23) and those not achieving competency (n = 26). Agencies achieving competency completed a greater proportion of implementation phase activities and had a significantly longer Stage 7 duration. These agencies participated in significantly more individual and group coaching calls, attended more in-person meetings, implemented more change projects, and spent approximately 64 more days, on average, engaging in all NIATx activities.

Conclusions

Organizational participation in dissemination and implementation research requires a significant investment of staff resources. The inability of an organization to achieve competency when utilizing a set of implementation strategies waste an opportunity to institutionalize knowledge of how to apply implementation strategies to future change efforts. The NIATx-SIC provides evidence that competency is not an attribute of the organization but rather a result of the application of the NIATx implementation strategies to improve agency co-occurring capacity.

Trial Registration

ClinicalTrials.gov, NCT03007940. Registered January 2, 2017, https://clinicaltrials.gov/ct2/show/NCT03007940

Plain Language Summary

Access to integrated services for persons with co-occurring substance use and mental health disorders is a long-standing behavioral health problem. Evidence-based practices (EBPs) that focus on patient needs are effective in improving care for persons with co-occurring disorders. The stages of implementation completion (SIC) is a measure that assesses the process that organizations go through when implementing a new EBP and can be used to compare differences between organizations in their fidelity to recommended processes. To implement, organizations use specified strategies to integrate EBP into the care process. These strategies require a significant investment of staff resources. When organizations struggle to achieve competency with a set of implementation strategies, resources are wasted impacting the ability to use the strategies in future change efforts. As such, it is critical to measure organizational efforts to achieve competency, but instruments to do so are lacking. The SIC was adapted for a proven implementation strategy, NIATx, to address this gap. The NIATx strategy provides outside support and coaching to facilitate the implementation of a new EBP. Results from this study indicated that the NIATx-SIC could distinguish between addiction treatment agencies that applied NIATx implementation strategies with competency, versus those that did not, in the context of a multilevel randomized control trial. Study results provide evidence for the utility of adapting the SIC to specific implementation strategies and the benefit that the NIATx-SIC could provide for similar studies involving the use of NIATx to implement EBPs.

Keywords

stages of implementation completion NIATx implementation strategies

Introduction

Dissemination and implementation frameworks provide the scaffolding to frame proposed research and explore the effectiveness of the proposed interventions. The utilization of multiple implementation strategies is a proven and effective approach to support organizational champions or change teams to implement evidence-based practice (EBP; Caton et al., 2021; Garner et al., 2020; Perry et al., 2019). NIATx is a multicomponent set of implementation strategies developed to guide organizational champions and change teams to improve processes of care and adoption of EBPs in substance abuse treatment agencies (Ford et al., 2017; Gustafson et al., 2013; Hoffman et al., 2008; McCarty et al., 2007; Molfenter et al., 2019; Roosa et al., 2011). Although recent efforts link implementation strategies to frameworks (Waltz et al., 2019; Weir et al., 2021), few instruments exist that allow for a closer look into the “black box” of implementation to examine fidelity to the implementation process in the adoption of an EBP and how organizations differ in their approach to implementation. One exception is the stages of implementation completion (SIC) instrument (Chamberlain et al., 2011; Saldana, 2014).

The SIC is an eight-stage assessment tool designed to measure and compare implementation strategies for scaling up proven interventions (Brown et al., 2014). Stages range from Engagement (Stage 1) with the EBP developers/purveyors, to achievement of program delivery with Competency (Stage 8). Each stage maps onto three implementation phases: Preimplementation, Implementation, and Sustainment (Saldana et al., 2020). SIC data include a log of activities that operationalize the implementation process necessary to move toward successful program start-up, competency, and sustainment, and their completion dates. Two scores are calculated for each SIC stage and phase. The proportion score calculates the proportion of key activities completed within a stage. The duration score is calculated by date of first through date of final scored activity completed. The duration score can account for activities not completed sequentially and for being in multiple stages at a given time. A third final stage score indicates the final stage achieved in the implementation process (Stage 1–8). The SIC is psychometrically valid and reliable (Chamberlain et al., 2011, 2012; Saldana, 2014; Saldana & Chamberlain, 2012; Saldana et al., 2015).

The SIC has been adapted or customized for multiple EBPs, through a clearly defined adaptation process (Dubowitz et al., 2020; Nadeem et al., 2018; Saldana et al., 2012, 2020; Sterrett-Hong et al., 2021; Watson et al., 2020). Several applications of the SIC have or are being evaluated in system settings focused on improving child or youth services (Aalsma et al., 2019; Brown et al., 2014; Palinkas et al., 2017). In addition, the SIC has been utilized to assess the cost associated with implementation (Saldana et al., 2014) and to develop interview guides to obtain a greater understanding of the implementation process within an organization (Palinkas et al., 2018). However, the SIC has not been utilized to explore organizational participation and adherence to specific implementation strategies.

Within the context of a multisite randomized control trial, 53 community addiction treatment agencies located in the State of Washington were trained how to use NIATx, a proven evidence-based implementation strategy (Gustafson et al., 2013; Hoffman et al., 2012; McCarty et al., 2007), to improve their dual capacity to provide integrated care for individuals with co-occurring addiction and psychiatric disorders (Ford, Osborne, et al., 2018; Ford, Stumbo, et al., 2018; Ford, Zehner, et al., 2018). These agencies successfully utilized NIATx implementation strategies to improve their dual diagnosis capability to provide integrated care (Assefa et al., 2019; Chokron Garneau et al., 2022) and increase access to medications (Ford et al., 2021, 2022). In this manuscript, we report on the adaptation of the SIC and the development of the NIATx-SIC to track each organization's fidelity to NIATx implementation strategies.

Method

Design and Setting

A cluster randomized waitlist control group design was used to examine the extent that the NIATx implementation strategies improved agency co-occurring capacity including access to medication-related treatment for individuals with a co-occurring disorder (COD). Community addiction treatment agencies throughout Washington state were eligible to participate. Eligibility criteria included: tax-exempt status, providing outpatient (or intensive outpatient) services, a governmental agency or agency that provides 50% or greater public funding, and no prior participation in NIATx research. Recruitment letters were sent to 468 eligible addiction treatment agencies. Fifty-three agencies (11.3%) agreed to participate and four withdrew prior to randomization, resulting in a final sample of 49 agencies.

Agencies were randomized to Cohort 1 (NIATx implementation strategies, n = 25) or Cohort 2 (waitlist control group, n = 24). Throughout the first year, NIATx implementation strategies (Table 1) were provided to individual agencies assigned to Cohort 1 to support their efforts to implement changes to improve capacity for co-occurring treatment and client access to integrated services. Cohort 2 agencies randomized to the waitlist period, had their performance monitored to assess baseline standard operations related to co-occurring capacity but they were not provided any implementation support. After 12 months, Cohort 2 agencies transitioned from waitlist to support with the NIATx strategies, while Cohort 1 transitioned from receiving support to a 1-year sustainment period of monitoring only. The study protocol and primary outcome results are described elsewhere (Assefa et al., 2019; Ford, Osborne, et al., 2018; Ford, Stumbo, et al., 2018; Ford, Zehner, et al., 2018).

Table 1

NIATx Implementation Strategies

Discrete strategies	Description of the NIATx implementation strategy
Conduct local consensus discussions (coaches)	The research team and coaches met monthly to review agency progress, discuss specific issues, and develop plans to ensure that the multicomponent implementation strategies were delivered with fidelity
Identify and prepare champions	Each participating agency was encouraged to identify a project champion (e.g., a change leader), who participated in the learning collaborative and was the primary contact between the provider and their assigned NIATx coach
Organize implementation team meetings	Each participating agency was encouraged to identify and develop a change team. Led by the change leader, the teams were encouraged to meet regularly to implement changes to improve co-occurring capacity, keep meeting minutes, and participate in the site visit and other learning collaborative opportunities as warranted
Recruit, designate, and train for coaching	Identified individuals to act as NIATx coaches and provide with a one-day coach training to prepare these individuals to offer external facilitation to assigned agencies
Use an implementation advisor	Each agency was assigned a NIATx-trained coach. The coach conducted a site visit with their assigned providers and guided the team in selecting and reviewing change projects
Facilitation (external)	The agency worked with their assigned external facilitator (coach) in a process of interactive problem solving where the coach provided support through an interpersonal relationship to guide agency staff through efforts to improve processes of care
Conduct educational outreach visits	The coach conducted a 3- to 4-h site visit with the participating agencies. Following a standard agenda, the coach meets with different staff, reviewed DDCAT results, and planned for implementation projects
Create a learning collaborative	Established learning collaborative opportunities allowed participating agencies to come together to share ideas and learn about how to apply the NIATx implementation strategies in their organization
Develop educational materials (NIATx)	Developed NIATx tools (walkthrough template, change project forms, change team meeting templates, etc.) for coach use
Distribute educational materials (NIATx)	Coaches were provided with common NIATx tools (walkthrough template, change project forms, change team meeting templates, etc.) to share and provide instruction on use with the agencies during the site visit
Make training dynamic	The in-person learning collaborative meetings (Months 3/4 and 12) integrated didactic and experiential learning opportunities to allow participating agencies to learn about how to apply NIATx implementation strategies in their agency
Provide ongoing consultation	The NIATx-trained coach conducted monthly calls with their assigned agencies. During these calls, the coach and change team review change projects; discuss successes; and identify ideas for future change projects
Audit and provide feedback	After each yearly assessment, agencies received DDCAT feedback report that graphically represented performance in each domain over time and a summary from the assessment team. The report guided change efforts in the agency
Conduct cyclical small tests of change	Participating agencies utilized Plan-Do-Study-Act (PDSA) change cycles to implement changes to improve co-occurring capacity based on their DDCAT assessment scores

Adaptation of the SIC (NIATx-SIC)

The process for reviewing and adapting the SIC to measure process of the NIATx implementation strategies occurred iteratively. The initial step was a review of the planned NIATx implementation strategies (JHF and MEZ). The NIATx team then met virtually with the SIC team for a series of adaptation meetings. The SIC team interviewed the NIATx team to understand the various paths of how the NIATx implementation strategies could unfold in a participating community addiction treatment agency. When adapting the SIC tool, our team had to account for the change in focus of the SIC from implementing an EBP to implementing a multicomponent set of NIATx implementation strategies (Ford et al., 2016). Specifically, we addressed questions such as:

How do you measure implementation process when the implementation is under a fixed time period versus an open-ended implementation or intervention process?

Given that each organization may implement variable numbers of change projects, and have varying levels of contact with their implementation coach, are these variations important to capture for thoroughness of completion? How does the process differ when the number of events (e.g., change projects) is not fixed but depends on the efforts of the organization?

Given the NIATx model, would it be appropriate to consider the start and end dates of all change cycles as an activity in the implementation process or is it better to focus on the start and end dates associated with various change projects?

Although operationalizing the NIATx implementation process demanded some unique variations from the standard SIC. For example, it was necessary to determine how to best capture participation in coaching calls or to document the number of change projects when these activities could vary by the participating agency. In addition, it was necessary to build in variables to account for the delayed start in Stage 3 for Cohort 2 agencies. However, the most significant challenge for the adaptation was defining competency of sites (Stage 8). The NIATx fidelity instrument, developed for this study, became the basis for determining competency. The instrument utilized a 5-point scale to assess agency adherence to the NIATx principles across 24 items in seven different domains (Supplemental File 1). Using interviews and project documentation (e.g., change project forms, walkthrough reports) supplied by the agency or coach, NIATx fidelity was determined by an independent rating of two external evaluators (JHF and MEZ) by averaging the scores within each of the seven domain and then adding those scores across domains and dividing the result by seven (Ford, Osborne, et al., 2018; Ford, Stumbo, et al., 2018; Ford, Zehner, et al., 2018). Although individual item interclass correlation coefficients ranged from 0.25 to 0.98, the overall individual item correlation between the two evaluators was highly positive (r = .863, p < .001). For the NIATx fidelity instrument, the Cronbach alphas for the seven individual domains ranged from 0.83 to 0.93 with an overall Cronbach alpha of .926. The overall fidelity scores were rank ordered across all agencies and those whose score was greater than or equal to the median were considered to have exhibited competency in the NIATx model.

Data for the final NIATx-SIC (Supplemental File 2) were collected from information tracked by State of Washington staff, coaching interactions between NIATx experts and agency documentation. NIATx-SIC data were entered into the standard online SIC reporting system. Following standard SIC scoring procedure, for each stage and phase of the NIATx-SIC, the proportion and duration scores were calculated for each agency and averaged across each respective cohort. The NIATx-SIC calculated a third final stage score that indicates the final implementation stage achieved in the implementation process (Stage 1–8). The NIATx-SIC also monitors standard key SIC implementation milestone outcomes: agency achievement of (a) program start-up and (b) competency. Important for this study, the NIATx-SIC includes an assessment of baseline organizational characteristics such as primary funding source, population served, and geographic size.

Analysis

Descriptive statistics on the proportion and duration of activities within each phase and agency attributes were calculated. Differences between agency cohort assignment were evaluated. Independent t-tests compared the proportion of events completed and the duration of activities by phase and stage within the NIATx-SIC for agencies that achieved competency versus agencies that did not. A similar analysis examined differences in participation in NIATx implementation strategies (e.g., coaching calls) and change projects initiated in agencies that achieved competency versus agencies not achieving competency.

Results

Figure 1 shows the Extended CONSORT Diagram (Campbell et al., 2012) for study recruitment. Agencies assigned to Cohort 1 (n = 25) and Cohort 2 (n = 24) did not differ by agency characteristics, however, agencies that achieved competency were more likely to be nonprofit addiction treatment agencies (Table 2).

Figure 1

Extended CONSORT Diagram

Table 2

Agency Attributes by Cohort and Competency

Agency characteristics	Assigned cohort^a		Achieved competency^b
Agency characteristics	Cohort 1N (%)	Cohort 2N (%)	NoN (%)	YesN (%)	TotalN (%)
Population size
<25,000	6 (24.0)	9 (37.5)	7 (26.9)	8 (34.8)	15 (30.6)
25,000–99,999	11 (44.0)	6 (25.0)	12 (46.2)	5 (21.7)	17 (34.7)
100,000–250,000	6 (24.0)	7 (29.2)	5 (19.2)	8 (34.8)	13 (26.5)
>250,000	2 (8.0)	2 (8.3)	2 (7.7)	2 (8.7)	4 (8.2)
Agency attributes (% of agencies responding yes)
Private	14 (55.0)	8 (33.3)	13 (50.0)	9 (39.1)	22 (44.9)
Nonprofit	13 (52.0)	16 (66.7)	12 (46.2)	17 (73.0)^c	29 (59.2)
Agency focus
Addiction treatment services	9 (36.0)	12 (50.0)	12 (46.2)	9 (39.1)	21 (42.9)
Addiction and mental health services	16 (64.0)	12 (50.0)	14 (53.8)	14 (60.9)	28 (57.1)
Medically underserved	13 (52.0)	13 (54.2)	14 (53.8)	12 (52.2)	26 (53.1)
Healthcare provider shortage area—primary care	16 (54.0)	19 (79.2)	21 (80.8)	14 (60.9)	35 (71.4)
Healthcare provider shortage area—behavioral health	18 (72.0)	19 (79.2)	22 (84.6)	15 (65.2)	37 (75.5)
Prior NIATx experience	5 (20.0)	10 (41.7)	7 (26.9)	8 (34.8	15 (30.6)
Addiction levels of care (% of agencies responding yes)
Outpatient	13 (52.0)	13 (54.2)	14 (53.8)	12 (52.2)	26 (53.1)
IOP/partial hospitalization	15 (60.0)	14 (58.3)	13 (50.0)	16 (69.6)	29 (59.2)
Residential/inpatient	9 (36.0)	11 (45.8)	13 (50.0)	7 (30.4)	20 (40.8)
Medically managed intensive inpatient (hospital)	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)
Opioid maintenance	1 (4.0)	1 (4.2)	0 (0.0)	2 (8.7)	1 (4.1)
Detoxification	1 (4.0)	0 (0.0)	0 (0.0)	1 (4.3)	1 (2.0)
Mental health level of care (% of agencies responding yes)
Outpatient	10 (40.0)	8 (33.3)	10 (38.5)	8 (34.8)	18 (36.7)
Partial hospital/day program	1 (4.0)	1 (4.2)	2 (7.7)	0 (0.0)	2 (4.1)
Inpatient	3 (12.0)	1 (4.2)	2 (7.7)	2 (8.7)	4 (8.2)

Number of agencies in Cohort 1 = 25 and in Cohort 2 = 24.

Number of agencies achieving competency: yes = 23 and no = 26. Of the agencies achieving competency, 13 were in Cohort 1 and 10 were in Cohort 2.

p = .048.

Table 3 provides descriptive statistics associated with the proportion of activities completed and the duration of time associated with each phase and stage depending on competency status. Of the 53 agencies that agreed to participate in this study, 23 agencies ultimately achieved competency, while 30 agencies discontinued their implementations. However, 41 agencies reached program start-up indicating that of those that discontinued, the majority did so during the active implementation phase. As shown in Table 3, although there was no significant variation in the amount of time spent on implementation processes, agencies that achieved competency completed a significantly greater proportion of activities along the implementation process from feasibility (Stage 2) to ongoing service delivery and monitoring (Stage 7), suggesting a stronger implementation process from start to finish. Supplemental File 3 shows similar descriptive information by assigned cohort.

Table 3

Proportion of Events Completed and Duration by Competency Achievement

Duration of scored activities completed (days)	Competency not achieved			Competency achieved			p
Duration of scored activities completed (days)	M	SD	N	M	SD	N	p
Stage 1	119.03	9.9	30	121.43	14.94	23	.485
Stage 2	43.73	18.65	30	50.52	16.15	23	.170
Stage 3	99.85	68.96	26	79.00	40.76	23	.212
Stage 4	40.74	41.9	23	23.70	19.32	23	.083
Stage 5	42.90	74.73	21	30.83	33.74	23	.487
Stage 6	46.32	46.76	19	35.65	28.27	23	.367
Stage 7	170.17	87.12	23	248.30	58.21	23	.001
Preimplementation Phase 1	286.83	72.34	30	275.87	37.78	23	.512
Implementation (Phase 2)	210.61	77.89	23	306.91	56.26	23	<.0001
Duration of scored activities completed for the entire SIC	436.00	129.77	30	551.96	62.98	23	<.0001
Proportion of scored activities completed
Stage 1 ^a	1.00	0.00	30	1.00	0.00	23	N/A
Stage 2	0.83	0.31	30	1.00	0.00	23	.013
Stage 3	0.78	0.37	30	0.99	0.03	23	.010
Stage 4	0.66	0.41	30	0.91	0.16	23	.007
Stage 5	0.55	0.41	30	0.90	0.22	23	.001
Stage 6	0.53	0.40	30	0.98	0.07	23	<.0001
Stage 7	0.22	0.18	30	0.49	.017	23	<.0001
Preimplementation (Phase 1)	0.83	0.29	30	0.99	0.02	23	.009
Implementation (Phase 2)	0.31	0.22	30	0.60	0.14	23	<.0001

t-test cannot be computed because standard deviation of both groups is zero.

NIATx-SIC Stage 7 activities focused on the use of NIATx implementation strategies to drive change in each agency. Table 4 identifies specific NIATx implementation strategies that were completed more thoroughly by agencies achieving competency versus those that discontinued. Agencies that achieved competency showed greater participation in activities related to coaching calls, site visits, and change projects. Specifically, agencies that achieved competency participated in three more coach calls and were more actively engaged in the group coaching calls and in-person meetings than those that did not achieve competency. In addition, these agencies implemented two more change projects than noncompetent agencies (p < .0001). The total time engaged with the NIATx implementation strategies was 64 days longer for agencies that achieved competency (289.7 days) versus agencies that discontinued (225.4 days).

Table 4

Comparison of Engagement in NIATx Implementation Strategies by Agency Competency

NIATx implementation strategies	Competency not achieved			Competency achieved			p
NIATx implementation strategies	M	SD	N	M	SD	N	p
Number of coach calls	3.57	3.26	30	7.04	2.44	23	<.0001
Average time between coach calls (days)	43.30	19.85	20	36.05	11.55	23	.145
Time to first coach contact (days)	17.14	13.68	21	11.70	7.90	23	.11
Time to presite visit call (days)	15.55	11.72	20	13.96	16.20	23	.717
Time to site visit (days)	21.59	17.97	22	34.70	21.77	23	.033
Time from walk-through introduced to completed (days)	27.70	26.55	10	36.07	24.02	15	.422
Time from walk-through introduced to share with coach (days)	76.42	91.37	12	43.16	30.41	19	.151
Time from change project introduced to first change project started (days)	43.00	55.38	13	39.13	45.30	23	.822
Number of group coaching calls attended	0.60	0.67	30	1.26	0.69	23	.001
Number of in-person meetings attended	0.77	0.63	30	1.57	0.59	23	<.0001
Total time from first NIATx engagement to last NIATx engagement (days)	225.41	65.87	22	289.70	57.75	23	.001
Number of change projects	1.00	0.98	30	2.52	1.27	23	<.0001
Average change project duration	117.75	88.30	4	101.22	67.64	16	.684
Proportion of meetings attended	0.49	0.25	23	0.71	0.22	23	.003

Discussion

Dissemination and implementation (D&I) research involves the introduction of specific implementation strategies related to an EBP or organizational support offered through external facilitation (Powell et al., 2015; Waltz et al., 2015, 2019). Organizational participation requires a significant investment of staff resources to engage with the different implementation strategies. For example, staff may need to invest significant time to receive training in an EBP (e.g., motivational interviewing) or set aside time for external coaching calls and/or internal change team meetings to design, implement, and evaluate the impact of Plan-Do-Study-Act change cycles. The inability to sustain a particular practice (e.g., motivational interviewing) represents a direct waste of invested resources, has costs associated with missed opportunities, and may affect an organization's ability to sustain future changes (Ford, Osborne, et al., 2018; Ford, Stumbo, et al., 2018; Ford, Zehner, et al., 2018). A similar argument can be made for organizations that do not achieve competency when utilizing a set of implementation strategies. By not achieving competency, the staff within an organization waste, the opportunity to institutionalize the knowledge of how to apply specific implementation strategies to future change efforts. As such, the ability to measure and assess organizational competency for implementation strategies represents an important advancement in D&I research.

Our results provide evidence that the SIC can be adapted for use with a specific set of implementation strategies. Specifically, the NIATx-SIC provides a tool to measure agency progression through the Preimplementation, Implementation, and Initiation to Sustainment phases when utilizing the NIATx implementation strategies—designed to improve and sustain access to care across multiple healthcare settings—to introduce change within an organization (Belenko et al., 2017; Gustafson et al., 2013; Hoffman et al., 2008, 2011; McCarty et al., 2007; Pankow et al., 2018; Quanbeck et al., 2011). This was the first adaptation of the SIC to measure change cycles, including the possibility for variable numbers of iterations between agencies, and time spent in each cycle. This application of the NIATx-SIC was able to distinguish between agencies that achieved competency in their application of the NIATx implementation strategies versus those agencies determined not to have achieved competency. This finding is important because these agencies did not differ in terms of organizational attributes except for nonprofit status. Thus, competency is not an attribute of the organization but rather a result of the application of the NIATx implementation strategies to improve agency co-occurring capacity. The differences in the proportion of activities completed during the implementation phase and the duration (i.e., time spent) in engaging in Stage 7 activities such as individual coaching calls, group coaching calls, in-person meetings, or implementation of change projects provide evidence that implementation strategy competency is a meaningful construct in D&I research.

Limitations

The study had several limitations. Information collection efforts for the implementation phase relied on self-reporting from the coach, in particular recall of when certain events occurred, and the availability of information (e.g., change project forms or walkthrough reports from the agency). As such, the dates associated with these activities could not be recorded for the SIC. These data gaps may have impacted proportion and duration scores. In addition, both the delay in defining competency in the NIATx-SIC and missing agency information to support agency self-report (e.g., change project forms) impacted the independent assessment of fidelity to the NIATx implementation strategies. As a result, the final determination of agency competency may have differed. Finally, agency attributes such as climate, leadership support, or staff commitment to the change process were not assessed as a part of this project. It is possible that one or more of these attributes may be associated with whether an agency achieved competency as measured by the NIATx-SIC instrument.

Conclusions

The development of instruments to assess organizational competency when the adoption and utilization of implementation strategies to drive change represents a significant advancement in dissemination and implementation research. Organizations that can apply implementation strategies with competence move one step closer to institutionalizing this knowledge and to fulfilling the sentiments of the adage that if you “teach a man to fish, you feed him for a lifetime.” Within D&I research, you are teaching an organization the skills of how to successfully implement change and hopefully sustain change so that they have the knowledge and skills to navigate current and future challenges.

Supplemental Material

sj-docx-1-irp-10.1177_26334895231200379 - Supplemental material for Adapting the stages of implementation completion to an evidence-based implementation strategy: The development of the NIATx stages of implementation completion

Supplemental material, sj-docx-1-irp-10.1177_26334895231200379 for Adapting the stages of implementation completion to an evidence-based implementation strategy: The development of the NIATx stages of implementation completion by James H. Ford, Mark E. Zehner, Holle Schaper and Lisa Saldana in Implementation Research and Practice

Supplemental Material

sj-docx-2-irp-10.1177_26334895231200379 - Supplemental material for Adapting the stages of implementation completion to an evidence-based implementation strategy: The development of the NIATx stages of implementation completion

Supplemental material, sj-docx-2-irp-10.1177_26334895231200379 for Adapting the stages of implementation completion to an evidence-based implementation strategy: The development of the NIATx stages of implementation completion by James H. Ford, Mark E. Zehner, Holle Schaper and Lisa Saldana in Implementation Research and Practice

Footnotes

Authors’ Note

Holle Schaper and Lisa Saldana's current affiliation is Chestnut Health Systems, Lighthouse Institute – Oregon Group, USA.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This manuscript was supported by funding from NIDA-funded research (R01DA037222 and R01DA052975-01A1, Drs. Ford and McGovern, principal investigators) and by grant R01DA044745 from NIDA (Dr. Saldana, principal investigator).

ORCID iD

Mark E. Zehner

James H. Ford

Supplemental Material

Supplemental material for this article is available online.

References

Aalsma

M. C.

Dir

A. L.

Zapolski

Hulvershorn

L. A.

Monahan

P. O.

Saldana

Adams

Z. W.

(2019). Implementing risk stratification to the treatment of adolescent substance use among youth involved in the juvenile justice system: Protocol of a hybrid type I trial. Addiction Science & Clinical Practice, 14(1), 36. https://doi.org/10.1186/s13722-019-0161-5

Assefa

M. T.

Ford

J. H.

2nd Osborne

McIlvaine

King

Campbell

McGovern

M. P.

(2019). Implementing integrated services in routine behavioral health care: Primary outcomes from a cluster randomized controlled trial. BMC Health Services Research, 19(1), 749. https://doi.org/10.1186/s12913-019-4624-x

Belenko

Visher

Pearson

Swan

Pich

O’Connell

Dembo

Frisman

Hamilton

Willett

(2017). Efficacy of structured organizational change intervention on HIV testing in correctional facilities. AIDS Education and Prevention, 29(3), 241–255. https://doi.org/10.1521/aeap.2017.29.3.241

Brown

C. H.

Chamberlain

Saldana

Padgett

Wang

Cruden

(2014). Evaluation of two implementation strategies in 51 child county public service systems in two states: Results of a cluster randomized head-to-head implementation trial. Implementation Science, 9, 134. https://doi.org/10.1186/s13012-014-0134-8

Campbell, M. K., Elbourne, D. R., Altman, D. G., & CONSORT group. (2004). CONSORT statement: extension to cluster randomised trials. BMJ, 328(7441):702–708. https://doi.org/10.1136/bmj.328.7441.702

Caton

Shen

Miele

G. M.

Darfler

Sandoval

J. R.

Urada

McGovern

M. P.

(2021). Opening the “black box”: Four common implementation strategies for expanding the use of medications for opioid use disorder in primary care. Implementation Research and Practice, 2. https://doi.org/10.1177/26334895211005809

Chamberlain

Brown

C. H.

Saldana

(2011). Observational measure of implementation progress in community based settings: The stages of implementation completion (SIC). Implementation Science, 6, 116. https://doi.org/10.1186/1748-5908-6-116

Chamberlain

Roberts

Jones

Marsenich

Sosna

Price

J. M.

(2012). Three collaborative models for scaling up evidence-based practices. Administration and Policy in Mental Health, 39(4), 278–290. https://doi.org/10.1007/s10488-011-0349-9

Chokron Garneau

Assefa

M. T.

Ford

J. H.

2nd Saldana

McGovern

M. P.

(2022). Sustainment of integrated care in addiction treatment settings: Primary outcomes from a cluster-randomized controlled trial. Psychiatric Services, 73(3), 280–286. https://doi.org/10.1176/appi.ps.202000293

10.

Dubowitz

Saldana

Magder

L. A.

Palinkas

L. A.

Landsverk

J. A.

Belanger

R. L.

Nwosu

U. S.

(2020). Protocol for comparing two training approaches for primary care professionals implementing the safe environment for every kid (SEEK) model. Implementation Science Communications, 1, 78. https://doi.org/10.1186/s43058-020-00059-9

11.

Ford

J. H.

2nd Abraham

A. J.

Lupulescu-Mann

Croff

Hoffman

K. A.

Alanis-Hirsch

Chalk

Schmidt

McCarty

(2017). Promoting adoption of medication for opioid and alcohol use disorders through system change. Journal of Studies on Alcohol and Drugs, 78(5), 735–744. https://doi.org/10.15288/jsad.2017.78.735

12.

Ford

J. H.

2nd Kaur

Rao

Gilson

Bolt

D. M.

Garneau

H. C.

Saldana

McGovern

M. P.

(2021). Improving medication access within integrated treatment for individuals with co-occurring disorders in substance use treatment agencies. Implementation Research and Practice, 2. https://doi.org/10.1177/26334895211033659

13.

Ford

J. H.

2nd Osborne

E. L.

Assefa

M. T.

McIlvaine

A. M.

King

A. M.

Campbell

McGovern

M. P.

(2018). Using NIATx strategies to implement integrated services in routine care: A study protocol. BMC health Services Research, 18(1), 431. https://doi.org/10.1186/s12913-018-3241-4

14.

Ford

J. H.

2nd Rao

Gilson

Kaur

Chokron Garneau

Saldana

McGovern

M. P.

(2022). Wait no longer: Reducing medication wait-times for individuals with co-occurring disorders. Journal of Dual Diagnosis, 18(2), 101–110. https://doi.org/10.1080/15504263.2022.2052225

15.

Ford

J. H.

2nd Saldana

Schaper

Campbell

Zehner

McGovern

(2016). Adaptation of the stages of implementation completion for NIATx implementation strategies. Addictions Health Services Research.

16.

Ford

J. H.

2nd Stumbo

S. P.

Robinson

J. M.

(2018). Assessing long-term sustainment of clinic participation in NIATx200: Results and a new methodological approach. Journal of Substance Abuse Treatment, 92, 51–63. https://doi.org/10.1016/j.jsat.2018.06.012

17.

Ford

J. H.

2nd Zehner

Hoffman

K. A.

McGovern

(2018, December 3–5). Development and testing of a tool to assess fidelity to NIATx implementation strategies [Paper presentation]. Proceedings of the 11th Annual Conference on the Science of Dissemination and Implementation in Health, Washington DC, United States.

18.

Garner

B. R.

Gotham

H. J.

Chaple

Martino

Ford

J. H.

Roosa

M. R.

Speck

K. J.

Vandersloot

Bradshaw

Ball

E. L.

Toro

A. K.

Griggs

Tueller

S. J.

(2020). The implementation and sustainment facilitation strategy improved implementation effectiveness and intervention effectiveness: Results from a cluster-randomized, type 2 hybrid trial. Implementation Research and Practice, 1. https://doi.org/10.1177/2633489520948073

19.

Gustafson

D. H.

Quanbeck

A. R.

Robinson

J. M.

Ford

J. H.

2nd Pulvermacher

French

M. T.

McConnell

K. J.

Batalden

P. B.

Hoffman

K. A.

McCarty

(2013). Which elements of improvement collaboratives are most effective? A cluster-randomized trial. Addiction, 108(6), 1145–1157. https://doi.org/10.1111/add.12117

20.

Hoffman

K. A.

Ford

J. H.

2nd Choi

Gustafson

D. H.

McCarty

(2008). Replication and sustainability of improved access and retention within the network for the improvement of addiction treatment. Drug and Alcohol Dependence, 98(1–2), 63–69. https://doi.org/10.1016/j.drugalcdep.2008.04.016

21.

Hoffman

K. A.

Ford

J. H.

Tillotson

C. J.

Choi

McCarty

(2011). Days to treatment and early retention among patients in treatment for alcohol and drug disorders. Addictive Behaviors, 36(6), 643–647. https://doi.org/10.1016/j.addbeh.2011.01.031

22.

Hoffman

K. A.

Green

C. A.

Ford

J. H.

2nd Wisdom

J. P.

Gustafson

D. H.

McCarty

(2012). Improving quality of care in substance abuse treatment using five key process improvement principles. The Journal of Behavioral Health Services & Research, 39(3), 234–244. https://doi.org/10.1007/s11414-011-9270-y

23.

McCarty

Gustafson

D. H.

Wisdom

J. P.

Ford

Choi

Molfenter

Capoccia

Cotter

(2007). The network for the improvement of addiction treatment (NIATx): Enhancing access and retention. Drug and Alcohol Dependence, 88(2–3), 138–145. https://doi.org/10.1016/j.drugalcdep.2006.10.009

24.

Molfenter

McCarty

Jacobson

Kim

J. S.

Starr

Zehner

(2019). The payer’s role in addressing the opioid epidemic: It’s more than money. Journal of Substance Abuse Treatment, 101, 72–78. https://doi.org/10.1016/j.jsat.2019.04.001

25.

Nadeem

Saldana

Chapman

Schaper

(2018). A mixed methods study of the stages of implementation for an evidence-based trauma intervention in schools. Behavior Therapy, 49(4), 509–524. https://doi.org/10.1016/j.beth.2017.12.004

26.

Palinkas

L. A.

Campbell

Saldana

(2018). Agency leaders’ assessments of feasibility and desirability of implementation of evidence-based practices in youth-serving organizations using the stages of implementation completion. Frontiers in Public Health, 6, 161. https://doi.org/10.3389/fpubh.2018.00161

27.

Palinkas

L. A.

Saldana

Chou

C. P.

Chamberlain

(2017). Use of research evidence and implementation of evidence-based practices in youth-serving systems. Children and Youth Services Review, 83, 242–247. https://doi.org/10.1016/j.childyouth.2017.11.005

28.

Pankow

Willett

Yang

Swan

Dembo

Burdon

W. M.

Patterson

Pearson

F. S.

Belenko

Frisman

L. K.

(2018). Evaluating fidelity to a modified NIATx process improvement strategy for improving HIV services in correctional facilities. The Journal of Behavioral Health Services & Research, 45(2), 187–203. https://doi.org/10.1007/s11414-017-9551-1

29.

Perry

C. K.

Damschroder

L. J.

Hemler

J. R.

Woodson

T. T.

Ono

S. S.

Cohen

D. J.

(2019). Specifying and comparing implementation strategies across seven large implementation interventions: A practical application of theory. Implementation Science, 14(1), 32. https://doi.org/10.1186/s13012-019-0876-4

30.

Powell

B. J.

Waltz

T. J.

Chinman

M. J.

Damschroder

L. J.

Smith

J. L.

Matthieu

M. M.

Proctor

E. K.

Kirchner

J. E.

(2015). A refined compilation of implementation strategies: Results from the expert recommendations for implementing change (ERIC) project. Implementation Science, 10(1), 21. https://doi.org/10.1186/s13012-015-0209-1

31.

Quanbeck

A. R.

Gustafson

D. H.

Ford

J. H.

2nd Pulvermacher

French

M. T.

McConnell

K. J.

McCarty

(2011). Disseminating quality improvement: Study protocol for a large cluster-randomized trial. Implementation Science, 6, 44. https://doi.org/10.1186/1748-5908-6-44

32.

Roosa

Scripa

J. S.

Zastowny

T. R.

Ford

J. H.

2nd (2011). Using a NIATx based local learning collaborative for performance improvement. Evaluation and Program Planning, 34(4), 390–398. https://doi.org/10.1016/j.evalprogplan.2011.02.006

33.

Saldana

(2014). The stages of implementation completion for evidence-based practice: Protocol for a mixed methods study. Implementation Science: IS, 9(1), 43. https://doi.org/10.1186/1748-5908-9-43

34.

Saldana

Bennett

Powers

Vredevoogd

Grover

Schaper

Campbell

(2020). Scaling implementation of collaborative care for depression: Adaptation of the stages of implementation completion (SIC). Administration and Policy in Mental Health, 47(2), 188–196. https://doi.org/10.1007/s10488-019-00944-z

35.

Saldana

Chamberlain

(2012). Supporting implementation: The role of community development teams to build infrastructure. American Journal of Community Psychology, 50(3–4), 334–346. https://doi.org/10.1007/s10464-012-9503-0

36.

Saldana

Chamberlain

Bradford

W. D.

Campbell

Landsverk

(2014). The cost of implementing new strategies (COINS): A method for mapping implementation resources using the stages of implementation completion. Children and Youth Services Review, 39, 177–182. https://doi.org/10.1016/j.childyouth.2013.10.006

37.

Saldana

Chamberlain

Wang

Hendricks Brown

(2012). Predicting program start-up using the stages of implementation measure. Administration and Policy in Mental Health, 39(6), 419–425. https://doi.org/10.1007/s10488-011-0363-y

38.

Saldana

Schaper

Campbell

Chapman

(2015). Standardized measurement of implementation: The universal SIC. Implementation Science, 10(1), A73. https://doi.org/10.1186/1748-5908-10-S1-A73

39.

Sterrett-Hong

E. M.

Saldana

Burek

Schaper

Karam

Verbist

A. N.

Cameron

(2021). An exploratory study of a training team-coordinated approach to implementation. Global Implementation Research and Applications, 1(1), 17–29. https://doi.org/10.1007/s43477-020-00003-y

40.

Waltz

T. J.

Powell

B. J.

Fernández

M. E.

Abadie

Damschroder

L. J.

(2019). Choosing implementation strategies to address contextual barriers: Diversity in recommendations and future directions. Implementation Science, 14(1), 42. https://doi.org/10.1186/s13012-019-0892-4

41.

Waltz

T. J.

Powell

B. J.

Matthieu

M. M.

Damschroder

L. J.

Chinman

M. J.

Smith

J. L.

Proctor

E. K.

Kirchner

J. E.

(2015). Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: Results from the expert recommendations for implementing change (ERIC) study. Implementation Science, 10(1), 109. https://doi.org/10.1186/s13012-015-0295-0

42.

Watson

D. P.

Snow-Hill

Saldana

Walden

A. L.

Staton

Kong

Donenberg

(2020). A longitudinal mixed method approach for assessing implementation context and process factors: Comparison of three sites from a housing first implementation strategy pilot. Implementation Research and Practice, 1. https://doi.org/10.1177/2633489520974974

43.

Weir

Presseau

Kitto

Colman

Hatcher

(2021). Strategies for facilitating the delivery of cluster randomized trials in hospitals: A study informed by the CFIR-ERIC matching tool. Clinical Trials, 18(4), 398–407. https://doi.org/10.1177/17407745211001504

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.07 MB

0.06 MB