Sage Journals: Discover world-class research

Abstract

Background

Strategies to implement evidence-based practices often require modifications. A systematic approach to documenting these changes was not widely adopted until the Framework for Reporting Adaptations and Modifications to Evidence-Based Implementation Strategies (FRAME-IS) emerged in 2021, enabling researchers to characterize both proactive and reactive implementation changes. While publications demonstrating the FRAME-IS's application are emerging, few have reflected on the use of the tool itself. The National Institutes of Health-funded Deidentified Opioid Initiative R01 trial, testing strategies to implement the Centers for Disease Control and Prevention guidelines on opioid prescribing, offered a timely chance to assess the FRAME-IS's utility in a multisite, hybrid type-3 trial.

Method

An interdisciplinary team of researchers, clinicians, and implementers documented modifications using the FRAME-IS across four implementation strategies that comprise an implementation package called systems consultation: (1) audit and feedback, (2) educational meetings, (3) practice facilitation, and (4) prescriber peer consulting. Modifications were needed due to COVID-19, the rise in telemedicine, changes in opioid prescribing, and healthcare system variations.

Results

The Deidentified Opioid Initiative was implemented in 32 clinics within two Midwestern healthcare systems using a sequential, multiple-assignment randomized trial. The implementation team completed the FRAME-IS's seven modules for each strategy's modifications and reflected on the process of using the tool, strengths, and limitations.

Conclusions

The team found the FRAME-IS is practical, comprehensive, and user-friendly. It effectively documents modifications and fosters reflection, raising critical questions about implementation. Challenges included role blurring (i.e., researcher/implementer/coordinator), capturing the complexity of cascading modifications (i.e., how one modification leads to another), and a lack of reporting options to capture modifications in a clustered, multisite trial (i.e., clinical staff nested in clinics nested in healthcare systems). Considerations and recommendations from this case study can enhance the FRAME-IS, guide other scholars in its use, and improve the research community's ability to measure the dynamic evolution of implementation strategies systematically. Future research should explore how documented modifications impact implementation outcomes.

Plain Language Title

Using and improving a framework to document modifications to implementation strategies: Lessons learned from opioid prescribing

Plain Language Summary

Implementation science increases the use of best practices known to improve healthcare and public health, ensuring that all people can have access to the best care and prevention services. Implementation strategies are the specific methods or approaches used to help clinics and communities put these best practices into routine use. Numerous implementation strategies have been used to help different clinics and communities implement these best practices. However, these strategies are rarely conducted exactly as intended, with unforeseen circumstances requiring modifications to implementation strategies. In our study, the Deidentified Opioid Initiative, we tested how four implementation strategies impact how clinicians adhere to opioid prescribing guidelines from the Centers for Disease Control and Prevention. These strategies involved education (educational meetings), providing data on different opioid prescribing metrics (audit and feedback), a guided facilitation process with clinic teams to improve adherence to prescribing guidelines (practice facilitation), and opportunities for physicians to discuss challenges when implementing guidelines (prescriber peer consulting). COVID-19 and other challenges required us to modify these implementation strategies. Before we can study how these modifications influenced implementation, we needed to clarify what changes were made and document them systematically. To do so, we used a tool called the Framework for Reporting Adaptations and Modifications to Evidence-Based Implementation Strategies (FRAME-IS). We completed all the FRAME-IS modules to document modifications to each strategy. Our findings provide insights into how the FRAME-IS can be applied in real-world settings and offer suggestions to make the tool more useful for researchers and practitioners.

Keywords

implementation science modification implementation strategy audit and feedback educational meetings practice facilitation hybrid intervention-implementation research design opioid prescription opioid

Introduction

Modifications are impromptu changes to programming due to unexpected circumstances (Mark, 2021). In implementation science, modifications to evidence-based programs and practices—and the strategies to implement them—are common, but detailed documentation has been virtually nonexistent in the literature (Miller et al., 2021). Additionally, journal requirements limit scholars’ ability to provide rich detail of modifications with study outcomes. Without knowing how interventions are modified, other implementers cannot understand—and other researchers cannot study—how interventions are implemented in real-world settings.

To examine this, we used the Framework for Reporting Adaptations and Modifications to Evidence-based Implementation Strategies (FRAME-IS; Miller et al., 2021) to document modifications made to implementation strategies in the Deidentified Opioid Initiative (DOI), a hybrid type-3 implementation-effectiveness trial (Quanbeck at al., 2020). The FRAME-IS provides modules to detail the nature, rationale, scope, and potential impact of each modification to an implementation strategy (Miller et al., 2021). Documenting modifications using the FRAME-IS allows for transparency and helps clarify how strategies influence implementation outcomes (Miller et al., 2021).

In hybrid type-3 trials, the primary function is to test implementation strategies while gathering information on the impacts of the clinical intervention itself (Curran et al., 2012). In DOI, the innovation/intervention was the “CDC Guideline for Prescribing Opioids for Chronic Pain—United States, 2016” (Dowell et al., 2016), and the goal was to test different implementation strategies to increase the use of its 12 recommendations among primary care clinicians who prescribe opioids for chronic pain.

The goals of this report are to systematically describe the modifications made when implementing the DOI and to explore the FRAME-IS as a tool to provide transparency in study execution for the research team, funders, collaborators, and other researchers looking to understand and replicate implementation strategies. There is a growing literature using the FRAME-IS to document modifications (Miller et al., 2024; Schoenthaler et al., 2024), and this paper provides a novel perspective by using a team-based approach to complete the FRAME-IS and providing feedback to improve the framework within the context of a multilevel, multisite, hybrid type-3 National Institutes of Health (NIH)-funded R01 study.

Context of This Study: The DOI—An NIH-Funded Clinical Trial

The DOI was an NIH-funded, 5-year clinical trial (trial number NCT 04044521) that aimed to increase adherence to the 2016 Centers for Disease Control and Prevention (CDC) opioid prescribing guidelines in primary care through a multilevel deimplementation strategy named systems consultation (Quanbeck at al., 2020; Dowell et al., 2016). The clinical goal of this hybrid trial was to improve adherence to the intervention/innovation of “opioid prescribing guidelines” within primary care; the implementation goal was to test systems consultation as the strategy to achieve this. Systems consultation is a combination of four implementation strategies intended to increase guideline-concordant opioid prescribing for patients with chronic pain: educational meetings, audit and feedback reports, practice facilitation, and prescriber peer consulting. Table 1 describes each implementation strategy that comprises systems consultation using Proctor et al.'s (2013) reporting recommendations, including actor, action, action target, temporality, dose/timeline, implementation outcome affected, and justification. A narrative overview of each strategy follows below.

Table 1.

Implementation Strategies Comprising Systems Consultation Before Modifications.

Systems Consultation Strategy Specification^a	Educational Meetings	Audit and Feedback	Practice Facilitation	Prescriber Peer Consulting
Name it (ERIC name)^b	Conduct educational meetings	Audit and provide feedback	Facilitation	Provide ongoing consultation
Define it (ERIC definition)^b	Hold meetings targeted toward different stakeholder groups to teach them about the clinical innovation	Collect and summarize clinical performance data over a specified time period and give it to clinicians and administrators to monitor, evaluate, and modify provider behavior	A process of interactive problem solving and support that occurs in a context of a recognized need for improvement and a supportive interpersonal relationship	Provide ongoing consultation with one or more experts in the strategies used to support implementing the innovation
Define it (level)	System-level strategy, provided to all clinic change teams; low-intensity strategy	System-level strategy, provided to all clinic change teams; low-intensity strategy	Clinic-level strategy, randomized to clinics; medium-intensity strategy	Prescriber-level strategy, randomized at the clinic-level to all prescribers within each clinic; high-intensity strategy
Specify it: The actor	Various physicians experienced with opioid prescribing in primary care	Data manager for the electronic medical records and implementation program manager	External facilitation team with experience in healthcare	Physicians experienced in opioid management
The action	Provides education on the current state of opioid prescribing in-person and through webinars	Customizes and provides performance feedback reports on opioid prescribing metrics to each clinic	Targets clinic processes and workflows related to opioid prescribing	Utilizes consultation between physicians with challenging panels of patients and the consultant
Action target	Clinic change teams^c	Clinic change teams^c	Clinic change teams^c	Prescribers
Temporality	Once change teams are assembled (beginning of the implementation process in February 2020)	Once change teams are assembled (beginning of the implementation process in February 2020)	Three months after educational meetings and audit and feedback begin (randomized in October 2020)^d	Nine months after educational meetings and audit and feedback begin (randomized in July 2021)^d
Dose	Six quarterly meetings	Monthly reports	Six monthly sessions followed by four quarterly meetings	Four quarterly sessions
Implementation Outcome Affected	Uptake of the innovation, fidelity to the protocol of the clinical innovation, acceptability of the innovation based on science	Uptake of the innovation, fidelity to the protocol of the clinical innovation, penetration among eligible providers to their patients	Uptake of the innovation, fidelity to the protocol of the clinical innovation, feasibility within unique clinic settings, decrease of the implementation cost of personnel time by streamlining processes, sustainability of processes to support the innovation	Uptake of the innovation, fidelity to the protocol of the clinical innovation, penetration among eligible providers to their patients
Justification	Low-intensity strategies, including educational meetings and audit and feedback, yield small to modest changes in practice within primary care (2%–9%; Lau et al., 2015). However, due to wide variations in these effects, it is possible that some clinics may be successful with only low-dose strategies, and when used in combination with higher-intensity strategies educational meetings and audit and feedback may help facilitate change (Quanbeck et al., 2020)		Practice facilitation with continuous follow-up contributes strongly to successful organizational change in healthcare settings and has a stronger evidence base than educational meetings and audit and feedback reports alone (Alagoz et al., 2018)	Our preliminary research suggests that physician-to-physician interaction may the most effective way to change prescribing behavior (Quanbeck et al., 2020)

Specified using Proctor et al. (2013).

Defined using Powell et al. (2015).

Clinic change teams consist of a change team leader, frequently the clinic's medical director or clinic manager, along with clinic staff involved in the opioid prescription process, including physicians, nurse practitioners, physician assistants, nurses, medical assistants, front-line staff, lab techs, and others.

These timeframes are delayed, accounting for the pause in the study from March 2020 to July 2020 due to COVID-19.

Implementation Process

Before implementing systems consultation, each clinic's medical director appointed a change team leader and three to seven staff members to form an implementation team to increase uptake to the innovation/intervention of the study, defined as improved adherence to opioid prescribing guidelines. Members of the clinic change team, including primary care prescribers, were the intended audience of systems consultation. Implementation outcomes were measured at the clinician level and aggregated by clinic (i.e., the percentage of patients per clinician treated according to each recommendation of the clinical guidelines, aggregated at the clinic level so all clinician data for each clinic was presented together).

Implementation Strategies

The first implementation strategy of systems consultation included six quarterly educational meetings focused on understanding guidelines for opioid prescribing. Meetings were implemented separately within the two participating healthcare organizations and were provided to the clinic change teams of each clinic. The first meeting was intended to be in-person, with all subsequent meetings held online. Meetings focused on sharing opioid prescribing guideline content, building enthusiasm for improving adherence to guidelines, and using expert physician trainers.

The second strategy, audit and feedback, involved monthly clinic-level feedback reports on opioid prescribing metrics reflecting CDC guidelines aggregated at the clinician level. Quanbeck et al. (2024) include the metrics corresponding to the CDC guidelines provided in each report. The core components of audit and feedback included the delivery of clinic-specific monthly reports customized for each clinic with metrics on opioid prescribing guidelines aggregated at the clinic level (e.g., number of opioid patients, the average opioid dosage per clinician, etc.).

Practice facilitation targeted clinic processes and workflows using an external change agent to set and monitor goals with clinic change teams in six monthly meetings followed by four quarterly meetings. Core components involved a facilitator working with each clinic change team to develop an opioid refill workflow, review audit and feedback reports, create goals from each clinic change team, and review progress on meeting those goals.

Prescriber peer consulting involved consultation sessions with a physician experienced in opioid prescribing to help prescribers manage their patients on long-term opioids using four videoconferencing sessions. Core components included a protocol process to share individual cases of patients from clinicians and a discussion on the challenges and possible solutions to improve adherence to clinical opioid prescribing guidelines.

Combined together, these implementation strategies comprised systems consultation. Systems consultation was implemented between 2020 and 2022 using a sequential, multiple-assignment randomized trial design to determine the best sequence of strategies for different clinics in the real world based on intensity. Participation included 32 primary care clinics within two healthcare systems in Wisconsin and Michigan located in multiple rural communities and two metropolitan areas. Specific clinic inclusion criteria are detailed in Quanbeck et al. (2024) and generally required clinics to employ clinicians prescribing opioids to adults and show room for improvement on opioid prescribing guidelines (all recruited clinics demonstrated room to improve adherence). Clinics were recruited via volunteer sampling and included 268 clinicians in permanent roles with opioid prescribing privileges and 8,978 eligible patients. Patients met eligibility criteria if there was documentation of at least one opioid prescription within the most recent 12 months in the electronic health record (patients with cancer or in hospice were excluded per the 2016 opioid prescribing guidelines that exclude this population). Based on electronic medical record data, the mean age of patients was 58.3 years (SD = 14) and 57.3% of patients were female. Patients self-identified as 0.5% American Indian or Alaska Native, 0.8% Asian or Pacific Islander, 4.6% Black, 2.1% Hispanic or Latino, and 90.5% White, while 1.5% were unreported or unknown. Additional study population details can be found in Quanbeck et al. (2024).

Clinics received different implementation strategies of systems consultation based on randomization. All clinics received quarterly educational meetings on opioid prescribing and monthly customized audit/feedback reports on opioid metrics. At month three, half the clinics were randomized to receive practice facilitation. At month nine, half the clinics were re-randomized to prescriber peer consulting. This resulted in four groups. See Figure 1 for the study design and Quanbeck et al. (2020) for the protocol and additional details on implementation strategies. Clinical outcomes of the DOI are reported in Quanbeck et al. (2024).

Figure 1.

The Deidentified Opioid Initiative Study Design.

Factors Influencing the Need for Modification

There were multiple factors across societal, system, and clinic levels that created the need for modifications in the DOI. These factors, particularly COVID-19, led to ad hoc changes. COVID-19 postponed many studies and demanded virtual methods of conduct in many proceeding studies (Audisio et al., 2022). The rise of telemedicine, or phone/video appointments due to the pandemic, shifted physician–patient contact from primarily in-person to increased virtual care, disrupting workflows and creating challenges in achieving guideline adherence. While COVID-19 led to many modifications to the study, modifications were also required to accommodate variations between the two healthcare systems that implemented the DOI along with evolving opioid prescribing guidelines, as these 2016 guidelines were updated in 2022 during our study. The DOI presented a unique opportunity to utilize the FRAME-IS to explore how four different implementation strategies were modified to increase the uptake of guideline-concordant opioid prescribing practices in primary care, while those guidelines were actively being modified. Each of these factors influencing the need for modification is described further in the Supplemental Appendix.

Method

The DOI was implemented from February 2020 to March 2022 (with a pause from March to July 2020 due to COVID-19). In April 2022, three key team members met to discuss the modifications required for the implementation strategies within systems consultation and desired a way to examine these changes more systematically. The PI sought out and shared the FRAME-IS, published in 2021 (Miller et al., 2021). The research team conceptualized the paper in the following month and scheduled a full team meeting with implementers of systems consultation. In June 2022, a team of seven researchers, clinicians, and implementers of systems consultation met to retrospectively apply the FRAME-IS to document modifications to the implementation strategies.

Team members represented most implementation staff across both healthcare systems. For educational meetings and audit and feedback, all implementers directly participated in the documentation of modifications. The implementers who documented modifications to practice facilitation and prescriber peer consulting had either implemented the strategies or directly observed all other implementers within every participating clinic. At one point, the primary facilitator in one healthcare system ended up facilitating at least one session with every participating clinic due to staff turnover. For sessions he did not facilitate, he trained, observed, and directly supervised the employees who conducted the facilitations. Both facilitators in the other healthcare system documented modifications with the full team. Every prescriber peer consulting session in one healthcare system was observed and documented by a member of the research team. In the other healthcare system one physician consultant documented the FRAME-IS directly and consulted the other physician consultant in this healthcare system. In summary, modifications were recalled by the primary implementers of each strategy, all of whom also observed or directly interacted with the other implementers not on the research team.

Information was validated in two ways, through group consensus in full team meetings and document review. First, the full team refined the documented modifications in meetings. Second, previous meeting notes were consulted to clarify discussions and identify additional modifications not recalled by implementers. The research team conducted meetings for both educational meetings and to update audit and feedback reports. Practice facilitators across both healthcare systems met monthly to discuss implementation challenges and successes and their session notes were reviewed for modifications. Prescriber peer consultants had similar full group meetings. Reviewing meeting notes from these sessions did not yield additional modifications beyond those already discussed by the team completing the FRAME-IS.

Before the full team meeting of seven implementers/researchers, all members were encouraged to read about the FRAME-IS to understand the framework and prepare questions (Miller et al., 2021). The team found it helpful to first discuss the primary needs for modifications and subsequently outline each major modification made to systems consultation. The FRAME-IS modules were then completed separately for each implementation strategy by a lead implementer, including the program coordinator/researcher for audit and feedback, a researcher for educational meetings who had a key role in outlining and compiling content from physician trainers, a pharmacist for practice facilitation, and a physician for prescriber peer consulting. After initially completing the modules, the team met again to provide feedback, clarify questions, and discuss strengths, challenges, and recommendations for the framework and the study. The modifications, core modules, and optional modules are provided for the four implementation strategies below (see Figure 2 reproduced from Miller et al. [2021] for a summary of each module). The DOI study was approved by the IRB of the institution of the corresponding author. Informed consent was obtained from clinic staff members participating in the research. A waiver of informed consent was obtained at the patient level for outcomes assessment in the parent trial (Figure 2).

Figure 2.

The FRAME-IS: (a) core modules and (b) optional modules. Note. Figures Reproduced Directly From Miller et al. (2021) Under the Creative Commons Attribution 4.0 International License: https://creativecommons.org/licenses/by/4.0/. Note. FRAME-IS = Framework for Reporting Adaptations and Modifications to Evidence-Based Implementation Strategies.

Results

Modifications to Educational Meetings

Educational meetings were implemented system-wide by experienced opioid-prescribing physicians. The first educational meeting occurred remotely in one healthcare system. See the completed FRAME-IS modules in Table 2.

Table 2.

Educational Meeting Modifications with the FRAME-IS.

FRAME-IS Module or Subcomponent		Modification
Module 1
	The evidence-based practice (EBP) being implemented is:	Adherence to opioid prescribing guidelines through systems consultation
	The implementation strategy being modified is:	Educational meetings
	The modification(s) being made is/are:	Remote delivery for the first educational meeting
	The reason(s) for the modifications(s) is/are:	COVID-19 lockdowns and distancing recommendations
Module 2
	What is modified?	Context: Format (intended online interaction); Setting (remote and recorded); Personnel (multiple trainers)
Module 3
	What is the nature of the content, evaluation, or training modification?	Changes in packaging or materials (remote delivery); Adding elements (additional trainers)
	What is the relationship to core elements?	Fidelity Consistent/Core elements or functions preserved
Module 4
	What is the goal?	Increase reach of the EBP (the number of clinic staff and physicians receiving the educational meeting); Other: Increase safety during COVID-19
	What is the LEVEL of the rationale for modification?	Sociopolitical level (national and state mandates)
Module 5
	When is the modification initiated?	Implementation phase
	Is the modification planned?	Planned/Reactive (reactive adaptation)
Module 6
	Who participates in the decision to modify?	Political leaders (mandating COVID-19 lockdowns)
	Who makes the ultimate decision?	Research team (modifying content to online delivery)
Module 7
	How widespread is the modification?	Organization (to all clinic change teams within one healthcare system)

Note. Regular text indicates predetermined categories in the framework. Italic text is written by the authors. FRAME-IS = Framework for Reporting Adaptations and Modifications to Evidence-Based Implementation Strategies.

Before the study was paused, the first meeting was conducted in-person for one healthcare system as intended. However, the second healthcare system held its first meeting remotely due to government-mandated COVID-19 lockdowns that changed the context of delivery. In lieu of in-person conversations, trainers and participants interacted in the chat and with unmuted dialogue. Personnel also increased since the remote format allowed multiple physicians to present simultaneously. The research team determined that the remote delivery of the first education meeting retained fidelity as measured by adherence (content, coverage, frequency, dosage) and by retaining core concepts (see Carroll et al. [2007] for this approach to assessing fidelity). However, some moderators of implementation fidelity (such as quality of delivery or participant responsiveness) may have decreased in an online format. This modification only applied to the first meeting; all subsequent meetings were planned online.

Modifications to Audit and Feedback Reports

Monthly audit and feedback reports were provided to each clinic with aggregated clinic-level metrics reflecting opioid prescribing guidelines. Report content and delivery were modified over time. See the completed FRAME-IS modules in Table 3.

Table 3.

Audit and Feedback Report Modifications with the FRAME-IS.

FRAME-IS Module or Subcomponent	Modification
Module 1
The evidence-based practice (EBP) being implemented is:	Adherence to opioid prescribing guidelines through systems consultation
The implementation strategy being modified is:	Audit and Feedback
The modification(s) being made is/are:	Changes to monthly reports: Additional data provided and consolidated clinic reports
The reason(s) for the modifications(s) is/are:	Consolidation/deconsolidation of clinics; Increased use of telemedicine
Module 2
What is modified?	Content (additional data); Context Other: Consolidation and deconsolidation of which clinics are included in each report
Module 3
What is the nature of the content, evaluation, or training modification?	Tailoring/tweaking/refining (adding/removing clinics); Adding elements (in-person visits/all visits)
What is the relationship to core elements?	Fidelity consistent/Core elements or functions preserved
Module 4
What is the goal?	Increase the acceptability, appropriateness, or feasibility of the implementation effort
What is the LEVEL of the rationale for modification?	Organizational level (telemedicine and consolidations)
Module 5
When is the modification initiated?	Implementation phase
Is the modification planned?	Unplanned/Reactive (modification)
Module 6
Who participates in the decision to modify?	Researcher;Implementer or implementation strategy expert; Clinicians being asked to use the EBP
Who makes the ultimate decision?	Research team; Clinicians being asked to use the EBP
Module 7
How widespread is the modification?	Network system/community (telemedicine); Clinic/unit (consolidations)

Reports were provided automatically via electronic medical record (EMR) in one system and manually by the program coordinator in the other, for every monthly report. Additionally, the consolidation and deconsolidation of some clinics within one healthcare system required the research team to regularly update reports to reflect patients moving to nonparticipating clinics. The clinic change updates were ongoing throughout the study. Consolidated clinics requested clarification and updates on metric calculations, prompting participant-initiated, unplanned modifications that were completed by the program coordinator. An updated metric was included to distinguish in-person versus virtual visits at the request of clinic change team members, due to the rise of telemedicine. Telehealth affected other metrics, especially completion of annual treatment agreements, urine drug screens, and pain/functionality screening during virtual “rooming.” Additionally, some clinics chose to update telemedicine workflows for opioid refill processes. The new metric distinguishing in-person from telehealth visits allowed staff to better track guideline implementation during high telemedicine use. This specification was implemented by the program coordinator after requesting additional data from the healthcare system's EMR data manager. This request was implemented in 2020 and continued throughout systems consultation. Our team determined these content changes maintained fidelity to the core components of audit and feedback while enabling clinicians and staff to monitor prescribing and inform implementation in real time.

Modifications to Practice Facilitation

Practice facilitation was implemented to review clinic metrics from audit and feedback reports, document prescribing workflows, set goals for improvement, and check on progress monthly for 6 months and then quarterly for 1 year. Two major modifications were made: all sessions were switched to remote delivery across both healthcare systems and external practice facilitators were replaced with novice facilitators of varying professional expertise in each healthcare system. See the completed FRAME-IS modules in Table 4.

Table 4.

Practice Facilitation Modifications with the FRAME-IS.

FRAME-IS Module or Subcomponent		Modification 1	Modification 2
Module 1
	The evidence-based practice (EBP) being implemented is:	Adherence to opioid prescribing guidelines through systems consultation	Adherence to opioid prescribing guidelines through systems consultation
	The implementation strategy being modified is:	Practice facilitation	Practice facilitation
	The modification(s) being made is/are:	All remote practice facilitation	Using internal facilitators (instead of external facilitators): Pharmacists in one healthcare system and researchers in another healthcare system
	The reason(s) for the modifications(s) is/are:	COVID-19	Internal relationships and resource consolidation
Module 2
	What is modified?	Context:Format (intended in-person); Setting (remote, virtual meeting); Personnel (multiple trainers) Training: Modifications to the ways implementers are trained	Context:Personnel (change in facilitators)
Module 3
	What is the nature of the content, evaluation, or training modification?	Tailoring/tweaking/refining; Loosening structure; Shortening/condensing	Substituting; Tailoring/tweaking/refining; Loosening structure
	What is the relationship to core elements?	Fidelity Consistent/Core elements or functions preserved	Fidelity Consistent/Core elements or functions preserved
Module 4
	What is the goal?	Increase the acceptability, appropriateness, and feasibility of the implementation effort	Decrease costs of the implementation effort; Increase the acceptability, appropriateness, or feasibility of the implementation effort; Improve sustainability of the EBP
	What is the LEVEL of the rationale for modification?	Sociopolitical level	Implementer level
Module 5
	When is the modification initiated?	Implementation phase	Implementation phase
	Is the modification planned?	Planned/Reactive (Reactive adaptation)	Planned/Reactive (Reactive adaptation)
Module 6
	Who participates in the decision to modify?	Political leaders	Program leader, Manager, or Administrator; Implementer or implementation strategy expert; Researcher
	Who makes the ultimate decision?	Research team	Research team
Module 7
	How widespread is the modification?	Network system/community	Implementation/facilitation team

All activities were conducted remotely by facilitators due to COVID-19 regulations. The first facilitation activity was intended to be an “in-person walkthrough” of the opioid prescription refill process. In this process, the facilitator and change team leader would act as one mock patient requesting a refill for their opioid prescription to experience the process firsthand. Instead, clinics described the steps verbally. We believe this substitution introduced recall and reporting bias, as descriptions likely reflected what was “supposed” to happen rather than what actually occurred. Bias was also possible because facilitators only heard from a few knowledgeable staff. To help reduce bias, facilitators shared the developed refill workflows with the clinic change teams to gather feedback and updates. Facilitators also encouraged change teams to share their workflows (in additional outside meetings) to collect feedback from all clinic staff involved in the opioid refill process.

This remote format also applied to all subsequent sessions of practice facilitation and may have improved or inhibited different aspects of practice facilitation. Online practice facilitation improved scheduling and attendance, especially benefiting clinics with remote workers who would not have joined in-person. This format also allowed for greater flexibility to reschedule due to greater patient demand or staff illness. However, technological issues occasionally limited participation, for example, when individuals shared devices, had audio issues, or left cameras off. This made it difficult for facilitators to engage all participants and gauge nonverbal cues. Despite these changes, all core components of practice facilitation were maintained. These included developing the opioid refill workflow, reviewing monthly audit and feedback reports, creating clinic-specific goals to increase guideline adherence, and progressing on goals set by each change team.

The second modification to practice facilitation was a change in facilitator roles. Before facilitation started, the principal investigator chose to replace an external facilitation team with facilitators already involved in the study due to budget and contract necessities. In one health system, a lead pharmacist was selected due to his existing membership on primary care teams. He recruited additional pharmacists from his pharmacy. Two members of the research team who worked in the other healthcare system served as facilitators within that system. Facilitator roles remained consistent throughout: only pharmacists facilitated in one system and only researchers in the other. Since none of the pharmacists or researchers had experience with practice facilitation, facilitator training and check-ins were added. A previous practice facilitator from the DOI pilot study (Brown et al., 2018) trained the new team and offered suggestions on adapting practice facilitation to a remote experience. Mock facilitation sessions were conducted, templates were shared, and questions were answered. Practice facilitators continued to meet throughout the study without the trainer to discuss challenges and progress. Pharmacist facilitators turned over, leaving the lead pharmacist to facilitate many additional sessions and onboard new pharmacists. During check-in meetings among facilitators, differences emerged between researcher- and pharmacist-led facilitation experiences. Reported differences included the types of goals set, questions asked, facilitators’ ability to answer those questions, power dynamics between clinic change team members and facilitators, and the overall communication between meetings. Despite these differences, the core components of facilitation were maintained across both healthcare systems. However, the team acknowledges that facilitator background—pharmacist versus researcher—possibly influenced the moderators of implementation fidelity such as participant responsiveness or quality of delivery.

Changes in Prescriber Peer Consulting (Consulting)

Consulting was a prescriber-level intervention that targeted the management of challenging patients or panels of patients through consultation between primary care clinicians (Delbecq et al., 1976) and expert physicians with extensive experience in chronic pain management. Four major modifications were made: (1) research staff supported and observed virtual sessions, (2) some sessions deviated from the original case protocol used to guide the session delivery, (3) some sessions were provided to multiple clinics at once, and (4) updated prescribing guidelines changed treatment messaging. See the completed FRAME-IS modules in Table 5.

Table 5.

Prescriber Peer Consulting Modifications with the FRAME-IS.

FRAME-IS Module or Subcomponent		Modification 1	Modification 2	Modification 3	Modification 4
Module 1
	The evidence-based practice (EBP) being implemented is:	Adherence to opioid prescribing guidelines through systems consultation	Adherence to opioid prescribing guidelines through systems consultation	Adherence to opioid prescribing guidelines through systems consultation	Adherence to opioid prescribing guidelines through systems consultation
	The implementation strategy being modified is:	Prescriber peer consulting	Prescriber peer consulting	Prescriber peer consulting	Prescriber peer consulting
	The modification(s) being made is/are:	All remote (with researchers/implementers corunning sessions)	Condensed clinics in one healthcare system	Deviations from case protocol provided used in sessions	Changes to messaging about treatment goals
	The reason(s) for the modifications(s) is/are:	COVID-19	COVID-19	Difficulty obtaining participant volunteers, adapting to individual needs and input from clinics	Changes in CDC opioid prescribing guidelines
Module 2
	What is modified?	Context: Setting (remote); Personnel (include research team at sessions to help with notetaking and organization)	Context: Format: (larger consolidated clinics participating)	Content: (cases from trainers vs. participants; Variations by trainers)	Content:(priorities and group discussion)
Module 3
	What is the nature of the content, evaluation, or training modification?	Tailoring/tweaking/refining (for online delivery); Changes in packaging or materials (including new model for practice facilitation, i.e., Star method for case study discussion); Adding elements (requiring participant volunteers, use of study team assistance)	Shortening/condensing (fewer total consulting sessions and working with consolidated clinics resulting in larger groups and fewer consulting sessions)	Substituting (cases); Loosening structure (shift from formal structure to more informal case discussion)	Substituting (refining messaging to emphasize updated guidelines)
	What is the relationship to core elements?	Fidelity Consistent/Core elements or functions preserved	Fidelity Consistent/Core elements or functions preserved	Fidelity Consistent/Core elements or functions preserved	Unknown
Module 4
	What is the goal?	Increase the acceptability, appropriateness, or feasibility of the implementation effort	Increase the acceptability, appropriateness, or feasibility of the implementation effort	Increase the clinical effectiveness of the EBP; Increase the acceptability, appropriateness, or feasibility of the implementation effort	Increase the clinical effectiveness of the EBP
	What is the LEVEL of the rationale for modification?	Sociopolitical level	Organizational level; Implementer level	Implementer level; Clinician or teacher level; Patient or Other Recipient level	Sociopolitical level
Module 5
	When is the modification initiated?	Implementation phase	Implementation phase	Implementation phase	Implementation phase
	Is the modification planned?	Planned/Reactive (Reactive adaptation)	Unplanned/Reactive (modification)	Unplanned/Reactive (modification)	Unplanned/Reactive (modification)
Module 6
	Who participates in the decision to modify?	Political leader(s)	Program Leader, Manager, or Administrator	Implementer or implementation strategy expert; Clinician(s) or teacher(s) who are being asked to use the EBP being implemented	Other: National researchers and developers of the CDC opioid prescribing guidelines
	Who makes the ultimate decision?	Researcher team	Administrator	Implementer (physician consultants)	Research team
Module 7
	How widespread is the modification?	Network system/community	Organization	Clinic/unit; Specific implementer/facilitator	Network system/community

Consulting sessions were originally designed as 1-hour, in-person lunchtime meetings for individual clinics, with food provided as an incentive to attend. All sessions were modified by the research team to be conducted virtually through Webex to comply with COVID-19 regulations. The virtual format had disadvantages in terms of clinician recruitment and retention, making it more difficult to develop a cohesive group with clear, shared goals. Also, participant engagement varied. Thus, some clinics were combined to increase participation. As determined by the research team and principal investigator, a member of the research team attended each session in one healthcare system to assist with setup, address technology challenges, and observe sessions. Their involvement in the sessions was necessary to use IRB-approved web conferencing tools that were not available in one healthcare system. Initially, study members had a larger role in guiding the online sessions while physicians learned the new online web conferencing program, but over time, physicians took on more responsibility. The researchers took big-picture notes on the case discussed, participants’ opioid prescribing goals, and clinical questions or challenges for future sessions. In the other healthcare system, the physician consultants had direct access to web conferencing tools and ran sessions independently.

Further, researchers communicated with consulting participants in both healthcare systems in preparation for the session and solicited one case per session. This support helped the consultant stay prepared for sessions. However, using an online format and having an additional researcher present may have made participants less comfortable discussing sensitive cases, even though no identifying notes were taken and sessions were not recorded.

A second modification to consultation involved deviation from the original protocol guiding prescriber peer consulting sessions. Consultants and study team members attempted to solicit cases in advance by having participants complete a one-page case presentation form detailing the patient's history and clinical challenge. However, it was challenging to solicit volunteers to submit cases and to obtain follow-through. As a result, the physician consultants made the decision to occasionally present cases from their own practices or those reflecting common primary care opioid prescribing challenges. Further, due to limited cases and low participation, the physician consultants and study team combined some of the clinics, allowing physicians to attend joint sessions. This occurred in only the third and fourth sessions for small clinics in both healthcare systems. Providing consulting to multiple clinics at once introduced new cases, but it may have limited participation from physicians hesitant to discuss sensitive cases with colleagues from other clinics.

The last modification to prescriber peer consulting was a change in messaging provided by the physician consultants due to a change in the innovation/intervention itself. The CDC's 2016 guidelines on opioid prescribing were updated in 2022 due to new evidence (Dowell et al., 2022), with the draft released near the end of our consulting sessions. Physician consultants and the research team decided to discuss these updates within the sessions. While the changing clinical guidelines did not fundamentally change the process of consulting, they were reflected in the content of consulting as an implementation strategy.

Overall, the team determined that the core components of consulting were maintained: a physician consultant met with opioid prescribers to discuss sensitive cases and problem solve around clinical guidelines in four quarterly sessions. As with other facilitation strategies, it is likely that moderators to implementation fidelity were impacted by the online format.

Discussion

Our multidisciplinary DOI team completed all core and optional modules of the FRAME-IS. This exercise allowed us to document modifications during the trial and consider the factors that made them necessary. Our team's reflection on the FRAME-IS and process suggested clear advantages and some remaining concerns. The process of completing the tool also raised broader questions relevant to the field of modifications in implementation science.

The Value of Using the FRAME-IS

The FRAME-IS was relatively simple to understand and easy to use. The seven modules allowed for comprehensive descriptions of modifications, and the template was flexible enough to document singular modifications across levels of influence (e.g., healthcare system-wide, clinic-wide). The tabular format yielded a visual that can easily be shared with other researchers and implementers interested in replication. In the future, compiling FRAME-IS tables across projects may support comparative analyses, such as identifying modification types across contexts and conditions.

Challenges Using the FRAME-IS

While generally easy to use, the FRAME-IS would have benefited from clear documentation of all aspects of implementation strategies that directly align with the FRAME-IS before completion, and ideally before implementation. Miller et al. (2021) recommend defining implementation strategies before completing the FRAME-IS and cite Powell et al. (2015) as a helpful resource and Proctor et al. (2013) as useful “if feasible.” Before completing the FRAME-IS our team had used the Expert Recommendations for Implementing Change (ERIC) compilation by Powell et al. (2015) to define our implementation strategies, but we did not use the specific reporting framework by Powell et al. (2015). We found that ERIC definitions alone were insufficient to complete the FRAME-IS, as they do not document further dosage, actors, justifications, core components, and so on. After completing Proctor et al.'s (2013) reporting recommendations and consulting our protocol paper (Quanbeck et al., 2020), we could easily reference information corresponding with FRAME-IS modules. One exception was optional Module 3, which requests the relationship of modifications to fidelity/core components. Core components and fidelity measurement plans are not included in Proctor et al.'s reporting specifications.

A concern arose when specifying primary modifications and the subsequent modifications that often resulted. Should certain modifications be documented together or separately? For example, consulting was modified to be delivered remotely (primary modification). As a result of remote delivery, we held joint sessions for multiple clinics (subsequent modification). These modifications happened sequentially; if documented together they became blurred and harder to interpret in other modules of the FRAME-IS, but if documented separately, it was unclear that they were sequential, interconnected, and that the first modification was necessary for the second modification to occur (i.e., remote sessions were necessary for physicians from different clinics to attend together). Cascading effects across the levels of influence can lead to modifications that are not clearly captured by the FRAME-IS. For example, changes to audit and feedback reports in the DOI were required because clinics were consolidated, but this consolidation occurred because of COVID-19 and its impact on the healthcare system. “Consolidated clinics” does not allow a reader to understand this chain of causation.

We encountered a similar quandary when completing Module 7, which asks, “How widespread is the modification? (i.e., for whom/what is the modification made?).” Options include singular units of “clinic/unit” or “organization.” In a multilevel and multisite implementation study, it is unclear if only one clinic/unit is impacted, if some of the clinics are impacted, or if all the clinics are impacted. This became problematic for our study, when we needed to describe changes to audit and feedback that occurred only within a handful of clinics and only within one healthcare system. In addition to how widespread a modification is, Module 7 may also be a good place to share how frequently the modification occurs to clarify whether modifications occurred once, multiple times, or became a permanent change for continuous implementation strategies that occurred at ongoing intervals.

We found the FRAME-IS to fall somewhat short in representing the complexity inherent in many implementation trials. The DOI was marked by blurred roles that complicated module completion. For example, our project manager was also a researcher on the study and an implementer of implementation strategies. However, listing all these roles when documenting “who” made the modification decisions, would suggest that multiple people were involved. Similarly, because implementation science is naturalistic and pragmatic, the stages and phases of an implementation trial are not always clearly delineated, with changes to the study design and the evidence-based practice occurring concurrently with changes to the implementation strategies. Modifications across the study design, the evidence-based practice, and the implementation strategies were intertwined, exceeding what the tabular format could support.

Additional Questions Raised by Completing the FRAME-IS

Completing the FRAME-IS raised critical questions for our future work. For example, while we considered it successful to continue the study and maintain fidelity to core components despite exigent circumstances, COVID-19 had an overwhelming impact on the DOI. We find ourselves asking if we should have changed the overall study instead of making modifications to keep it going.

Module 6 asks who makes the ultimate decision to modify. In most cases, decisions were made by our research team, with occasional assistance from other implementers (i.e., other facilitators, presenters, and physician consultants). How might other participants be able to inform modifications in real time? We argue that who makes the “ultimate decision” should be a required, not optional, module question to promote transparency and encourage stakeholder and participant-led decision-making. One exception to the top-down modification decision-making in the DOI was the modifications to audit and feedback reports, which were almost exclusively participant-driven. These critical modifications raise the question of how to create more natural feedback loops that allow participants to inform modifications.

Finally, completing the FRAME-IS led us to consider the overall philosophy of modification and how well the US medical research enterprise supports real-time modification. Health service researchers and their funders claim to value stakeholder decision-making. However, would major modifications requested during an active, costly, multiyear clinical trial be welcomed?

Future Directions: Recommendations and Research Questions

Our recommendations for the FRAME-IS are to:

Create a premodule to be completed before implementation. This module would request information describing the original implementation strategies, making this documentation required rather than optional, and guide the implementers to complete the FRAME-IS. For example, it could request the core components and specifications for each implementation strategy, drawing on Proctor et al.'s (2013) recommendations, and include plans for measuring fidelity. Documentation of implementation strategies, fidelity measurement, and the FRAME-IS process before implementation could increase transparency (by requiring this information to be completed first) and help reviewers interpret the remaining modules.

Clarify the roles of key personnel and stakeholders. Identify the main implementers for each strategy and clarify their roles and titles. This could be included in a premodule or within Module 6.

Support the documentation of multilevel reasons for modifications. The end of Module 1 requests the reasons for modifications. A multilevel rationale could list the reasons at varying levels. In the audit and feedback example described earlier, this list would include COVID-19 at the societal level, reallocation of healthcare resources for COVID care and vaccines at the institutional level, and clinic consolidation at the clinic level. Allowing users to list reasons at multiple levels would better reflect how interconnected and multilevel factors require modifications.

Clarify reporting on the spread and frequency of modifications. Module 7 could be improved with follow-up questions or write-in options to indicate how many individuals/units were impacted by modifications or other prompts to quantify the spread of the modification, as demonstrated in an example by Miller et al. (2021) but not made explicit within the reporting table. In addition to spread, Module 7 could be expanded, or a new module added, to clarify modification frequency (e.g., a one-time modification vs. an ongoing change within continuous implementation strategies).

Encourage interdisciplinary teams to discuss modifications at defined intervals during implementation. It is unlikely that one individual alone can know all modifications happening in an implementation trial. A team approach improves the ability to identify and document modifications and reduces bias compared to reporting from one individual. Teams could document in advance when and how often they would complete and update the FRAME-IS together. However, completing the template itself is only the first step. We found that the team discussions that followed were most helpful in raising questions about choices we made, adding research questions, and determining next steps. It could be helpful to develop guiding questions for teams to use during such discussions.

Future research should focus on how to incorporate the FRAME-IS into systematic analysis of the question, “How much do modifications matter and in what ways?” Implementation science lacks a systematic approach to understanding the impact of responsive modifications. One option is combining the FRAME-IS with evaluation tools like the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation, and Maintenance) (Gaglio et al., 2013). For example, our study modified facilitator roles, by using pharmacists as practice facilitators in one healthcare system and researchers in another. To test how this modification impacted evaluation outcomes, we could assess RE-AIM categories to ask questions such as, “Were clinics with one type of facilitator more likely to have higher attendance at practice facilitation sessions?” or “Does the type of facilitator impact whether clinics met and maintained their goals?” Finally, we agree with scholars who have applied the FRAME-IS (Miller et al., 2024; Schoenthaler et al., 2024) that studying modification impacts can clarify which modifications lead to which impacts. For example, do responses to Module 5 (when the modification is made) and Module 6 (who participates in the decision to modify) influence study outcomes differently?

Limitations

The modifications were self-reported by the research team. Despite observations and multiple interactions with implementers, we may have missed modifications that occurred without our knowledge, especially given the number of individuals implementing various strategies. Given the definitional and application challenges discussed, we may not have always made the “correct” choices as intended by the FRAME-IS authors within the modules. The FRAME-IS was published at the end of our trial, so we could not use it prospectively, as recommended by Miller et al., and instead applied it postintervention for implementation reflection. Thus, we may have been limited by recall bias over time and possibly institutional knowledge loss due to minor staff turnover.

Conclusions

We used the FRAME-IS to document modifications to four implementation strategies comprising systems consultation at varying levels of influence. We found the FRAME-IS tool to be easy to understand and use, and we recommend it to future researchers and practitioners. We provide several recommendations to improve the FRAME-IS and its use. Future research can assess how to study outcomes of documented modifications and how the FRAME-IS can further advance the field of implementation science.

Supplemental Material

sj-docx-1-irp-10.1177_26334895251407338 - Supplemental material for Using the Framework for Reporting Adaptations and Modifications to Evidence-Based Implementation Strategies (FRAME-IS) to Document Modifications to a Multilevel Deimplementation Strategy Aimed at Reducing Opioid Prescribing for Chronic Pain

Supplemental material, sj-docx-1-irp-10.1177_26334895251407338 for Using the Framework for Reporting Adaptations and Modifications to Evidence-Based Implementation Strategies (FRAME-IS) to Document Modifications to a Multilevel Deimplementation Strategy Aimed at Reducing Opioid Prescribing for Chronic Pain by Rose Hennessy Garza, Nora Jacobson, Andrew Cohen, Jillian Landeck, Nicholas Schumacher, Rachel Lundwall and Andrew Quanbeck in Implementation Research and Practice

Footnotes

ORCID iDs

Rose Hennessy Garza

Nora Jacobson

Andrew Cohen

Jillian Landeck

Andrew Quanbeck

Ethical Considerations

This study was approved by the University of Wisconsin–Madison's Health Sciences Institutional Review Board, submission 2018-1276-CP008.

Consent to Participate

Informed consent was obtained from clinic staff members participating in the research. A waiver of informed consent was obtained at the patient level for outcomes assessment in the parent trial.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: National Institute on Drug Abuse R01DA047279 (U.S. NIH Grant/Contract).

Declaration of Conflicting Interests

The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Dr. Quanbeck is a shareholder of CHESS Health, a small business devoted to marketing information technologies to agencies that deliver addiction treatment. Dr. Quanbeck also provides consulting on process improvement and implementation research through the nonprofit NIATx Foundation. The other authors declare that there is no conflict of interest.

Data Availability Statement

Access to data may be arranged by sending a request to corresponding author Dr. Andrew Quanbeck (andrew.quanbeck@fammed.wisc.edu) and completing a data sharing agreement and required onboarding steps with the Health Innovation Program ().

Trial Registration

NCT 04044521 (ClinicalTrials.gov). Registered on 05 August 2019. Available at:

Supplemental Material

Supplemental material for this article is available online.

References

Alagoz

Chih

M. Y.

Hitchcock

Brown

Quanbeck

(2018). The use of external change agents to promote quality improvement and organizational change in healthcare organizations: A systematic review. BMC Health Services Research, 18(1), 42. https://doi.org/10.1186/s12913-018-2856-9

Audisio

Lia

Robinson

N. B.

Rahouma

Soletti

Cancelli

Perezgrovas Olaria

Chadow

Tam

D. Y.

Vervoort

Farkouh

M. E.

Bhatt

D. L.

Fremes

S. E.

Gaudino

(2022). Impact of the COVID-19 pandemic on non-COVID-19 clinical trials. Journal of Cardiovascular Development and Disease, 9(1), Article 1. https://doi.org/10.3390/jcdd9010019

Carroll

Patterson

Wood

Booth

Rick

Balain

(2007). A conceptual framework for implementation fidelity. Implementation Science, 2(1), 40. https://doi.org/10.1186/1748-5908-2-40

Curran

G. M.

Bauer

Mittman

Pyne

J. M.

Stetler

(2012). Effectiveness-implementation hybrid designs: Combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care, 50(3), 217–226. https://doi.org/10.1097/MLR.0b013e3182408812

Delbecq

A. L.

Van de Ven

A. H.

Gustafon

D. H.

(1976). Group techniques for program planning: A guide to nominal group and Delphi processes (S. Foresman, Ed.). Scott Foresman and Company.

Dowell

Haegerich

T. M.

Chou

(2016). CDC guideline for prescribing opioids for chronic pain—United States, 2016. JAMA, 315(15), 1624. https://doi.org/10.1001/jama.2016.1464

Dowell

Ragan

K. R.

Jones

C. M.

Baldwin

G. T.

Chou

(2022). CDC clinical practice guideline for prescribing opioids for pain—United States, 2022. MMWR Recommendations and Reports, 71(3), 1–95. https://doi.org/10.15585/mmwr.rr7103a1

Gaglio

Shoup

J. A.

Glasgow

R. E.

(2013). The RE-AIM framework: A systematic review of use over time. American Journal of Public Health, 103(6), e38–e46. https://doi.org/10.2105/AJPH.2013.301299

Lau

Stevenson

Ong

B. N.

Dziedzic

Treweek

Eldridge

Everitt

Kennedy

Qureshi

Rogers

Peacock

Murray

(2015). Achieving change in primary care—Effectiveness of strategies for improving implementation of complex interventions: Systematic review of reviews. https://doi.org/10.1136/bmjopen-2015-009993

10.

Mark

M. M.

(2021). Adding adaptation/modification size and scope to frameworks for classifying changes to an intervention. Prevention Science, 22(7), 923–927. https://doi.org/10.1007/s11121-021-01248-1

11.

Miller

C. J.

Barnett

M. L.

Baumann

A. A.

Gutner

C. A.

Wiltsey-Stirman

(2021). The FRAME-IS: A framework for documenting modifications to implementation strategies in healthcare. Implementation Science, 16(1), 36. https://doi.org/10.1186/s13012-021-01105-3

12.

Miller

C. J.

Sullivan

J. L.

Connolly

S. L.

Richardson

E. J.

Stolzmann

K. L.

Kim

(2024). Adaptation for sustainability in an implementation trial of team-based collaborative care. Implementation Research and Practice, 5, 1–11. https://doi.org/10.1177/26334895231226197

13.

Powell

B. J.

Waltz

T. J.

Chinman

M. J.

Damschroder

L. J.

Smith

J. L.

Matthieu

M. M.

Proctor

E. K.

Kirchner

J. E.

(2015). A refined compilation of implementation strategies: Results from the expert recommendations for implementing change (ERIC) project. Implementation Science, 10(1), 21. https://doi.org/10.1186/s13012-015-0209-1

14.

Proctor

E. K.

Powell

B. J.

McMillen

J. C.

(2013). Implementation strategies: Recommendations for specifying and reporting. Implementation Science, 8(1), 139. https://doi.org/10.1186/1748-5908-8-139

15.

Quanbeck

Almirall

Jacobson

Brown

R. T.

Landeck

J. K.

Madden

Cohen

Deyo

B. M. F.

Robinson

Johnson

R. A.

Schumacher

(2020). The Balanced Opioid Initiative: Protocol for a clustered, sequential, multiple-assignment randomized trial to construct an adaptive implementation strategy to improve guideline-concordant opioid prescribing in primary care. Implementation Science, 15(1), 26. https://doi.org/10.1186/s13012-020-00990-4

16.

Quanbeck

Brown

R. T.

Zgierska

A. E.

Jacobson

Robinson

J. M.

Johnson

R. A.

Deyo

B. M.

Madden

Tuan

W. J.

Alagoz

(2018). A randomized matched-pairs study of feasibility, acceptability, and effectiveness of systems consultation: A novel implementation strategy for adopting clinical guidelines for opioid prescribing in primary care. Implementation Science, 13(1), 21. https://doi.org/10.1186/s13012-018-0713-1

17.

Quanbeck

Robinson

Jacobson

Hennessy Garza

Landeck

Cohen

Madden

Pulvermacher

Brown

(2024). Strategies to Deimplement Opioid Prescribing in Primary Care: A Cluster Randomized Trial. JAMA Network Open, 7(10), e2438325. https://doi.org/10.1001/jamanetworkopen.2024.38325

18.

Schoenthaler

De La Calle

Leon

Garcia

Colella

Nay

Dapkins

(2024). Application of the FRAME-IS to a multifaceted implementation strategy. Research Square, rs.3.rs-3931349. https://doi.org/10.21203/rs.3.rs-3931349/v1

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB