Sage Journals: Discover world-class research

Abstract

The Opioid Response Project (ORP) was designed as an intensive 2-year health promotion learning collaborative grounded in the Collective Impact Model (CIM) to prepare ten local communities to address the opioid crisis. The purpose of this evaluation was to describe the ORP implementation, provide a summary of the evaluation results, share insights, and discuss implications. Results were informed by multiple data sources, including project documents, surveys, and interviews with members of the ORP and community teams. Based on process evaluation results, 100% of community teams reported being satisfied with the ORP and recommended this experience to others. ORP participation outputs ranged from new opioid response programs, to strengthened community teams, to receipt of additional funding. Based on the outcome evaluation, the ORP was effective at increasing community knowledge and capacity, promoting collaboration, and facilitating sustainability. This initiative is an example of an effective learning collaborative to curb the opioid epidemic at the community level. Participating communities found great value in working together as part of a larger cohort and reported benefits from the peer learning and support provided by the ORP. In particular, access to technical assistance, identification of engagement strategies within and across community teams, and a focus on sustainability are key practice components to be included in learning collaboratives designed to address large-scale public health issues.

Keywords

opioid community learning collaborative collective impact model technical assistance

Recently declared a national public health emergency (U.S. Department of Health and Human Services, 2017), the opioid epidemic has impacted communities across the United States. The effects of this crisis have been experienced at individual-, family-, and community levels. Individuals may face chronic health problems, increased risk for overdose, and oftentimes homelessness (National Alliance to End Homelessness, 2016). Families often experience stress due to their loved one’s opioid use, and struggle to help. Impact on the community is reflected through increased demand for services including health care, social services, and involvement with the criminal justice system (Vincenzes et al., 2019).

Although this epidemic is far reaching, North Carolina (NC) has been particularly affected, with an average of three daily deaths (North Carolina Department of Health and Human Services, 2017). In 2016, four NC cities ranked in the nation’s top 20 highest for opioid misuse rates (Knopf, 2017). The North Carolina Opioid Action Plan was released in 2017 in an effort to identify achievable steps with the greatest potential impact. Efforts centered on prevention and treatment as evidenced by the Plan priorities (e.g., reduce the oversupply of prescription opioids). Although progress is being made at the state level, a number of challenges (e.g., spikes in demand for local foster care, number of overdoses addressed by local law enforcement) remain for local implementation of policies and solutions. One promising, yet severely underutilized, community and systems-level strategy is the coordinated use of multidisciplinary community teams or coalitions (Haegerich et al., 2019).

The Opioid Response Project (ORP) was designed as an intensive health promotion learning collaborative to address the opioid crisis by providing support to 10 local NC community teams. This initiative was meant to complement current policy efforts at the state and local levels by offering resources to communities preparing for implementation of best practices within a coordinated, centralized initiative. The teams were asked to plan, implement, and evaluate local policy and practice initiatives of their choosing in their respective communities. The learning collaborative was grounded in the Collective Impact Model (CIM; Kania & Kramer, 2011), with community teams preparing for action through a systematic approach that included a series of forums, webinars, activities, and meetings, as well as access to resources and technical assistance.

Funding for this 2-year project was awarded to the University of North Carolina (UNC) School of Government (SOG) by BlueCross BlueShield of North Carolina (Blue Cross NC) in February 2018. The UNC Eshelman School of Pharmacy Center for Medication Optimization (CMO) group was subcontracted in June 2019 to evaluate this initiative. Of note, the project extended to August 2020 due to COVID-19 to allow the community teams’ members to attend to more urgent pandemic-related priorities.

The purpose of this article is to describe the ORP implementation, provide a summary of the evaluation results, share insights, and discuss implications for health promotion practice. Given the dearth of literature on this type of coordinated, systems-level strategy to curb the opioid epidemic at the local level (Haegerich et al., 2019), this evaluation adds to the literature by offering an example of an organized, multidisciplinary, and collaborative community-based approach.

The ORP Initiative

Conceptual Model: Collective Impact Model

The CIM was selected because of evidence suggesting its promise in addressing other complex challenges (Edmondson & Hecht, 2014; Kania & Kramer, 2011). Complex problems, such as the opioid epidemic, require system-wide change that target multiple root causes involving a diverse set of actors working collectively to address the problem. Developed by the Forum for Community Solutions (FSG), the CIM is designed as a collaborative approach to create large-scale social change (Kania & Kramer, 2011). Collective impact is defined as “the commitment of a group of important actors from different sectors to a common agenda for solving a specific problem” (Kania & Kramer, 2011). The model assumes a joint effort that involves a centralized infrastructure, dedicated staff, and a structured process that leads to collective impact. The five major elements of the CIM include: creation of a common agenda to facilitate shared understanding among participants; development of shared measures to ensure that individual actions within the collective remain aligned; execution of mutually reinforcing activities that are coordinated across participants and aligned with the common agenda; continuous communication to build trust and assure mutual objectives; and support of a backbone organization that can provide expertise, planning, and coordination. These tenets were weaved into the ORP collaborative learning content and structure in support of the project stated goals (see description below).

Who Implemented the ORP?

The ORP team was composed of faculty and staff from UNC SOG and coordinated by the ncIMPACT Initiative within the UNC SOG. The ncIMPACT Initiative helps local communities use data and evidence to improve conditions and inform decision-making. UNC SOG and the ncIMPACT Initiative served as the backbone organization supporting the community teams throughout the ORP, with SOG faculty leading and ncIMPACT Initiative coordinating the project through an experienced program manager. The ORP team involved 15 faculty and staff with a diversity of expertise (e.g., law, public health, and organizational change), with a subset of five comprising the ORP core team. The ORP team met once a month to discuss progress, problem-solve, and facilitate execution of the project. Eight faculty members from the ORP team also served as faculty liaisons and designated points of contact to the 10 community teams. They supported the teams by encouraging progress, identifying needs and connecting them with resources, providing assistance as needed, and strengthening the relationship between the community teams and the SOG.

How Were the Community Teams Recruited and Selected to Participate in the ORP?

Any NC community interested in engaging in the project to “enact an integrated and innovative policy and practice response to their local opioid crises” was provided an opportunity to apply. The project was advertised through a variety of methods, including presentations at City and County Managers Associations meetings, emails to local government managers, communications with the NC Association of County Commissioners and NC League of Municipalities, interactions with state agencies such as the Department of Health and Human Services, and a webinar hosted for prospective applicants (200+ participants). The 10 finalists were selected by a panel of ORP team members based on considerations of the following factors: geographical, economic, and demographic diversity; prior experience with opioid mitigation efforts (a diversity of experience levels was intentionally sought across teams); the magnitude of the crisis in their community; identification of a core team composed of three individuals from different local government(s) and/or organizations; and demonstration of community commitment through at least three letters of support. The size of the community teams ranged from 9 to 152 participants, with representation from a wide diversity of community organizations depending on the focus of the team (see below for additional details).

What Were the Goals of the ORP?

The ORP purpose was to support and enhance local community efforts to respond to the opioid crisis. More specifically, the goals of this project were to: (a) plan, deliver, and refine activities in support of community teams responding to the crisis; (b) strengthen the capacity of the local teams to plan, implement, monitor, and sustain their selected innovations; (c) strengthen connections and partnerships around the opioid response efforts at the local level; and (d) disseminate resources and findings to a larger stakeholder audience.

How Was the ORP Structured?

The initiative was structured around five regional forums convened across NC. During the forums, the community teams were guided through the CIM, established goals, developed plans for implementation and sustainability, collaborated across fields and jurisdictions, and learned from experts about opioid-related content. Between forums, community team members completed project exercises and action plans, further developed and executed on their opioid response plans, engaged in six webinars, had at least six contact points with their faculty liaison, and participated in community team meetings. In addition, the community teams’ project managers attended monthly calls with the ncIMPACT Initiative program manager to report on progress, share learnings, and discuss ORP updates. Materials and resources, including an online educational software platform and monthly newsletters, were provided throughout the project.

Method

Overview and Approach

The evaluation was guided by the Utilization-Focused Evaluation (U-FE) principles (Patton, 2002), as well as the Centers for Disease Control and Prevention (CDC) six-step Framework for Program Evaluation in Public Health (Milstein & Wetterhall, 1999). These collaborative approaches were selected to ensure that data gathered would provide accurate, and most importantly, useful information to the ORP team. The CMO evaluation group worked with a four-member ORP subgroup to plan and conduct the ORP evaluation. This collaborative strategy facilitated the CMO evaluation group’s understanding of the initiative, allowed for co-creation of evaluation goals, and led to the development of the evaluation plan.

In brief, the objectives of the evaluation were to describe the project, determine its impact, and identify lessons learned. These objectives assume a mixed data approach that integrated elements of formative, process, and outcome evaluation. For sake of brevity, the formative evaluation results are reported elsewhere (Livet & Richard, 2020). This evaluation was reviewed and deemed non-human subjects research by the UNC IRB prior to data collection.

Data Collection and Analysis

The results summarized in this manuscript were informed by multiple data sources, including existing project documents (i.e., community team rosters, community team learning journeys, and ORP program manager meeting notes), surveys, interviews with ORP team members, and interviews with representatives from each community team. Table 1 aligns the evaluation questions with the data types, indicators, and sources, identifies the respondents, and specifies the data collection timeline. Table 2 describes the surveys used in this evaluation. All information was analyzed by data source and combined as relevant for ease of reporting.

Table 1

Evaluation Data Collection Plan

Evaluation approach	Evaluation question	Data type	Indicators	Data source	Collected from	Collection timeline
Process evaluation	Who were the community teams?	Project documents	Number of team members and agencies represented	Opioid community team rosters	Community teams—project managers (N = 10/10)	January 2020
	Who were the community teams?	Survey	Participant demographics	Demographics survey results	Community teams—all team members (N = 132/584)	July–August 2020
	What were the teams’ experience with the ORP?	Survey	Satisfaction levels	Core Team Survey—satisfaction Items	Community teams—core team members (N = 21/64)	July–August 2020
	What were the teams’ experience with the ORP?	Survey	Satisfaction insights	Core Team Survey—satisfaction Items (open ended responses)	Community teams—core team members (N = 21/64)	July–August 2020
	What were the resulting outputs of participation for the community teams?	Interviews	Outputs	Community team interviews	Community teams—team representatives (N = 10)	August 2020
		Interviews		ORP team interviews	ORP team members (N = 5)	July 2020
		Project documents		Learning journeys	Community teams (N = 10)	July–August 2020
		Project documents		ORP program manager meeting notes	ORP program manager	Ongoing
Outcome evaluation	Was the ORP effective at strengthening capacity to plan, implement, and monitor opioid response efforts among participants?	Survey	Significant changes in own individual capacity	Core Team Survey—individual capacity items	Community teams—core team members (N = 21/64)	July–August 2020
		Survey	Significant changes in team capacity	Core Team Survey—team capacity items	Community teams—core team members (N = 21/64)	July–August 2020
	Was the ORP effective at strengthening connections among and across participants?	Survey	Significant changes in partnering relationships with organizations within community team	Core Team Survey—Collaboration Among Agencies Within Community Teams	Community teams—core team members (N = 21/64)	July–August 2020
		Survey	Significant changes in partnering relationships with organizations outside of community team	Core Team Survey—Collaboration Items Among Agencies Outside Community Teams	Community teams—core team members (N = 7/21)	July–August 2020
		Survey	Significant changes in partnering relationships across community teams	Team Survey—Collaboration Items Across Community Teams	Community teams (N = 10)	July–August 2020
	Was the ORP effective at preparing the community teams to sustain their efforts?	Survey	Evidence supporting sustainability	Team Survey—Sustainability Items	Community teams (N = 10)	July–August 2020

Note. This table provides an overview of the data collection plan used in this evaluation, including evaluation questions, data types, indicators, data sources, participants, and data collection timeline. ORP = Opioid Response Project.

Table 2

Survey Measures

Survey name	Number of items	Response options	Example items		Survey source
Core Team Survey—Satisfaction Items	Up to 56, with 4 items for each of the 14 ORP activities (e.g, forum 1, assistance from faculty liaison; quantitative)	Rate your level of agreement on a 6-point scale ranging from “strongly disagree” to “strongly agree”	This [activity] (e.g., assistance from my faculty liaison): • Provided me with useful information • Met my expectations • Was worthwhile to receive/participate in • Was beneficial to advance the work of my opioid response team		Survey created for this evaluation (Cronbach’s alpha=.96)
Core Team Survey—Satisfaction Items	3 (qualitative)	Open ended	Overall, the thing that I liked best about participating in the ORP was. . ..		N/A
Core Team Survey—Individual capacity items	18 across 3 domains (processes, opioid response policies and landscape, communication and relations)	Retrospective pre-post; 6-point scale ranging from “strongly disagree” to “strongly agree”	Prior to participating in the ORP, I was able to. . .	After participating in the ORP, I is able to. . ..	Survey created for this evaluation (Cronbach’s alpha=.95)
Core Team Survey—Individual capacity items			• Understand how to develop and use Key Performance Indicators (KPIs) • Understand North Carolina policies currently being used to address the opioid epidemic • Develop meaningful relationships with members of my community team		Survey created for this evaluation (Cronbach’s alpha=.95)
Core Team Survey—Team capacity items	11	Retrospective pre-post; 6-point scale ranging from “strongly disagree” to “strongly agree”	Prior to participating in the ORP, my team was able to. . .	After participating in the ORP, my team is able to. . ..	Survey created for this evaluation (Cronbach’s alpha=.94)
Core Team Survey—Team capacity items	11		• Identify and prioritize the problems most impacting the opioid epidemic in our community • Create a plan of actionable steps to address the opioid epidemic in our community		Survey created for this evaluation (Cronbach’s alpha=.94)
Core Team Survey—Collaboration Among Agencies within Community Teams	1 item for up to 20 organizations	Retrospective pre-post; 5-point scale ranging from “no relationship” to “networking” to “ cooperation” to “coordination” to “collaboration”;	Prior to the ORP, the relationship between my organization and this other organization [name] could be best described as. . .	Prior to the ORP, the relationship between my organization and this other organization [name] can be best described as. . .	(Seybolt et al., n.d.)
Core Team Survey—Collaboration Items Among Agencies Outside Community Teams	1 item for up to 10 organizations	4-point scale ranging from “networking” to “ cooperation” to “coordination” to “collaboration”	As a result of participating in the Opioid Response Project did your organization create any new relationships (i.e., networking, cooperation, coordination, or collaboration; see definitions above) with organizations not represented on your opioid response community team. Please list below up to ten organizations. After listing, hover over the words “list organization” to drag and drop each organization to the appropriate box (networking, cooperation, coordination, collaboration)		(Seybolt et al., n.d.)
Team Survey—Collaboration Items Across Community Teams	1 item, one for each of the other 9 community teams	Retrospective pre-post; 5-point scale ranging from “no relationship” to “networking” to “ cooperation” to “coordination” to “collaboration”	Prior to the ORP, the relationship between my community team and this other community team [name] could be best described as. . .	Prior to the ORP, the relationship between my community team and this other community team [name] can be best described as. . .	(Seybolt et al., n.d.)
Team Survey—Sustainability Items	8 items	3-point scale, including “not yet started,” “in the early stages,” “well established or completed”	Please select the option that best describes your community team’s progress on the following activities: • Identifying funding to continue my community team’s work beyond the ORP • Demonstrating sufficient progress to justify my community team’s existence		Coalitions Work (2012)

Note. ORP = Opioid Response Project.

The project document review involved extracting information from the relevant data sources above and organizing it into the categories of interest, namely understanding the community team members and agencies represented and identifying evaluation outputs. For the quantitative survey data, descriptive statistics, paired t-tests, and Wilcoxon signed-ranked tests were conducted as appropriate. As indicated in Table 1, a retrospective pre–post survey method was used to assess changes in individual and team capacity, as well as changes in partnering relationships within and across community teams, to control for response-shift bias (Thomas et al., 2019). All surveys were completed within the last 2 months of the project. All quantitative survey data were analyzed using IBM SPSS Statistics v26.

Open-ended survey questions and interview data were analyzed by the second author using content analysis with the appropriate a-priori coding structure (Hsieh & Shannon, 2005). Open-ended survey questions provided information about the community teams’ most and least favorite aspects of the ORP. The interviews with both members of the community teams and members of the ORP team were specifically reviewed for information related to outputs resulting from participation in this project. These data were ultimately synthesized across interview type for ease of reporting. The analysis was reviewed by the second author, with any disagreements discussed until consensus was reached. QSR International’s NVivo 12 software was utilized to facilitate the interview coding process.

Results

Process Evaluation

Who Were the Community Teams?

The teams were structured to include a set of core members and affiliate community members. Based on demographic information (N = 132), the majority of community team members were female (67%), white (75%), and above 45 years of age (60%), with more than half holding a master’s degree or above. Of the organizations that were represented across the teams (N = 132), the majority were either mental health/behavioral health/substance misuse providers (20%), governmental county and city organizations (16%), or public health organizations (e.g., county health departments, public health departments; 12%).

What Were the Teams’ Experience With the ORP?

Based on the quantitative results from the Core Team Survey satisfaction items (N = 21; Tables 1 and 2), 100% of respondents involved in ORP activities reported “moderately” to “strongly” agreeing being satisfied with the ORP. This trend was observed across the different types of activities, including forums, webinars, project managers’ conference calls, materials provided, faculty liaison meetings, and worksheets. Of note, the most highly rated activity was the assistance provided by the faculty liaisons, with all but one participant giving it the highest possible score. In addition, when asked to reflect on their experience with ORP, 100% of respondents (N = 18) stated that they would recommend this experience to others.

When asked to identify their favorite aspect of the ORP on the Core Team Survey satisfaction (i.e., the one thing they liked best about participating in the ORP), respondents identified the following: the ability to learn from experts and gain a broader perspective of the opioid crisis; the opportunity to network and collaborate with other teams across the state working on similar issues; the development and building of their own teams and communities; and the benefits of having access to a backbone organization that provided support (e.g., faculty liaisons), structure (e.g., forums), and expertise. When asked about what could have been done differently, respondents pointed to the need for: increased community team engagement from the outset and help with orienting new members; additional time to complete the project and collaborate with other groups; access to a calendar of project activities (i.e., forum and webinar dates) ahead of time to facilitate consistent attendance; additional opportunities to be strategically connected to teams with similar goals and initiatives; and additional coordination across participating teams in support of statewide change.

What Were the Resulting Outputs of Participation for the Community Teams?

Based on the qualitative data gathered during interviews and the project documents review (Table 1), the outputs fell into one of four categories: additional community-based opioid response programs and services (e.g., MAT in jails program, distribution of Narcan kits, drug take back program); opportunities to develop communication, planning, and structure within teams (by creating a shared focus and clarify team leadership, for instance); pursuit and receipt of additional funding (with all 10 teams pursuing or receiving additional funding due to their involvement in the ORP); and products created as a result of community programming (e.g., messages, resource guides, and other materials to be shared through local TV stations, bus wraps, public service announcements, media campaigns).

Outcome Evaluation

Was the ORP Effective at Strengthening Capacity to Plan, Implement, and Monitor Opioid Response Efforts Among Participants?

Based on the Core Team Survey capacity items (Tables 1 and 2), respondents (N=21) reported significantly higher levels of individual capacity after participating in the ORP compared to before (t(20) = 6.53, p < .01). This increase in knowledge and skills was significant in all three content domains targeted by the ORP: (a) planning, implementation, and evaluation processes (t(20) = 6.48, p < .01), (b) opioid response policies, programs, and landscape (t(20) = 5.26, p < .01), and (c) communications and relations (t(20) = 5.12, p < .01).

When asked about capacity of their respective community teams, survey participants (N = 20) also reported significantly higher team capacity to plan, implement, and evaluate their opioid initiative after participating in the ORP (t(19) = 6.52, p < .01). Similar to individual capacity, participants reported that their teams had significantly higher levels of skills in: (a) planning (t(19) = 7.11, p < .01), (b) implementation and monitoring (t(19) = 4.85, p < .01), and (c) ability to share results (t(19) = 4.19, p < .01).

Was the ORP Effective at Strengthening Connections and Partnerships Among and Across Participants?

Based on the Core Team Survey collaboration items (assessing relationships among agencies within the community teams; Tables 1 and 2), survey respondents (N = 21) reported significantly higher levels of collaboration between their own organizations and other community organizations within the team post-ORP compared to pre- (Z = −3.92, p < .01). Review of descriptive statistics indicated that these relationships strengthened from “cooperation” (informal relationships for a common purpose) to “coordination” (formal relationships in which partners have compatible missions). In addition, 33% (N = 7) of core team members who completed the survey collaboration items (assessing relationships among agencies outside of community teams) had created new relationships with organizations within their community that were not represented on their community teams as a result of implementing their local opioid response initiatives.

Finally, based on the results from the Team Survey collaboration items (to assess relationships across community teams), community teams (N = 10) reported significantly higher levels of collaboration with other community teams at the end of the project compared to the beginning (Z = −2.81, p < .01). The community teams were able to create “networking” (i.e., formal or informal connections between individuals and groups) relationships with other community teams with whom they had no prior relationships prior to the ORP.

Was the ORP Effective at Preparing the Community Teams to Sustain Their Efforts?

Based on the Team Survey sustainability items, 100% of teams (N = 10) reported completing at least two of the eight activities listed toward sustainability of their opioid response efforts by the end of the project. The highest rated activities for completion were demonstrating sufficient progress to justify my community team’s existence (N = 10/10) and establishing shared values reflected in the community team’s priorities, strategies, and day-to-day operations (N = 10/10). The least frequently completed activity was establishing access to funding sources beyond traditional funding such as in-kind contributions, fundraising events, or membership dues. (N = 4/9).

Discussion

Based on the evaluation results, the ORP was a highly successful initiative. Participation in the ORP was perceived by participants as beneficial to advance their community work and worthwhile to attend. In addition, this learning collaborative model was effective at increasing community knowledge and programming capacity, promoting collaboration within and across teams, allowing for expansion of local opioid response efforts, and facilitating sustainability beyond the project period. The participating teams found value in working as part of a larger cohort of teams and reported benefits from the peer learning and support provided during the ORP. These results are consistent with previous literature highlighting collaboratives and community teams as effective organizational structures to promote change at the local level (Hajjar et al., 2020).

Key Insights and Implications for Practice and Research

Resulting from the evaluation were three key insights, with implications for health promotion practice and research. First, the supports provided by the ORP team were perceived as essential by the community teams. More specifically, the individualized assistance provided the faculty liaisons was the most highly rated support strategy. Customized technical assistance has been consistently identified as a facilitative strategy in support of community-level public health efforts (Flaspohler et al., 2008; Leeman et al., 2015; Livet et al., 2005; Livet & Wandersman, 2005, 2018). In this context, its focus should be on proactively facilitating application of knowledge acquired through trainings or forums, maintaining the teams’ motivation and commitment, assisting with access to needed resources and information, and supporting local problem-solving efforts (Durlak, 2015; Durlak & DuPre, 2008; Meyers et al., 2012). A formal technical assistance approach that is tailorable to the needs of each participating team should be included as an implementation support strategy in future applications of any learning collaborative of this kind. Given the importance of technical assistance as a promising support strategy, further research also needs to be conducted to identify the essential features and mechanisms of technical assistance to optimize its effectiveness at different developmental stages of a community team (Scott et al., 2022).

Second, the ability to facilitate and maintain community team engagement both with the collaborative and within individual community teams was perceived as critical to success. Previous research supports the role of member engagement as mediator of coalition effectiveness (Butterfoss & Kegler, 2009; Kegler & Swan, 2012). However, with team members oftentimes managing multiple demands and competing priorities, reliable attendance was challenging, making it difficult to keep all team members up-to-date and hindering timely decision-making. Previous literature on effective community coalitions highlight the need to promote a culture of inclusion, trust, respect, and meaningful engagement, as well as attend to the motivational factors for participation (Hajjar et al., 2020; McCloskey et al., 2011; Wolff, 2001). Clearly outlining expectations of participation at the outset, sharing requirements and timeline as a condition of participation, and offering additional incentives were identified as potential recommendations. It might also be helpful to expand the collaborative curriculum to include sessions on community engagement and team dynamics based on best practices literature. Explorations of the most effective strategies and mechanisms for strengthening member engagement need to continue as a promising mediator of community team effectiveness (Kegler & Swan, 2012).

Finally, the collaborative intentional focus on sustainability led to some notable benefits for the community teams, including additional funding to continue past the project period. Oftentimes, without the backbone organization providing structure and expertise, community teams dissolve (Turner et al., 2012). In this case, all 10 teams reported intent to continue efforts to address the opioid crisis in their community. Strategies identified by team members to increase the likelihood of sustainability were to pair teams with common agendas and enact the CIM as a cohesive state-level initiative to create statewide change and impact. Demonstrating the value of local opioid response efforts to policymakers through quality of evidence and additional research will more likely lead to improved funding prospects with strength in numbers (Bailit & Dyer, 2004; Jacobson & Neumann, 2009; Practical Playbook et al., 2017).

Limitations

Although the ORP evaluation generated insights into successful learning collaboratives, it is not without limitations. The data collection relied heavily on the community teams’ core team members and the ORP team members, a relatively small sample size in comparison to the total number of participating individuals. Second, although the diversity of community teams (size, demographics, community environment) supports generalizability of lessons learned, the teams were selected into the project, thereby assuming a certain level of readiness to engage in this work. Finally, due to the advent of COVID and efforts by the ORP team to reduce burden on the community teams, community-level outcome data (to assess the impact of individualized opioid response strategies used by a community team on the target populations) were not collected. Instead, the work accomplished by each community was showcased through reported outputs.

Conclusion

In conclusion, the ORP was an effective example of an organized, multidisciplinary, and systematic learning collaborative to curb the opioid epidemic by focusing on community-level efforts. Given the dearth of literature on the use of coordinated efforts by communities to address the opioid crisis, the ORP adds to the existing evidence highlighting learning collaboratives as a promising strategy. Not only can it serve as a model for other states attempting to address this issue but it could also be tailored to respond to other public health concerns. Ultimately, continued and expanded use of the CIM should be considered to solve large-scaled problems that would benefit from a well-coordinated interdisciplinary collaborative effort.

Footnotes

Authors’ Note:

The authors gratefully acknowledge members of the 10 community teams engaged in this project for their valuable work and insights. In addition, this work would not have been possible without the contributions of the UNC-SOG faculty and staff involved with this project, as well as the ncIMPACT Initiative team. Finally, the authors acknowledge the generous support provided by Blue Cross NC. This work was supported by funding from Blue Cross Blue Shield of North Carolina (Blue Cross NC) awarded to the University of North Carolina School of Government (UNC SOG). The University of North Carolina Eshelman School of Pharmacy Center for Medication Optimization (CMO) group was awarded a subcontract from UNC SOG to evaluate the initiative.

ORCID iD

Melanie Livet

References

Bailit

Dyer

M. B.

(2004). Beyond bankable dollars: Establishing a business case for improving health care (pp. 1–12). The Commonwealth Fund.

Butterfoss

F. D.

Kegler

M. C.

(2009). The community coalition action theory. In DiClemente

R. J.

Crosby

R. A.

Kegler

M. C.

(Eds.), Emerging theories in health promotion practice and research (2nd ed., pp. 237–276). Jossey-Bass.

Coalitions Work. (2012). Coalition sustainability characteristics [Internet]. https://wholespire.org/pdf/OFA/COALITION-SUSTAINABILITY-CHARACTERISTICS.pdf

Durlak

J. A.

(2015). What everyone should know about implementation. In Durlak

J. A.

Domitrovich

C. E.

Weissberg

R. P.

Gullotta

T. P.

(Eds.), Handbook of social and emotional learning: Research and practice (pp. 395–405). The Guilford Press.

Durlak

J. A.

DuPre

E. P.

(2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3–4), 327–350.

Edmondson

Hecht

(2014). Defining quality collective impact. Stanford Social Innovation Review, 12(4), A6–A7. https://doi.org/10.48558/TW78-RG69

Flaspohler

Duffy

Wandersman

Stillman

Maras

M. A.

(2008). Unpacking prevention capacity: An intersection of research-to-practice models and community-centered models. American Journal of Community Psychology, 41(3–4), 182–196.

Haegerich

T. M.

Jones

C. M.

Cote

P. O.

Robinson

Ross

(2019). Evidence for state, community and systems-level prevention strategies to address the opioid crisis. Drug and Alcohol Dependence, 204, Article 107563.

Hajjar

Cook

B. S.

Domlyn

Ray

K. A.

Laird

Wandersman

(2020). Readiness and relationships are crucial for coalitions and collaboratives: Concepts and evaluation tools. New Directions for Evaluation, 165, 103–122.

10.

Hsieh

H. F.

Shannon

S. E.

(2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288.

11.

Jacobson

P. D.

Neumann

P. J.

(2009). A framework to measure the value of public health services. Health Services Research, 44(5p2), 1880–1896.

12.

Kania

Kramer

(2011). Collective impact. Stanford Social Innovation Review, 9, 36–41. https://ssir.org/articles/entry/collective_impact#bio-footer

13.

Kegler

M. C.

Swan

D. W.

(2012). Advancing coalition theory: The effect of coalition factors on community capacity mediated by member engagement. Health Education Research, 27(4), 572–584. https://doi.org/10.1093/her/cyr083

14.

Knopf

(2017, July 27). Four North Carolina cities make top 25 list for opioid abuse. North Carolina Health News. https://www.northcarolinahealthnews.org/2017/07/27/four-north-carolina-cities-make-top-25-list-opioid-abuse/

15.

Leeman

Calancie

Hartman

M. A.

Escoffery

C. T.

Herrmann

A. K.

Tague

L. E.

Moore

A. A.

Wilson

K. M.

Schreiner

Samuel-Hodge

(2015). What strategies are used to build practitioners’ capacity to implement community-based interventions and are they effective? A systematic review. Implementation Science, 10(1), Article 80.

16.

Livet

Richard

(2020). The opioid response project: Evaluation report. https://ncimpact.sog.unc.edu/wp-content/uploads/sites/1111/2021/05/ORP-Evaluation-Report_FINAL-updated.pdf

17.

Livet

Richter

Ellison

Dease

McClure

Feigley

Richter

D. L.

(2005). Emergency preparedness academy adds public health to readiness equation. Journal of Public Health Management and Practice, 11(6), S4–S10.

18.

Livet

Wandersman

(2005). Organizational functioning: Facilitating effective interventions and increasing the odds of programming success. In Fetterman

Wandersman

(Eds.), Empowerment evaluation principles in practice (pp. 123–154). Guilford Press.

19.

Livet

Yannayon

Sheppard

Kocher

Upright

McMillen

(2018). Exploring provider use of a digital implementation support system for school mental health: A pilot study. Administration and Policy in Mental Health and Mental Health Services Research, 45(3), 362–380.

20.

McCloskey

D. J.

McDonald

M. A.

Cook

Heurtin-Roberts

Updegrove

Sampson

Gutter

Eder

(2011). Community engagement: Definitions and organizing concepts from the literature. https://www.atsdr.cdc.gov/communityengagement/pdf/PCE_Report_Chapter_1_SHEF.pdf

21.

Meyers

D. C.

Durlak

J. A.

Wandersman

(2012). The quality implementation framework: A synthesis of critical steps in the implementation process. American Journal of Community Psychology, 50(3–4), 462–480.

22.

Milstein

Wetterhall

S. F.

(1999). Framework for program evaluation in public health. MMWR. Recomm Rep, 48(Rr-11), 1–40.

23.

National Alliance to End Homelessness. (2016). Opioid abuse and homelessness. https://endhomelessness.org/resource/opioid-abuse-and-homelessness/

24.

North Carolina Department of Health and Human Services. (2017). North Carolina’s opioid action plan 2017-2021. https://www.ncdhhs.gov/media/1585/download

25.

Patton

M. Q.

(2002). Utilization-focused evaluation (U-FE) checklist. Evaluation Checklists Project.

26.

Practical Playbook, deBeaumont Foundation, Duke Community, & Family Medicine. (2017). Demonstrating value in population health projects. https://www.practicalplaybook.org/api/sites/default/files/imce/PPB-strategy-brief-demonstrating-value-in-population-health-projects.pdf

27.

Scott

V. C.

Jillani

Malpert

Kolodny-Goetz

Wandersman

(2022). A scoping review of the evaluation and effectiveness of technical assistance. Implementation Science Communications, 3, Article 70. https://doi.org/10.1186/s43058-022-00314-1

28.

Seybolt

Hein

Wandersman

(n.d.). A systematic look at collaboration within a mental health public-academic consortium: Strategies, measurement, and lessons learned in collaboration. Unpublished manuscript.

29.

Thomas

E. V.

Wells

Baumann

S. D.

Graybill

Roach

Truscott

S. D.

Crenshaw

Crimmins

(2019). Comparing traditional versus retrospective pre-/post-assessment in an interdisciplinary leadership training program. Maternal and Child Health Journal, 23(2), 191–200.

30.

Turner

Merchant

Kania

Martin

(2012). Understanding the value of backbone organizations in collective impact: Part 1. Stanford Social Innovation Review. https://doi.org/10.48558/X3KJ-BS10

31.

U.S. Department of Health and Human Services. (2017). HHS acting secretary declares public health emergency to address national opioid crisis. https://public3.pagefreezer.com/browse/HHS.gov/31-12-2020T08:51/https://www.hhs.gov/about/news/2017/10/26/hhs-acting-secretary-declares-public-health-emergency-address-national-opioid-crisis.html

32.

Vincenzes

K. A.

McMahon

Lange

Forziat-Pytel

(2019). Systemic issues in the opioid epidemic: Supporting the individual, family, and community. International Journal of Mental Health and Addiction, 17(5), 1214–1228.

33.

Wolff

(2001). Community coalition building—contemporary practice and research: Introduction. American Journal of Community Psychology, 29(2), 165–172.

The Opioid Response Project: An Effective Learning Collaborative for Local Communities?

Abstract

Keywords

The ORP Initiative

Conceptual Model: Collective Impact Model

Who Implemented the ORP?

How Were the Community Teams Recruited and Selected to Participate in the ORP?

What Were the Goals of the ORP?

How Was the ORP Structured?

Method

Overview and Approach

Data Collection and Analysis

Results

Process Evaluation

Who Were the Community Teams?

What Were the Teams’ Experience With the ORP?

What Were the Resulting Outputs of Participation for the Community Teams?

Outcome Evaluation

Was the ORP Effective at Strengthening Capacity to Plan, Implement, and Monitor Opioid Response Efforts Among Participants?

Was the ORP Effective at Strengthening Connections and Partnerships Among and Across Participants?

Was the ORP Effective at Preparing the Community Teams to Sustain Their Efforts?

Discussion

Key Insights and Implications for Practice and Research

Limitations

Conclusion

Footnotes

Authors’ Note:

ORCID iD

References