Open accessMeeting reportFirst published online 2026-4
Proceedings of the Seventh Biennial Conference of the Society for Implementation Research Collaboration (2024): Strategic Synergy: Implementation Research,Practice,and Policy for Impact
Proceedings of the Seventh Biennial Conference of the Society for Implementation Research Collaboration (2024): Strategic Synergy: Implementation Research, Practice, and Policy for Impact
Geetha Gopalan1, Bo Kim2, McKenzie LeTendre3, William A. Aldridge II4, Stephanie Brooks5, Margaret Crane6, Sapana Patel7, Mónica Pérez Jolles8, Taren Massey-Swindle9, Tyler Frank,10 Humphrey Beja 11,12,13,14, Hyunjin Kang15, Tessa Palafu16, Jacob Tempchin17, and Suzanne E.U. Kerns18
1Silberman School of Social Work, Hunter College, City University of New York, NY, NY, USA
2Center for Healthcare Organization and Implementation Research (CHOIR), VA Boston Healthcare System; Department of Psychiatry, Harvard Medical School, Boston, MA, USA
3Graduate School of Social Work, University of Denver, Denver, CO, USA
4The Impact Center at Frank Porter Graham Child Development Institute, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
5School of Public Health, University of Alberta, Edmonton, Alberta, Canada
6BRIDGE Program, Department of Psychiatry and Human Behavior, Brown University Providence, RI, USA
7Center for Practice Innovations, New York State Psychiatric Institute; Department of Psychiatry, Columbia University Irving Medical Center, NY, NY, USA
8Department of Pediatrics, University of Colorado School of Medicine; Adult and Child Center for Outcomes Research and Delivery Science (ACCORDS), Aurora, CO, USA
9Department of Pediatrics, University of Arkansas for Medical Sciences, Little Rock, AR, United States, Arkansas Children's Nutrition Center, Little Rock, AR, USA; Arkansas Children's Research Institute, Little Rock, AR, USA
10School of Public Health, Washington University in St. Louis, USA
11Faculty of Public Health, Nursing and Midwifery, Uganda Christian University (UCU), Mukono Campus, Uganda
12National Executive Council, National Midwives Association of Uganda (NMAU), Kampala, Uganda
13Department of Midwifery, Faculty of Nursing and Midwifery, Lira University, Lira, Uganda
14Society for Implementation Research Collaboration (SIRC)
15Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
16Department of Psychology, University of Oregon. Eugene, OR, USA
17Department of Psychology, University of Memphis, Memphis, TN, USA
18Kempe Center for the Prevention and Treatment of Child Abuse and Neglect, Department of Pediatrics, University of Colorado Anschutz Medical Campus, Aurora, CO, USA
Corresponding author:
Geetha Gopalan, LCSW, PhD, Silberman School of Social Work at Hunter College, City University of New York, 2180 3rd Avenue, New York, NY 10035, USA. Email: ggopalan@hunter.cuny.edu
Acknowledgements
We deeply appreciate all the efforts made by the 20 on-site conference volunteers, 97 volunteer abstract reviewers, 18 volunteer board members, the awards committee's 5 volunteer core members and 24 volunteer reviewers, the administrative director, multiple partner organizations, and several sponsors who made the SIRC 2024 conference possible. A special thanks to our conference committee members: Gregory Aarons, Caryn Blitz, Alicia Bunger, Darin Carver, Kate Comtois, Elizabeth Connors, and Nicole Stadnick. We also extend our sincere gratitude to HPN Global Meeting Services for their exceptional support in coordinating and managing the SIRC 2024 conference. The authors acknowledge the use of Microsoft Copilot in the preparation of this manuscript. Copilot was used to assist with drafting text and developing tables. All AI-assisted content was critically reviewed and edited by the authors to ensure accuracy, clarity, and alignment with the journal's academic and ethical standards.
Funding
Partial sponsorship supported the SIRC 2024 conference; however, the majority of funding was provided through registration fees. The conference was organized and executed through the volunteer efforts of conference organizers, leaders, and planning committee members.
Stephanie Brooks is supported by the Canadian Institutes of Health Research (CIHR) CGS-D and the University of Alberta Killam Doctoral Program.
Monica Perez Jolles is supported by the Agency for Healthcare Research and Quality under Award Number R01HS029858-01A1, PCORI SOE -2022C2-28909 and AD-2021C1-22459, NIDDK 5R01 DK130176-03, NCI R01 CA282292, and Internal University of Colorado Denver Funding.
Taren Swindle is supported by the National Institute of General Medical Sciences of the National Institutes of Health under Award Number 5P20GM109096, NIH R37CA252113, NIH NCATS 1UM1TR004909, the Arkansas Biosciences Institute, Vitamix Foundation, and USDA OSQR 6026-10700-001-000D.
Declaration of Conflicting Interests
Bo Kim serves as an Associate Editor of Implementation Research and Practice. Geetha Gopalan and Sapana Patel are members of the journal's Editorial Board. None of these individuals were involved in the review or editorial decision-making process for this manuscript.
Introduction
The field of implementation science holds promise for ensuring that evidence-based interventions and aligned strategies effectively reach the populations who need them. While the origins of implementation science date back centuries, the field has been growing exponentially since the early 2000s (Chambers & Emmons, 2024). The Society for Implementation Research Collaboration (SIRC) is dedicated to help keep pace with this rapid growth. SIRC held its seventh biennial conference in September, 2024 in Denver, Colorado, USA, attended by over 400 researchers, practitioners, policymakers, students, and trainees (https://societyforimplementationresearchcollaboration.org/conference-information/conference-7-september-2024-2/). The conference provided an exciting opportunity to converge on the most novel and groundbreaking implementation science work, with a particular focus on behavioral and mental health services across multiple settings.
This gathering of ideas and energy occurred against a backdrop of significant political anticipation: notably, the SIRC 2024 conference took place just over a month before the 2024 presidential election in the United States. At the time, we could not have anticipated the seismic shifts in the federal research landscape that would unfold in the ensuing months (Beidas et al., 2025). This shift, while directly focused on funding in the United States, sent ripples throughout the globe and stands to impact the next generation of research. In this context, the conference's theme—Strategic Synergy: Implementation Research, Practice, and Policy for Impact—felt especially salient. The emphasis on aligning research, practice, and policy to accelerate the uptake of evidence-based practices into real-world settings resonates deeply with the challenges and opportunities emerging from this new funding landscape. Implementation science as a field may be well positioned to play a critical role in this shifting environment. Many of us were drawn to the field of implementation science because it is designed, at its very core, to create efficiencies and maximize the return on investments in evidence-based approaches.
In this paper, we briefly summarize the history of SIRC, delineate ongoing SIRC initiatives, and report on the seventh biennial SIRC Conference. These collective efforts provide a platform for various stakeholders within the implementation ecosystem to come together and enhance the quality of mental/behavioral health, healthcare services, prevention efforts, and community-based initiatives (e.g., education, child welfare) worldwide. As a result, SIRC is strategically situated to realize the vision of implementation science as a “big tent” (Chambers & Emmons, 2024) inclusive of individuals with lived experience, communities, practitioners, policymakers, and researchers.
History of SIRC
SIRC initially started as a conference grant, the Seattle Implementation Research Conference, awarded in 2010 (NIMH 1R13MH086159-01A1; PI: Comtois). This three-conference series sought to elevate the role of implementation science in supporting the expansion and dissemination of evidence-based practices within behavioral health contexts. The first conference (October 2011) theme was “Key Issues in Evidence-Based Psychosocial Intervention (EBP) Implementation Methods and Research.” The second conference (May 2013) centered on the theme “Solving Implementation Research Dilemmas.” The third conference (September 2015) focused on “Advancing Efficient Methodologies Through Community Partnerships and Team Science.” As each conference progressed, attendance increased and capitalized on the emerging enthusiasm for implementation science in behavioral health services. Thus, at the conclusion of the original grant funding, we decided to continue SIRC under the auspices of a new society and, since that time, have hosted four additional biennial conferences. As a result, SIRC has since evolved to become an influential society in the field of implementation science via its biennial conference as well as additional initiatives described in Table 1 below.
Overview of SIRC Initiatives.
Initiative
Description
Implementation Research and Practice (IRP)
SIRC's open-access journal focused on implementation research in mental and behavioral health. Launched in 2020, it has published 183 articles and four special collections.
Implementation Development Workshops (IDWs)
Small-group feedback sessions for implementation projects. Offered in-person at conferences and virtually throughout the year.
Mentorship
Structured individual, group, and peer mentorship programs to support professional development across career stages.
Networks
Includes Investigator Networks and the Implementation Practice and Support Network (IPSN), fostering community among researchers, practitioners, and policymakers.
Implementation Infrastructure Initiative (I3)
Supported by the Annie E. Casey Foundation, I3 develops resources to strengthen implementation infrastructure across sectors.
SIRC City
Local and virtual social gatherings to build community among implementation professionals. Open to members and non-members.
Mechanisms Network of Expertise (MNoE)
A network of 26 experts advancing research and policy on implementation mechanisms.
D&I Resources and Training
A curated repository of training opportunities, fellowships, degree programs, and online resources in dissemination and implementation science.
Webinars
Regular webinars on topics such as implementation infrastructure, human-centered design, and expert office hours.
Community Engagement Initiative Action Council
Sets priorities, builds foundational structures to broaden membership, facilitates safe, action-oriented dialogue, and embeds equity in implementation research and practice.
Conference Proceedings
Planning Committee
The conference planning committee included the SIRC 2024 Conference Chair (Geetha Gopalan), student co-chair (McKenzie LeTendre), and former SIRC president (Suzanne Kerns) as main conference organizers, as well as 14 SIRC members (including board members and the SIRC 2022 conference chair). From September 2023 through September 2024, the planning committee met regularly to review progress in conference planning, discuss program options (e.g., keynote speakers, pre-conference workshops), and assist with different workflows embedded within the planning process (e.g., setting up conference social events). An initial task for the planning committee involved addressing feedback and recommendations from the SIRC 2022 Conference. In addition to retaining many aspects of the 2022 conference that were very well received, we aimed to be responsive to specific recommendations and preferences. This process included ensuring that the SIRC 2024 conference site be hosted in a walkable location that offered a variety of affordable accommodations and food, facilitating greater representation of practice and policy-oriented presentations, as well as incorporating more interactive session formats. Conference planning and implementation was further supported by a professional conference planning service.
Abstract Procedure and Review
To increase the number of submissions from practitioners and policymakers, the planning committee made substantial revisions to the conference announcement. Updates involved adopting more inclusive language (i.e., replacing “abstract” with “submission”) throughout the announcement, as well as equally highlighting the importance of implementation practice, research, and policy. Planning committee members developed two submission tracks: one for practice/policy-oriented submissions with newly developed review criteria, as well as a track for research-oriented submissions using existing review criteria from prior conferences. This process took into consideration the constraints and functionalities of the conference planning software. The planning committee encouraged non-traditional submission formats to diversify conference offerings with presentations that would be more accessible to non-researchers. The SIRC 2024 Call for Submissions introduced Slams (5-minute brief presentations) and Fishbowls (https://www.betterevaluation.org/methods-approaches/methods/fishbowl-technique; BetterEvaluation, n.d.) as new formats, in addition to traditional oral, poster, and symposium formats. Conference organizers engaged additional practice- and policy-oriented reviewers to evaluate practice/policy-oriented submissions with the newly developed review criteria, created video tutorials on the submission and review process, and participated in specialized SIRC webinars for practice and policy-oriented events. These efforts resulted in an increased percentage of practice/policy-oriented submissions (2022; 19%; 2024: 29%) and accepted presentations (2022: 15%; 2024: 32%), as seen in Figure 1. Table 2 demonstrates that, although SIRC 2024 had fewer overall submissions, more presentations were accepted, including a greater number of posters and slams.
Comparison of SIRC 2022 and SIRC 2024 conference submission and acceptance rates. Note. Submission rates reflect the percentage of research or practice/policy submissions among all submitted abstracts. Acceptance rates reflect the percentage of accepted research or practice/policy submissions among all accepted submissions.
Comparison of SIRC 2022 and SIRC 2024 submission and acceptance rates.
SIRC 2022
SIRC 2024
# of total submissions
355
335
# accepted submissions
182
209
# Oral presentations
52
51
# Posters
108
128
# Symposium
18
18
# Pre-conference workshops
4
4
# Slams
N/A
12
Pre-Conference Activities
Prior to the start of the official conference, pre-conference activities described below were offered on Thursday, September 26th, 2024 to enhance networking opportunities within the implementation community, facilitate the acquisition of new competencies, and provide space to workshop implementation projects in development.
Topical Preconference Workshops
Four pre-conference workshops provided experiential and skills-based learning opportunities for attendees across a spectrum of topics relevant to implementation policy, practice, and research. Across all workshops, attendance ranged from 26 to 53 participants, with a total of 104 preconference workshop attendees.
Participants at the “Considering Context Across the Lifespan of Implementation” workshop gained hands-on experience with practical strategies for understanding and addressing contextual (i.e., system and organizational) factors impacting implementation. Delivered by experts from The Center for Implementation (Julia Moore, PhD; Sobia Khan, PhD, MD; https://thecenterforimplementation.com/what-we-do), the workshop elevated the central importance of considering context across all phases of implementation, including readiness, adaptation, and sustainability. This workshop introduced the “Context Compass,” an interactive online tool designed to help users assess and respond to contextual factors in real-world implementation settings.
In the second workshop, “Advancing Health Equity through Co-Creation across the Spectrum of Research, Practice, and Policy,” participants engaged in interactive and didactic activities focused on applying novel dissemination and implementation science tools (e.g., co-creation guidebook) and strategies to co-create health equity-oriented policy, services, and research projects. Experts from the University of Utah (Paul Estabrooks, PhD), University of Colorado Anshutz Medical Campus (Amy Huebschmann, MD, MSc; Mónica Pérez Jolles, PhD, MA; Katy Trinkley, PharmD, PhD), and the University of California San Diego (Borsika Rabin, PhD, MPH) led activities focused on co-creation processes specific to engaging stakeholders, co-designing new intervention and/or implementation strategies, and evaluating efforts.
Presenters for the third workshop, “Partnering for Sustainable Financing of Evidence-Based Practices in Implementation Research and Practice: Tools to Navigate the Storm” included experts from the RAND Corporation (Alex Dopp, PhD, MA; Marylou Gilbert, JD, MA) and San Antonio AIDS foundation (Cherise Rohr-Allegrini). Participants in this hands-on workshop learned about utilizing the Public Health Sustainability Framework and the Fiscal Mapping Process to guide assessment and planning of financial strategies that support sustainment of evidence-based practice implementation. Representatives from the San Antonio AIDS foundation provided a case example of efforts to sustain motivational-based brief intervention (MIBI) for substance use.
Finally, experts from the SUSTAIN Wellbeing COMPASS Coordination Center, Eastern Connecticut State University (Megan Stanton, PhD, MSW), University of Texas at Austin (Katie McCormick, MSW), and University of Houston (Samira Ali, PhD, MSW) convened the workshop, “Justice-Centered Implementation Research and Practice: Laying the Foundation and Building a Collective Vision.” In this workshop, participants took part in didactic and collaborative activities that centered equity within implementation practice, research, and policy.
Implementation Development Workshops (IDWs)
Three half-day Implementation Development Workshops (IDW's) convened on Thursday, September 26th, 2024. The IDW's aimed to support SIRC members with ideas or current projects to receive high quality feedback from their peers in synchronous, small-group settings. IDW's were open to practitioners, students, as well as new and established investigators, who sought feedback on projects or proposals in development. These discussions placed a strong emphasis on the methodological rigor of implementation projects or proposals, while also fostering collaborative engagement among participants.
Conference Summary
In total, 475 participants attended the 2024 SIRC Conference over two days (Friday September 27th through Saturday, September 28th, 2024), representing 11 countries (Australia, Canada, Chile, Ireland, Norway, Singapore, Spain, Sweden, Switzerland, United Kingdom, United States). The majority of attendees (61%) resided in the United States, representing 38 states, and Washington, D.C. Most (n = 267, 56%) identified as researchers, while practitioners and trainees represented 17% (n = 77) each. An additional 11% of attendees (n = 54) identified as Chief Executive Officers, consultants, academic administrators, journal editors, federal funders, intermediaries, training coordinators, research staff, and state government administrators. Over half (57%, n = 269) indicated they were first-time SIRC conference attendees.
Presentations
Table 3 summarizes author-reported characteristics among submitted and accepted presentations. Prominent themes represented at the SIRC 2024 conference included: (1) local implementation efforts in the conference location of Colorado, (2) strategic synergy between practice, policy, and research; (3) prevention; (4) including practitioners; (5) emphasis on equity; (6) decision-making; and (7) partnerships, financing, and sustainability.
Characteristics of submitted and accepted presentations.
Submissions (n = 335)
Accepted (n = 209)
Topic
Research: 71% Practice/Policy: 29%
Research: 68% Practice/Policy: 32%
Global Projects (any part of work conducted
outside the United States)
22%
22%
Collaboration between researchers, practitioners,
and community members
81%
81%
Health Equity/Social Justice Focus
59%
57%
Submissions Involving Student Authorship
19%
16%
Project Phase
Completed 33% In process: 36% N/A: 15% Planned: 1.5% Planning for next phase: 15%
Completed: 33% In process: 33% N/A: 16% Planned: 1.5% Planning for next phase: 17%
Local Implementation Efforts. The keynote panel “Implementation Science in Colorado: A Panel Using the Translational Science Benefits Model to Describe Impact” showcased a diverse group of Colorado community practitioners, academic researchers, and their work. Experts shared how the Translational Science Benefits Model (TSBM; Luke et al., 2018) could be applied to bridge research, policy, and practice in service of advancing health equity. Panelists included Amy Huebschmann, Meredith Fort, and Lily Cervantes from the University of Colorado - Anschutz, with Roger Low (Colorado Equitable Economic Mobility Initiative; CEEMI) and Tiffany Madrid (Colorado Implementation Science Unit; CISU) moderating discussion. Examples from Colorado illustrated how implementation frameworks have been used to inform local policy changes, improve community outcomes, and embed equity considerations into decision-making. Panelists emphasized the importance of cross-sector collaboration, meaningfully addressing health equity through strategic use of implementation frameworks, and effective communication with policymakers to translate research into action. Drawing on insights from government, academia, and healthcare, the panel illustrated how aligning strategies across sectors can lead to impactful and equitable systemic change.
Strategic Synergy Between Practice, Policy, And Research.
Keynote: “A Movable Feast: Working at the Boundary to Support the Implementation of Science into Policy and Practice”. In her keynote, Annette Boaz explored the complex and relational nature of translating research into policy and practice. Drawing on her work in United Kingdom (UK) government and academia, she challenged linear models of implementation and emphasized the importance of boundary work—collaborative efforts that bridge the distinct cultures of research, policy, and practice. Boaz highlighted tools like the UK's Areas of Research Interest and the U.S. Learning Agendas as mechanisms for aligning research with policy needs. Her keynote further underscored the value of trust, contextual awareness, and relational competencies in effective implementation.
Invited Fishbowl Session. An invited fishbowl session brought together implementation researchers and practitioners for critical conversations about the current status of the field, organized by William Aldridge, Chair of SIRC's Practitioner Network, and Stephanie Brooks, Trainee Chair of SIRC's Practitioner Network. The session, entitled “Contending with the Implementation Research-to-Practice Gap: Thematic Report and A Fishbowl Conversation,” engaged conference attendees across the implementation research-practice spectrum in conversations to identify and advance strategies to help close the gap between research and practice. Fishbowl participants included active discussants seated in an inner circle and observers in an outer circle, with the flexibility for individuals to 'tag in' or be invited into the conversation. This dynamic structure allowed for a rotating group of contributors, fostering inclusive dialogue and enabling diverse perspectives to enter the discussion organically. A distinguished panel – including former SIRC presidents Cara Lewis and Byron Powell, Julia Moore and Sobia Khan from The Center for Implementation, and SIRC 2024 Conference keynote speaker Rohit Ramaswamy – initiated the discussion-portion of the session. Panelists reflected on their own experiences with the divide between implementation research and practice, explored new perspectives on its full scope, and considered strategies to bridge this disconnect. Remaining tensions between research, practice, and community experience were identified and explored. Participants and audience members alike reflected on the importance of continuing these brave conversations intentionally, and the key role played by the SIRC community in providing space for these critical discussions and interactions. This conference session served as the capstone event for a year-long SIRC Practitioner Network series of events on the theme “Contending with the implementation research-practice gap.”
Slams. Slam presentations offered a series of rapid, engaging, 5-minute talks showcasing innovative projects across varied implementation contexts and were designed to reach a broad audience. Among other topics, presentations addressed the limited visibility of practitioners in dominant frameworks, offered novel perspectives such as applying entrepreneurial thinking to implementation science, and explored whether agreement between leaders and providers on organizational climate served as a potential mechanism for facilitating uptake of evidence-based practices. These newly introduced formats spotlighted the creativity that emerges when research, policy, and practice are brought into purposeful and accessible dialogue.
The Inaugural SIRC Cup. The inaugural SIRC Cup, renamed the Katherine Anne Comtois SIRC Cup during the SIRC 2024 conference awards ceremony, brought together trainees (e.g., graduate students, post-baccalaureate staff, and postdoctoral fellows), practitioners, and researchers from the implementation community. In Spring 2024, teams with mixed membership of researchers, practitioners, and trainees developed a collaborative proposal to respond to a community-based implementation scenario framed through a case study shared across all participating teams. Four finalist teams were selected to fully develop an integrated implementation and evaluation plan based on the case study over the summer of 2024. At the lunchtime session on Friday, September 27th, 2024, finalists presented their plans to conference attendees and a panel of judges from across SIRC's networks of expertise. Members across all teams, including those not selected as finalists, commented about the value of the partnerships, learning, and new perspectives they developed working in teams spanning research, practice, and trainee boundaries. This new competition draws inspiration from the Society for Prevention Research's (SPR) Sloboda and Bukoski SPR Cup (https://preventionresearch.org/2021-annual-meeting/sloboda-and-bukoski-cup/).
Scientist-Practitioner Model. The conference featured a compelling focus on the scientist-practitioner model (Ramaswamy et al., 2019) as a strategy for strengthening implementation capacity and accelerating equitable impact. In his keynote, “Developing Scientist Practitioners in Global Settings,” Rohit Ramaswamy explored the critical role of implementation science training in low- and middle-income countries, emphasizing the need to cultivate professionals who both generate and apply evidence in complex, real-world settings. The session emphasized the value of capacity-building efforts led by local stakeholders, grounded in contextual knowledge and strengthened through community partnerships.
The symposium “Strategic Synergy: Can Implementation Science Benefit from the Scientist-Practitioner Model?” expanded on these themes. Presenters discussed the scientist-practitioner model as more than a professional identity, but a collaborative infrastructure for translational success. Presentations examined how blended roles and teams can improve alignment, promote mutual learning, and support the development of adaptive, responsive implementation strategies.
Implementation Ecosystem. The 2024 conference provided a venue to recognize and increase alignment among various components and stakeholders within the implementation ecosystem (Chambers & Emmons, 2024), which includes research, synthesis and translation, intermediary/support systems, service provider organizations, and community members/service recipients. Challenging siloed roles, both the content and format of presentations illustrated synthesis and translation with practical applications in hospital systems, education, and public health settings. For example, Kim Dumont emphasized the importance of attending to the social dynamics of implementation and evidence use in the final keynote presentation, “The Social Side of Research Use: What Else Is Needed, and Where Do We Go From Here.” She highlighted the importance of trust, relationships, and context-sensitive partnerships between researchers, practitioners, and communities, as well as institutional reforms—particularly within universities—to support collaborative, co-created approaches that value diverse forms of evidence. As a result, the SIRC 2024 conference advanced an inclusive and applied vision of the implementation ecosystem, echoing calls to move beyond technical solutions and toward relational, contextualized, and equity-driven approaches.
Co-design. Co-design was elevated as a critical strategy and a foundational approach to equitable outcomes. The pre-conference workshop, “Advancing Health Equity Through Co-Creation Across the Spectrum of Research, Practice, and Policy,” brought together scholars and practitioners to share practical frameworks for engaging stakeholders from the outset. Presenters emphasized that co-creation is essential not only for relevance, but for sustainable, equity-centered impact. This theme was further echoed in the symposium, “Co-Design: What if We Didn't Create the Research/Practice Gap to Begin With?”, which challenged attendees to reimagine implementation work as a collaborative, iterative process shaped by community wisdom and lived experiences rather than top-down dissemination. Sessions throughout the program reinforced this theme, featuring examples of co-design in behavioral health, youth services, and healthcare delivery. Presentations illustrated that embedding co-creation within implementation systems can mitigate power imbalances, increase contextual fit, and generate sustainable change.
Prevention. Prevention emerged as a prominent theme with sessions exploring both the promise and complexity of implementing upstream interventions across various systems. Multiple presentations focused on embedding prevention strategies into public systems, particularly those serving youth, families, and historically underrepresented communities. For example, symposia explored the role of school-based mental health interventions, suicide prevention programs tailored to tribal nations, and multi-level strategies to reduce adverse childhood experiences. Several discussions focused on integrating preventive care within healthcare systems, such as screening and brief interventions for substance use and chronic disease prevention in primary care settings. These presentations emphasized co-design with frontline providers and communities, the use of culturally responsive approaches, and the challenge of sustaining preventive efforts in reactive systems. A common theme across prevention domains focused on how equitable implementation requires shifting from downstream responses to upstream capacity-building and environmental change.
Including Practitioners. Nearly a third of all accepted SIRC 2024 submissions were submitted under the category of “Practice/Policy” submissions (See Table 3). SIRC's Practitioner Network curated 20 practice-oriented sessions—comprising 35 presentations, workshops, and panels—across preconference, invited, and peer-reviewed session formats. This represented a meaningful increase in the presence of accepted practice/policy-oriented sessions at a SIRC conference (from 15% in 2022 to 32% in 2024), likely supported by a more inclusively written Call for Submissions, separate Practice/Policy-oriented review criteria and reviewer pool, and increased efforts to engage practice/policy-oriented professionals within and on the periphery of the SIRC community.
Themes represented among the practice/policy-oriented sessions at SIRC 2024 included co-creation and community involvement in implementation, increasing the integration of implementation research and practice, advancing equity, the development of intermediary organizations and other regional implementation support structures to catalyze large-scale systems change, and implementation practitioner workforce development models and strategies. These themes all touched on the importance of moving out of existing silos through innovative collaboration, training experiences, and translational infrastructure. Presentations and discussions highlighted the importance of building community and trusted relationships, setting ideal conditions for healthy conflict, listening and moving through defensiveness, and working together with a sense of vulnerability, humility, and accountability. Altogether, these themes may represent key current and future directions within the implementation science field, requiring attention from implementation researchers, policymakers, and practitioners alike.
Emphasis on Equity. Across SIRC 2024 conference presentations, equity served as an integral driver of implementation design, strategy, and evaluation. For example, the pre-conference workshop, “Advancing Health Equity Through Co-Creation Across the Spectrum of Research, Practice, and Policy,” shared applied methods for promoting equity in implementation efforts by structurally embedding community voice and addressing systemic power imbalances. Other conference sessions explored equity-focused adaptations of implementation frameworks, strategies to engage historically underrepresented populations, and policy-level mechanisms to drive justice-oriented change. This emphasis reflected a maturing field that is not only asking what works, but also for whom, in what context, and with what structural accountability.
Decision-Making. Centering evidence, context, and stakeholder insight emerged as key considerations in decision-making within varied implementation environments. Several presenters explored how implementation teams navigate trade-offs in the allocation of resources and selection of strategies and intervention adaptation. Sessions highlighted guidance and tools that support structured decision making such as rapid-cycle evaluation and participatory design. Presentations underscored the importance of inclusivity, local expertise, transparency, as well as data-informed and context-sensitive decision processes to enhance effective implementation.
Partnerships, Financing, and Sustainability. Throughout the conference, presentations highlighted the critical role of strategic partnerships in advancing implementation science. Sessions delved into topics such as community-academic collaborations, innovative financing models, and strategies for sustaining evidence-based practices in diverse settings. By bringing together researchers, practitioners, and policymakers, the conference facilitated discussions on aligning implementation efforts with policy and practice to achieve lasting outcomes.
Additional Conference Offerings
In addition to over 200 peer-reviewed presentations, conference offerings included opportunities for networking, development, and recognition. These included lunchtime networking meetings for Special Interest Groups, meeting implementation journal editors, social activities, as well as ongoing SIRC initiatives: Community Engagement Initiative and Implementation Infrastructure Initiative (I3). Finally, the SIRC 2024 conference concluded with awards honoring the contributions of numerous individuals within the implementation community.
Special Interest Group Networking Lunches. The SIRC 2024 conference offered a general Networking Lunch on Friday, September 27th, 2024. Attendees were able to identify others with common interests (e.g., healthcare services, education) based on signs posted at different tables. Additionally, a separate lunch specifically for Student and Trainees was scheduled for Saturday, September 28th, 2024.
Meet the Editors Lunch. A Meet the Editors luncheon took place on Saturday, September 28th, 2024. This event encouraged interaction between conference attendees and members of the editorial boards of prominent journals in the field, such as Global Implementation Research and Applications (GIRA; Rosalyn Bertram), Implementation Research and Practice (IRP; Bianca Albers, Byron Powell, Sonja Schoenwald) and Implementation Science and Implementation Science Communications (Gregory Aarons).
Social Activities. SIRC 2024 conference social events fostered partnerships between researchers and practitioners, providing attendees with opportunities to connect and collaborate. The Practitioner Network Social Event held on the evening of Thursday, September 26th, 2024, brought together implementation practitioners to exchange insights, discuss challenges, and explore opportunities for collaboration. The well-attended event highlighted the growing engagement of implementation practitioners, inclusive of those working in areas related to policy, intermediary support, and provider organizations, within the broader SIRC community. William Aldridge, Chair of SIRC's Practitioner Network, and Stephanie Brooks, Trainee Chair of SIRC's Practitioner Network, provided welcoming remarks, a guiding vision for implementation practitioner engagement and recognition through SIRC, and a conference guide highlighting the numerous practice-oriented sessions at SIRC 2024. Concurrently, a New Attendee welcome event was also held on Thursday, September 27th, 2024 in the evening. This informal event facilitated initial networking and orientation for both new and returning attendees. Additional social events included a Happy Hour following the second poster session on the evening of Friday, September 27th, 2024, a “Fitness with the Franks” class on the morning of Friday, September 27th, 2024, and a guided morning fun run (attended by 24 runners and walkers) on Saturday, September 28th, 2024. Through these social events, SIRC reinforced its commitment to bridging the gap between research and practice by fostering relationships that will continue beyond the conference.
Community Engagement Initiative. A dedicated lunch session on Saturday, September 28th, 2024, provided updates on the Community Engagement Initiative Action Council's accomplishments and initiatives. More recent efforts to support the SIRC community amid evolving diversity efforts and changes in federal funding have included listening sessions, revisions to SIRC's mission, vision, and values statements, as well as guidance from peer professional societies and non-profit organizations. These efforts underscore SIRC's steadfast dedication to cultivating an inclusive and welcoming professional community and advancing equitable implementation of social service and health innovations.
Implementation Infrastructure Initiative (I3). On Friday, September 27th, 2024, leaders of the SIRC Implementation Infrastructure Initiative (I3) provided an update to their work during a lunchtime meeting. They reported on a survey conducted with 178 implementation support practitioners to identify where these individuals worked and what kinds of support would help them do their job. Findings indicated that implementation support practitioners worked in universities (40%), healthcare settings (26%), intermediary organizations (18%), and community agencies (15%). Moreover, individuals play many roles, including being an implementation support practitioner (79%), technical assistance provider (40%), intervention developer (42%), administrator (22%), provider (17%), funder (6%), and community members (7%). Implementation support practitioners requested (1) practical tools to assist with implementation, (2) training in implementation science and managing interpersonal dynamics, (3) coaching through external mentorship or organized peer learning, (4) supports to assess programs and implementation, (5) a researcher-practitioner matching program to aid mutual learning, collaboration, and evaluation between these roles, and (6) that SIRC serve an agenda-setting role for the future of evidence-based implementation supports. These findings suggest a growing demand for evidence-based implementation supports. Further, the distinction between implementation “researchers” and “practitioners” is not always clearcut as individuals often possess diverse responsibilities.
Awards. At the conference conclusion on Saturday, September 28th, 2024, SIRC 2024 honored individuals and their teams with awards to recognize their contributions to implementation research and practice, showcased at the conclusion of the SIRC 2024 conference (See Table 4). The review process engaged volunteer reviewers with diverse expertise from the SIRC Community. For the symposium, poster, and presentation awards, the presenters for the top three (or four, where there were ties) highest-scoring abstracts in each award category were notified prior to the conference that their presentation had been selected as an award finalist. The finalists’ presentations were attended and rated by the awards committee's volunteer reviewers. Review criteria covered a presentation's (i) incorporation of community partners’ perspectives, (ii) bridging of science and practice, (iii) methodological strength, (iv) importance to the field, and (v) extent of innovation. For the SIRC Mission Award, in the weeks leading up to the conference, nominations were solicited from the field for collaborative teams (with at least one team member being a SIRC member) contributing to advancing implementation research and practice consistent with the spirit of SIRC's mission. Nominations were reviewed and ranked by the awards committee's volunteer reviewers. Review criteria covered a nominated team's (i) bridging of the gap between research and practice, (ii) multi-stakeholder collaboration, (iii) impact, and (iv) diversity and innovation of the funding portfolio.
Summary of presented awards.
Award Name
Recipient(s)
Description
Katherine Anne Comtois SIRC Cup
“Stop, Collaborate, and Listen” team (Anna Bartuska, Dan Cheron, Rachel Kim, Kelsie Okamura, Alayna Park)
Recognized for outstanding team collaboration and contribution to implementation research.
Outstanding Symposium Award
Nathaniel Williams
For work on leadership and organizational change in implementing evidence-based practices.
Outstanding Poster Award
Alex Kent and colleagues
Project “Kwiis-hen-niip (Change)” focused on improving emergency care access for First Nations people.
Outstanding Practice-Focused Oral Presentation Award
Angelina Ruiz and colleagues
Highlighted community-academic partnerships in addressing adverse childhood experiences.
Outstanding Research-Focused Oral Presentation Award
Ulrica von Thiele Schwarz and colleagues
Examined patient perceptions and evidence use.
Outstanding Slam Presentation Award
Melanie Barwick and colleagues
Developed and evaluated the usability of "The Implementation Playbook" software platform.
Outstanding Student Presentation Award
Vivian Byeon and colleagues
Work on aligning the implementation climate between organizational leaders and providers.
SIRC Mission Award Finalist
CEASE-HIV
Advanced implementation of evidence-based HIV care engagement strategies for underserved populations.
SIRC Mission Award Finalist
U.S. Department of Veterans Affairs Suicide Prevention 2.0 Clinical Telehealth Team
Accelerated suicide prevention through interdisciplinary practice-research partnership.
SIRC Mission Award Winner
Child Health and Development Institute (CHDI)
Recognized for nearly 20 years of public-private partnership to disseminate evidence-based treatments for youth.
Conclusion and Future Directions
The 2024 Society for Implementation Research Collaboration (SIRC) Conference underscored the field's continued evolution toward a more integrated, equity-driven, and practice-informed implementation science. The conference advanced discourse on bridging the persistent research-to-practice gap through expanded practitioner and policy-maker engagement and novel presentation formats, while also retaining the essential features of SIRC's commitment to rigorous evaluation and research communication. Notably, this was reflected in proportional acceptance rates regardless of topic area (see Table 3). Presentation themes further reflected a maturing field increasingly attuned to contextual complexity and systemic change.
Given the recent profound shifts in the U.S. federal research funding landscape and resulting global implications, the conference's theme – Strategic Synergy: Implementation Research, Practice, and Policy for Impact – aligns with calls for greater efficiency and maximizing the real-world impact of investments in research, policy, and practice. Concurrently, there is substantial uncertainty regarding how to advance equity-oriented and global initiatives, as well as addressing the unique needs of marginalized populations, given the deprioritization within U.S. federal research funding. As SIRC moves forward, its strategic initiatives aim to deepen cross-sector partnerships, enhance capacity-building infrastructures, and broaden access to implementation research and practice resources. These collective endeavors establish SIRC as a pivotal entity promoting an inclusive and impact-driven implementation science ecosystem.
Looking ahead, SIRC is poised to expand its impact by deepening its commitment to interdisciplinary collaboration and capacity building. Key initiatives include expanding strategic partnerships with organizations aligned with SIRC's mission to improve public health through implementation research and practice and supporting researchers and practitioners by providing structured feedback and mentorship to advance their projects. These efforts reflect SIRC's dedication to fostering a robust community of practice that bridges research and real-world application.
In addition to these initiatives, SIRC is enhancing its focus on sustainable implementation efforts targeting populations in need. SIRC plans to expand the reach of its Implementation Practice & Support Network, facilitating greater engagement with practitioners and policymakers to ensure that evidence-based practices are effectively integrated and sustained across various settings. Furthermore, SIRC is exploring innovative dissemination strategies, including webinars and continued SIRC City gatherings, to broaden access to implementation science resources and foster inclusive dialogue among diverse collaborators. These future directions underscore SIRC's commitment to advancing implementation research and practice that impactfully meet the needs of our communities.
References
Beidas, R. S., Aarons, G. A., Geng, E. H., Sales, A. E., Wensing, M., Wilson, P., & Xu, D. R. (2025). Implementation science grant terminations in the United States. Implementation Science, 20(1), 20, s13012-025-01434–01437. https://doi.org/10.1186/s13012-025-01434-7
Chambers, D. A., & Emmons, K. M. (2024). Navigating the field of implementation science towards maturity: Challenges and opportunities. Implementation Science, 19(1), 26, s13012-024-01352–0. https://doi.org/10.1186/s13012-024-01352-0
Luke, D. A., Sarli, C. C., Suiter, A. M., Carothers, B. J., Combs, T. B., Allen, J. L., Beers, C. E., & Evanoff, B. A. (2018). The Translational Science Benefits Model: A New Framework for Assessing the Health and Societal Benefits of Clinical and Translational Sciences. Clinical and Translational Science, 11(1), 77–84. https://doi.org/10.1111/cts.12495.
Ramaswamy, R., Mosnier, J., Reed, K., Powell, B. J., & Schenck, A. P. (2019). Building capacity for Public Health 3.0: Introducing implementation science into an MPH curriculum. Implementation Science, 14(1). https://doi.org/10.1186/s13012-019-0866-6
The following section presents information on Pre-Conference workshops, as well as a selection of accepted abstracts presented at the SIRC 2024 Conference. Abstract inclusion for the SIRC 2024 Proceedings adhered to Implementation Research and Practice guidelines. Upon initial abstract submission to be considered for the SIRC 2024 conference in Spring 2024, corresponding authors indicated whether they were willing to have their abstract published in potential future proceedings. Only those who selected “yes” were contacted to confirm whether the abstract or related work had since been published or was under review elsewhere. Abstracts were excluded only when corresponding authors confirmed concurrent publication or review. For corresponding authors who did not respond by February 28, 2026, abstracts were included unless there was clear evidence of concurrent publication or review. All decisions were based on information available at the time of finalization and were documented accordingly.
Pre-Conference Workshops
Partnering for Sustainable Financing of Evidence-Based Practices in Implementation Research and Practice: Tools to Navigate the Storm
Authors
Alex R. Dopp, PhD, MA - RAND Corporation
Marylou Gilbert, JD, MA - RAND Corporation
Cherise Rohr-Allegrini, PhD, MPH - San Antonio AIDS Foundation
Sustained delivery of evidence-based practices (EBPs) is essential to their large-scale impact. Unfortunately, limited and fragmented funding is a major barrier to EBP implementation and sustainment in community-based service organizations (Dopp et al., 2020). This workshop aims to inform partnered strategic planning efforts among researchers, implementation practitioners (e.g., trainers, facilitators), and service organizations to identify sustainable financing strategies for EBPs. The workshop will first discuss eight domains of EBP sustainment capacity from the Public Health Sustainability Framework (Dopp et al., 2023, with a particular emphasis on the strategic planning and financing stability domains. Attendees will learn to self-assess their programs’ sustainment capacities, using tools based on the framework (e.g., Program Sustainability Assessment Tool; www.sustaintool.org). We will then present the Fiscal Mapping Process, a structured strategic planning tool that service organizations and coaches can use to select the optimal combination of financing strategies for EBP sustainment (Schell et al., 2013). Attendees will review the five steps of the Fiscal Mapping Process (identify resources needed, specify funding objectives, select financing strategies, create fiscal map, monitor/sustain) and consider how the tool could be applied to plan for sustainable funding in their own projects and partnerships. As a case example, we will discuss the experiences of an HIV service organization currently working to sustain a motivational interviewing-based brief intervention (MIBI) for substance use. The presenters will include researchers, facilitators, and representatives from the HIV service organization. Throughout the workshop, the service organization representatives will share invaluable perspectives on collaborating with implementation researchers and practitioners to address issues of sustainable funding, including application of the Program Sustainability Assessment Tool and the Fiscal Mapping Process. We will also highlight the process of adapting these resources for application in the novel setting of HIV service organizations. Overall, the workshop will be structured, but include ample opportunities for discussion, questions, and active reflection among attendees and presenters.
Learning Objectives:
Attendees will be able to describe EBP implementation and sustainment capacities that inform strategic planning and financing priorities
Attendees will be able to identify key steps and tools, derived from implementation theory and community-partnered practice, that can guide planning for sustainable financing of EBPs
Attendees will be able to approach community-academic partnerships in ways that effectively support and study issues of sustainable financing for EBPs
References
Dopp, A. R., Narcisse, M. R., Mundey, P., Silovsky, J. F., Smith, A. B., Mandell, D., Funderburk, B. W., Powell, B. J., Schmidt, S., Edwards, D., Luke, D., & Mendel, P. (2020). A scoping review of strategies for financing the implementation of evidence-based practices in behavioral health systems: State of the literature and future directions. Implementation Research and Practice. https://doi.org/10.1177/2633489520939980
Dopp, A. R., Gilbert, M., North, M., Martineau, M., & Ringel, J. S. (2023). The Fiscal Mapping Process: A Strategic Planning Tool for Sustainable Financing of Evidence-Based Treatment Programs in Youth Behavioral Health Services. RAND Corporation. https://doi.org/10.7249/TLA2678-1
Schell, S. F., Luke, D. A., Schooley, M. W., Elliott, M. B., Herbers, S. H., Mueller, N. B. & Bunger, A. C. (2013). Public health program capacity for sustainability: a new framework. Implementation Science, 8:15. https://doi.org/10.1186/1748-5908-8-15
Disclosures of interest: This workshop was supported by grants from the U.S. National Institute of Mental Health (R21MH122889; PI: Dopp) and the U.S. National Institute on Drug Abuse (R01DA052294; PI: Garner).
Advancing Health Equity Through Co-Creation Across the Spectrum of Research, Practice, and Policy
Authors
Paul Estabrooks, PhD - University of Utah
Mónica Pérez Jolles, PhD, MA - University of Colorado, Anschutz Medical Campus
Katy Trinkley, PharmD, PhD - University of Colorado, Anschutz Medical Campus
Borsika Rabin, PhD, MPH, PharmD - University of California, San Diego;
Amy G. Huebschmann, MD, MSc - University of Colorado, Anschutz Medical Campus
This workshop will leverage a variety of didactic, discussion, and interactive activities to support participants in applying novel dissemination and implementation science (DIS) tools and methods to co-create research projects, services, and policy that advance health equity. The ideal participants in this workshop are those with a basic understanding of DIS principles, theories, models, and frameworks (TMFs), and research methods. As envisioned, this will be an intermediate level workshop focused on co-creation processes of (1) engaging representatives from a diverse set of key community or clinical partners, (2) planning and designing new or adapted interventions and/or implementation strategies that allow all participants (organizational, community, and individual) an opportunity to benefit, and (3) iterative evaluation strategies that ensure relevant outcomes are assessed for all partners to allow for advances in local practice improvement and policy development to achieve sustained, generalizable public health impact. Suited participants for this workshop are IS researchers and implementation practitioners with a basic understanding of DIS principles, TMFs, and research methods. For those in community or clinical practice settings and are interested in implementation, we anticipate some collaborative experience with research or program/policy design would be of benefit.
Learning Objectives:
Describe how to apply a DIS TMF, using the Practical, Robust Implementation and Sustainability Model (PRISM) which includes the RE-AIM outcomes as an example, to the process of co-creation and planning for impact
Develop an assessment of an example project using the iPRISM webtool to optimize program impact, equity and sustainability
Apply knowledge gained from learning about a newly developed co-creation guidebook to consider co-creation of interventions for advancing health equity
Disclosures of interest: None
Considering Context Across the Lifespan of Implementation
Authors
Julia Moore, PhD - The Center for Implementation
Sobia Khan, PhD, MPH - The Center for Implementation
This workshop will explore practical ways to understand and address contextual factors (organizational and system factors), including considerations for readiness, adaptations, and sustainability. There are many frameworks for assessing context, but relatively little practical guidance exists on selecting strategies to address contextual factors. This workshop will focus on these factors. This workshop will introduce Context Compass, an interactive online tool that participants can use to assess and address context in their work. Workshop participants will leave with approaches, a tool, and strategies that they can use to address contextual challenges.
Learning Objectives:
Apply a framework for assessing context across the lifespan of implementation
Identify factors impacting implementation (barriers and facilitators) at multiple levels
Consider the importance of different factors over the lifespan of implementation (e.g., decision to adopt, assessing fit, readiness, implementation, sustainability)
Select appropriate strategies to address challenges and leverage facilitators at different points during the implementation process.
References
Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., Lowery, J., et al. (2022). The updated Consolidated Framework for Implementation Research based on user feedback. Implementation Science, 17, Article 75. https://doi.org/10.1186/s13012-022-01245-0
Fagan, A. A., Bumbarger, B. K., Barth, R. P., Bradshaw, C. P., Cooper, B. R., Supplee, L. H., & Walker, D. K. (2019). Scaling up evidence-based interventions in U.S. public systems to prevent behavioral health problems: Challenges and opportunities. Prevention Science, 20(8), 1147–1168. https://doi.org/10.1007/s11121-019-01048-8
Feldstein, A. C., & Glasgow, R. E. (2008). A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Joint Commission Journal on Quality and Patient Safety, 34(4), 228–243. https://doi.org/10.1016/S1553-7250(08)34030-6
Fleuren, M., Wiefferink, K., & Paulussen, T. (2004). Determinants of innovation within health care organizations: Literature review and Delphi study. International Journal for Quality in Health Care, 16(2), 107–123. https://doi.org/10.1093/intqhc/mzh030
Flottorp, S. A., Oxman, A. D., Krause, J., Musila, N. R., Wensing, M., Godycki-Cwirko, M., Baker, R., & Eccles, M. P. (2013). A checklist for identifying determinants of practice: A systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implementation Science, 8, Article 35. https://doi.org/10.1186/1748-5908-8-35
Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4), 581–629. https://doi.org/10.1111/j.0887-378X.2004.00325.x
Gurses, A. P., Marsteller, J. A., Ozok, A. A., Xiao, Y., Owens, S., & Pronovost, P. J. (2010). Using an interdisciplinary approach to identify factors that affect clinicians’ compliance with evidence-based guidelines. Critical Care Medicine, 38(8 Suppl), S282–S291.
Harvey, G., & Kitson, A. (2016). PARIHS revisited: From heuristic to integrated framework for the successful implementation of knowledge into practice. Implementation Science, 11, Article 33. https://doi.org/10.1186/s13012-016-0398-2
Kenworthy, T., Domlyn, A., Scott, V. C., Schwartz, R., & Wandersman, A. (2023). A proactive, systematic approach to building the capacity of technical assistance providers. Health Promotion Practice, 24(3), 546–559. https://doi.org/10.1177/15248399221080096
Means, A. R., Kemp, C. G., Gwayi-Chore, M. C., et al. (2020). Evaluating and optimizing the Consolidated Framework for Implementation Research (CFIR) for use in low- and middle-income countries: A systematic review. Implementation Science, 15, Article 17. https://doi.org/10.1186/s13012-020-0977-0
Nilsen, P., & Bernhardsson, S. (2019). Context matters in implementation science: A scoping review of determinant frameworks that describe contextual determinants for implementation outcomes. BMC Health Services Research, 19, Article 189. https://doi.org/10.1186/s12913-019-4015-3
Wensing, M., & Grol, R. (2013). Determinants of change. In R. Grol, M. Wensing, M. Eccles, & D. Davis (Eds.), Improving patient care: The implementation of change in clinical practice (pp. 139–150). Wiley.
World Health Organization. (2011). Identifying and addressing barriers to implementing policy options. In SURE guides for preparing and using evidence-based policy briefs. https://epoc.cochrane.org/sure-guides
Disclosures of interest: None
Justice-Centered Implementation Research and Practice: Laying the Foundation and Building a Collective Vision
Authors
Megan Stanton PhD, MSW - Eastern Connecticut State University
Katie McCormick, PhD, MSW - University of Texas at Austin
Samira Ali, PhD, MSW - University of Houston
Are you interested in promoting equitable implementation in your work? Are you keenly aware of the ways power and social injustice impact implementation decision-making in research, policy, and practice, and want to learn more? Though, increasingly, implementation researchers and practitioners identify equity and justice as critically important considerations in implementation, there is less guidance regarding how to integrate an equity focus into implementation research and practice and there are limited opportunities to learn specific skills and acquire conceptual tools necessary to center equity in implementation. Further, though individual researchers and practitioners have advanced equity-centered implementation science, there have been few opportunities to gather as a community to take stock of current gaps and opportunities regarding equity in implementation. In this workshop, presenters will review the current research and theory related to power and implementation, including their work articulating a typology of power generated through implementation. Presenters will then discuss their use of structured critical questions to guide analysis of power in implementation projects. In doing so, presenters will draw on the practical experiences of over 75 HIV service organizations they have coached through the implementation process. Attendees will be asked to bring a project – in any stage of implementation – and will work through a guided reflection process to analyze the role of power in their work and build strategies to enhance equity and justice in their projects moving forward. The second half of the pre-conference workshop will be devoted to building a collective vision of justice-centered implementation. We will engage in a facilitated, collaborative process to identify strengths, gaps, and future actions to promote equity and justice in implementation research and practice, culminating in an action and advocacy agenda. This agenda will be taken up as part of the launch of the ‘Health Justice Implementation Lab”, an interdisciplinary collective of researchers and practitioners committed to justice-centered implementation research and practice. This workshop will be led a group of scholars associated with the SUSTAIN Wellbeing COMPASS Coordinating Center and experts at the Eastern Connecticut State University, the University of Houston, and the University of Texas at Austin
Learning Objectives:
Describe how power and social injustice impact implementation
List and define three types of power that impact implementation
Apply critical questions to identify how participants’ implementation projects generate and are influenced by epistemic, material, and discursive power
Develop a plan to enhance the equity potential of their projects
Engage in collective vision building for justice-centered implementation
Build a justice-centered implementation action agenda for improvements on your project, future projects and the broader field of implementation science
Disclosures of interest: None
Symposia
Policy/Practitioner Track
Strategic Synergy: Can Implementation Science Benefit from the Scientist-Practitioner Model?
Authors
Will Aldridge - The Impact Center at FPG Child Development Institute, University of North Carolina at Chapel Hill
Sarah Walker - University of Washington
Capri Banks - The Impact Center at FPG Child Development Institute, University of North Carolina at Chapel Hill
Jessica Reed - The Impact Center at FPG Child Development Institute, University of North Carolina at Chapel Hill
Stephanie Brooks - University of Alberta
Denise Thomson - Alberta SPOR SUPPORT Unit, University of Alberta
Tracy Wasylak - Alberta SPOR SUPPORT Unit, University of Calgary
Stephanie Montesanti - School of Public Health,
University of Alberta and the Alberta SPOR SUPPORT Unit
Rohit Ramaswamy - James M. Anderson Center for Health Systems Excellence, Cincinnati Children's Hospital Medical Center
Symposium Overview Summary
Over the past year, the SIRC Practitioner Network has hosted a special series entitled, Contending with the Implementation Research-Practice Gap. This series has explored lived experience of the implementation research-practice gap across diverse professional identities, conversed about the promises and pitfalls of implementation science and how implementation practitioners may contribute to the acceleration of promises and mitigation of pitfalls, and sought to increase collaboration among researchers and practitioners.
One model for increasing the strategic synergy of implementation research, practice, and policy that has gotten cursory discussion is the scientist-practitioner model, originally developed in the field of clinical psychology. Outlined in 1949 in Boulder, Colorado, the main goal of the model is to develop professionals who can apply scientific knowledge to their practice work and who can generate new knowledge, empirical findings, or interventions to move the field forward (Jones & Mehr, 2007). Mallinckrodt, Miles, & Levy (2014) more recently proposed expanding the model by adding a social justice advocacy component, which is intended to draw attention to addressing the social context, including policy, implicated in creating presenting needs and gaps in practice settings.
This symposium will illustrate three variations on how the scientist-practitioner model can be applied in implementation science. The first presentation will focus on the application of the model from the perspective of an individual who primarily identifies as a researcher. The second presentation will focus on the application of the model among a group of individuals primarily working in implementation practice. The third presentation will focus on an organizational application of the model intended to leverage collaboration among teams of implementation researchers and practitioners. Following these presentations, a discussant with a broad perspective on both implementation research and practice will summarize key takeaways and discuss implications for the future development of the field.
Title 1: The Benefits of Policy Practice Expertise for D&I Scientists
Background: The challenges of sharing health knowledge across universities and government are well documented. Each sector has a unique language and routines the other has to learn in order to effectively collaborate. Being a part of or leading teams that beneficially influence real world processes outside of research-funded studies greatly enhances depth of insight for scientist-practitioner researchers studying implementation strategies at the policy and system level.
Approach: The CoLab for Community and Behavioral Health Policy (CoLab) aims to achieve real world policy and systems impact through the use of a variety of strategies known to implementation science and knowledge mobilization science. The CoLab actively pursues competitive research funding to study novel strategies as well as document outcomes from the use of known strategies from research funders including William T Grant, NIDA, Annie E Casey, and the Robert Wood Johnson Foundation.
Outcomes: Maintaining credibility in the research sector and government sector as an effective policy intermediary (think thank) is challenging. It requires team infrastructure that can produce rigorous research alongside the dynamic and fast-moving pace of policy cycles. At the same time, doing so enhances the depth of insight within both activities: Policy products are informed by up-to-date research evidence and active debates in the academic field, while research is informed by the realities of the policy making context.
Next Steps: An agenda to build more scientist-practitioners pursuing competitive research in policy and systems implementation science would greatly benefit by a rethinking of the promotion process within research institutions. University departments aiming to foster these careers should support interested faculty to develop a blend of government, philanthropic and research funding, produce a scholarly portfolio of policy and peer reviewed publications, and participate on government commissions and committees.
Title 2: Integrating the Scientist-Practitioner Model into Implementation Support Practice
Background: There is an identified gap between research and practice in implementation science. Implementation support practitioners (ISPs; e.g., Albers et al., 2020) translate theories, models, frameworks, and strategies from implementation science to improve capacity and performance for the use of evidence-based interventions in health and human services. ISPs are actively learning, collecting data, and developing questions that should be informing future implementation research.
Approach: Over the past seven and a half years, our team has developed and refined a theoretically grounded, evidence-informed implementation support practice model based in core practice components and activities (Aldridge et al., 2023a & 2023b). The model is backed by eight practice principles, including the implementation scientist-practitioner model, which advocates that ISPs continually advance their competency in the transdisciplinary science of implementation, including the dual roles of implementation research and practice.
Outcomes: Promoting the development of competencies across implementation research and practice has required several structural supports for our team. We have developed in-service professional development structures and supports to ensure that ISPs continually incorporate developments in implementation research. We have built evaluation and improvement capacity, including mixed methods data collection systems, to both improve our practice activities and contribute new learning to the field while formulating novel questions that can advance practical and useful research. We have developed professional networks across research and practice that allow us to be bridge builders. Our pursuit of the scientist-practitioner model has impacted how we show up in practice settings. Examples will be shared.
Next Steps: Our experience demonstrates the potential value of pursuing the scientist-practitioner model through in-service, and possibly pre-service, professional development and teaming structures. Building individual and team competencies across implementation research and practice may lead to more practical and impactful improvements in health and human service systems and the easing of the research-practice gap.
Title 3: Applying the Principles of the Scientist-Practitioner Model at the Organization Level: Alberta, Canada's Implementation Science Collaborative
Background: The Alberta Strategy for Patient-Oriented Research Support Unit (AbSPORU) is an intermediary mandated to strengthen Alberta, Canada's evolving learning health system. One of AbSPORU's strategies to meet its mandate is to increase the use of implementation science in the province. AbSPORU's Implementation Science Collaborative is a flagship initiative that integrates implementation practice and research expertise to facilitate health innovation spread, scale and sustainment.
Approach: The Implementation Science Collaborative brings together leaders from various health communities (e.g., clinical settings, primary health, public health) to tap into local practical knowledge of widespread implementation barriers. AbSPORU identifies implementation initiatives that have the potential to generate knowledge of how to overcome the barriers identified by health leaders. A working group, comprised of local implementation practice experts, co-designs implementation research questions with the innovation teams. The innovation teams are then connected with a panel of international implementation science leaders to develop associated implementation research designs and measures. AbSPORU then catalogues lessons learned to inform future implementation province-wide.
Outcomes: This model gives innovation teams access to practical and scientific implementation expertise. These embedded teams are PhD-trained with qualitative and quantitative expertise but have limited exposure to implementation science. The Implementation Science Collaborative 1) gives innovation teams on-site experience with implementation science, 2) generates local implementation science evidence, and 3) provides a community for people to learn about implementation research and practice from one another. Thus, when applied at the organization level, the scientist-practitioner model performs as implementation capacity and research infrastructure.
Next Steps: Using a transdisciplinary approach, AbSPORU's Implementation Science Collaborative promotes a model of intermediaries supporting the effective implementation of initiatives and the development of implementation capacity and research infrastructure. An impact assessment is currently in preparation to measure the Collaborative's success and inform future adaptations.
References
Aldridge, W. A., Roppolo, R. H., Brown, J., Bumbarger, B. K., & Boothroyd, R. I. (2023). Mechanisms of change in external implementation support: A conceptual model and case examples to guide research and practice. Implementation Research and Practice, 4, 26334895231179761-26334895231179761. https://doi.org/10.1177/26334895231179761
Aldridge, W. A., Roppolo, R. H., Chaplo, S. D., Everett, A. B., Lawrence, S. N., DiSalvo, C. I., Minch, D. R., Reed, J. J., & Boothroyd, R. I. (2023). Trajectory of external implementation support activities across two states in the United States: A descriptive study. Implementation Research and Practice, 4, 26334895231154285-26334895231154285. https://doi.org/10.1177/26334895231154285
Jones, J. L., & Mehr, S. L. (2007). Foundations and Assumptions of the Scientist-Practitioner Model. American Behavioral Scientist, 50(6), 766-771. https://doi.org/10.1177/0002764206296454
Mallinckrodt, B., Miles, J. R., & Levy, J. J. (2014). The scientist-practitioner-advocate model: Addressing contemporary training needs for social justice advocacy. Training and Education in Professional Psychology, 8(4), 303-311. https://doi.org/10.1037/tep0000045
Advancing Implementation Policy-Practice-Research Synergy at Scale: A Panel Discussion with a National Network of State Intermediary Organizations
Authors
Brian Bumbarger - Science, Systems & Communities Consulting, LLC
Cynthia Weaver - Annie E. Casey Foundation
Stephen Phillippi - Louisiana State University
Michael Southam-Gerow - Virginia Commonwealth University
Jeff Vanderploeg - Child Health and Development
Institute
Sarah Walker - University of Washington
Robert Franks - The Baker Center for Children and Families/Harvard Medical School
Kate Volti - Meadows Mental Health Policy Institute
Rafaella Sale - Virginia Commonwealth University
Symposium Overview Summary
Evidence-based practice sits within broader, complex policy and funding systems (i.e. “infrastructure”) that represent enabling or inhibiting contexts. Implementation efforts get “swallowed up” in larger complex policy landscapes or are ill-fit to existing workforce or funding infrastructures. A number of states have established system innovation centers (SICs) – state intermediary organizations working synergistically across policy, practice, and research to create sustainable, equitable behavioral health solutions at scale. Seven such SICs have recently come together to form a learning collaborative, build from existing implementation science, and advance the pace of innovation across states. Each SIC developed uniquely within the specific context of their states’ resources and priorities, creating a test-bed for policy and practice innovation informed by research and a rich diversity of ideas and approaches that often challenge assumptions about what is required to move science to policy and practice.
This symposium will explore the diverse structures of SIC and their cutting edge approaches to creating infrastructure necessary to address the most pressing challenges to evidence-based behavioral health policy and practice. The symposium chair, from a national philanthropy, will guide discussion with SIC leaders to discuss tools, resources, and strategies they’ve developed and their approaches to innovating policy and practice within their states, including:
How they engage "consumers" or communities, policy makers, and researchers as co-creators?
How research is embedded in their work and how they maintain a focus on scientific rigor with policymakers and practitioners?
How implementation models, theories or frameworks influence the intermediary's design or their thinking or approach to policy work in particular?
How they are working to center equity across these three potentially synergistic constituencies?
How have process and outcomes measurement approaches synergistically impacted policy?
An internationally-recognized subject matter expert will reflect on the panel and generalizable lessons from this collective that can push the field forward.
Title 1: Centered on relationships to promote access to quality services in Virginia
Background: Partnerships among stakeholders in behavioral health established on trust and respect and sustained through ongoing relationship and collaboration have potential to begin unraveling longstanding public mental health challenges concerning access to quality services across multiple communities. The Center for Evidence-based Partnerships in Virginia (CEP-Va) was created to leverage this notion.
Approach: Beginning at its founding, CEP-Va has sought to deliver on its vision, mission, and values in working with its multiple partners and collaborators, including state agencies, provider companies, purveyors, MCOs, local workers, and others. Although dedicated to the projects and deliverables in its state contracts, CEP-Va has maintained a focus on the overall goals of the state to improve access to services through multiple initiatives.
Outcomes: Several projects exemplify the approach CEP-Va has taken, including (a) needs assessment for implementation of evidence-based programs (EBPs) pursuant to the Family First Prevention Services Act; (b) statewide fidelity monitoring of key EBPs; and (c) projects to synergize state investments in specific EBPs, like multisystemic therapy, functional family therapy, and high fidelity wraparound. Specifically, these projects resulted in (a) a series of policy and practice recommendations to the state, some of which have already been implemented; (b) development of fidelity reporting to meet Family First Prevention Services Act requirements that also provide key data for policymakers and administrators; and (c) inception of several cross-agency and multi-group conversations leading to development and testing of solutions to key barriers to implementation of quality services.
Next Steps: Although we have experienced notable successes, we remain keenly aware of the challenges facing Virginia and our own work. We will outline a few of our aspirational future directions.
Title 2: Reaching Beyond Political Term Limits: A University Intermediary for Behavioral Health Policy & Practice Development & Sustainability
Background: The need to support the development of quality behavioral healthcare in Louisiana was demonstrated in a 2016-17 survey that found over 40% of Medicaid mental health service providers not using research informed practices. But as policy makers and state leadership changes, who remains to plan, sustain, and critically evaluate policy changes handed off to new administrations? One answer is university intermediaries. Universities have long stood fast amidst the winds of change and are, in fact, the developers of future government leaders. The role to inform and educate state policy makers and support practical approaches to implementation are a natural fit for university collaboration.
Approach: The Center for Evidence to Practice is a collaboration between Louisiana State University Health Sciences Center, School of Public Health – Behavioral and Community Health Sciences Program and the Louisiana Department of Health – Office of Behavioral Health. The Center was created to collaboratively improve access to quality care and support the state's focus on provider readiness, supportive policies, implementation, and evaluation.
Outcomes: The Center has continued to expand efforts with the state to review policy and execute new approaches. This includes expanding eight evidence based behavioral health programs by 400% in just five years; co-leading the rapid shift of policy and care to telehealth, with fidelity, during COVID; co-developing and executing policy and practice for a new statewide workforce to deliver Louisiana's behavioral health crisis response system; and more.
Next Steps: We’re focused on bringing more technology to this collaboration between the university, state department of health, policy makers, providers, and managed care organizations, using more public facing data visualization about access and outcomes related to behavioral healthcare. We stand ready to provide objective information about policy recommendations and critically evaluate those that are implemented, even long after many policy makers have moved on.
Title 3: Key Infrastructure Elements for Advancing System of Care Implementation in Children's Behavioral Health: The Critical Role of the Intermediary Organization
Background: Systems require significant infrastructure to be able to provide effective interventions. In the area of children's behavioral health, infrastructure is proposed as the link between system of care values and principles and the implementation of a comprehensive service array. However, there is a need for consensus on the critical elements of system-level infrastructure and how to deploy it to support specific implementation initiatives. Within large public systems, infrastructure development is often achieved through synergistic public-private partnership involving state agency policy makers and one or more intermediary organizations.
Approach: We propose five critical elements of system-level infrastructure: 1) integrated governance and decision-making structure; 2) structures and processes for braiding funding; 3) central point of access for information and referral; 4) workforce development, training, and coaching in effective practices; 5) data and quality improvement.
Outcomes: The presentation will share examples from an intermediary organization that has experienced success in partnering with the state to build core system infrastructure and leverage it to advance a more comprehensive and effective service array. Three system and practice level initiatives will be used to highlight the approach: 1) development of a statewide strategic plan for behavioral health workforce enhancement; 2) development of a system-level data dashboard; 3) design and implementation of crisis-oriented services (i.e., mobile response, urgent crisis centers).
Next Steps: Insufficient investment in system infrastructure may be one of the reasons children's behavioral health systems do not produce better outcomes for youth and families. Future research is needed to confirm the most critical elements of system-level infrastructure, determine effective approaches for developing and deploying infrastructure, and investigate whether and how infrastructure contributes to more equitable access, quality, and outcomes of care.
Title 4: Why not Just Hire McKinsey? A Critical Reflection on the Purpose of a State-Supported Policy Intermediary
Background: Technical policy products, such as policy briefs and strategic plans, center epistemic knowledge (scientific and technical information) but do not guarantee praxis (thoughtful action) which is needed for successful policy design and implementation. Scholarly work on evidence use supports the establishment of sustained partnership between research and government institutions. This is because the practical knowledge of how individuals within complex systems like to work and make decisions is critical to effective collaboration and action. This paper will explore learnings from a decade of work as a policy intermediary, highlighting where epistemic knowledge (evidence) was enhanced by relational knowledge of the state political ecosystem to create public value and impact.
Approach: The CoLab for Community and Behavioral Health Policy (CoLab) is a youth wellness policy think tank operating out of an academic medicine environment. A contract from the state Health Care Authority makes up less than a fourth of the CoLab's annual operating budget. However, the role as a state contractor positions CoLab faculty and staff to work closely with state government staff and other system partners involved in mental health policymaking (insurance payors, service providers, etc).
Outcomes: These relationships have spawned innovations not funded or only partially funded by HCA, including efforts related to workforce access, value-based care, and the scale-up of evidence informed treatments (among others). The sustained funding of a policy intermediary by the state for activities that could have otherwise been subcontracted to consulting agencies maximized public value because it sustained collaborative relationships over time that produce more innovation in the ecosystem.
Next Steps: As governments look for external expertise to supplement internal capacity, decision-makers should consider funding sustained, intermediary partners as part of their expert portfolio. Supporting sustained relationships with policy intermediaries over time will maximize the value of government investment by fostering innovative ecosystems.
Conceptualizing Technical Assistance Implementation for Broad Scale Dissemination: Three Practical Examples from Regional and Statewide Prevention Support Systems
Authors
Jochebed Gayles - Pennsylvania State University
Janet Welsh - Penn State
Roger Spaw - Pennsylvania State University
Jordan Joyce - Pennsylvania State University
Kristin Camplese - Pennsylvania State University
Kathryn Bruzios - Washington State University
Brittany Cooper - Washington State University
Michelle Frye-Spray - University of Washington
Kevin Haggerty - University of Washington
John Briney - University of Washington
Sarah Chilenski - Pennsylvania State University
Melissa Tomascik - Pennsylvania State University
Daniel Perkins - Pennsylvania State University
Will Aldridge - University of North Carolina at
Chapel Hill
Symposium Overview Summary
Training and technical assistance centers are crucial in supporting and enhancing individuals’, groups’, and organizations’ prevention capacity and planning efforts. These prevention support systems provide resources, expertise, and guidance to communities in need, helping them build capacity and implement evidence-based practices for preventing various social issues such as substance abuse, mental health disorders, and community violence. However, many challenges exist to delivering technical assistance at a large scale. Often, geographical barriers and reach pose a challenge to underserved areas that may not have access to traditional in-person support. Challenges to delivering relevant and digestible information to a varied audience also come into play. Using digital platforms allows for scalability and cost-effectiveness, making disseminating technical assistance services to a larger audience easier. One of the key challenges is ensuring equal access and digital literacy across all communities. Another challenge includes maintaining stakeholder engagement and participation over time. This session presents new innovative approaches for the design and wide-scale dissemination of training and technical assistance aimed at enhancing prevention delivery systems knowledge and general prevention capacity. Drawing on three distinct examples addressing the aforementioned challenges, our session will dissect the methodologies, successes, challenges, and lessons learned faced by each project in achieving broad-scale dissemination of support services. Through these practical examples, we aim to generate a robust discussion on the best practices, lessons learned, and future directions for conceptualizing and implementing effective technical assistance at a broad scale to reach greater numbers of service providers within the prevention workforce.
Title 1: Development and Implementation of a Prevention Learning Portal and Strategic Marketing Campaign for Workforce Development: A State-University Partnership that is Taking Technical Assistance to Scale
Background: Workforce development among prevention professionals remains a central challenge within implementation practice. Training and technical assistance (TTA) is a key support mechanism that can enhance the prevention capacity of human service providers. However, moving the needle in workforce development and reaching a broader audience across diverse communities requires innovative approaches to TTA dissemination. This presentation describes the efforts of the Evidence-based Prevention and Intervention Support (EPIS) project, formed in 2008 from a state-university partnership, in designing, implementing, and evaluating broad-scale TTA across the state of Pennsylvania.
Approach: The Prevention Learning Portal (PLP) was created to address workforce challenges and expand TTA reach. The goals include (1) providing resources on scientific approaches to primary prevention and (2) proliferating information for multiple sectors and systems. To achieve our goals, EPIS developed four objectives: 1) Build a repository of resources; 2) Establish a platform to house, distribute, and track utilization; 3) Distribute a marketing campaign to raise awareness/engagement; and 4) Develop a monitoring and evaluation system. EPIS engaged external consultants with expertise in media development, instructional design, learning management, and communications to help achieve these goals.
Outcomes: The PLP went live on October 31st, 2023. Eight online courses are available, with ten more slated for distribution this year. We ran a 16-week marketing campaign that ended in February 2024. We currently have over 300 registered users and high satisfaction ratings.
Next Steps: We are currently evaluating data about reach (who and where) and use to identify which resources are most widely used and by whom and which are less useful. We plan to adjust the material's content and format to better meet stakeholder groups’ needs. We are also using data to identify underserved sectors, demographic groups, and regions throughout the state to strategically tailor communication, outreach, and content to improve engagement among these groups.
Title 2: SAMHSA's Northwest Prevention Technology Transfer Center: Developing a Prevention Support System to Enhance Prevention Workforce Capacity
Background: The effective delivery and broad scale impact of evidence-based programs aimed at improving public health requires multi-layered training and technical assistance (TTA). The Interactive Systems Framework (ISF) and evidence-based system for innovation support (EBSIS) represent two models whose goal is to address this need and emphasize moving research into practice. However, limited practical information exists on how such systems are developed and maintained. Expanding upon the ISF and EBSIS, this presentation will describe the intentionality behind building a regional TTA center (Support System) with a goal of enhancing prevention workforce (Delivery System) capacity.
Approach: In 2019, SAMHSA funded 10 regional Prevention Technology Transfer Centers (PTTC) to provide TTA to enhance prevention workforce capacity. The Northwest PTTC (NWPTTC), aims to help communities in Alaska, Idaho, Oregon, and Washington, reduce substance misuse by advancing prevention workforce capacity to utilize prevention science principles. This occurs through the development and dissemination of tools and strategies via a three-tier conceptual model that includes basic-, targeted-, and intensive-TTA.
Outcomes: We integrated the EBSIS logic model with the NWPTTC tiered-TTA model to develop an enhanced model that guides our regional approach. The model has three iterative steps: 1) Identifying Goals/Current Capacity, 2) Developing and Delivering Support Components, and 3) Assessing Support Components. Our presentation will describe the processes within these steps, offer a visual depiction of the model, and describe lessons learned from five years of using our model to guide TTA provided to 11,983 prevention practitioners.
Next Steps: This presentation improves our understanding of the ISF and EBSIS by illustrating how they align with a regional tiered-TTA system. Through multisector-coordinated efforts, the NWPTTC has been able to extensively reach the prevention workforce and support equitable implementation of prevention efforts. However, more work is needed to determine to what degree support systems are reaching their goals.
Title 3: Coordination and Dissemination of Technical Assistance to Support Rural Communities: A Five-State Steering Committee Approach
Background: Given data on rural substance misuse, evidence-based solutions regarding mental health and substance misuse need to reach rural populations. For instance, in 2022, the rate of drug overdoses was 20% higher in rural communities compared to urban communities in Pennsylvania, and in Maryland, rural residents were twice as likely to overdose as those in urban areas. As a result, the Substance Abuse and Mental Health Services Administration released the Rural Opioid Technical Assistance – Regional funding effort to support partnerships with the Cooperative Extension Systems in land grant universities to better reach rural populations.
Approach: Health and Human Services Region 3, including Delaware, Maryland, Pennsylvania, Virginia, and West Virginia, created a region-wide steering committee in response to this call. The committee includes Extension, Prevention Science, and Public Health faculty and staff across the five states. They are disseminating three training and technical assistance opportunities to rural substance use disorder and mental health stakeholders: (1) the Prevention Learning Portal, a series of asynchronous, web-based modules and a resource library focused on principles of intervention, prevention, and implementation science; (2) monthly webinars on special topics; and (3) training in Mental Health First Aid. Evaluation and monitoring data is routinely collected.
Outcomes: The infrastructure supporting region-wide operations was developed and the committee meets monthly to review and make decisions. Progress has been made across all three training and technical assistance areas, and about half of training attendees reported living in rural areas. Recently, the steering committee collected a training, and technical assistance needs survey for Extension faculty and staff. These data are being reviewed.
Next Steps: The Region 3 Steering Committee is continuing its work. Discussion will focus on how to apply learnings from the Extension Survey as well as other evaluation data into future regionwide training and technical assistance activities.
Co-Design: What if we didn't create the research/practice gap to begin with?
Authors
Mike Kenney - Public Children Services Association of Ohio
Sarah Kaye - Kaye Implementation & Evaluation
Lara Laroche - Franklin County Children Services
Symposia Overview Summary
Historically, implementing programs, models, and other change efforts in child welfare has focused on rules and mandates designed for more compliance and consistency. Child welfare is not short on rules and mandates. And yet, we continue to achieve the same results, year after year, and we continue to create more federal, state, and local rules and mandates.
In 2021, PCSAO set out on a journey to develop a Shared Practice Model for Ohio's 88 public children services agencies. The approach began with curiosity, humility, and the foundational belief that co-designing change with families, caseworkers, and leaders would result in a more equitable and effective model. The model developed by this team in 2022 is called Practice in Action Together (PACT), which represents the working alliance among caseworkers and families. The theory of change for PACT is rooted in the belief that relationships are the leading indicator of any and all systemic outcomes.
PCSAO, Kaye Implementation and Evaluation, and eight Ohio public child welfare organizations have co-designed a novel theoretical framework called Cultural Implementation and are currently in the installation stage of implementation. This symposium will include three unique oral presentation, each aligned and complementary to one another.
PACT Co-Design – The unique process and approach utilized to address power differential and establish strong working alliances among families, caseworkers, and leaders, resulting in a model designed to elevate healing, build relationships, and achieve equity.
Behavioral Research Project – The findings of this study support the validation of the PACT model framework and was utilized to define and refine the model outcomes.
Cultural Implementation – The operationalization of seven guiding principles, based upon the understanding that how we provide services is more impactful than solely what services are provided.
Title 1: Everything is a parallel process: Aligning our implementation with our service delivery
Background: The objectives of this design effort were to 1) develop a shared practice model for Ohio's public child welfare agencies, 2) to establish a team of diverse voices including families, caseworkers, leaders, and subject matter experts to co-design the model and implementation plan, and 3) ensure scalability, sustainability, and evidence were key drivers of the model and strategies.
Approach: The initial approach included an environmental scan, qualitative and quantitative reviews of 15 other state child welfare practice models, and more than 40 stakeholder interviews. Champions were nominated and recruited to represent rural, urban, and geographically diverse parts of Ohio, and a third-party facilitator was hired to allow for full participation of all members of the group and to ensure power differentials were addressed. A 12-month co-design process was facilitated using techniques including, shared learning, brokering, teaming, cultivating champions, communication, and relationship development
Outcomes: The primary outcome of the co-design process was the development of a practice model comprised of a purpose/vision, values, behaviors, and outcomes which families, caseworkers, and leaders all “saw themselves” reflected in. The second outcome was that over 90% of the design team members volunteered to spend an additional 12 months helping to co-design the implementation team. They found their work together rewarding, rejuvenating, and empowering. After the completion of the plan and strategies, eight agencies applied to be a part of the initial cohort and initial installation activities began in January 2024.
Next Steps: In a parallel process consistent with the statewide co-design approach, Local Advisory Groups comprised of families, caseworkers, and leaders are being developed within the eight agencies. Similar implementation support strategies that were utilized at the state level are being conducted at the local level with an emphasis on capacity building and sustainability.
Title 2: Proof of concept: Co-designing measures to co-defined relational outcomes
Background: The purposes of the Behavioral Research Project were to (1) co-define the model's outcomes with caseworkers and family members, and (2) test the relationship between the PACT behaviors and co-defined outcomes.
Approach: The research team hosted focus groups with family members and caseworkers to co-define and operationalize PACT outcomes. Using participants’ words and concepts, researchers compiled the PACT survey. Together, the research team and pilot reviewers finalized two parallel versions of the survey: one for caseworkers, one for family members. Workers in 32 agencies were asked to complete a survey, and to invite families on their caseload. Almost 250 individuals, including 162 workers and 85 family members participated. Respondents answered questions based on the individual with whom they had the best working relationship.
Outcomes: Family members and workers reported high levels PACT behaviors. Workers reported slightly higher rates of PACT behaviors than family members. Most workers and family members identified Respecting, Listening, and Being Vulnerable as the most important worker behaviors for achieving PACT outcomes. Family members and workers reported high levels of PACT outcomes. Families reported significantly higher levels of mutual honesty than workers. All PACT behavior sets were correlated* with each relational outcome among both workers and families. Correlations between PACT behaviors and relational outcomes were stronger among families than workers, suggesting that PACT behaviors might be particularly important to families.
Next Steps: This study offered encouraging initial data to support the PACT program theory. This study intentionally focused on gathering input from the strongest worker/family member relationships to maximize learnings from successful working relationships. The survey sample was predominantly white and workers and family members of color were underrepresented in this study. It is unknown whether the results observed in this specific sample would generalize to broader and more diverse populations, so continued research is needed.
Title 3: A theoretical framework contending that how you implement is far more important than what you implement
Background: Culture lives in habits and is driven by what a group of people believe and the behaviors they engage in on a daily basis. A 2020 MIT study found no correlation between official values and actual corporate culture. In child welfare we call this the values-practice gap. This gap is the space between what we believe about the families we serve, and our behaviors demonstrated in response to their needs and challenges. To address this gap, the PACT partner organizations came together to develop an innovative, adaptive, and responsive implementation framework.
Approach: The stage-based approach of Cultural Implementation includes a set of initial installation steps focused on ensuring agency leadership is aligned with the values, behaviors, and implementation strategies of PACT, followed by steps focused on building the capacity and ownership of Local Advisory Groups. The two defining components of Cultural Implementation are 1) the focus on empowering caseworkers to debate, discuss, practice, and teach others the practice behaviors within the model, and 2) operationalizing seven guiding principles at the local level. An example of these two components is the use of a see one – do one – teach one approach to learning and development.
Outcomes: To date, the primary outcome achieved has been the application and full commitment of eight public child welfare organizations.
Next Steps: This theoretical framework has wide-ranging implications within the Implementation Science field relating to adaptive capacity, power-sharing, and collaborative decision making as primary drivers of scalability and sustainability of change efforts. The research and implementation teams are conducting extensive and ongoing process documentation and have a shared agreement with the Local Advisory Groups to test outcome data collection methods and Cultural Implementation strategy effectiveness. 2024 installation activities also include the development of a comprehensive evaluation plan.
From application to adaptation: 10 years of evolution in applying implementation science
Authors
Jenna McWilliam - Triple P International
Sara van Driel - Triple P America
Courtney Towne - Triple P America
Veronica Villablanca - Triple P Latin America
Jacquie Brown - Triple P International
Symposia Overview Summary
Triple P International is a purveyor organisation that has invested in the application of implementation science for more than 10 years. From the development of the initial Triple P Implementation Framework, TPIF 1.0, to the current Triple P Implementation Framework TPIF 3.0, there has been an ongoing process of refining and revising implementation support provided by the purveyor organisation in response to the emerging science and the field experience.
The three presentations in this symposium will provide insight into the key learnings and developments of applying implementation science, describe TPIF 3.0 – the resources and supports that are provided, the use of core components in supporting quality and fidelity monitoring and provide the current experience of the use of TPIF3.0 in a broad rollout of Triple P in over 30 municipalities in Chile.
3 presentations:
Evolving the application: 10 years of application of IS through the Triple P Implementation Framework - TPIF 3.0.
– Explore the application of implementation science through the development of a tailored implementation support system for the implementation of a global parenting program.
– Describe the 10-year journey from the development of the initial Triple P implementation support framework to the recent adaptation of a continuum of implementation support from self-directed to high-intensity.
– Demonstrate the application of implementation science through the use of the Triple P Implementation Resource with intensive support in Chile.
Supporting quality and fidelity: Core components and quality and fidelity monitoring.
– Articulate the core components of all Triple P programs
– Develop a quality and fidelity monitoring process that implementing organisation could tailor with support from the Implementation Consultant
Putting TPIF3.0 into practice: Using the TPIR with intensive IC support in 41 municipalities in Chile.
– Introduce the TPIR as the foundation for implementation support.
– Explore the experience of users.
– Assess achievement of implementation outcomes
Title 1: 10 years of application of IS through the Triple P Implementation Framework - TPIF 3.0
The Triple P Implementation Framework (TPIF), supported by Triple P implementation consultants (IC) has been in use since 2013. The goal of TPIF is to increase the uptake and sustainability of the program over time and develop an understanding of effective use of implementation science at the implementing organisations. TPIF was revised and refined in 2018 in response to the experience of the ICs and feedback from implementing organisations – TPIF 2.0. In 2022 Triple P International embarked upon a process of assessing the effectiveness of TPIF 2.0 and IC support. Organisations suggested a need for simpler tools, more flexibility in the support available and supports that are adaptable to various implementation approaches. This information has been used to develop TPIF 3.0, through which a continuum of support can be provided from self-directed Implementation, adapted and simplified tools and IC support more reflective of minimal sufficiency. TPIF 3.0 is based on and still demonstrates the research and body of knowledge from implementation science.
Approach: 10 years of experience and information from implementing organisations, implementation consultants and the purveyor organisation was gathered and analysed. Information and data collected indicated the need for simplification of the process and tools, more flexibility in the application of what implementation science has indicated as essential, and a range of support.
Outcomes: A user survey will be concluded by May 31st 2024 which will inform the success of TPIR in the effectiveness of applying implementation science (content), useability of TPIR and effectiveness in engaging in an IS based implementation process.
Next Steps:
Revise TPIR in response to the user survey
Contribute to the discussion between implementation researchers and practitioners to bridge the gap between implementation research and application of implementation science.
Title 2: Core components and quality and fidelity monitoring.
Background: Between 2013 and 2018 Triple P International was frequently asked by initiative funders how fidelity of Triple P was monitored. These questions led to consideration at TPI through which a number of factors were identified:
As the purveyor, TPI does not have agency over the trained practitioners performance (over 90,000 worldwide)
As implementation supports TPI had a responsibility to support organisations to develop effective quality and monitoring processes as part of their implementation process
Any process suggested must reflect the Triple P principals of self-regulation and minimal sufficiency.
The TPI implementation team initiated a process to address these considerations and provide a tool that would support implementers to develop a tailored quality and fidelity monitoring process as part of their implementation process.
Approach: The first step in the process was to identify and articulate the core components for Triple P programs. There are over 25 programs and it was determined that the core components should be common to all programs. An extensive process of analysis led to confirmation of 7 core components that are non-negotiable components for any Triple P program. This process included discussion between the implementation team and the program developers. The process described by the National Implementation Research Network identifying core components and suggested indicators for fidelity of content, competency of delivery and effective supportive context was used.
Outcomes: A descriptive tool with suggested indicators, The Quality and Fidelity Monitoring Process, is an integral part of the implementation process supported by TPI implementation consultants. The tool includes definitions of the core components as well as suggested indicators in 3 domains (content, competency and context) and checklists that promote quality improvement both at an organisational level and individual level.
Next Steps: Integrate the ongoing use of the Quality and Fidelity Monitoring Process tool into the TPI.
Title 3: Using the TPIR with intensive IC support in thirty-five municipalities in Chile
Background: Triple P Latin America in Chile has been providing implementation support for municipalities to implement Triple P since 2019. In 2023 Subsecretaría de Prevención del Delito (SPD) integrated Triple P into the LAZOS project, funding the expansion into more than 50 municipalities. SPD values highly implementation support and recognises its significant contribution to the successful delivery and sustainment of Triple P, and its return on investment.
Approach: Five Triple P Implementation consultants have been trained in the use of the TPIR and, with ongoing support from the implementation manager. Implementation support is being provided to clusters of five municipalities each cluster has a dedicated IC. SPD were involved in discussions about the provision of support through clusters of municipalities. They voiced confidence in their support of the process and the development of clusters of municipalities. Intensive support is being provided over twelve months through 14 facilitated sessions including an orientation session for all municipalities. These topic-specific sessions support the implementers to work through the TPIR. In addition, each individual organisation participates in individual consultations monthly to address any specific challenges they are encountering, allowing for tailoring and support that addresses the individual needs of each municipality. It also allows for the ICs to identified themes that may be common in the use of the TPIR or in the implementation process.
Outcomes: All municipalities are actively engaged and are developing support networks between municipalities, which provides for implementation capacity building across the system. A TPIR User Survey is being conducted in May. These results will be reported at the presentation.
Next Steps: Apply the information from the TPIR User Survey and make adaptations that are suggested. Following this TPIF 3.0 will be implemented by all TPI ICs globally by September 2024.
Strategies for implementing and sustaining integrated care for substance use disorders within HIV care settings: Policy, intermediary, practice, and research perspectives
Authors
Harold Phillips - National Minority AIDS Council
Heather Gotham - Stanford University
Laura Branch - AID Atlanta
Alonzo Martinez - San Antonio AIDS Foundation
Bryan Garner - The Ohio State University
Mathew R. Roosa - Roosa Consulting, LLC
Tom J. Donohoe - University of California, Los Angeles
Beth A. Rutkowski - University of California, Los Angeles
Thomas E. Freese - University of California, Los Angeles
AJ Martinez - Los Angeles County Department of Health Services
Cherise Rohr-Allegrini - San Antonio AIDS Foundation
Kent Montgomery - AID Atlanta
Symposium Overview Summary
Per the 2024 Society for Implementation Research Collaboration (SIRC) conference theme, this symposium includes perspectives from research, practice, and policy, but also includes the intermediary support system perspective given the Interactive Systems Framework for Dissemination and Implementation posits this as one of the critical systems for successfully bridging the research-to-practice gap. Each of these four related but distinct perspectives will center on strategies for implementing and sustaining integrated care for substance use disorders (SUDs) within HIV service organizations (HSOs), which is a major public health need given SUDs among people with HIV is both highly prevalent and problematic. The first presentation will provide the symposium's policy perspective and will be presented by Mr. Harold Phillips, who served as director of the White House Office of National AIDS Policy June 2021 to January 2024 and is now the Deputy Director of Programs for the National Minority AIDS Council. The second presentation will provide the symposium's intermediary perspective and will be presented by Dr. Heather Gotham, who co-directs a Technology Transfer Center funded by the Substance Abuse and Mental Health Services Administration. The third presentation will provide the symposium's practice perspective and will be co-presented AID Atlanta's Ms. Laura Branch and San Antonio AIDS Foundation's Mr. AJ Martinez, both of which have been helping their respective HSOs implement and sustain a motivational interviewing brief intervention for SUDs. The fourth presentation will provide the symposium's research perspective and will be presented by Bryan Garner, who is Professor and Director of Dissemination and Implementation Science at The Ohio State University and has been leading a program of implementation research focused on the integration of SUD services within HSO since 2014. By combining these four perspectives this symposium has the potential to have a strategic synergistic impact on the symposium attendees and its presenters.
Title 1: Strategies for implementing and sustaining integrated care for substance use disorders within HIV care settings: Policy perspectives
Background: For more than a decade, the HIV response in the United States has been guided by the National HIV/AIDS Strategy (NHAS). This presentation will highlight the goals and strategies of 2022-2025 NHAS that are related to improving integrated care for substance use disorders (SUD), as well highlight harm reduction as a key strategy of President Biden's National Drug Control Strategy and a Plan to Address Methamphetamine Supply, Use and Consequences.
Approach: To improve HIV-related health outcomes of people with HIV (NHAS Goal 2), expand implementation research to successfully adapt effective evidence-based interventions to local environments to facilitate uptake and retention of priority populations is a key strategy. To achieve integrated, coordinated efforts that address the HIV epidemic among all partners and interested parties (NHAS Goal 4), implement a no-wrong-door approach to screening and linkage to services for HIV and comorbidities like SUD is a key strategy. Another key strategy is a harm reduction approach as a critical intervention to reduce both overdose deaths and communicable disease rates. Harm reduction is an approach that emphasizes working directly with people who use drugs to prevent overdose and infectious disease transmission, improve the physical, mental, and social wellbeing of those served, and offer flexible options for accessing SUD treatment and other health care services.
Outcomes: Among the key lessons to be shared as part of this presentation are how state laws and regulations challenge national implementation of these strategies, as well as the ways courts are threatening access to preventative health services and the impacts for integrating services for both HIV and SUD.
Next Steps: Given these challenges, innovative policy strategies are needed. Strategies that include collaborations between public health and law enforcement or public safety, which are often overlooked and under-utilized partner in this work, are promising directions to explore.
Title 2: Implementing and sustaining integrated care for substance use disorders within HIV care settings: Intermediary perspectives
Background: Comorbid HIV and substance use disorder (SUD) is a major public health concern. Therefore, there is an urgent need to identify effective strategies to improve integration of SUD services with HIV service organizations (HSOs). Wandersman and colleagues’ (2008, 2012) Interactive Systems Framework for Dissemination and Implementation (ISF4DI) highlights the support system as key to helping service delivery organizations (i.e., the delivery system) enhance their capacity for quality implementation of evidence-based interventions. This presentation will share perspectives from two national intermediary support systems, the Addiction Technology Transfer Center (ATTC) and the AIDS Education and Training Center (AETC) regarding multifaceted strategies for implementing and sustaining integrated care for SUDs within HSOs across the United States.
Approach: As part of a program of research funded by the National Institute on Drug Abuse, a complementary set of strategies were developed and experimentally tested to improve integration of SUD services within HSOs. Strategies included 1) a motivational interviewing brief intervention (MIBI) training manual, 2) in-person and asynchronous online training for the MIBI, 3) a team-focused Implementation and Sustainment Facilitation strategy, and 4) a staff-focused Pay-For-Performance strategy.
Outcomes: Consistent with ISF4DI, we found an additive set of tools, training, technical assistance, and quality improvement/assurance incentives are necessary for optimizing implementation effectiveness (i.e., the consistency and quality of implementation by targeted users).
Next Steps: Although ATTC and AETC collaboration is appropriate, acceptable, and feasible such cross-system collaborations are rare. As such there remains a need to identify effective strategies at the policy level and/or support system level to increase the likelihood of this cross-system collaboration that are necessary to support the implementation and sustainment of SUD services with HSOs across the United States.
Title 3: Implementing and sustaining integrated care for substance use disorders within HIV care settings: Practice perspectives
Background: Community-based HIV service organizations (HSOs) provided multiple services to people with HIV. Substance use disorders (SUDs) are especially prevalent and problematic for this vulnerable population, yet the integration of SUD services within HSOs remains very limited. This presentation shares the perspectives of two HSOs that have been working to integrate an evidence-based screening and brief intervention (SBI) program.
Approach: Following participation in a cluster-randomized type 3 hybrid trial funded by the National Institute on Drug Abuse, where a staff-focused pay-for-performance strategy was shown to be a highly effective adjunct strategy to a staff-focused training, feedback, consultation strategy + team-focused facilitation strategy, AID Atlanta and San Antonio AIDS Foundation agreed to participate in a subsequent Quality Improvement (QI) initiative. As part of the QI initiative the HSOs were awarded a subgrant to financially support their assistance in translation of the project's SBI research protocol into a sustainable SBI practice protocol.
Outcomes: Implementation research testing different strategies for implementing an evidence-based practices in “real-world” practice settings often requires elements that are not part of a truly real-world practice protocol, such as the research informed consent process and the research assessments that are for research purposes only. In addition to presenting on this important lesson, one representative from each HSO will highlight their respective HSO's research to practice translation process, status regarding the development of a pragmatic SBI practice protocol, and current level of SBI implementation effectiveness (i.e., consistency and quality of SBI implementation by targeted HSO staff).
Next Steps: Key next steps will include finalization of HSO-specific SBI practice protocols, as well as the development of a national SBI-focused toolkit to help facilitate exploration, preparation, implementation, and sustainment of a motivational interviewing-based SBI practice protocol by other HSOs across the United States.
Title 4: Implementing and sustaining integrated care for substance use disorders within HIV care settings: Research perspectives
Background: Since 2014, the National Institute on Drug Abuse has funded multiple large-scale implementation projects focused on understanding how best to address substance use disorders (SUDs) for people with HIV. In addition to detailing the strategies tested and shown effective as part of this research, this presentation will detail the research designs that were used and proposed for use as part of future research.
Approach: There are two key settings for which improvements in integrated care for comorbid HIV and SUD are urgently needed. One is SUD service organizations, while a second is in HIV service organizations (HSOs). The program of research described as part of this presentation is focused on the latter. More specifically, this program of research focused on testing strategies to improve integration of a single session motivational interviewing brief intervention for SUDs within HSOs across the United States. Considering the staff-focused training, feedback, and consultation strategy used by the Addiction Technology Transfer Centers (ATTC) network as a necessary but not sufficient control strategy, a type 2 hybrid trial was used to demonstrate the incremental effectiveness of a team-focused strategy called Implementation and Sustainment Facilitation, with a subsequent type 3 hybrid trial being used to demonstrate the additional incremental effectiveness of a staff-focused strategy called pay-for-performance.
Outcomes: The most effective strategy for implementing a motivational interviewing brief intervention for SUD within HSOs was a strategy that combined the team-focused ISF with staff-focused training, feedback, consultation, and pay-for-performance. However, we also found that the ATTC network lacks sufficient resource capacity to provide this strategy to HSOs given their focus is on supporting addiction treatment organizations.
Next Steps: Because the AIDS Education & Training Center (AETC) network primarily supports HSOs, next steps included making AETCs aware of this strategy and/or creating cross-system collaborations between these two support networks.
References
Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., Blachman, M., Dunville, R., & Saul, J. (2008). Bridging the gap between prevention research and practice: The interactive systems framework for dissemination and implementation. American Journal of Community Psychology, 41(3-4), 171–181. https://doi.org/10.1007/s10464-008-9174-
Wandersman, A., Chien, V. H., & Katz, J. (2012). Toward an evidence-based system for innovation support for implementing innovations with quality: Tools, training, technical assistance, and quality assurance/quality improvement. American Journal of Community Psychology, 50(3-4), 445–459. https://doi.org/10.1007/s10464-012-9509-7
Research Track
Implementation Science and Health Informatics: Tensions and Symbiosis
Authors
Mustafa Ozkaynak - University of Colorado, Anschutz Medical Campus
Saira Haque - Pfizer
Craig Kuziemsky - MacEwan University
Kim Unertl - Vanderbilt University Medical Center
Symposium Overview Summary
Health informatics is the science of how to use data, information and knowledge to improve human health and delivery of health care services. Implementation of interventions that support this mission is an important component of health informatics. However, the interaction between the implementation science and health informatics research has been limited. This symposia aims to enrich the dialogue between these two fields that will be mutually beneficial. More specifically, this symposia will highlight how each of these fields can inform research in the other domain and the methodological, theoretical and cultural challenges for further collaboration. Implementing health informatics applications into health settings remains a significant challenge and requires new approaches to overcome implementation issues. This workshop will describe the value of integrating implementation science and health informatics to enhance future HIT implementation. All four presenters of this symposia have expertise in both health informatics and implementation science.
This symposia will include three parts. The first part (40 minutes) will include brief presentations by each of the four presenters. The second part (10 minutes) will include a Q&A session to answer audience questions related to the presentations. The last 25 minutes will involve an interactive session, in which the audience and panelists will prioritize related challenges regarding collaboration of health informatics and implementation science experts. Ozkaynak will moderate this session. Because of its interactive nature, the symposia will serve as a forum in which members of the audience and panelists can collaboratively advance the dialogue between implementation science and health informatics. The sharing of experiences stimulated by this panel will serve as a foundation for generating and prioritizing future collaboration initiatives to examine and support the implementation of novel health information technologies such artificial intelligence.
Title 1: Implementation as a System of Systems
The digitization of the health care sector has been challenging despite years of research and many examples of system implementation. The COVID-19 pandemic substantially stimulated the digital care delivery in a very short time and it gave us a glimpse of how digital care delivery could be an essential part of the health delivery system. However, digital health implementation often struggles in implementing and integrating informatics models, tools, and approaches into actual care delivery settings so that meaningful outcomes are achieved. Implementing digital heath applications into health settings remains a significant challenge. Post-implementation issues such as unintended consequences (UICs) including workflow, usability, communication, and social and organizational issues are common occurrences post-implementation. Systems thinking approaches can help us understand the integrated nature of digital health systems and the broader systems and contexts where digital health is used. This session will discuss how systems thinking approaches could enhance digital health implementation.
Title 2: EHR Audit logs as an opportunity to track adaptations
This presentation will focus on leveraging Electronic Health Records (EHR) audit logs to track adaptations related to implementation of various interventions. Two types of adaptations will be covered: intervention adaptation and context adaptation. Tracking adaptations can be a data intensive effort. Traditional approaches to track adaptations rely on qualitative methods such as surveys and interviews. These methods are useful to capture the context of the adaptation; however, they do not necessarily provide sufficient direct data on adaptation itself. This presentation will explore the opportunities and challenges of using EHR audit logs to track adaptations particularly related to informatics interventions (e.g., decision support tools such as order sets, alerts, pathways or communication tools such as inbox). EHR audit logs provide time-stamped data about how these Informatics interventions are used, for example time takes to use, frequency of use or navigation patterns. Completely benefit from the audit logs requires close collaboration between implementation researchers and research informatics departments of the health systems. Moreover, due to large amount of data, advanced analytics methods are also needed to make sense of data. Collaboration can include many challenges due to expectations, field related assumptions and terminology. In this presentation we will also provide a case study in which we utilized audit logs to understand adaptation of an antibiotic order set. We will discuss how Audit logs can be used as a standalone approach to track intervention and context adaptation as well as how audit logs can be used complimentary with traditional approaches. The results suggested that audit logs are effective in capturing essential details of intervention adaptation and can provide limited information on context adaptations.
Title 3: Technology as both a factor and the intervention
One important aspect of implementation science is understanding the internal and environmental context in which the intervention takes place. In the case of Health IT, the technology is a mechanism which is used to enact organizational improvement as well as part of the infrastructure and external context with respect to information sharing across organizations. Thus, there are interactions between the technology, external and internal environment, users, and actions. All of these interactions influence how an intervention is disseminated and used across the organization.
Some interactions in particular to consider include the degree to which the organization participates in health information exchange efforts, the degree of technical integration in the organization, the digital literacy of end users, usability of the technology at hand and feasibility. In addition, if the intervention involves features that involve a level of trust such as automated measurement or interpretation of results, then that must also be addressed. This section will focus on implications of the technology as both an internal and external infrastructure factor as well as a driver for change.
Title 4: Applying organizational theory in technology-based interventions
Unintended consequences of interventions, particularly of interventions with a technology component, are a widely experienced phenomenon. Previous implementation studies have repeatedly demonstrated that negative consequences of innovations are both possible and likely, without thoughtful design and implementation planning. For example, landmark studies in the early 2000s demonstrated that electronic order entry systems, with the laudable goals of increased patient safety and order accuracy, led to additional work, changes in communication, and even new errors (Campbell et al., 2006). While we have a foundation now to understand potential unintended consequences, healthcare continues to experience issues related to implementation of innovations, such as the many unintended consequences related to workload, burnout, and note bloat that emerged from the expansion of electronic health record usage in response to policy changes (Colicchio et al., 2019). Although innovations will always have some degree of unintended consequences, applying the robust evidence base from organizational theory in practice can contribute to proactively anticipating and addressing these potential issues. Combining implementation science, informatics, and organizational theory is key to navigating intervention processes with technology components.
Multi-level Implementation Strategies used in School-based Research
Authors
Jill Locke - University of Washington
Aubyn Stahmer - University of California, Davis
Karolyn Maurer - University of California, Los Angeles
Miguel Villodas - San Diego State University
Aaron Lyon - University of Washington
Kelsey Dickson - San Diego State University
Sara Chung - University of California, San Francisco
Lauren M. Haack - University of California,
San Francisco
Feion M. Villodas - San Diego State University
Linda J. Pfiffner - University of California, San Francisco
Karolyn Maurer - Minnesota State University
Aubyn Stahmer - University of California, Davis
Wendy Shih - University of California, Los Angeles
Connie Kasari - University of California, Los Angeles
Cathy M. Corbin - University of Florida
Roger Goosey - University of Washington
Vaughan K. Collins - University of Washington
Mark G. Ehrhart - University of Central Florida
Kurt Hatch - University of Washington Tacoma
Christine Espeland - University of Washington Tacoma
Olivia G. Michael - University of Miami
Tana Holt - San Diego State University
Amy Drahota - University of California, Los Angeles
Over one third of youth receive mental health care in schools (Ali et al., 2019). Implementation of evidence-based practices (EBP) in schools is frequently incomplete, inconsistent, and occurs with insufficient fidelity to ensure positive effects for youth due to significant implementation barriers across multiple levels (Durlak & DuPre, 2008; Lyon et al., 2019; Moore et al., 2023). Despite the known importance of fidelity for achieving positive outcomes, deficits in malleable provider- (e.g., attitudes, knowledge), organizational- (e.g., leadership, climate), and district-level factors (e.g., operating policies and procedures) frequently stymie efforts to implement EBPs with fidelity amidst perpetual shifts in staff, students, extra-school policies, and emergent crises. Utilizing implementation strategies (i.e., “methods or techniques used to enhance the adoption, implementation, and sustainment of a clinical program or practice”) is critical to support providers’, schools’, and districts’ use of EBPs. School-based implementation strategies often need to address schools’ unique professional roles, leadership structures, and operational features (e.g., 9-month school year) that can significantly differ from other mental health settings (Owens et al., 2014). We will present four studies that use novel implementation strategies to facilitate EBP delivery and enhance access to quality care in public school settings. The four studies apply different implementation strategies across multiple levels (e.g., school mental health provider, school-based teams, school leadership, and district administrators) to: 1) strengthen school-based teams delivering behavioral interventions for children with ADHD; 2) support the uptake and implementation of three EBPs that focus on early intervention, school-aged children, and autistic adolescents with the goal of promoting access and services for underrepresented and under resourced communities; 3) enhance implementation leadership among building-level leaders and district administrators to support the use of a universal socio-emotional and behavioral intervention; and 4) develop a selection quality implementation toolkit for autistic youth in middle and high schools.
Title 1: Enhancing Teamwork to Support Family-School Partnerships in School-Based Interventions for ADHD
Background: The Collaborative Life Skills Program (CLS) is a school-based program for elementary school children with ADHD. In CLS, school mental health clinicians are trained to coordinate the concurrent implementation of three evidence-based behavioral interventions, behavioral parent training, daily behavioral report cards with teacher consultation, and child skills training. The impact of these interventions is optimized when they are implemented daily in synchrony across school and home settings by teams of clinicians, teachers, and parents. However, this requires clinicians, teachers, and parents to be “on the same page” about their expectations, goals, tasks, roles, responsibilities, and implementation process, which is challenging in schools. Team effectiveness research offers empirically-supported team development strategies from other organizational contexts that can be leveraged to enhance the implementation of school-based interventions like CLS. Team Charters are written documents that clarify expectations about team operating procedures. Team Handoff Protocols ensure clear and timely communication among team members during transitions in responsibility. Team Performance Monitoring provides critical feedback to maintain team motivation and initiate team adaptation. We will describe the development and integration of team-based implementation strategies to enhance CLS.
Methods: We conducted focus groups and interviews with 6 school principals, 10 school mental health clinicians, 10 teachers, and 6 parents of children with ADHD to obtain input about the use of Team Charters, Handoff Protocols, and Performance Monitoring in the CLS program.
Results: Focus groups and interviews were transcribed, summarized, and analyzed for themes using the Rapid Assessment Procedure. Themes reflected preferences for using electronic communication among clinicians, teachers, and parents about children's daily school and home behavior, monitoring team progress through shared electronic tracking sheets, and establishing shared team values and expectations, and clear roles for each team member at the start of the program.
Conclusions: Findings have implications for enhancing teamwork to improve school-based mental health intervention implementation.
Title 2: Using Novel Implementation Tools for Evidence-based intervention Delivery (UNITED): A Team-Based Implementation Strategy for Education
Background: Few evidence-based interventions (EBIs) for autistic individuals are successfully implemented in under-resourced communities. Many implementation strategies are specifically designed to a single EBI. It is not feasible to develop new implementation strategies for each EBI and community. Therefore, we developed and are testing the effectiveness and generalizability of an organizational implementation strategy with three EBIs concurrently to increase successful EBI use with under-resourced autistic youth (from birth to age 22) across service settings.
Methods: Using Novel Implementation Tools for Evidence-based intervention Delivery (UNITED) facilitates the implementation and sustainment of three EBIs in under-resourced schools and parent resource centers: Remaking Recess (RR), Self-Determined Learning Model of Instruction (SDLMI), and Mind the Gap (MTG). Community settings randomized to UNITED established an implementation team devised of leaders, interventionists, and/or other staff using an organizational social network analysis. Teams met regularly and identified goals to support EBI implementation at their unique site.
Results: We will examine the structure, participants, and goals across UNITED teams to understand the components they determined necessary to best support implementation and sustainment within their context. Qualitative methods will be used to examine each UNITED team's goals. The average number of team members was lower in schools (4) than community agencies (6.5). Teams included administrators, providers and support staff. Of all UNITED sites (Overall: 36; MTG: 9; RR: 16, SDLMI: 11), the average number of goals developed was 4 (MTG: 6.14, RR: 3.67; SDLMI: 3.09). Preliminary results show goals focused on 1) data collection and monitoring; 2) increasing awareness of the intervention throughout the site; 3) creation of internal processes; and 4) knowledge and capacity building for team members.
Conclusions: Results provide rigorous data about the team structure and implementation supports needed to effectively implement EBIs in diverse, under-resourced public service settings.
Title 3: Not Getting Better but Not Getting Worse: A Cluster Randomized Controlled Pilot Trial of a Leadership Implementation Strategy
Background: Implementation of evidence-based practices (EBP) that support youth social, emotional, and behavioral well-being in schools is fraught with challenges. Deterioration of implementation efforts (e.g., engagement or support, abandonment of new practices, etc.) often hinders long-term success of EBPs. School leadership behaviors can influence teachers’ EBP implementation. Our study tested an implementation strategy called Helping Educational Leaders Mobilize Evidence (HELM), iteratively adapted from the Leadership and Organizational Change for Implementation (LOCI) strategy, to enhance EBP implementation through improvements in school leadership teams’ implementation leadership and climate to buffer against the deterioration of implementation efforts. This study explores the impact of HELM on theorized mechanisms of change (i.e., implementation leadership, implementation climate), educator-level factors (i.e., teacher implementation citizenship), and implementation outcomes (i.e., fidelity, initiative stability).
Methods: One school district and 10 schools participated. Five of the schools were randomized to receive HELM and the remaining five schools received an alternative leadership training as an implementation attention control. Teachers at every school (n = 341) received training for an EBP called Positive Greetings at the Door that has been previously demonstrated to reduce student behavior problems. Principals and Assistant Principals (n = 18) received HELM or an alternative leadership training. Three district Administrators also participated.
Results: HELM significantly slowed the average decline of implementation leadership (perseverant leadership and communication), three dimensions of implementation climate (recognition, rewards, and existing supports) and total implementation climate, and one dimension of implementation citizenship (keeping informed). No significant effects were found with regard to implementation outcomes (i.e., fidelity, initiative stability).
Conclusions: HELM shows promise in buffering the deterioration of EBP implementation efforts in schools. Furthermore, the HELM strategy positively influenced implementation leadership and climate, which are the hypothesized mechanisms for promoting successful long-term implementation efforts. An appropriately powered trial is needed to determine the efficacy of HELM in the future.
Title 4: Iterative Refinement of a Selection Quality Implementation Toolkit For use in Schools: Engaging Community Partners to Optimize Implementation Strategy Impact
Background: Evidence-based practice (EBP) selection often is distributed across multiple educators and leaders in schools. Yet, EBP selection and implementation often are reactive in response to broader mandates or emergent crises. There is a need for multi-level decision-making tools that enable the systematic selection and adoption of EBPs in schools. This study describes specific modification and redesign areas to a selection quality implementation toolkit for use in schools and presents preliminary outcomes from the pilot test.
Methods: We describe specific changes to a multifaceted implementation strategy called, Autism Community Toolkit: Systems to Measure and Adopt Research-based Treatments (ACT SMART), for use in middle and high schools. We engaged in a community-partnered, multiphase iterative redesign process to redesign ACT SMART including focus groups, expert advisory boards, and Cognitive Walkthrough for Implementation Strategies (CWIS) sessions. We conducted a pilot trial of the redesigned ACT SMART toolkit in six schools. Within each phase, we collected implementation outcome data (feasibility, acceptability, appropriateness, and usability) regarding ACT SMART to inform iterative modifications and redesign.
Results: Educators and school administrators in the focus groups rated Phases 1, 2, and 3 of the ACT SMART toolkit to be acceptable (M=4.11, 4.04, and 4.21, respectively), appropriate (M=4.01, 4.03, and 4.09, respectively), and moderately feasible (M=3.72, 3.74, 3.86, respectively). Overall, usability scores for the focus groups (M=60) and CWIS sessions (M=50.3) were in the “marginally” usable range and emphasize the need for further redesign of the ACT SMART toolkit in schools. Modifications included integrating the toolkit with school structures and timelines, and content modifications such as altering language, shortening and targeting assessments, and incorporating toolkit engagement strategies to improve usability in schools. Additional redesigned areas identified following our feasibility pilot test will be discussed.
Conclusions: The current project highlights the importance and value of tailoring multilevel implementation strategies for use in schools.
References
Durlak, J.A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3-4), 327–350. https://doi.org/10.1007/s10464-008-9165-0
Lyon, A. R., Cook, C. R., Duong, M. T., Nicodimos, S., Pullmann, M. D., Brewer, S. K., … & Cox, S. (2019). The influence of a blended, theoretically-informed pre-implementation strategy on school-based clinician implementation of an evidence-based trauma intervention. Implementation Science, 14, 54. https://doi.org/10.1186/s13012-019-0905-3
Moore, S. A., Cooper, J. M., Malloy, J., & Lyon, A. R. (2023). Core components and implementation determinants of multilevel service delivery frameworks across child mental health service settings. Administration and Policy in Mental Health and Mental Health Services Research, 51(2), 172–195. https://doi.org/10.1007/s10488-023-01320-8
Owens, J. S., Lyon, A. R., Brandt, N. E., Warner, C. M., Nadeem, E., Spiel, C., & Wagner, M. (2014). Implementation science in school mental health: Key constructs in a developing research agenda. School Mental Health, 6(2), 99–111. https://doi.org/10.1007/s12310-013-9115-3
Economic Evaluation in Dissemination and Implementation Research: Identifying Value Through Integrated Outcomes
Authors
Todd Wagner - Stanford University
Jennifer Boggs - Institute for Health Research, Kaiser Permanente
Debra Ritzwoller - Institute for Health Research, Kaiser Permanente
Brian Ahmedani - Henry Ford Health System
Melissa Maye - Henry Ford Health System
Courtney Benjamin Wolk - University of Pennsylvania
Shari Jager-Hyman - University of Pennsylvania
Christina Johnson - Northwestern University Feinberg School of Medicine
Rinad Beidas - Northwestern University Feinberg School of Medicine
Andria Eisman - Wayne State University
Jacob Whitman - Wayne State University
Amy Kilbourne - University of Michigan, Ann Arbor
Lawrence Palinkas - University of California San Diego
Judy Fridline - Genesee Intermediate School District
Christina Harvey - Oakland County Schools
David Hutton - University of Michigan, Ann Arbor
Sean Murphy - Weill Cornell Medicine
Danielle Ryan - Weill Cornell Medicine
Thanh Lu - RTI International
E. Jennifer Edelman - Yale School of Medicine
Kathryn Hawk - Yale School of Medicine
Patrick O’Connor - Yale School of Medicine
Edouard Coupet - Yale School of Medicine
Ali Jalali - Weill Cornell Medicine
Patricia Owens - Yale School of Medicine
David Fiellin - Yale School of Medicine
Gail D’Onofrio - Yale School of Medicine
Andrew Quanbeck - University of Wisconsin–Madison
Xiang Li - University of Wisconsin–Madison
This symposium addresses the burgeoning need to identify economic outcomes best suited for evaluating the value of resources invested in dissemination and implementation (D&I) research. Traditional economic evaluations in health, such as cost-effectiveness and cost-benefit analyses, frequently neglect the cost of implementing and sustaining interventions, thus underestimating the actual investment required to successfully achieve intervention objectives. One unique feature of D&I studies is a focus on both clinical or behavioral outcomes and implementation outcomes. While economic evaluation centers on evaluating the costs and benefits, the evidence base on what constitutes value for the money in D&I research is still developing. To date, a limited number of D&I studies conduct comparative economic evaluations of implementation strategies, hampering innovations in the field.
Our symposium contributes to the field by presenting four economic evaluations of implementation strategies across various settings. All studies estimate implementation strategy costs, but vary on outcomes for the economic evaluation. One study compares two strategies for firearm storage using reach as the outcome. Another compares strategies for universal drug use prevention using student anxiety mapped onto health utilities. The third compares strategies for clinical practices related to opioid use disorder (OUD) treatment and use health utilities, opioid-free years, and community-based OUD care uptake. The fourth compares multiple strategies as part of an adaptive trial and uses fidelity as the outcome. These studies span diverse analytical methods and examine the economic impact on implementation and clinical outcomes. We will identify areas consistent with traditional health economic research and alignment across studies and highlight considerations unique to implementation science.
We aim to foster a robust dialogue on the future direction of economic evaluations in implementation science, encouraging researchers to focus on greater consistency and standardization in economic methods and integrate essential outcomes to accurately determine the value of implementation investments.
Title 1: Adolescent and child Suicide Prevention in Routine clinical Encounters (ASPIRE) study
There are evidence-based recommendations for pediatric primary care clinicians to counsel caregivers on secure firearm storage to prevent youth suicide, but this practice is not routinely done. The ASPIRE study, a cluster-randomized hybrid effectiveness-implementation type III trial (R01MH123491), conducted at 30 clinics across two learning healthcare systems, aimed to evaluate two implementation strategies to reduce this gap. All well-visits (5-17 years) were eligible to receive the S.A.F.E. Firearm program, which includes (1) a brief discussion on secure firearm storage with youth/caregivers and (2) offering a free cable lock. Two implementation strategies were tested: "Nudge,” a new EHR documentation template, vs. “Nudge+,” which includes Nudge plus practice facilitation delivered by a trained facilitator who checked in with clinics, troubleshot barriers, and provided audit/feedback reports. All sites received clinician training and collateral (e.g., handouts, posters). The primary outcome was clinician-reported delivery of firearm counseling and cable lock offered (reach) and secondary outcome was survey-reported parent storage behavior. There was a significant main effect for reach for both implementation strategies with a stronger effect for Nudge+. A cost-effectiveness analysis with activity-based costing was conducted, including time to implement S.A.F.E. Firearm (e.g., practice facilitator time, training time), materials (e.g., cable locks), and overhead. At the symposium, we will report on our process for defining the implementation outcome in the cost analysis. Specifically, is reach the right outcome for a cost-effectiveness analysis? Should we instead use a clinical outcome of parent storage behavior? How do we generalize reach to other cost analyses of these implementation strategies? Should reach be defined as cost per patient reached, cost per physician trained, cost per clinic? Our decision will be informed by collaborations with health care system decision makers for adoption of S.A.F.E. Firearm.
Title 2: Economic Evaluation of Enhanced versus Standard REP for School-Based Universal Prevention: a Pilot Study
Background: Universal prevention can avert behavioral health downstream consequences, including addiction, and are considered “stunningly” good investments. School-based prevention, such as the Michigan Model for HealthTM (MMH), however, is often delivered suboptimally. Implementation strategies, including Enhanced Replicating Effective Programs (REP), are needed to realize the public health benefits of school-based prevention, but they can be resource-intensive. High-resource strategies are most challenging for disadvantaged communities to deploy and sustain. Strategy return on investment is vital to decision-making but is understudied. Economic evaluation of implementation strategies may require assessments that are sensitive to changes in youth outcomes within a short-term implementation trial.
Methods: We collected data during a pilot cluster randomized implementation trial in 8 Michigan high schools (4 Enhanced REP, 4 standard implementation) with ≤ 20% of students eligible for free/reduced lunch during 2021-2022. We used activity-based costing to determine Enhanced REP cost (described elsewhere). To estimate effectiveness, we used the Generalized Anxiety Disorder-7 (GAD-7) scale, mapping to utilities to assess cost-effectiveness. We used deterministic and probabilistic sensitivity analyses and adopted a system perspective and one- and five-year time horizons.
Results: Enhanced REP cost per school was $8,659 (1-year) and $12,598[AE1] [JW2] (5-year). Student GAD-7 scores (N=279) were lower for the Enhanced REP versus the standard implementation group (= -1.56, p = .046); we used entropy balancing to reduce bias due to differential attrition. We found the cost for 1 unit GAD-7 decrease was $59.38/1-year and $17.34/5-year time horizon; cost per QALY was $39,648/1-year and $11,537/5-year time horizon.
Conclusion: Investing in implementation strategies such as the Enhanced REP can help schools use limited resources efficiently and contribute to student well-being and success. This research provides vital information for decision-makers and highlights the challenges and importance of implementation strategy costs and outcomes to promote school-based prevention's short- and long-term impact.
Title 3: An Economic Evaluation of Implementation Facilitation to Promote Emergency Department–1 Initiated Buprenorphine for Opioid Use Disorder
Background: Emergency department (ED) visits are common among persons with opioid use disorder (OUD); thus, the ED can serve as a critical touchpoint for OUD care. ED-initiated buprenorphine (an effective OUD medication) with referral to community-based care is an effective strategy to engage and treat patients with OUD; however, uptake has lagged, partly due to insufficient “readiness” of ED clinicians. The Project ED Health clinical trial tested an implementation facilitation (IF) intervention, guided by the PARiHS framework, to increase adoption of ED-initiated buprenorphine, and found it to be effective. However, understanding the resource/financial commitment of IF and the resulting program will be critical for health systems considering adoption, while insurers will want to understand the ongoing cost in the context of downstream cost-offsets.
Methods: A detailed microcosting analysis was conducted to identify the resources/costs needed to perform the IF intervention, and sustain the resulting ED-buprenorphine program. The average per-patient sustainment cost was then integrated into a cost-effectiveness analysis. Costs were estimated from a healthcare-sector perspective, and evaluated alongside three effectiveness measures: quality-adjusted life-years (QALYs), opioid-free years (OfYs), and engagement in community-based OUD care on the 30th day following the ED visit.
Results: The average, per-site, costs were: pre-implementation=$49,656, IF=$75,625, and sustainment=$108,687/year. The mean, per-person, healthcare-sector cost associated with ED-administered buprenorphine following IF did not differ significantly from that of standard-education, while the mean effectiveness for all 3 measures significantly favored the IF strategy. IF has a 71-76% probability of being considered cost-effective from a healthcare-sector perspective at the recommended value range of $100,000-200,000/QALY. ICERs estimated using secondary effectiveness measures had a 75% probability of being considered cost-effective at $25,000/OfY and $35,000/engagement.
Conclusion: Although IF requires a considerable up-front investment, relative to a standard educational strategy, maintaining the resulting program would likely be considered a cost-effective strategy from a healthcare-sector perspective.
Title 4: A cost-effectiveness analysis of telemedicine and mobile health interventions for patients with alcohol use disorder
Introduction: This paper presents the findings of a cost-effectiveness analysis conducted as part of a previously reported randomized clinical trial comparing telemedicine and digital health interventions for patients with alcohol use disorder (McKay et al., 2022).
Methods: The trial enrolled 262 participants (82% African American, 71% male) from four intensive outpatient programs in Philadelphia. The analysis compared a telemedicine intervention (TMC) and a mobile health intervention (A-CHESS) in a 2×2 factorial design. Intervention costs and effectiveness (measured by the number of reduced heavy drinking days) were assessed for each group and were compared to the control group.
Results: All 3 treatment groups (TMC, A-CHESS, and TMC+A-CHESS) showed a statistically significant reduction in heavy drinking days compared to the control group although there were no significant differences in effectiveness between the TMC, A-CHESS, and TMC+A-CHESS group). We utilized effect sizes from the primary study (percentage of heavy drinking days) to calculate the difference in estimates compared to the baseline heavy drinking days (PDHD). The TMC group achieved a mean reduction of 37 heavy drinking days over 12 months compared to baseline, while the ACHESS group achieved a mean reduction of 33 days. A-CHESS was slightly more expensive than TMC ($479 vs. $434 per patient). The incremental cost-effectiveness ratios were comparable between the two interventions. Labor costs predominated for both intervention groups.
Conclusion: Both A-CHESS and TMC were found to be cost-effective choices for providing continuing care in alcohol use disorder treatment, with no clear evidence favoring one over the other based on the trial results. Finally, in a model-based analysis, A-CHESS appears to be the more cost-effective choice as organizational size increases, allowing a given population to be managed (per protocol) using roughly ½ the labor costs of TMC.
Using Novel Methodologies to Optimize Scalability, Contextual Alignment, and Feasibility of Evidence-based Intervention Implementation
Authors
Rosemary Reyes - Department of Psychiatry & Behavioral Sciences, University of Washington
Bryan Weiner - University of Washington
Predrag Klasnja - University of Michigan, Ann Arbor
Celine Lu - University of Washington
Maria Hugh - Department of Special Education, University of Kansas
Michael Pullmann - Department of Psychiatry & Behavioral Sciences, University of Washington
Mallory Dobias - Appa Health
Aaron Lyon - Department of Psychiatry & Behavioral Sciences, University of Washington
Symposia overview summary
Background: While evidence-based interventions (EBIs) shown effectiveness in improving youth mental health, scaling them up can be challenging due to limited resources, contextual mismatch, and feasibility concerns. The University of Washington IMPACT Center has developed multiple toolkits for methodologies such as causal pathway diagramming (CPD), rapid analog methods (RAM) and implementation barrier prioritization (IBP), which can optimize the implementation of EBIs in resource-constrained and other settings. When it comes to understanding the intended effect of an implementation strategy, CPDs visually map out which mechanisms influence how an implementation strategy operates. This is useful when developing novel implementation strategies and for improving how multifaceted implementation strategies address context-specific barriers. To optimize strategies for EBI implementation, RAM can be used to empirically test strategies and mechanisms in artificial conditions (e.g., reading vignettes of strategies) comparable to the real world while limiting burden to community partners. Lastly, the implementation barrier prioritization process can assist in addressing the most critical context-specific barriers to meet community partners’ needs. Methods: Using several real-world examples, this symposium will discuss the utility and application of emergent methods in implementation science including CPDs, RAM and IBP. Results: Emergent methods in implementation science are useful for optimizing scalability, contextual alignment, and feasibility of EBI implementation. Conclusion: This symposium illustrates the application of novel methodological approaches to optimizing EBI implementation that can be used independently or in conjunction with one another. These findings will inform updates to existing toolkits for these methods. By showcasing the integration of these methodologies across diverse projects, this presentation advances implementation science by paving the way for future research on their broader application.
Title 1: Reconceptualizing Optimization for Implementation Science
Background: In her groundbreaking work on the Multiphase Optimization Strategy (MOST), Linda Collins introduced the notion of intervention optimization. Rather than moving quickly to efficacy or effectiveness trials, Collins argued that behavioral scientists should first conduct studies that generate evidence needed to improve their interventions—identifying active components, refining decision rules, etc.—which would maximize the likelihood that the final intervention package is effective. There is a growing interest in optimization in implementation science (IS) as well, mostly focused on the use of sequential multiple assignment randomization trials to optimize adaptive implementation strategies. However, the IS focus on organizational processes rather than individuals means that some of the established approaches to optimization of behavioral interventions, such as factorial experiments, may be less applicable in implementation science. To fully take advantage of MOST in IS, we need to refine how we think about optimization.
Methods: In this talk, Dr. Klasnja proposes that optimization in IS should focus on three core ideas: (1) conducting resource-efficient studies that generate the “right” level of evidence for the decisions on how to optimize implementation in the current context; (2) understanding how variations in context and operationalizations of implementation strategies impact strategy effectiveness; and (3) identifying processes through which strategies operate—including key moderators and preconditions—that are transportable across contexts.
Results: Focus on the appropriate—rather than best—level of evidence enables the use of a wide range of efficient methods, including single-case studies and analog experiments, to inform the design of implementation strategies, while the focus on evaluating distinct strategies, inherent in many optimization methods, enables building of a rigorous evidence base about how to design effective strategies for different contexts.
Conclusion: Optimization studies in IS can thus result both in more effective implementation initiatives and more rigorous generalizable knowledge about implementation.
Title 2: Optimizing implementation coaching to promote a positive implementation climate for cognitive behavioral therapy in community mental health: A pre-post experimental analog study
Background: Experimental analog study designs, in which participants review written passages that simulate real-world conditions, can optimize implementation interventions by having partners identify their most promising components before deploying in a trial. The current talk illustrates this IMPACT Center method within the context of Peer-Led Implementation Coaching, in which experienced community mental health (CMH) supervisors are trained to support—through Implementation Coaching—their peers in promoting a positive implementation climate with their clinicians (i.e., expect, support, and reward CBT). We compared clinician's perceptions of their agency's implementation climate before and after envisioning themselves in vignettes of specific implementation climate strategies.
Method: 155 CMH clinicians completed a pre-post online analog study. Implementation climate was assessed at baseline (i.e., extent to which CBT currently expected, supported and rewarded at their agency). They were then randomized to read 6 strategy vignettes – 2 from 10 possible within the Expect, Support and Reward categories. After imagining themselves in each vignette (e.g., Your supervisor connects CBT to your personal values), implementation climate was reassessed.
Results: One-way repeated measures ANOVAs (F(2,306.65)=73.73,p<.001), and post-hoc paired samples t-tests revealed that baseline Reward (M=5.09,SD=2.45) was the lowest rated subscale. Repeated measures ANOVAs revealed higher post-vignette ratings on each subscale: Expect (F(1,429.09)=11.13,p<.001), Support (F(1,427.22)=4.72,p=.03), and Reward (F(1,428.44)=237.03,p<.001). However, there were no significant interaction effects to indicate that one vignette performed significantly better compared to others (p's>=0.05).
Conclusion: Rapid analog methods can help identify target areas for implementation strategy focus, such as rewarding CBT in this case. Given no differences in clinicians’ perceptions on specific strategies, supervisors can instead opt for strategies best suited for their organization's needs and available resources. The analog experimental design was a cost-effective, low-clinician burden method for optimizing peer implementation support. Strategies are currently being used in our Peer-led Implementation Coaching randomized trial.
Title 3: Navigating Causal Pathways to Optimize a Multifaceted, Theoretically Informed Implementation Strategy
Background: In schools, where a significant percentage of youth receive mental health services but where the utilization of evidence-based practices (EBPs) remains low, effective implementation strategies are needed. We are optimizing a multifaceted implementation strategy for schools, Beliefs, and Attitudes for Successful Implementation in Schools (BASIS), to impact key, theoretically-derived individual-level mechanisms associated with clinician behavior change. BASIS is a four-hour live-facilitated group-based training. Although BASIS has demonstrated feasibility, appropriateness, and efficacy, its scalability could be improved through parsimonious and pragmatic approaches.
Methods: We employed Causal Pathway Diagramming (CPD) of BASIS components on key mechanisms and outcomes. We used these methods to address the following research questions: 1) Which components of BASIS are most logically related to their target mechanisms based on causal pathways? 2) Based on CPDs, how can we package a digitized BASIS to test via a RAM trial? Our interdisciplinary team, comprising four PhD-level researchers and three research team members, utilized the UW IMPACT Center toolkit and the online diagramming tool Miro to develop CPDs for mechanisms related to intention and implementation outcomes.
Results: Over a period of three months, we deconstructed BASIS into its 11 strategies and developed CPDs specific to each hypothesized mechanism. By merging CPDs across mechanisms and strategies and considering dependencies, we prioritized three BASIS components that targeted attitudes and self-efficacy with a plan to test digitized strategy to target subjective norms.
Conclusion: Our study highlights systematic diagramming via CPDs to refine implementation strategies like BASIS. IMPACT Center methods were useful and needed to prioritize components across multiple iterations considering key aspects of strategy delivery (e.g., chronology, dependencies). CPDs informed the development of 14 conditions to test for optimization and impact on mechanisms in a RAM trial.
Title 4: Addressing Barriers to Adolescent Engagement in Digital Mental Health using Emerging Methods
Background: Digital mental health services present solutions to challenges adolescents encounter in face-to-face therapy, such as access and lack of relatability and rapport between adolescents and therapists. Appa Health is a DMH tool for adolescents delivered via video chat and text that includes CBT-based content (tiktok-style short form videos by clinically trained social influencers) along with personalized near-peer mentoring. A pilot study found good youth engagement with mentoring, but less than optimal use of CBT-informed DMH tools and videos. This study aimed to identify barriers to engagement in short form CBT videos and develop a strategic plan to enhance engagement. It highlights how IMPACT Center methods can be used in partnerships to optimize implementation.
Methods: We identified determinants of engagement using a Rapid Evidence Review of published literature followed by a focus group with Appa users. A Barrier Prioritization process using Nominal Group Decision Making with Appa leadership followed, and these barriers were applied to building Causal Pathway Diagrams (CPD) of determinants and solution plans.
Results: Eighteen barrier/facilitator pairs were identified and eight of these were endorsed as most relevant by Appa leadership. A group card sorting process was used to collapse barriers into salient categories and prioritize, based on importance and feasibility, barriers related to: multiple steps to access the videos, difficulty finding videos on the platform, and forgetting to watch. CPDs were collaboratively built to identify moderators, preconditions, and measurable outcomes, and a strategic plan was developed based on these CPDs. Participants rated the CPD process with high satisfaction for utility, helpfulness, and ease of use, with lower ratings for efficiency and clarity. This collaborative pilot work served as the foundation for a successfully funded federal grant.
Conclusion: The IMPACT Center methods were a useful way to systematically consider and address barriers to implementation.
Enhancing Acceptability and Reach of Evidence-Based Practices: Aligning Implementation with Human-Centered Design with Diverse Populations in Real-World Contexts
Authors
Kathryn Macapagal - Northwestern University Feinberg School of Medicine
Juan Pablo Zapata - Northwestern University Feinberg School of Medicine
Ashley Knapp - Northwestern University Feinberg School of Medicine
Andy Rapoport - Northwestern University Feinberg School of Medicine
Ian Sotomayor - Northwestern University Feinberg School of Medicine
Jacob Gordon - Northwestern University Feinberg School of Medicine
Andrés Avila - Northwestern University Feinberg School of Medicine
Julianna Lorenzo - Northwestern University Feinberg School of Medicine
Bryant Norton - Northwestern University Feinberg School of Medicine
Eva Minahan - Northwestern University Feinberg School of Medicine
Laura Jans - Northwestern University Feinberg School of Medicine
Jessica Schleider - Northwestern University Feinberg School of Medicine
Miguel Herrera - Northwestern University Feinberg School of Medicine
C. Hendricks Brown - Northwestern University Feinberg School of Medicine
Simrandeep Kour - Northwestern University Feinberg School of Medicine
Shannon Hill - Northwestern University Feinberg School of Medicine
Arielle Smith - Northwestern University Feinberg School of Medicine
Overall Symposium Abstract
It's a well-documented fact that two-thirds of implementation efforts are unsuccessful (Damshroder et al., 2009; Lewis et al., 2018); despite increased focus, the gap between research and practice remains wide. One significant hurdle is that evidence-based practices (EBPs) often aren't developed in conjunction with their eventual users or within the actual context in which they'll be used. Human-centered design (HCD) offers a powerful solution to these implementation challenges by tailoring EBPs and/or their implementation strategies to meet the specific needs, learning preferences, and operational realities of the target audience (Bazzano et al., 2017; Johnson et al., 2021). This symposium will share insights from four distinct populations and real-world settings, showcasing effective strategies where implementation science and HCD intersect to enhance health and well-being via digital interventions. First, Dr. Macapagal will discuss the feasibility of using the collaborative platform of Discord to host a youth advisory council to inform the adaptation of a text-message based sexual health intervention for LGBT adolescents. Next, Dr. Zapata will discuss their collaboration with youth to develop dissemination strategies for optimizing engagement and acceptability of a digital single session intervention for major depression. Third, Knapp will present their collaborative work with teens to co-adapt digital mental health tools to be implemented within public library settings. Finally, Ms. Hill will discuss their co-design partnership with adult women living with ADHD and a non-profit mental health organization to adapt a single session intervention for mood troubles and anxiety. Each presentation will exemplify the different HCD techniques used to tailor an array of different digital health interventions. Dr. Macapagal will then synthesize the findings and discuss how implementation of digital health interventions and HCD methods can align to achieve impactful change in the lives of end-users within real-world contexts.
Title 1: Using Discord with an LGBT adolescent advisory council to increase acceptability and implementability of a digital sexual health intervention
Background: Digital interventions hold significant potential for reaching sexual and gender minority (SGM) adolescents with sexual health information. Community engagement using a human-centered design (HCD) approach can increase these interventions’ implementability and acceptability among end-users. Here we describe our use of the social platform Discord to host a youth advisory council (YAC) to inform adaptation of a text-message-based sexual health intervention that will be tested in an upcoming RCT.
Methods: Participants are 20 SGM adolescents (14-18yrs) across the USA who have engaged in our private YAC Discord server. Participants were invited to join Discord for 3 months, with the option to renew participation every 3 months. We posted weekly open-ended questions and polls to gather teen perspectives on different elements of the intervention, engaged teens through non-research-oriented conversations, and reported back to teens on how their feedback was used for intervention adaptation. After 1 year we assessed participants’ perspectives on the YAC.
Results: 17 of 20 participants consistently engaged in Discord over the year. Acceptability of the YAC Discord was high, with most reporting they felt comfortable sharing their thoughts, that the research team listened to their feedback, and that they felt this process improved the intervention overall. Deidentified screenshots of Discord YAC will be used to illustrate examples of how we iteratively sought teens’ feedback on intervention content, interactive components, and implementation strategies.
Conclusions: Discord is a feasible and collaborative setting to engage in sustained co-design for digital interventions with SGM adolescents. Our methods highlight the advantages of Discord compared to standard methods of HCD, such as one-time focus groups or infrequent in-person advisory committees, and how an ongoing process of community engagement can increase intervention acceptability and implementability.
Title 2: Enhancing Dissemination Strategies for Digital Single Session Interventions among Youth with Major Depression through Human-Centered Design
Background: Online single-session interventions (SSIs) provide immediate, evidence-based support for youth with major depression (MD), improving treatment access. These self-guided, 15–30-minute web-based sessions have shown in multiple RCTs to significantly reduce MD severity in teens over 3-9 months. Yet, outside of compensated RCTs, youth engagement with SSIs significantly decreases, underscoring the need for effective, sustainable dissemination strategies, particularly through social media. This study proposes leveraging human-centered design (HCD) to create direct-to-youth, online dissemination materials for SSIs, tailored to their needs and contexts. The goal of this study is to collaboratively develop dissemination strategies with youth, aiming to boost engagement and acceptability.
Methods: Using a human-centered design approach, qualitative data were collected through Focus Groups (FGs) (N=20) with youth recruited from a national advisory group. All transcripts were thematically analyzed.
Results: Participants were generally unaware of SSIs, with many believing they were suited for milder mental health issues rather than severe conditions like chronic depression. Skepticism about the intent and credibility of SSIs posed a significant barrier to engagement. Experiences of ineligibility or unreliable online mental health services further fueled this distrust. Although social media platforms were recognized as potential resources, there was a consensus on the need for dissemination strategies that connect youth with peers and trusted local healthcare professionals to promote SSIs.
Conclusions: Our research highlighted the effectiveness of human-centered design practices in designing dissemination strategies for SSIs with youth. These practices are fundamental implementation strategies that enhance engagement and reach of these interventions, while also boosting perceived effectiveness and promoting their sustained use.
Title 3: Co-Designing with Teens in the Adaptation of Digital Mental Health Resources within a Public Library
Background: Digital mental health (DMH) prevention services have high public health potential to extend reach and maximize impact if implemented into organizations highly accessed by marginalized populations. Over the past five years, our research team has partnered with a community-based teen program within a public library to adapt and implement DMH services with and for teen patrons who frequent the public library.
Methods: Two design workshops were conducted with different cohorts of teen patrons and focused on gathering teen patrons’ lived experiences around anxiety to be used in the co-adaptation of single session interventions (SSIs) teaching anxiety management skills. The first workshop asked teens to draw out how they experience anxiety, how they safely cope, and how they feel after. In the second workshop, researchers gathered feedback on improving the SSIs, with a focus on representing marginalized teens’ experiences related to anxiety. The teen sample (n=24) was highly diverse regarding race/ethnicity, gender identity, and SES.
Results: In the first workshop, teen patrons illustrated and provided text around their somatic, cognitive, and behavioral experiences of anxiety. They also discussed the current ways in which they safely manage their anxiety (i.e., breathing, social support, distraction, and problem solving). In the second workshop, teens provided feedback on adapting existing stories of teens that support the theme of each SSI (e.g., stories about avoiding anxiety-evoking situations).
Conclusions: Partnering with end-users and deployment organizations prior to and throughout the design process is critical to ensure uptake, effectiveness, and sustainability of the designed services. Themes focused on the implications of designing and implementing DMH services into community organizations serving teens with marginalized identities, and how teens’ feedback will be used in intervention adaptation, will be discussed.
Title 4: Non-Profit/Academic, Co-Design Partnership: Adapting Digital Single-Session Interventions for Women with ADHD
Background: Digital, single session interventions (SSIs) are well-positioned to address gaps in the mental health landscape: provider shortages, long wait times, and attrition within longer-term therapy. There is a significant need for human-centered design (HCD) methods to adapt SSIs for new populations and in specific clinical contexts. Adults living with Attention-Deficit / Hyperactivity Disorder (ADHD), particularly women, have distinct needs that HCD methodologies can address; they need interventions which are more effectively tailored to their requirements. In this project, we have partnered with a digital, non-profit mental health organization (Understood) to adapt our SSIs for young women, who are experiencing mood troubles and anxiety as they navigate life with ADHD.
Methods: Understood's online platforms were utilized to recruit 100 adult women aged 25-45 who self-identify with symptoms of ADHD. A diverse group of twenty participants took part in a focus group session lasting between 90-120 minutes, employing HCD methodologies across three stages. Initially, participants interacted with SSIs prototypes, offering feedback through the "think aloud" technique. This was followed by semi-structured interviews informed by the Theoretical Domains Framework to gather insights for refining the implementation of SSIs. Finally, discussions on preferences for online assessments and potential outcome measures were held to guide the design of a pilot study for adapted SSIs.
Results: Preliminary findings from focus groups (approximately 19% Black/African American, 19% Hispanic/Latinx; 38% LGBT) have included feedback on topics ranging from SSI content to length/format. Participants suggested advertisements which could facilitate implementation and identified preferred outcome measures for depression, anxiety, and ADHD symptoms.
Conclusions: This research demonstrates HCD methods to adapt SSIs to new populations; the goal will be to make our digital SSIs more specific, effective, and accessible, for young women living with ADHD.
Simulations, cocreations, and community rankings: Information integration strategies to promote synergy in policy making
Authors
Sarah Walker - University of Washington
Radley C. Sheldrick - University of Massachusetts
Gracelyn Cruden - Chestnut Health Systems
Annette Boaz - Kings College, London
Noah R. Gubner - University of Washington
David J. Vanness - Pennsylvania State University
Jason E. Chapman - Chestnut Health Systems, Lighthouse Institute
Lisa Saldana - Chestnut Health Systems, Lighthouse Institute
Health policy frequently fails to achieve hoped for health improvements. Studies of these failures frequently highlight the challenges of integrating multiple perspectives at the design stage (Hoagwood et al., 2020; Chriqui et al., 2023). For example, burdensome regulations or unfunded mandates can result in sabotage at the service provider level (Crable et al., 2022); or universal policies intended to improve the quality of care without attention to marginalized populations can increase health disparities (Fisher et al., 2022).
Implementation frameworks, such as the EPIS model, clearly highlight the value of intersectoral linkages from the outer to inner context through bridging factors, such as intermediaries (Lengnick-Hall et al., 2021). However, strategies for policy design in these linkages are understudied. As policy inevitably involves trade-offs in desired outcomes (e.g., reach vs. equity), strategies specifically designed to elicit diverse perspectives are expected to improve clarity in policy preferences. When applied in group settings, these strategies support the development of shared goals and synergy across sectors.
This symposium will highlight the acceptability and impact of strategies applied at different phases of policy design. The session will describe three methods, each presented interactively with the audience. The first presentation will demonstrate a simple computer simulation a decision-maker can use to clarify their desired tradeoffs for policy impact. The second presentation will showcase the use of participatory community policy ranking across diverse communities as a precursor to more technical phases of policy design. The third presentation will demonstrate the application of multi-criteria decision analysis (MCDA) with multisectoral teams (government, service, community) to support selection of policy options. All presentations will focus on the usability and transportability of the policy design tools. The discussant will draw out themes related to defining evidence in policy design and the integration of information and values in policy D&I science.
Title 1: A policy simulator to help researchers develop policy-relevant research
Background: Methodological choices depend on scientific values. For example, scientists may justify the expense of randomized trials by emphasizing their value for attributing causal effects, and model selection is widely informed by the epistemic value of parsimony (reflected in principles like Occam's Razor and metrics like the Aikeke Information Index). Likewise, methodological choices can have ethical implications. For example, the population impact of public health interventions is widely operationalized as the product of reach and effectiveness—a parsimonious outcome metric that is consistent with the utilitarian principle of achieving the greatest good (i.e., effectiveness) for the greatest number (i.e., reach). While other options are available—e.g., maximizing benefit among those in greatest need, consistent with Rawl's theory of justice as fairness—they are seldom considered by implementation scientists. Novel methods are needed to evaluate the policy implications of methodological choices regarding outcome metrics.
Approach: We developed a computer simulation that places researchers in the position of a policymaker who must decide how to allocate funds for a hypothetical multilevel, multicomponent intervention. Funds can be used to support services that extend reach in the population and/or that increase effectiveness among those served; furthermore, such services can be supported in either of two populations, one of which experiences greater symptoms and higher costs.
Outcomes: Results demonstrate important tradeoffs in decision-making. For example, budgets that maximize the product of reach and effectiveness in the overall population can also increase disparities. In contrast, budget allocations that most improve outcomes among those in greatest need can ameliorate disparities but achieve lower impact in the overall population.
Next Steps: Little is known about how researchers, policymakers, and the general public evaluate the desired outcomes of complex policy interventions. Policy simulators provide a unique opportunity to evaluate decision-making and to design research trials that meaningfully inform this process.
Title 2: Participatory policy ranking in diverse communities to inform county health equity policy
Background: Two contributors to observed failures in evidence-informed policy implementation are poorly designed policy ideas and insufficient political will. Democratic participation in the policy development phase is a promising approach for addressing both of these problems as well as being inherently. However, democratic participation in policy design poses considerable recruitment and information synthesis challenges. Participatory policy ranking is a promising strategy for overcoming these challenges.
Approach: The project took place in a mid-sized county in Washington state. The policy ranking activity was informed by previously conducted community listening sessions held to obtain information about health needs in geographical areas with high health disparities as well as culturally affiliated groups. Twelve listening sessions were coded for health themes. These thematic areas were cross walked with two sources: 1) policy levers extracted from the scholarly and gray literature focused on building health equity and resilience; and 2) health priority areas identified by the health department. The research team then collaborated with the health department to develop and implement policy ranking sessions. Acceptability and feasibility were assessed using mixed methods.
Outcomes: The process of implementing policy ranking sessions was largely acceptable to health department staff who co-facilitated or solely facilitated sessions using the policy ranking guide. More experienced facilitators made in-session adaptations to the guide to respond to participant engagement, but adaptations were in line with the purpose and key activities of the sessions. The research-health department was able to successfully implement 10 sessions, 83% of the original groups.
Next Steps: The feasibility of the current project was greatly facilitated by already established relationships between the health department and community groups. Similarly structured strategies, such as citizen panels, could be alternatives to this process.
Title 3: User-Centered Refinement of a Decision Tool to Converge Diverse Perspectives and Support Transparent Intervention Selection
Background: Many policy implementation decisions, such as intervention adoption, are group decisions. Diverse perspectives must converge to yield decisions that individuals feel good about and are eager to implement. Further, policy makers are often interested in understanding how their priorities and evaluations compare to those of their constituents. This presentation will present a web-based tool that elicits individuals preferences and evidence evaluations to facilitate transparent, focused conversations about which interventions to adopt, and why.
Approach: Multi-criteria decision analysis (MCDA) is both a framework and methodological approach to deconstruct a decision, transparently represent how a decision was reached, and facilitate the convergence of diverse perspectives. A MCDA tool is being refined while leveraging state-level implementation efforts for a federal policy to prevent child maltreatment. Individuals: a) rank which criteria (N = 18) most influence their decisions, b) use a Likert scale to rate interventions on their most influential criteria. A weighted-sum approach merges users’ preferences (criteria rankings) and expertise (intervention evaluations) to produce intervention rankings. Individuals can compare their quantified preferences, intervention evaluations, and intervention rankings with other policy actors, community members, and implementers. User-centered design methods, including concurrent and retrospective think-aloud, are being used to refine the tool's interface and functionality with target end users (e.g., policy makers, technical assistance providers; n = 12).
Findings: User feedback prompted efforts to clarify criteria, improve quantitative results visualizations, and develop a webpage to introduce how and when to incorporate the tool into implementation decision-making.
Is it Truly Pragmatic? Lessons Learned from Clinical & Implementation Researchers Using the Pragmatic Implementation Strategies Reporting Tool
Authors
Brittany Rudd - University of Illinois Chicago Department of Psychiatry
Catalina Ordorica - University of Illinois Chicago
Yashashwi Pokharel - Wake Forest University School of Medicine
Vidhya Suresh - Wake Forest University School of Medicine
Rahma Ajja - Wake Forest University School of Medicine
Sean K. Wang - Wake Forest University School of Medicine
Adrianna L. Elashker - Wake Forest University School of Medicine
Hayden B. Bosworth - Department of Population Health Sciences, Duke University
Justin B. Moore - Wake Forest University School of Medicine
Byron Powell - Washington University in St. Louis
Symposium Overview
Background: The Pragmatic Implementation Strategy Reporting Tool (Rudd et al., 2020) was developed to support clinical and implementation researchers in reporting their use of implementation strategies to help accelerate the implementation of evidence-based practices into real-world contexts. An important goal of the tool was to increase clinical researcher's considerations of implementation processes in their work. However, this impact will only be felt if clinical researchers adopt and use these implementation research frameworks in their work. Upon examination, the Pragmatic Implementation Strategy Reporting Tool has been cited 74 times since first being developed and disseminated in 2020. However, the vast majority of scientists using the Reporting Tool have been implementation researchers.
Methods: In this symposium, we will review the uses of the Pragmatic Implementation Reporting Tool to date, provide perspectives on its use by implementation and clinical researchers, and describe adaptations we needed to make it more useful for clinical researchers.
Results: The low frequency of the tool being used to drive clinical and implementation research, as measured via a scoping review, was further supported by one exemplar use of the tool by clinical researchers to execute a systematic review on telemedicine trials of hypertension management.
Conclusions: When developing implementation research frameworks and tools like the Pragmatic Implementation Reporting Tool, deliberate efforts are needed to increase the likelihood of adoption by the clinical scientist community with limited implementation science training. As such, there is continued opportunity for cross-disciplinary collaboration to ensure such endeavors are tailored to the clinical researcher's context and that there is shared understanding of the purpose of the tool.
Title 1: Use of the Pragmatic Implementation Strategy Reporting Tool: Lessons Learned
Introduction: The Pragmatic Implementation Strategy Reporting Tool (PISRT; Rudd et al., 2020) was developed with the goal of accelerating the implementation of evidence-based practices into the settings for which they were designed. As implementation strategies are often used at the clinical research stage, and communicating their use may help to encourage rapid adoption of the practice, the tool was designed to make implementation strategy reporting more pragmatic for clinical researchers. We sought to understand the ways in which the PISRT has been used to date to determine if it is meeting its stated goal.
Methods: We conducted a scoping review using PubMed and Google Scholar to assess the frequency of the tool's use in published literature. We reviewed each paper citing the PISRT to date to determine the extent to which it was used to support implementation strategy reporting. We also assessed the degree to which it was used by clinical versus implementation researchers by reviewing the biography of each co-author and characterizing their expertise (e.g., primarily implementation, mixed, primarily clinical).
Results: Analyses are underway and will be complete by SIRC. To date, the tool has been cited 74 times. Preliminary results suggest that the PISRT is primarily being used by implementation researchers, suggesting that only half of the target audience is utilizing the tool. Further, this suggests that the tool is not being used as it was intended.
Discussion: To date, the PISRT has mostly been adopted by implementation researchers, suggesting that it might need refinement to increase adoption by clinical researchers. The presenter, an implementation scientist and developer of the tool, will present an overview of how to use the tool, her method of using it to report on implementation strategies from published manuscripts, and lessons learned.
Title 2: Optimizing Use of the Pragmatic Implementation Strategy Reporting Tool: Insights from Clinical Science and Implementation Science
Background: Despite having tools to guide Implementation Strategy (IS) reporting, clinical researchers encounter practical challenges in systematically describing IS from efficacy and effectiveness studies due to the complexities of implementation science concepts and terminologies. This systematic review retrospectively examined the report of IS from 13 telemedicine trials of hypertension management.
Methods: To address the confusion of using IS as clinical researchers, we describe our study methods and provide a practical manual to support clinical researchers in systematically reporting IS and pragmatic attributes of trial design from pre-implementation clinical studies, using the Pragmatic Implementation Strategy Reporting Tool. Our approach involved clinical researchers (new and familiar to implementation science) undergoing short training on implementation science methods, including how to use the Pragmatic Implementation Strategy Reporting Tool. After individual data abstraction, the group convened to select and operationalize IS and trial design domains along efficacy to effectiveness spectrum using group discussion and consensus. Two implementation science experts reviewed clinical researchers’ IS reporting and provide written assessment about any discordance.
Results: Upon review by the implementation science experts, there was clear discordance in the selection and operationalization of IS, which suggests lack of ease in using the Pragmatic Implementation Strategy Reporting Tool by clinical researchers without further adaptation and support from implementation researchers. As such, the two implementation science experts and lead clinical researchers on the project collaboratively adapted the tool for ease of use.
Discussion: The challenges, lessons learned, and discordant ratings will be provided from both a clinical researcher and implementation researcher perspective. One key lesson learned was the importance of distinguishing between intervention components and implementation strategies. Early involvement from cross-disciplinary researchers while developing implementation tools and conversely when clinical researchers embark on implementation work are important.
References
1. Brittany N Rudd, Molly Davis, Rinad S Beidas (2020) Integrating implementation science in clinical research to maximize public health impact: a call for the reporting and alignment of implementation strategy use with implementation outcomes in clinical research. Implement Sci. 2020 Nov 25;15:103. doi: 10.1186/s13012-020-01060-5
Understanding the role of readiness for implementation through implementation trials of collaborative care models
Authors
Shawna Smith - University of Michigan, Ann Arbor
David Kolko - University of Pittsburgh
Ian Bennett - University of Washington
Bryan Weiner - University of Washington
Amy Kilbourne - University of Washington
Elizabeth A. McGuier - University of Washington
Eileen Thompson - Pennsylvania Chapter of the American Academy of Pediatrics
Renee Turchi - Drexel University
Byron Powell - Washington University in St. Louis
Kimberly Hoagwood - New York University
Symposium Overview:
In this panel, experienced investigators will discuss key findings and “lessons learned’ from the conduct of three large-scale implementation trials of collaborative and chronic care models (CCMs). Findings and lessons will focus specifically on issues related to implementation readiness, accommodating and addressing early implementation challenges, and practical strategies for maintaining rigor in the face of unexpected challenges and rapidly changing contexts. The three implementation trials presented are diverse with respect to collaborative care intervention models, practice settings, and trial designs. The first presentation focuses on practical and methodological challenges experienced early in an ongoing trial aimed at optimizing implementation support for collaborative care models in pediatric settings. The second study discusses findings from a pragmatic effort to assess readiness and adapt implementation support accordingly when implementing a collaborative care model. The last presentation focuses on adaptive implementation strategies, specifically using data on early response to inform adaptations to implementation support provided for sites that are slower to respond. Our expert discussant, Bryan Weiner, will share perspectives on lessons learned and future directions for understanding the role of implementation readiness in implementation projects, as well as for potentially accommodating varying levels of implementation readiness through implementation strategies and adaptations.
Title 1: Studying team- and leadership-level implementation strategies for a pediatric chronic care model: Early practical and methodological challenges
Background: CCM adoption in pediatric primary care is challenging. This hybrid type 3 effectiveness-implementation trial of Doctor-Office Collaborative Care (DOCC), a CCM for disruptive behavior disorders and ADHD, evaluates implementation strategies targeting the care team (TEAM) and leadership levels (LEAD) in pediatric primary care settings. This presentation describes early challenges related to practice engagement and readiness, and how they were addressed to balance relevance and rigor.
Methods: The trial uses a 2×2 factorial design to evaluate the effects of TEAM and LEAD. TEAM facilitation targets provider clinical competencies and team functioning/integration. LEAD facilitation targets implementation climate and implementation leadership. The study is currently recruiting practices/providers and caregivers. We report baseline data for the Mental Health Practice Readiness Inventory (provider survey; AAP 2010; Beers et al., 2017) which includes 5 CCM competency domains (e.g., help for kids/FAMs; clinical info syst; decision support; Beers et al., 2017) and facilitator completed Readiness Checks after CCM training ended (2, 1-hr. meetings with both trainers; based on Bennett/Kilbourne/Smith)
Results: On the MHPRI, sites reported modest CCM competencies but there was variation within sites across domains 1.3–2.7) and across sites. The overall mean score was 1.8 (range: 1-3). Lowest scores were found for financing and clinical information systems. We also found variability across sites on the readiness checks; we report the highest (e.g., leadership is supportive of DOCC and study participation, Leader is prepared to provide administrative support, Staff are ready to complete self-reports) and lowest rated items (e.g., prepared to use web-based registry; scheduled team CCM meetings). Modifications to improve practice recruitment and engagement included flexible timelines, randomization by practice, expanded study role definitions, and adaptive nature and timing of pre-implementation tasks sites.
Conclusions: Readiness challenges in this trial included the many “touches” needed to support necessary research and clinical operations, and the administrative and legal/research burdens created by using telehealth and a web-based data system. We will discuss how to balance practicality and rigor in the face of a rapidly changing clinical and financial context and its associated demands.
Title 2: Assessment strategies for readiness to implement collaborative care for common mental disorders
Background: The CCM for common mental disorders is a complex intervention requiring the use of a psychiatric consultant and specialized care manager who provide support to primary care providers. The implementation of this model has proven to be difficult in part because the practice change required is novel and requires interdisciplinary work between behavioral and primary care teams. The assessment of readiness for implementation has been suggested to be critical to the tailoring of implementation support for complex interventions. We wished to assess the ability of a systematic checklist of readiness by implementation teams with external implementation facilitators to reduce variation in early implementation stages.
Methods: Twenty clinics participating in the Maternal Infant Dyad Implementation (MInD-I; NCT02976025) Type-III Implementation-Effectiveness trial (2017-2022) were included. Sites were identified from the OCHIN national network of community health centers as well as primary care settings providing prenatal care to at least 50 patients annually. External implementation facilitators utilized a checklist of key elements of implementation readiness and tailored their support based on the results. A CoCM specific version of the Stages of Implementation Completion (CoCM-SIC) was used to assess implementation outcomes across 8 stages with measures of proportion of tasks completed and duration of each stage.
Results: Among the 20 sites that participated in this study, all had high proportions of task completion (>80%) in the pre-implementation stages (1-5). However, wide variation in proportion of tasks completed was found in later stages of the CoCM-SIC.
Conclusion: While a systematic checklist approach to assessment of readiness was associated with high rates of task completion in early stages of completion, there was a wide range in final implementation success later in the process. Additional work is needed to assess whether distinct strategies to address variation in readiness might improve implementation success of the CoCM intervention.
Title 3: Learning how to adapt: Lessons learned from a clustered non-responder sequential multiple assignment randomized trial implementing a collaborative care model in primary care and community mental health clinics
Background: CCM adoption in lower-resourced community-based practices has been particularly challenging. Adaptive implementation efforts that adapt implementation support to accommodate low implementation readiness or address early implementation failure may aid scale-up and equitable reach. We discuss lessons learned from a trial comparing options for tailoring implementation support for CCM “slower-responder” community-based sites.
Methods: Sites initially received support using a low-intensity implementation strategy, Replicating Effective Programs (REP). After 6 months, slower-responder clinics were randomized to add either External Facilitation (REP+EF) or External+Internal Facilitation (REP+EF/IF). Comparative effectiveness and moderators of REP+EF vs. REP+EF/IF were examined for both implementation (CCM delivery) and effectiveness (mental health quality of life, depression) outcomes. Qualitative data on facilitation engagement and fidelity was also examined.
Results: N=43 slower-responder clinics were randomized to receive REP+EF (N=21) or REP+EF/IF (N=22). Overall, EF/IF practices saw more CCM delivery than EF practices (ΔEF/IF-EF = 4.4 patients, 95% CI = 1.87-6.87). Moderation analyses revealed that only sites with pre-randomization CCM delivery benefitted from EF/IF (ΔEF/IF-EF = 9.2 patients, 95% CI: 5.72, 12.63). Only 27 sites (63%) identified patients suitable for CCM pre-randomization; patients at EF clinics (vs. EF/IF) showed significantly larger improvement in functioning clinics (ΔEF/IF-EF =8.38; 95%CI=3.59, 13.18). Process data indicated challenges in implementing IF with fidelity at many clinics, notably those with low implementation readiness.
Conclusion: Lessons learned included: (1) for clinics struggling with implementation, higher-intensity implementation support did not outperform lower-intensity; (2) early indicators of readiness (e.g., patient identification) and success (e.g., pre-randomization delivery) were useful in tailoring later implementation support; and (3) more intensive strategies requiring substantial buy-in or oversight may be less effective when implementation readiness is lower. While readiness for implementation has traditionally been considered an important precursor to implementation, differential levels of readiness may also signal how best to support implementation to ensure equitable implementation efforts.
Oral Presentations
Policy/Practitioner Track
Crafting shared knowledge: Creative evidence synthesis and presentation practices for policy codesign in adolescent behavioral health
Authors
Chris Ackerley - University of Washington
Noah Gubner - University of Washington
Alya Azman - University of Washington
Kym Ahrens - Seattle Children's Hospital and Research Institute
Mandy Owens - Addictions, Drug & Alcohol Institute, University of Washington
Joe Langley - Lab For Living, Sheffield Hallam University
Sarah Walker - University of Washington
Background: Policy is an important tool to support the adoption and sustainment of evidence-based behavioral health practices. Yet, aligning policy intent, policy design, and successful policy implementation is a well-known challenge in the literature. A blended approach of coproduction and codesign — called Policy Codesign — has been iteratively developed over multiple years to address problems in policy formation contributing to poor implementation outcomes. Since Policy Codesign involves responsive cycles of evidence synthesis and presentation via interactive workshops, its implementation requires a practical understanding of how to create evidence syntheses that support rather than constrain creative innovation.
Approach: Drawing on strategies from coproduction and codesign, Policy Codesign employs varied formats to elicit and synthesize different types of evidence, to support the emergent needs of multisector codesign teams. By explicating design choices and sharing concrete examples, this presentation offers practice-focused insights from Policy Codesign for adolescent behavioral health in two diverse counties. Various approaches — including participatory mapping, tabletop games, tailored rapid reviews, and facilitation — enriched dialogue among the multisector teams. Transcripts and observational notes were analyzed after each workshop, to pragmatically assess alignment of the materials with the workshops’ goals and iteratively refine design principles to inform the final policy recommendation.
Outcomes: Incorporating and reflecting participants’ professional and lived experiences alongside academic evidence sources supported new and shared team understandings. While each Policy Codesign process has different needs, practical design principles, and required skillsets, templates were distilled from these two case studies to inform future creative evidence synthesis and presentation across Policy Codesign settings.
Next Steps: These Policy Codesign practices will be evaluated as part of a larger single-arm, longitudinal with staggered implementation study to examine their acceptability and short-term impacts. The group will continue exploring interactive ways to synthesize evidence for use with local, multi-sector groups of policymakers, practitioners, and community members.
Surfacing Implementation Research and Practice Tensions at the Crossroads of Research-Initiated Implementation in Practice Settings: A New Determinant Domain
Authors
Melanie Barwick - Hospital for Sick Children / University of Toronto
Andrea Greenblatt - Hospital for Sick Children
Ashleigh Miatello - Hospital for Sick Children
Kadia Petricca - Hospital for Sick Children
Background: Implementation research frequently occurs in real-world settings where the intention is to advance the science while successfully and sustainably implementing innovations in a practice setting. Tensions can emerge when implementing evidence-based innovations (EBIs) in real-world settings in the context of research studies. We call these Implementation Research and Practice Tensions (IRPTs). IRPTs occur when research methods conflict with how things are done in implementing organizations. Deficient collaboration or limiting the autonomy and decision latitude of the implementing organization may result in an implementation approach that does not consider real-world barriers, may be misused, and cannot be replicated. While determinant frameworks have long-identified factors acting as implementation barriers or facilitators, we suggest IRPTs are a separate determinant domain. IRPTs reflect a strained state between individuals or groups and the resulting problematic circumstances when research methods clash with practice realities to skew power dynamics and practice-based implementation experiences.
Approach: Members of a Canadian implementation research lab undertook a group brainstorming exercise to identify IRPTs that can derail implementation efforts and hinder optimal outcomes, producing sub-optimal experiences for practice settings. OUTCOMES: We identified eight tensions across our implementation practice-based studies: 1) misaligned EBI core components, 2) premature implementation, 3) undefined EBI core components, 4) misaligned timelines and expectations, 5) poor organizational fit, 6) misaligned clinical and research procedures, 7) blurred division of responsibilities, and 8) misaligned measurement and monitoring.
Next Steps: Detecting IRPTs early means they may be avoided or mitigated to circumvent detrimental effects on the implementation process and outcomes. Implementation conducted in the context of research studies has implications for the implementing organization's feasibility, timeline, outcomes, and innovation sustainability. This work is the first step in soliciting input from the field via dialogue and a future e-Delphi study toward improving effective implementation.
Developing a Resource Guide for Building Capacity for Refugee Research Partners
Authors
Sarah Brewer - Adult and Child Center for Outcomes Research and Delivery Science (ACCORDS); Department of Family Medicine; University of Colorado Anschutz Medical Campus
Farduus Ahmed - Refugee Review Board and the University of Colorado Anschutz Medical Campus
Mahri Hader - Refugee Review Board and University of Washington, Harborview Medical Center
Ola Alani - SRHP Refugee Review Board
Yusef Al-molieh - SRHP Refugee Review Board and University of Toronto
Jacob Atem - Southern Sudan Healthcare Organization
Anisa Ibrahim - Refugee Review Board and University of Washington, Harborview Medical Center
Russel Jabber - SRHP Refugee Review Board
Noor Almusahwi - SRHP Refugee Review Board
Taryn Bogdewiecz - University of Colorado Anschutz Medical Campus
Sarah Kimball - Boston University
Background: The Research Agenda Subcommittee of the Society of Refugee Healthcare Providers’ Research Evaluation and Ethics Committee (REEC) recently conducted a research agenda setting process to establish key areas in need of focus in refugee health. Engaging refugee community members as research partners was identified as a priority in the field. However, few resources are available to build the capacity of refugee partners as co-equal members on research teams. The Refugee Review Board (RRB) sought to fill this gap by creating a guidebook for refugee community members to build knowledge and skills as research partners.
Methods: The RRB identified topics for inclusion based on their experiences in research, a survey of Society members, and discussion with the REEC. We conducted a thorough review of available resources for community partners in research. Resources were evaluated for their content, user-friendliness, and cultural competence for refugee communities. For each topic, RRB members used a standardized template to provide brief content, recommend and summarize selected resources, and propose reflection questions for the refugee and their research partner.
Results: The RRB created “A Guide To Engaging In Patient Centered Research: Resources For Refugee Partners” with 3 units and 11 topic areas. The guide is being made publicly available to researchers and their community partners.
Conclusion: The RRB Guide provides a unique resource tailored to refugee communities to support capacity building and engagement of those with lived experience as refugees to become co-equal research partners. The RRB is seeking funding to offer facilitated training using the Guide.
A day in the life an implementation support practitioner: Navigating core competencies with new implementation initiatives
Authors
Ashley Charbonneau - Atlas Institute for Veterans and Families
Shannon Tracey - Atlas Institute for Veterans and Families
Tara McFadden - Atlas Institute for Veterans and Families
Background: In 2020, Metz and colleagues published the Implementation Support Practitioner Profile which outlines guiding principles and competencies of the role. Around the same time, a new intermediary organization, the Atlas Institute for Veterans and Families built an Implementation team intended to support movement of research into practice within the Veteran and Family mental health system in Canada. The initial few years focused on forming trusted relationships to build awareness and support engagement with implementation support services. The team is now engaged in the exploratory phase of its first two implementation practice projects, each aimed at supporting the implementation of an evidence-based mental health treatment into practice in external organizations. The projects provide an opportunity to share a practical example of how the Implementation Support Practitioner Profile is operationalized.
Approach: We will compare and contrast the experience of two implementation practitioners as they engage in the exploratory phase of their respective initiatives (implementation of 1. Written Exposure Therapy and 2. Neurofeedback treatment). The Implementation Support Practitioner Profile (Metz et al., 2020) will be used to guide the discussion, with a focus on one core competency from each of the three domains (Co-creation & Engagement; Ongoing Improvement; and Sustaining Change). We will lean into exploring co-design, understanding context, and developing transdisciplinary teams within the context of the projects’ exploratory phases.
Outcomes: Implementation practitioners will reflect on lessons learned from experience in this role. Facilitators and challenges relating to establishing the role, collaborating with other professionals, and initiating implementation projects will be discussed.
Next Steps: We will offer considerations and recommendations for those who are establishing themselves in the role of Implementation Support Practitioner within an organisation, including how to approach educating other professionals about the role and other tips for effective team building and collaboration when initiating an implementation project.
Universal Communication and Networks: Normalizing the Use of Implementation Best Practices in the Triple P Positive Parenting Program
Authors
Sandra Diehl - University of North Carolina at Chapel Hill
Will Aldridge - University of North Carolina at Chapel
Hill
Margaret Sullivan - University of Minnesota
Background: Universal implementation support has the potential to greatly accelerate the awareness of and normalization of implementation best practices by extending reach, modeling challenges and successes, and tapping into key implementation network influencers to share information, advice and expertise related to program scale-up via universal communication strategies. Universal support adds a multi-level approach to complement direct implementation support efforts.
Methods: We developed universal communication strategies using a theory-based conceptual model, survey data, and qualitative data from members of our stakeholder audiences (implementation teams, leadership teams, funders, system partners, community partners, support providers and policymakers). Strategies include social media, podcasts, electronic newsletters with capacity-building information, and participation in a Triple P learning collaborative. We use longitudinal social network analysis data to evaluate change in network composition, network metrics to evaluate collaboration, and the 20-item scale of perceived network support (SPANS) to evaluate change in network attitudes and behaviors toward implementation and scale-up over time.
Results: We created two network maps representing individual-level connections within and across two states. Comparisons of timepoints 1 and 2 were the following: The Triple P network remained a similar size across timepoints (n=72 and n = 67). Network density increased from .038 to .075. The number of unidirectional ties and number of reciprocal ties both increased, from 151 to 194 and from 22 to 29, respectively. The network reflected increased representation of African American respondents (19.1% to 26.9%), and a decrease in Hispanic respondents (7.4% to 4.5%). The ratio of female to male respondents remained consistent. Finally, average overall SPANS scores decreased from 4.1 to 3.9 on a scale from 1 (low) to 5 (high). These differences were not statistically significant.
Conclusion: We offer a theory and data-based rationale for using a multi-level approach that includes universal communication and networking to accelerate the use of best practices in the implementation of evidence-based programs.
Building Bridges: Strengthening Ohio's Child Welfare System through Implementation Support Infrastructure
Authors
Fawn Gadel - Public Children Services Association of
Ohio
Jennifer Millisor - Public Children Services Association of Ohio
Alicia Bunger - Ohio State University Wexner Medical Center
Rebecca Smith - University of North Carolina at Chapel Hill
Background: In response to a need for system-level implementation support for Ohio's child welfare agencies, the Public Children Services Association of Ohio (PCSAO) created a technical assistance (TA) and training infrastructure in 2019 to support statewide scale-up of the START (Sobriety, Treatment and Recovery Teams) model. Consistent with intermediary organizations, our goal is to lay the foundation for expanding implementation TA to support scale-up of other models in Ohio's child welfare system.
Approach: PCSAO developed the TA program to support agencies working toward model fidelity while striving to do what is best for families. In practice, implementation often occurs slowly and is disjointed; the TA program was built to be flexible, creative, and not time limited. We partnered with state agency funders and university-based implementation scientists to develop a program plan, review evidence about the implementation support professional role, secure funding, define TA consultant position descriptions, and design an evaluation. Simultaneously, we contracted with implementation TA consultants to help us co-develop the program content and format. Currently, the TA program supports model implementation in 53 of Ohio's 85 county child welfare agencies, 4 of which have been nationally certified by the National START Training and Technical Assistance Program.
Outcomes: We identified key lessons for designing and developing a TA implementation program: Invest in TA consultants with industry experience and train in coaching and facilitation skills; Cultivate strong relationships between and within the START teams and their TA consultants; Encourage TA consultants to tailor their delivery approaches based on local need and capacity; and Support the many roles TA consultants serve (e.g., educator, coach, mediator, boundary-spanner, champion).
Next Steps: We will continue to evaluate and refine the TA program to scale and sustain Ohio START. Our next step is to replicate the TA program to implement other evidence-based child welfare interventions.
How New Jersey Department of Children and Families Applied Implementation Theory and Research to the Statewide Scale-up of High-Quality Child Welfare Programming
Authors
Nancy Gagliano - New Jersey Department of Children & Families
Pamela Lilleston - New Jersey Department of Children & Families
Michael Doyle - New Jersey Department of Children & Families
Background: Over a decade of research points to the critical role of effective implementation and enabling contexts in ensuring evidence-based services achieve their intended outcomes. However, the reality of translating implementation research into practice can be challenging, particularly within large, complex systems like state government. As a result, many proven approaches fail to achieve their intended outcomes when scaled up in real-world settings.
Approach: Since 2017, New Jersey's Department of Children and Families (NJDCF) has systematically applied National Implementation Research Network's Active Implementation Frameworks to the statewide scale-up of services for children and families. We describe the organizing framework, internal infrastructure development, and innovative, practical strategies NJDCF used to translate implementation research into its infrastructure and norms. Examples include: investments in internal program development and research offices; development of multilevel teaming structures to facilitate collaboration and communication among Executive Leaders, program staff, researchers, implementation specialists, community providers and stakeholders; creation of structured programmatic plans to ensure that, for each program, implementation supports are tended to in alignment with the program's stage of development; and intentional collaboration among researchers, implementation specialists and program partners to facilitate evidence-driven decision-making.
Outcomes: NJDCF has successfully embedded implementation science best practices into its efforts to improve its contracted community-based services, totaling approximately $300 million. We describe lessons learned along the way, including the need to start small, strategically build internal capacity, learn and innovate, simplify processes, focus on sustainability, and build trust and accountability with diverse partners.
Next Steps: There is a gap between implementation research and practice. We describe a practical approach used within a state child welfare agency to apply implementation research to effectively execute and sustain evidence-based child welfare, behavioral and mental health and prevention-focused programs and practices. This approach can be feasibly adopted in other settings.
Operationalizing Implementation Science to Improve Student Mental Health
Authors
Abigail Gray - Transforming Research into Action to Improve the Lives of Students (TRAILS)
Karen Ware - Transforming Research into Action to Improve the Lives of Students (TRAILS)
Background: TRAILS (Transforming Research into Action to Improve the Lives of Students) is an innovative school-based mental health partner whose mission is to make effective mental health services accessible in all schools. TRAILS is rapidly expanding in response to demand from states and districts for its cost- and time-efficient programs. Preliminary research finds medium-sized effects on student depression and anxiety from TRAILS’ school-based programming.
Approach: TRAILS is rigorously developing and refining an evidence-based implementation model and compendium of implementation strategies by systematically tailoring, developing, testing, and improving supports in response to implementer needs and insights from implementation science research. We will detail TRAILS’ Center the Implementer process, which engages all staff and members of implementer groups in identifying new strategies for development, and our use of Implementer roadmaps to guide this process. We will also share our process for prioritizing, developing, and piloting new implementation strategies.
Outcomes: Short-term outcomes include insights about the processes, systems, and organizational commitments and realignments required to facilitate a deep, strategic focus on building implementation evidence while improving practice. Concrete outcomes of the early phase of this effort will be shared, including: TRAILS’ Implementation Map (iMap), which delineates roles, knowledge needs, barriers, and implementation strategies for each implementer group. We will discuss examples of new and evolving implementation strategies, including technology solutions and flexible implementation education and data-collection modules aimed at specific implementer groups (e.g. school principals).
Next Steps: In the long term, TRAILS’ investment in implementation science-driven innovation is expected to strengthen program implementation in schools and magnify program impacts. Next steps include: the development of a compendium of adapted implementation strategies and outcomes for school-based mental health programs; and research to examine relationships among TRAILS’ implementation strategies, implementation outcomes, and program outcomes.
Leveraging Policy Windows in Implementation Science to Maximize Impact
The past few decades have seen an increase in evidence-supported policy solutions to various public health problems. However, the availability of effective policy solutions to health problems by itself does not lead to policy change and subsequent improvements in population health. Opportunities to for policy change, also known as policy windows, open predictably or unpredictably and last for a brief period. These windows can facilitate knowledge uptake and evidence-supported policymaking. Strategies to identify and effectively use policy windows have received minimal attention in the field of implementation science. Tools are needed to effectively leverage policy windows. In this paper, we propose a guide based on Kingdon's policy window framework. The guide includes: 1) Ways to identify an open policy window—understand that solutions become policies not necessarily because they are the best, but sometimes because they are positioned to appear to be the only solution at the opening of a policy window; 2) engage key leaders and community voices; 3) propose an evidence-based solution—implementation scientists should have the readiness to offer solutions supported by the best available evidence; 4) provide clear strategies, outcomes and indicators for implementation success—offering a roadmap for implementation success; and 5) utilize technology effectively to advance a policy agenda—these could be achieved through the use of search terms, trending hashtags, and alerts on Google.
Advancing Implementation Practice and Research through Community-Academic Partnership
Authors
Yukihiro Kitagawa - University of Oregon
Alayna Park - University of Oregon
Anna Bartuska - University of Oregon
Tessa Palafu - The Baker Center for Children and Families/Harvard Medical School
Catherine Waye - The Baker Center for Children and Families/Harvard Medical School
Daniel Cheron - The Baker Center for Children and Families/Harvard Medical School
Kelsie Okamura - The Baker Center for Children and Families/Harvard Medical School
Rachel Kim - The Baker Center for Children and Families/Harvard Medical School
Background: Community-academic partnerships (CAPs) are mutually beneficial for advancing implementation practice and research. The Plan-Do-Study-Act (PDSA) cycle is a useful framework for facilitating implementation of evidence-based interventions (EBIs), enhancing the quality of healthcare, and improving patient outcomes. Leveraging their respective expertise, implementation practitioners and researchers can collaboratively build structures for collecting programmatic data (plan); implementation researchers can analyze programmatic data to inform quality improvements and extend implementation science (do); implementation practitioners and researchers can then review and interpret findings (study); and lastly, implementation practitioners can tailor implementation strategies (e.g., training and consultation) based on findings (act).
Approach: This presentation will describe the PDSA cycle completed in a CAP between a nationally recognized center facilitating implementation of EBIs for youth across the country and an implementation research lab at a major R1 university.
Outcomes: Our collaborative efforts have generated substantial outputs across all PDSA phases since 2020. The CAP “planned” the redesign of a measurement feedback system. The “do” phase involved analysis of programmatic data from 19 organizations, 1,181 clinicians, 11,802 patients, 9,763 sessions, 67,659 client survey records, and 89,457 caregiver survey records. Results were “studied” during weekly CAP “Think Tank” meetings to identify implementation barriers, generate novel implementation strategies and improve quality of youth mental health care. These strategies have been “enacted” in two new clinician trainings for community-based providers and ongoing consultation led by the center. Findings have been disseminated through one peer-reviewed publication and seven conference presentations to date.
Next Steps: The CAP members have applied for additional grant funding to further apply the PDSA cycle to iteratively refine implementation strategies and promote quality care, specifically by improving routine outcome monitoring, fidelity to evidence-based interventions, and cultural adaptation of evidence-based interventions. Strategies and lessons learned for building strong CAPs will be discussed to foster similar future partnerships.
A Model for Scaling the Mental Health Workforce in Sub-Saharan Africa
Author:
Aparna Kumar - The Carter Center
Eve Byrd - The Carter Center
Benedict Dossen - The Carter Center
Cecelia Flomo - Liberian Board of Nursing and Midwifery
Gemane Getteh - United Methodist University
Jerry Kulah - United Methodist University
Liberia has a significant need for mental health services with up to 20% of Liberians experiencing a mental health disorder. Prior experience with a prolonged civil war, Ebola outbreak, COVID-19 pandemic and severe shortage of mental health care workers contribute challenges to care-seeking and service delivery. In this context, we describe efforts to sustainably develop a community-based mental health workforce.
The Carter Center (TCC), government of Liberia, and local partners launched a partnership to develop a cadre of mental health clinicians comprising nurses and physicians’ assistants through a 6-month post-baccalaureate training. TCC supported stakeholder-led legal and regulatory changes to create an environment of practice for clinicians following training. To embed training locally, TCC partnered with United Methodist University (UMU) to create a two-year interdisciplinary, competency-based master's program. The curriculum was tailored to the Liberian context and supports applied learning, ongoing continuing education, and supportive clinical supervision.
Over 360 mental health clinicians across Liberia have successfully completed the 6-month training. Of these, 25 students are now enrolled in the master's program. Challenges to implementing the master's program include having appropriate competency assessments, securing adequate clinical training, and ensuring ongoing student support, particularly for students from rural areas. Advantages include tailored training to scale up the workforce and ensuring a long-term pipeline of mental health workers who can meet population demands. Building on these efforts, TCC is using the ECHO model to reach mental health and non-specialist workers in rural areas to enhance learning in priority topics such as substance use management.
Next steps involve engaging with African ECHO partners and creating a virtual mental health center for excellence to support regional training, continuing education, and clinical supervision. TCC will target training over 300 healthcare workers in two years with the goal of harmonizing the workforce to meet global mental health demands.
Integrating Systems Science and Participatory Modeling in Implementation Mapping for Chronic Disease Prevention
Authors
Travis Moore - Tufts University
Yuilyn A. Chang Chusan - Tufts University
Background: Addressing complex public health challenges such as chronic diseases require methodologies that adapt to their complexities. Traditional implementation mapping, although systematic, often overlooks the multifaceted nature of these public health issues. This study introduces a novel approach integrating systems science and participatory modeling, a combined method that employs participatory approaches to model systems, into implementation mapping to address the complexities in chronic disease prevention more effectively (Wang et al., 2021). In this oral presentation, we will highlight how we changed traditional implementation mapping steps to integrate systems thinking and modeling.
Approach: Our study employed participatory systems modeling, engaging community partners and researchers in creating both conceptual and simulated models of complex systems. This approach emphasizes the identification, selection, and operationalization of context-specific and relevant interventions to explore intervention effects and system interactions that can be accounted for and integrated into implementation (Rees, 2013).
Outcomes: Outcomes highlighted the effectiveness and challenges of participatory modeling in identifying and addressing emergent system behaviors and complex interdependencies that traditional methods may overlook. Specifically, the conceptual models facilitated the visualization of feedback loops and nonlinear dynamics, aiding in the formulation of quantitative models tailored to simulate intervention strategies that are sensitive to the underlying dynamics of chronic disease prevalence and management.
Next Steps: Integrating systems science and participatory modeling within implementation mapping provides a robust framework for developing, testing, and refining public health interventions. This approach not only enhances the adaptability and effectiveness of implementation strategies but also supports the decision-making processes necessary for complex public health challenges. Additionally, this approach could leverage social network data to investigate the dissemination of information to enhance practice or policy implementation. Future research should explore the application of this integrated approach across different public health issues and contexts to further validate its effectiveness and adaptability (Baugh Littlejohns & Wilson, 2019).
References
Baugh Littlejohns, L., & Wilson, A. (2019). Strengthening complex systems for chronic disease prevention: A systematic review. BMC Public Health, 19(1), 729. https://doi.org/10.1186/s12889-019-7021-9
Rees, D. (2013). Developing a Theory of Implementation for Better Chronic Healthcare Management: A Cognitive Mapping and System Dynamics Approach [Thesis]. https://doi.org/10.26686/wgtn.17004958.v1
Wang, Y., Hu, B., Zhao, Y., Kuang, G., Zhao, Y., Liu, Q., & Zhu, X. (2021). Applications of System Dynamics Models in Chronic Disease Prevention: A Systematic Review. Preventing Chronic Disease, 18, E103. https://doi.org/10.5888/pcd18.210175
Community-Academic Partnership in Leveraging Applied Implementation Data
Authors
Angelina Ruiz - The Baker Center for Children and Families/Harvard Medical School
Rachel Kim - The Baker Center for Children and Families/Harvard Medical School
Tessa Palafu - The Baker Center for Children and Families/Harvard Medical School
Katherine Corteselli - The Baker Center for Children and Families/Harvard Medical School
Catherine Waye - The Baker Center for Children and Families/Harvard Medical School
Alayna Park - University of Oregon
Yanchen Zhang - University of Iowa
Anna Bartuska - University of Oregon
Kelsie Okamura - The Baker Center for Children and Families/Harvard Medical School
This presentation will describe a unique community-academic partnership to leverage applied implementation data. The goal of this collaboration is to bridge research and practice by analyzing data collected from community partner agencies at multiple levels (initiative, organization, clinician, and client) during comprehensive implementation initiatives in order to inform both current and future implementation practice. This endeavor aligns with calls from SIRC to capitalize on practice partnerships to contribute to the field of implementation science (Connors et al., 2023).
Datasets come from diverse comprehensive implementation initiatives (N = 19) to implement three evidence-based child psychotherapy treatments (MATCH-ADTC, PCIT, and TF-CBT). Initiatives included 89 partner agencies, representing children's hospitals, community-based mental health agencies, and school-based mental health agencies from 12 states. Initiative, organization, and clinician-level data are collected through survey databases and client-level data are routinely collected through a web-based progress monitoring system. This nested data structure allows for exploration of the within and between effects of implementation-related factors across organizations and implementation initiatives.
Aggregated datasets across the 19 initiatives include 1,759 clinician background surveys, 11,802 client background surveys, 83,102 treatment session checklists, 67,659 youth client outcome surveys, and 89,457 caregiver client outcome surveys. Additional datasets including clinician-reported organizational readiness, self-efficacy for evidence-based psychotherapies, attitudes about evidence-based psychotherapies, and job satisfaction will be merged by summer 2024 and included in this presentation.
This collaboration provides an example of a community-academic partnership that strives to inform the field of implementation science. This presentation will include the following: 1) an overview of the process of developing data infrastructure and management for data collected from implementation initiatives, 2) an explanation of how these data are used to address the immediate needs of partner agencies, and 3) an outline of the broader conclusions from these data about implementation strategies to inform future implementation initiatives.
Reference
Connors, E. H., Martin, J. K., Aarons, G. A., Barwick, M., Bunger, A. C., Bustos, T. E., Comtois, K. A., Crane, M. E., Frank, H. E., Frank, T. L., Graham, A. K., Johnson, C., Larson, M. F., Kim, B., McHugh, S. M., Merle, J. L., Mettert, K., Patel, S. P., Patel, S. V., Swindle, T., Saldana, L., Stadnick, N. A., Viglione, C., & Powel, B. J. (2023). Proceedings of the Sixth Conference of the Society for Implementation Research Collaboration (SIRC) 2022: From Implementation Foundations to New Frontiers. Implementation Research and Practice, 4. https://doi.org/10.1177/26334895231173514
Implementation of Cure Violence in Southwest Philadelphia: A Community-Engaged, Equity-Centered & Adaptive Approach
Authors
Sara Solomon - University of Pennsylvania
Background: Despite a decline in violence since 2021, community violence in Philadelphia remains pervasive, particularly in areas of high concentrated disadvantage. Cure Violence (CV) is among the scant community violence interventions with partial effectiveness. However, implementation research methods have not been leveraged to guide communities in adapting interventions. The goal of this research is to develop and implement a stakeholder-engaged, local adaptation of the CV Intervention for Southwest Philadelphia.
Approach: We employed community-engaged implementation research methods to tailor the Penn Community Violence Prevention (PCVP) program, targeting the high-violence area of Southwest Philadelphia. This adaptation utilized an equity-focused framework and tools like concept mapping and SenseMaker® software, a mixed-methods, novel mobile app. Predominantly involving PCVP staff and a community advisory board—comprising residents, business owners, and community violence (CV) programs—this process refined PCVP's approach for subsequent evaluation.
Outcomes: Findings underscore the importance of engaging community members to ensure interventions are culturally attuned and locally relevant. We observed a significant shift in how violence is perceived – from a law enforcement issue to a public health crisis. Conclusively, the integration of community insights into the implementation strategy has shown potential for improving the model's efficacy and acceptability. Both quantitative and qualitative results will be shared.
Next Steps: The outcomes lead us to consider scaling the adapted model to other communities and conducting longitudinal studies to assess its long-term impact. Unanswered questions include identifying the most effective methods for sustaining community engagement and measuring the long-term effects on violence reduction and community resilience.
EBP or Better: An Intervention-Agnostic Approach to Promote the Development of Evidence-based Clinical Decision-making by Community Mental Health Providers Implementing Multiple EBPs
Authors
Sean Wright - Lutheran Community Services Northwest
Elizabeth Jacobs - Lutheran Community Services Northwest
Psychotherapy practitioners in publicly-funded systems of care may experience limited voice in the interventions they are asked to implement. System pressures to implement a limited menu of evidence-based practices have been identified as driving practitioners to leave community care settings prematurely to pursue eclecticism. We describe the initial results of an alternative approach called "EBP or better," a practitioner-focused framework that allows practitioners and supervisors/agencies to pragmatically assess the evidence-base of practitioners’ interventions (ranging from clinical micro-skills to evidence-based treatment packages and principle-guided approaches) as well as the implementation practices associated with the clinical interventions used. We assessed the evidence-base at the level of cues (e.g., clinically relevant stimuli that practitioners respond to), rules (fidelity to specific interventions), and meta-rules (e.g., evidence-based policies that guide the selection of rules).
EBP or better is an extension of our participation in an ongoing statewide effort to build internal agency capacity for delivering an EBP through embedded coaching, which entails developing agency-facilitated EBP training that is accepted as equivalent to external EBP trainings (UWCoLab, n.d.). EBP or better is an example of embedded clinical decision-making. This intervention-agnostic approach can accommodate multiple psychotherapeutic interventions (including multiple EBPs), which makes it possible to explore and potentially incorporate clinicians’ personal interests without losing sight of the evidence-base. Conceptualizing professional development as an equitable partnership between practitioners and supervisors/agencies, where the clinical interest of the practitioner can be harmonized with the evidence-base to ensure sustained delivery of high-quality care across diverse caseloads over time, may be a key step in addressing the problem of experienced therapists leaving publicly-funded settings. We will present 1) examples of two pragmatic assessments 2) a qualitative case study of a supervisor-supervisee dyad using this approach and 3) completed and planned implementation activities in a community mental health clinic.
Policymakers’ Use of Evidence When Designing Substance Use Treatment Benefits
Authors
Erika Crable - University of California San Diego
Siena Fisk - University of California San Diego
Background: Dissemination science presents a useful, but underused tool to address the growing research to policy gap. Misaligned and non-evidence informed policies across states and health plans are exacerbating disparities in access to medications for opioid use disorder (MOUD) for Medicaid members across the U.S. This study investigates how policymakers use different information sources to inform MOUD benefit policies and restrictions.
Methods: We developed a national survey of policymakers at all state/territory Medicaid agencies and Medicaid-contracted managed care organizations (MCOs) to investigate their evidence use behaviors and preferences. Survey instruments investigated Exploration, Preparation, Implementation, Sustainment Framework domains: outer (e.g., partisanship, state budget) and inner (e.g., leadership/staff preferences, benefit review processes) contexts, bridging factors (e.g., trusted intermediaries, data-sharing processes), innovation factors (MOUD acceptability, stigma), and were pilot tested by Medicaid staff for relevance before launching the national survey. We computed descriptive statistics to summarize national survey responses.
Results: Pilot testing items revealed challenges in using traditional implementation science measures (e.g., acceptability, implementation leadership) with policymakers and the need to optimize political science measures for dissemination science. To date, 86 survey responses describe evidence use behaviors at 33 Medicaid agencies, 3 territories, and 8 MCOs. Respondents primarily include top agency/MCO leaders (41.5%) and program directors (25.6%). Most (76.6%) are frustrated by political disagreement around MOUD; 55.6% viewed MOUD as more effective than non-pharmacological approaches alone. Most (96%) policymakers trust NIH-funded research, but prefer to conduct their own literature reviews, seek federal guidance or convene experts rather than receiving evidence from researchers directly.
Conclusions: This study is producing a national dataset explaining key determinants and intermediaries influencing policymakers’ evidence use in MOUD benefit design. Future latent class analyses will identify subgroups of Medicaid agencies/MCOs with similar evidence use preferences and will be used to design tailored dissemination strategies to promote evidence-informed MOUD benefits.
Tailoring the Parts and Wholes of Implementation to Contextual Determinants and Dynamics: An Example from Norwegian School Health Services
Authors
Thomas Engell - Center for Child and Adolescent Mental Health
Silje Berg - Center for Child and Adolescent Mental Health
Line Brager Larsen - Center for Child and Adolescent Mental Health
Josefine Bergseth - Center for Child and Adolescent Mental Health
Inga Brenne - Fremsam, WHO Healthy Cities Network
John Kjøbli - Center for Child and Adolescent Mental Health
Siri Saugestad Helland - Center for Child and Adolescent Mental Health
Line Solheim Kvamme - Center for Child and Adolescent Mental Health
Anneli Mellblom - Center for Child and Adolescent Mental Health
Background: Implementing psychosocial interventions in educational settings involves complex dynamic processes with many moving parts. Implementation science highlights the importance of tailoring discrete implementation strategies to produce causal mechanisms that address contextual implementation determinants and outcomes. However, implementation strategies may interact with contextual elements and system dynamics in terms of how and when they influence implementation outcomes. This study used causal modeling from different theoretical perspectives to tailor the implementation of a transdiagnostic youth mental health intervention to the Norwegian school health system.
Methods: First, we conducted a theory-informed mixed-methods context analysis using dialectical pluralism. Second, we combined implementation strategy co-design with causal pathway diagramming and system dynamics modeling to iteratively tailor implementation strategies to address specific and system determinants of implementation over time. Third, a hybrid mixed-methods intensive longitudinal study was conducted to test the implementation and intervention.
Results: The mixed-methods context analysis identified 16 implementation determinants that needed to be addressed. Co-design and causal pathway diagramming resulted in eight discrete implementation strategies hypothesized to meet these needs. Subsequent causal loop diagramming indicated that, despite the potential effectiveness of these strategies, system dynamics and unintended consequences were likely to undermine implementation success. We adapted the intervention and implementation strategies, developing 11 interconnected strategies conditioned on meeting certain intervention and implementation parameters. The implementation ends in June 2024, and implementation outcomes will be presented.
Conclusions: This study demonstrates the scientific and practical value of combining different causal theories in understanding and planning implementation. It provides actionable tools for implementation researchers, policymakers, and practitioners to enhance the understanding and planning of implementation. Linear and discrete causal modeling can help tailor potentially effective parts of the implementation, while systems modeling can help understand how they fit and interact as a whole in their intended contexts.
Utilizing the Grasha-Riechmann Framework to Identify Facilitation Styles Used as Part of the Implementation and Sustainment Facilitation Strategy: Findings from the Substance Abuse Treatment to HIV Care Project
Authors
James Ford - University of Wisconsin-Madison
Aaron M Gilson - University of Wisconsin-Madison
Kim Hoffman - Oregon Health Science University/Portland State University School of Public Health
Martha Maurer - University of Wisconsin-Madison
Michele A Gassman - University of Wisconsin-Madison
Bryan Garner - The Ohio State University
Background: The Implementation & Sustainment Facilitation (ISF) strategy has been effective for improving implementation of a motivational interviewing-based intervention (MIBIs) for substance use disorders within HIV service organizations (HSOs). Using the Grasha-Riechmann model that classifies five teaching styles – Delegator, Expert, Facilitator, Formal Authority, and Personal Model – we sought to understand facilitation styles used when delivering the ISF strategy. The Grasha-Riechmann model has not been applied to a facilitation-based strategy.
Methods: A total of 137 ISF strategy meetings with leadership staff and MIBI staff from participating HSOs were recorded and transcribed. A deductive thematic coding strategy using the Grasha-Riechmann model was developed and applied to a purposively-selected sample of transcripts (n=66) using NVivo. A case study thematic analysis was utilized to explore facilitation style changes across three ISF phases – preparation, implementation, and sustainment.
Results: The Grasha-Riechmann model effectively identified the use of different facilitation styles and individual elements within the styles. Examples include promoting a mechanism that will enable immediate independent work (Delegator); heading off/foreseeing future problems (Expert); offering empathy and/or encouragement (Facilitator); setting expectations (Formal Authority); and providing explicit examples of how the facilitator or others have previously carried out a task (Personal Model). Case study analysis also revealed that facilitation styles varied by HSO based on number of MIBIs completed (e.g., an HSO with 7 MIBIs versus one with 72 MIBIs completed) and across ISF phases within the HSO.
Conclusion: The Grasha-Riechmann model can be applied to facilitation styles and provided evidence that facilitation styles varied temporally depending on the ISF phase (e.g., implementation, sustainment). Findings address a literature gap, since few studies have examined the facilitation styles that ISF facilitators employ to improve MIBI implementation and sustainment in HSO. Future research examining relationships between facilitation styles, implementation fidelity, and client outcomes is warranted.
Decolonizing Suicide Prevention Implementation in Nepal: A Mixed Methods Implementation Feasibility Trial of a Co-Designed Culturally Anchored Suicide Prevention Package in a Nepali Hospital
Authors
Ashley Hagaman - Yale University
Renu Shakya - Kathmandu University School of Medical Sciences
Anmol Shrestha - Dhulikhel Hospital
Roshana Shrestha - Kathmandu University School of Medical Sciences
Riya Bajracharya - Kathmandu University School of Medical Sciences
Ajay Risal - Kathmandu University School of Medical Sciences
Pratiksha Paudel - Kathmandu University School of Medical Sciences
Madeleine Sorenson - Yale School of Public Health
Lakshmi Vijayakumar - Volunteer Health Services Hospital
Background: Low and middle income countries (LMIC) hold 75% of the world's suicides. However, little research in non-western and low-income contexts has informed suicide-specific prevention interventions and how they may be implemented in complex and resource-strained health systems. This project is situated in Nepal, a country with high suicide burden and increased mental health infrastructure. This study investigated preliminary implementation of a suicide prevention package (SuPP) for individuals at high risk for suicide following discharge from a large peri-rural Nepali hospital.
Methods: SuPP was co-designed with a community advisory board of individuals with lived experience of suicide, hospital staff and our team of Nepali anthropologists, psychiatrists, and public health experts. SuPP included a culturally anchored decolonized approach to “safety planning” called aashako diyo (meaning lighting hope), and tapered phone calls to adjust aashako diyo, increase connectedness, and decrease helplessness over six months. We conducted a pilot hybrid type two open pilot trial testing SuPP. We culturally adapted implementation assessment tools to measure acceptability, appropriateness, and fidelity. We used mixed methods and the EPIS framework to design, adapt, and assess the feasibility and acceptability of SuPP. We conducted embedded ethnography and periodic reflections to examine implementation climate, barriers, and facilitators. We track implementation adaptations with the FRAME.
Findings: Thirteen implementers and 24 individuals at high-risk for suicide were enrolled and 92% and 91.7% remained at six months respectively. SuPP met all pre-determined milestones to establish feasibility, fidelity, and acceptability, however, several barriers existed that suggest the hospital may not be an appropriate delivery agent particularly across genders. The presentation will highlight mixed-methods findings, focusing on elements related to fidelity, adaptations to improve delivery (attending to salient EPIS sub-domains of innovation fit and patient/family characteristics), and recommendations for future suicide prevention implementation in resource-strained, culturally diverse settings, like Nepal
Bridging the Gap Between Implementation Science and Indigenous Science with the xaȼqanaǂ ʔitkiniǂ (Many Ways of Working on the Same Thing) Research Team
Authors
Alex Kent - University of British Columbia- Okanagan
Sana Shahram - University of British Columbia-
Okanagan
Christopher Horsethief - Ktunaxa Nation Council
Bernie Pauly - University of Victoria
Background: Despite calls for action for reconciliation, decolonization, and Indigenization in both healthcare and research, there are few documented examples of how these interventions are being implemented. Implementation science can offer valuable theories, models, frameworks, and evidence-based strategies to support these efforts; however, there is a paucity of Indigenous scholarship informing the field. xaȼqanaǂ ʔitkiniǂ (Many Ways of Working on the Same Thing) is an Indigenous Nation-led partnership with the Ktunaxa Nation Council, Interior Health Authority, and university partners. We are an integrated knowledge translation project where partners are co-designing and piloting interventions with the aim of re-orienting local health services to learn from Ktunaxa approaches to promoting and restoring community wellbeing.
Methods: We combine implementation science with Ktunaxa methodologies to examine implementation processes and measure implementation outcomes. Implementation frameworks such as the Transcreation Framework for Community-Engaged Behavioral Interventions to Reduce Health Disparities (Nápoles & Stewart, 2018), provide evidence-informed guidance for planning, delivering, and evaluating interventions. Ktunaxa methodologies, such as ʔa·kwiti̓s ktunaxa (the wing model of culture), ʔuk̓iniǂwitiyaǂa (a group thinking with one heart), and xaȼqanaǂ ʔitkiniǂ (many ways of working on the same thing) (Horsethief, 2021), are culturally-informed approaches to collaborative problem solving, as aligned with millennia-old protocols in the Ktunaxa Nation.
Results: So far, we have facilitated 11 community dialogues, with diverse representation from Nation administration, Band Council leadership, health service providers, residential school survivors, and youth (N=133). Qualitative data from the dialogues demonstrate colonial disconnection within the health system as a major barrier to equitable healthcare; data also highlight cultural connection as a source of resiliency and a key facilitator of community wellbeing.
Conclusions: This research advances implementation science by weaving in Indigenous science, knowledge systems, and ways of knowing. Transferable takeaways include insights for applying community-driven, locally-specific, and culturally-informed approaches to co-learning and co-creating knowledge.
Nápoles, A. M., & Stewart, A. L. (2018). Transcreation: an implementation science framework for community-engaged behavioral interventions to reduce health disparities. BMC health services research, 18(1), 710. https://doi.org/10.1186/s12913-018-3521-z
Using Synergistic Partnerships and Implementation Science to Develop and Scale-Up a Peer-Based Intervention to Support Justice-Involved Veterans
Authors
Beth Ann Petrakis - VA Bedford Healthcare System
Bo Kim - VA Boston Healthcare System
Justeen K. Hyde - VA Bedford Healthcare System
Thomas H. Byrne - VA Bedford Healthcare System
Vincent Song - VA Bedford Healthcare System
Maria D. Venegas - VA Bedford Healthcare System
Eric Richardson - VA Boston Healthcare System
D. Keith McInnes - VA Bedford Healthcare System
Background: VA's national Veterans Justice Programs (VJP) recently added peer specialists to provide social support, linkage and referral to services, and skill-building guidance to justice-involved veterans. This expansion evolved from our close collaboration with VJP leadership to design and implement a peer-based Post-Incarceration Engagement (PIE) intervention.
Approach: Our two-state pilot, guided by the Consolidated Framework for Implementation Research, signaled PIE's positive impact on engagement in healthcare, housing stability, and reduced recidivism. Post-implementation we further specified core PIE components to inform expansion. Our VA central office partners helped identify expansion sites. Formative interviews, guided by the Dynamic Sustainability Framework, informed tailoring of core components to local settings. Implementation strategies including facilitation and ongoing monitoring ensured fidelity and identified adaptations. Monthly meetings with partners supported information sharing and problem-solving. Discussions and site visits, including one with a VJP partner, increased shared understanding and informed VJP's subsequent nationwide spread plans.
Outcomes: Six expansion sites launched PIE. Four are currently implementing with high fidelity and two suspended implementation due to staffing challenges. Since June 2022, 643 incarcerated and 241 released veterans received VJP PIE peer services. Staff and veterans note the value-added peers bring to help address reentry veterans’ needs including increased capacity to address their complex healthcare needs, logistical support to complete critical community reintegration tasks, and increased sense of hope inspired by peers sharing their recovery stories. Our on-going evaluation and timely dissemination of findings supported VJP's efforts to fund peers and expand services. VA's VJP peer program is now a sustainable practice, with 56 peers and imminent plans to hire 38 more. An April 2024 policy directive delineates VJP peer specialist responsibilities.
Next Steps: We will create an implementation playbook that includes a manual, training materials, and evaluation measures. This will inform future implementation and understanding of peer-based justice-related interventions.
Practice Assessment of Change Engagement (PACE): A Pragmatic Measure from an EvidenceNOW Trial of Streamlined Practice Facilitation
Authors
Reid Parks - Indiana University
Madeline Moureau - Medical College of Wisconsin
Joanna Balza - Medical College of Wisconsin
Kathryn Flynn - Medical College of Wisconsin
Joan Neuner - Medical College of Wisconsin
Heidi Brown - Permanente Medicine
Edmond Ramly - Indiana University
Background: Clinicians and practice change teams have limited time and resources to engage with facilitated implementation projects. Efficient facilitation is important for implementing interventions into everyday practice without interrupting care. Measuring how engaged a practice is in an implementation project can help implementation facilitators provide practices with the tailored support they need. Research-oriented measures of practice engagement depend on clinicians and practice change teams completing time-consuming surveys. Within an AHRQ-funded EvidenceNOW trial of streamlined practice facilitation, we developed a pragmatic rubric for facilitators to perform an ongoing Practice Assessment of Change Engagement (PACE).
Methods: Our multidisciplinary team (practice facilitation, implementation science, primary care) iteratively developed the PACE rubric building on Normalization Process Theory (NPT) and established measures of organizational readiness, capacity, and capability. Facilitators assigned PACE scores to 33 participating clinics over multiple timepoints across 108 implementation meetings. We calculated longitudinal descriptive statistics and performed correlational analyses between PACE scores and baseline Organizational Readiness for Implementing Change (ORIC) scores.
Results: The PACE rubric includes 4 criteria to help facilitators consistently and rapidly assign practice engagement scores from 5 (exceptional engagement) to 0 (termination of engagement). The criteria, grounded in NPT, are: i) Coherent progression through planned tasks in the project schedule, ii) Participation in facilitation activities, iii) Dedication to collective action, and iv) Resourcefulness through problem-solving. The PACE scores from our 108 practice facilitation meetings averaged 3.27; mid-implementation scores were 1.4 points higher on average than pre-implementation scores. Average PACE scores were highly correlated with ORIC scores (correlation = 0.7).
Conclusions: This study established a pragmatic scoring rubric for practice engagement with implementation. Increased engagement from pre- to mid-implementation may capture the impact of ongoing engagement with the implementation effort. Future work includes evaluation of the measure for use in different contexts.
Outcome Preferences in Parenting Program Adaptations: A Discrete Choice Experiment
Authors
Kristoffer Pettersson - Mälardalen University
Aaron Lyon - University of Washington
Henna Hasson - Karolinska Institutet
Fabrizia Giannotta - Stockholm University
Pernilla Liedgren - Mälardalen University
Ulrica von Thiele Schwarz - Mälardalen University
Background: In recent years, the field of Implementation Science has started to consider the multitude of sometimes conflicting outcomes involved in managing fidelity-adaptation dilemmas. These decisions can be managed as part of implementation planning, but evidence suggests that applied contexts continuously require navigating fidelity-adaptation issues. This makes front-line practitioners’ prioritization of outcomes crucial for successful implementation. This study aims to investigate practitioners’ outcome preferences as predictors of fidelity-adaptation decisions in evidence-based parenting programs.
Methods: A Discrete Choice Experiment (DCE) was utilized to evaluate outcome preferences among practitioners trained in evidence-based parenting programs within Swedish welfare systems. Data analysis employed a Bayesian Hierarchical Logistic Regression model, accounting for the importance of outcome preferences for choices made in typical fidelity-adaptation scenarios and practitioners’ willingness to make trade-offs when outcomes conflict.
Results: When faced with typical fidelity-adaptation dilemmas, practitioners displayed preferences for options, as indicated by log-odds estimates, in descending strength: improved parent-child relationship (+4.56, 95% CI: [4.19, 4.94]), increased program satisfaction (+2.43, 95% CI: [2.10, 2.77]), reduction of value conflicts (-2.37, 95% CI: [-2.71, -2.03]), reduction of workload strain (-2.09, 95% CI: [-2.44, -1.75]), and increased program reach (+1.96, 95% CI: [1.63, 2.31]).
Conclusions: Practitioners’ outcome preferences were mainly in line with the goals of evidence-based parenting programs, emphasizing aspects related to efficacy, reach, and satisfaction. However, results also indicate that practitioners weigh these outcomes against those directly affecting themselves. When value conflicts or workload strains become too severe, these aspects might take the upper hand and influence practitioners to sacrifice intervention outcomes to reduce the burden put on them as individuals. There are limits to practitioners’ loyalty to intervention outcomes. This study offers an approach to quantify this limit in a way that could inform implementation projects to avoid unplanned modifications later on.
Health Equity Promotion through Community-Engaged Implementation: Co-Designing Multilevel Implementation Strategies to Promote Gender-Affirming Psychotherapy Adoption
Authors
Maggi Price - Boston College
Kara Johansen - Boston College
Marina Rakhilin - MIT Center for Constructive Communication
Patrick Mulkern - Boston College
Madelaine Condon - Boston College
John Pachankis - Yale University
Lisa Saldana - Chestnut Health Systems - Lighthouse Institute Oregon
Aaron Lyon - University of Washington
Background: Health inequities are exacerbated by multilevel barriers to both health and healthcare access. Despite the development of efficacious health equity interventions, their broader reach is often impeded by implementation barriers such as provider bias and insufficient leadership endorsement. Our study employs community-engaged methods to co-design multilevel implementation strategies to increase the adoption of Gender Affirming Psychotherapy (GAP; core skills and practices to improve mental health care for transgender clients). This presentation will outline the implementation strategy co-design process and offer concrete recommendations for implementation scientists committed to advancing health equity.
Methods: As part of a larger research-clinic partnership pilot study, we utilized the 3-stage human-centered design approach “Discover, Design/Build, and Test” to develop GAP and associated implementation strategies. The Discover stage involved a scoping review and identifying determinants of GAP implementation via 9 focus groups with transgender clients, their parents, and providers. The Design/Build stage included co-designing implementation strategies with these stakeholder groups across 6 meetings and 70 meetings with research-clinic providers, and refining strategies through 23 usability testing sessions. The Test stage (ongoing) involved collecting pre-implementation data to validate previously-identified determinants. Data were analyzed through traditional and rapid content analysis.
Results: We identified determinants of GAP implementation spanning several levels, including patient-level factors like unsupportive parents, provider-level issues such as lack of knowledge, and organizational barriers including non-inclusive intake forms. Tailored implementation strategies were developed to address these barriers, including patient and provider narratives to combat bias, practice activities to boost provider competency, and organization-level modifications to electronic health records. Preliminary pilot results suggest that these strategies led to significant improvements in GAP adoption among N=78 providers (d=0.63***).
Conclusions: Community-engaged methods can be leveraged by practitioners and implementation scientists to develop acceptable and scalable multilevel implementation strategies to promote health equity intervention implementation.
Development and Preliminary Validation for a 15-item Scale of Local-Regulation of Implementation Processes
Authors
Rebecca Roppolo - University of North Carolina at Chapel Hill
Will Aldridge - University of North Carolina at Chapel
Hill
Background: The success and sustainment of evidence-based programs/empirically-supported interventions (EBP/ESI) hinge in part upon the capacity of the teams responsible for day-to-day management and scale-up activities to locally regulate these implementation processes. However, the field lacks measures for these team-based abilities and resources. Using best practices in scale development, a 15-item measure of the resources and abilities of teams to apply implementation best practices has been developed.
Methods: Deductive methods generated an initial 104-item pool that was subjected to content expert validation. Items were deleted and refined using content adequacy assessments and content validity ratios to improve the reliability and validity of the measure, resulting in a preliminary 16-item scale. This scale was administered to a developmental sample of implementation support practitioners (ISPs) and regional support recipient teams in the Implementation Capacity for Triple P project through the Frank Porter Graham Child Development Institute at the University of North Carolina at Chapel Hill. An exploratory factor analysis (EFA) was conducted on the responses from ISPs (n=253) and support recipients (n=163) collected between July 2019 and June 2023 to assess the structure of the scale, examine its internal reliability, and improve its performance.
Results: The EFA yielded a three-factor solution that explained 74.9% and 70.8% of cumulative variance for ISPs and support recipients, respectively. Items that were poor indicators of the underlying dimensions were identified and removed. The final 15-item measure of the regulation of implementation processes was established using an iterative process to improve the representativeness, clarity, comprehensiveness, reliability and content validity of the scale.
Conclusion: Effective regulation of implementation processes is an important characteristic of teams that successfully scale-up and sustain EBP/ESIs. This measure may support the development of empirical knowledge about the correlates for successful EBP/ESI implementation and in the emergent literature on the sustainability of implementation.
Successes and Challenges in Recruiting School-Based Health Centers for Implementation Research Supporting LGBTQ+ Adolescent Health Equity
Authors
Daniel Shattuck - Pacific Institute for Research and Evaluation
Cathleen Willging - Pacific Institute for Research and Evaluation
Mary Ramos - University of New Mexico
Background: School-based health centers (SBHCs) are well-positioned to address the many unmet medical and behavioral health needs of LGBTQ+ adolescents. However, SBHCs often lack policies, procedures, training, and other structural elements that facilitate high-quality services for these youth. The “SBHCs Addressing Health Equity for LGBTQ+ Patients (SBHCs HELP)” study aims to engage SBHCs in New Mexico in applying LGBTQ+ inclusive practices to enhance patient care while examining implementation processes and impacts.
Methods: SBHCs HELP is a four-year stepped-wedge implementation trial that uses the Dynamic Adaptation Process in high-school SBHCs to operationalize the Exploration, Preparation, Implementation, Sustainment Framework and facilitate practice uptake. We used an iterative approach to code documents detailing SBHC recruitment processes (e.g., recruitment logs, emails, fieldnotes) to discern influential factors.
Results: The (mis)alignment of research funding, dictates of scientific rigor, and school calendars exerted a major influence on recruitment including adjusting anticipated timelines. Key outer-context factors included community opinions toward SBHC's confidential services that were shaped by conservative discourses about “parents’ rights.” Bridging factors included SBHCs relationships to the state-level school health system formalized through contracts, funding, and leadership structures. Inner-context factors included SBHCs’ organizational complexity, decision-making processes, and the capacity, attitudes, and beliefs of staff. Innovation characteristics—particularly the LGBTQ+ focus and potential to mitigate the negative effects of localized stigma and discrimination—emerged as a powerful motivation for participation. Despite potential barriers to participation, we successfully recruited over 53% (n=25) of eligible SBHCs.
Conclusions: This work illustrates our negotiation of the pragmatics of community-based work, funding-related restrictions, and ensuring scientific rigor in the process of recruitment. Second, it reveals a dramatic divergence from previous projects where influential factors typically construed as barriers (e.g., community and family resistance to LGBTQ+ supportive action) propelled SBHC decisions to participate in organizational change efforts.
Harnessing the Power of Implementation Research to Jumpstart Culturally-Informed Advanced Care Planning with Alaska Native and American Indian People in Primary Care
Authors
Jennifer Shaw - University of Alaska Fairbanks
Sonia Singh - University of Arkansas for Medical
Sciences
Christopher Piromalli - Southcentral Foundation
Sara Landes - University of Arkansas for Medical
Sciences
Solana Rollolazo - Southcentral Foundation
Christina Fieldhouse - Southcentral Foundation
Shannon Medlock - Southcentral Foundation
Ryan Mortenson - Southcentral Foundation
Background: Advance care planning (ACP), a cornerstone of palliative care, aims to increase quality-of-life and reduce suffering among people with serious illness and their families. Alaska Native and American Indian (ANAI) populations have higher rates of serious illness, with lower rates of ACP, than the general population. Previously, we culturally-tailored and piloted an ACP communication tool, JUMPSTART-ANAI, in Alaska and New Mexico. We are now planning a hybrid type 1 effectiveness-implementation trial of JUMPSTART-ANAI. This presentation presents results from qualitative data collected to guide integration of JUMPSTART-ANAI into routine primary care in the Alaska Tribal Health System.
Methods: At submission, semi-structured interviews and focus groups had been completed with 16 clinicians and administrators and 8 customer-owners in a large Tribal health organization in Anchorage. Guided by the Consolidated Framework for Implementation Research, we used template analysis to code interview data. Four analysts iteratively coded two interviews to establish consistency. Next, two analysts independently coded each interview and collaboratively reconciled discrepancies. Analysts sorted data into the structured template, allowing for rapid and rigorous data reduction, organization, and summarization.
Results: Participants endorsed implementing JUMPSTART-ANAI in the Tribal health system. Employees emphasized the need for multiple, flexible methods of delivery and evaluation of JUMPSTART-ANAI to maximize intervention reach. Customer-owners emphasized that implementation should incorporate culturally-centered strategies such as learning circles and shared story. Using qualitative findings and the StrategEase tool, we identified organizationally and culturally feasible and acceptable implementation strategies.
Conclusion: Next, the study team will collaborate with health system champions and internal facilitators to refine strategies for integrating JUMPSTART-ANAI into routine primary care in the Tribal health system. The final implementation plan will be piloted in preparation for a type 1 hybrid effectiveness-implementation cluster randomized trial of JUMPSTART-ANAI to increase ACP among ANAI people with serious illness in primary care.
Selection and Specifying of Implementation Strategies to Mitigate Barriers of n Integrated Community-Based Frailty and Intrinsic Capacity Management Program for Older Adults in Singapore: Using the CFIR-ERIC Matching Tool and Stakeholder-Driven Approach
Authors
Grace Sum - Geriatric Education & Research Institute, Singapore
Mimaika Luluina Ginting - Geriatric Education & Research Institute, Singapore
Zhen Sinead Wang - SingHealth Polyclinics, SingHealth, Singapore
Pei Yoke Tay - Community Nurse, Department of Nursing
Susan Ho - Bethesda CARE Centre (BCARE), Singapore
Yew Yoong Ding - Geriatric Education & Research Institute, Singapore
Laura Tay - Geriatric Medicine, Department of General Medicine, Sengkang General Hospital, Singapore
Background: The selection and tailoring of implementation strategies are crucial for successfully implementing evidence-based healthcare programs in local contexts. An integrated community-based frailty and intrinsic capacity screening and management program for older adults is piloted in Singapore. It is adapted from the World Health Organization Integrated Care for Older People (WHO ICOPE) framework. Our previous study identified the Consolidated Framework for Implementation Research (CFIR) version 2.0 barriers from focus group discussions with healthcare professionals. This study aims to select and specify Expert Recommendations for Implementing Change (ERIC) strategies for development to address contextual barriers.
Methods: CFIR determinants were matched to implementation strategies using the CFIR-ERIC matching tool. A stakeholder panel (SP) consisting of researchers and lead implementers (healthcare workers, community partners) participated in selecting ERIC strategies. Participants independently ranked strategies based on relevance, improvement opportunity, feasibility, difficulty, and validity. The stakeholder-driven decision-making process of systematically narrowing down the strategies was conducted via a meeting and emails. A series of questions guided the discussion to elicit the SP's considerations. Implementation research guidelines informed the domains for specifying strategies.
Results: After applying the matching tool and stakeholder-driven approach, this study selected 5 of 73 ERIC strategies. These included 1) Develop educational materials 2) Conduct educational meetings 3) Distribute educational materials 4) Develop a formal implementation blueprint 5) Identify and prepare champions. The SP revealed a need for an education-focused strategy package among the first three strategies. We specified each strategy in detail, contextualised strategies to practice settings, and developed indicators for tracking each strategy to evaluate the fidelity and success of implementation.
Conclusion: This study contributes towards the documentation of selecting and specifying implementation strategies that mitigate contextual barriers to successful implementation of a community-based program. Lessons learnt will improve the process and refine its documentation for reporting and replicability of results.
Applying Collaborative Implementation Science Approaches to Develop an Australian “Research-Ready” Precision Medicine Clinic
Authors
Skye McKay - Faculty of Medicine and Health, University of New South Wales, Sydney
Shuang Liang - Faculty of Medicine and Health, University of New South Wales, Sydney
Kathryn Leaney - Consumer Involvement in Research, Cancer Voices, Sydney
Jeffery Chan - Faculty of Medicine and Health, University of New South Wales, Sydney
Arya Shinde - Faculty of Medicine and Health, University of New South Wales, Sydney
Bridget Douglas - Precision Care Clinic, Prince of Wales Hospital, Sydney
Helen Ke - Precision Care Clinic, Prince of Wales Hospital, Sydney
Mandy Ballinger - Faculty of Medicine and Health, University of New South Wales, Sydney
David Thomas - Faculty of Medicine and Health, University of New South Wales, Sydney
Frank Lin - Garvan Institute of Medical Research,
Sydney
Milita Zaheed - Precision Care Clinic, Prince of Wales Hospital, Sydney
Kathy Tucker - Precision Care Clinic, Prince of Wales Hospital, Sydney
David Goldstein - Precision Care Clinic, Prince of Wales Hospital, Sydney
Natalie Taylor - Faculty of Medicine and Health, University of New South Wales, Sydney
Background: Despite significant advances in precision medicine technology and accessibility, real-world provision of genomics care continues to face multi-level barriers. These challenges are exacerbated by unsystematic implementation efforts, leading to resource wastage, and missed opportunities for patients. The Precision Care Clinic (PCC), the first of its kind in Australia, aims to address these issues by integrating genomic data into routine cancer care. The clinic's underlying model focuses on bridging knowledge-service gaps in precision medicine.
Methods: The aim of Phase 1 was to co-design the clinic (service intervention), the implementation platform (implementation intervention), and a suite of outcome measures. This was undertaken by clinicians, researchers, and consumers with expertise in precision oncology, implementation science, clinical informatics, cancer genetics, health economics and personal experience. Implementation researchers were integrated into clinical operations to observe and provide real-time feedback. Mixed methods, guided by implementation frameworks, were used to triangulate data from observational, qualitative, and quantitative sources. Implementation tools, such as process mapping and implementation research logic models, captured clinic process iterations and complex interrelationships. Phase 2, a Type 1 Hybrid effectiveness-implementation trial, is underway.
Results: Clinic co-design occurred April-December 2023, and the service publicly launched in March 2024. An implementation log systematically captured CFIR2.0 coded barriers, enablers, adaptations, and intuitive strategies. Inner setting determinants (e.g., hospital infrastructure) and outer setting barriers (e.g., communication with referring clinicians) were identified. The rapidly changing precision medicine landscape influenced the implementation process, necessitating a flexible and adaptive approach. Patient perspectives highlighted the value of accessibility, timeliness, understanding, and satisfaction, which were included in patient-reported outcomes.
Conclusion: Designing and implementing the PCC, a complex clinical intervention, requires careful integration of implementation research into clinical processes. Close collaboration with intervention deliverers and recipients allows real-time monitoring of precision medicine contextual factors. A flexible approach ensures systematic adaptation and informative data for scale-up and sustainability.
The Moderating Effect of Attitude toward EBP Adoption on Staff Perceived Implementation Leadership and Implementation Outcomes in California School Systems
Authors
Yue Yu - UC Davis MIND Institute
Melina Melgarejo - San Diego State University
Patricia Schetter - California Autism Professional Training and Information Network
Jessica Suhrheinrich - San Diego State University
Aubyn Stahmer - University of California Davis
Background: Ensuring effective use of evidence-based practices (EBP) for autism in schools is imperative. Implementation leadership and attitudes toward EBP adoption relate to successful EBP implementation (Lyon et al., 2019; Melgarejo et al., 2020). Little is known about how these two factors interact and impact implementation in schools. Using an implementation science framework (Aarons et al., 2011), we explored attitudes toward EBP as a moderator between implementation leadership and EBP implementation outcomes.
Methods: Data were collected from school-based providers (e.g., teachers) and administrators working in public schools supporting autistic students in California and aggregated at the district level (N = 490). Surveys included School-Implementation Leadership Scale (S-ILS), Evidence-based Practice Attitude Scale (EBPAS), and EBP Implementation Outcomes. Providers rated their primary leader in implementation leadership. Participants in a leadership role (e.g., principals) rated their own leadership. Moderation analysis was conducted using SPSS PROCESS macro (Hayes, 2013).
Results: At the district level, self-report implementation Leadership was slightly higher than staff-report (meanself = 10.15, SD = 2.97; meanstaff = 8.52, SD = 3.60). The average Attitude toward EBP adoption across districts was moderate (mean = 10.50, SD = 2.54). For the moderation analysis, Attitude significantly moderated the effect of staff-reported Leadership on implementation outcome (b = 0.01, SE = 0.004, t = 2.56, p = .01; R = .34, F(3, 291) = 13.04, p < .001; See Figure 1). The moderation effect on self-report Leadership was not significant.
Conclusions: Attitude toward adopting EBP moderates the effect of leadership on EBP implementation at the district level in California. When Attitude is low, the impact of Leadership on EBP implementation is limited. As Attitude toward EBP increases, the positive impact of Leadership on EBP implementation increases. Therefore, increasing Attitude toward EBP may help maximize the effectiveness of Leadership-focused implementation intervention in the school system.
References
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and policy in mental health and mental health services research, 38(1), 4-23. https://doi.org/10.1007/s10488-010-0327-7
Hayes, A. F. (2013). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach. The Guilford Press.
Lyon, A. R., & Bruns, E. J. (2019). From evidence to impact: Joining our best school mental health practices with our best implementation strategies. School Mental Health, 11(1), 106–114. https://doi.org/10.1007/s12310-018-09306-w
Melgarejo, M., Lind, T., Stadnick, N. A., Helm, J. L., & Locke, J. (2020). Strengthening capacity for implementation of evidence-based practices for autism in schools: The roles of implementation climate, school leadership, and fidelity. American Psychologist, 75(8), 1105. https://doi.org/10.1037/amp0000649
Observing Community Therapist Augmenting Adaptations in TF-CBT Sessions and Implications for Clinical Process Outcomes with Racial/Ethnic Minoritized Youth
Authors
Stephanie Yu - University of California, Los Angeles
Caroline Shanholtz - Lyra Health
Kenya Rodriguez-Montalvo - Alta California Regional Center
Qiwen Jin - University of Pennsylvania
Alyssa De Los Santos - California State University, Fullerton
Qiran (Cheryl) Sun - University of California, Los Angeles
Leila Malak - University of California, Los Angeles
Mia Flores - University of California, Los Angeles
Adriana I. Perez - California State University, Northridge
Eman Magzoub - University of California, Los Angeles
Xuanyao (Clark) Qian - University of California, Los Angeles
Topaz Tabibi - Alliant International University
Namrata Poola - University of California, Los Angeles
Sebastian Luque - University of California, Los Angeles
Brissa Bejarano - University of California, Los Angeles
Erica Jones - University of California, Los Angeles
Ana Berman - University of California Los Angeles
Umiemah Farrukh - University of Southern California
Xinran (Wendy) Wang - Vanderbilt University
Qasim Farrukh - University of California, Los Angeles
Joanna Kim - Arizona State University
Lauren Brookman-Frazee - University of California San Diego
Anna Lau - University of California Los Angeles
Background: Community therapists often adapt evidence-based practices (EBPs) when transported into public care systems to enhance their fit for the diverse communities they serve. Studies of therapist-reported adaptations have identified Augmenting adaptations as those that add to or tailor the EBP, such as modifying presentation of EBP content, integrating supplemental content, and extending through repetition. Community therapists hold valuable local expertise about what may promote client engagement in their care context. Yet, there may be risks to EBP integrity if adaptations are unsystematic. The study aims were to examine associations between Augmenting adaptations and (1) client engagement outcomes and (2) adherence in sessions of trauma-focused cognitive behavioral therapy (TF-CBT).
Method: Community therapists (n=46) provided 190 TF-CBT session audio recordings delivered to 82 youth (Mage=10.30; 67.07% Latine, 21.95% Black, 7.32% Asian American/Pacific Islander, 2.44% White, 1.22% Multiracial). Two observational coding systems were developed to index Augmenting adaptations (MICC=.76) and adherence to TF-CBT (MICC=.85) at the session-level. Two independent teams of coders (n=18) were trained to rate the extensiveness of each adaptation type and TF-CBT component, respectively, from session audio recordings (0=no occurrence, 6=great extent). Client disengagement was reported by therapists and client engagement behaviors were coded observationally from a prior study. Multilevel modeling was used, controlling for session, client, and therapist factors.
Results: Modifying presentation adaptations were associated with lower odds of therapist-reported client disengagement in session (OR=0.41, 95% CI [0.15, 0.85]). Integrating adaptations were associated with higher odds of therapist-reported client disengagement in session (OR=2.10, 95% CI [1.08, 5.03]). Repeat adaptations were associated with more observed client engagement behaviors in session (β=0.13, 95% CI [0.01, 0.26]). No Augmenting adaptations were related to adherence to TF-CBT at the session-level.
Conclusion: This study holds translational value for improving EBP implementation and delivery in community mental health settings serving structurally marginalized communities.
Slam Presentations
Policy/Practitioner Track
The Implementation Playbook: Usability of Software Supporting Digital Facilitation in Service Settings
Authors
Melanie Barwick - The Hospital for Sick Children/U of Toronto
Kadia Petricca - The Hospital for Sick Children
Jacquie Brown - Triple P International
Jill Shakespeare - Centre for Addiction and Mental Health
Emily Seto - University of Toronto
Bonnie Stevens - The Hospital for Sick Children/U of Toronto
Byron Powell - Washington University in St. Louis
Alexia Jaouich - Stepped Care Solutions
Michele Sparling - Family Advocate
Evidence-based innovations (EBIs) promote effective outcomes, but effective implementation remains problematic despite advances in implementation science. Organizations often need help accessing, understanding, and using implementation evidence. Implementation support is typically static, cumbersome, overly academic, and rarely evaluated. Technical assistance is soft-funded, costly, and inequitably accessible. In response, we developed first-in-kind software called The Implementation Playbook (TIP) to provide practical, empirically based guidance and project management functionality. TIP is intended for practice-based users implementing any EBI.
Approach: Healthcare organizations were recruited globally via networks and social media to test TIP in their setting while implementing an EBI of their choosing. We invited organizations whose implementation timelines aligned with our evaluation funding and where the EBI was sufficiently complex. Organizations secured ethics approval, established and consented their implementation teams, and completed surveys about readiness (ORIC) and their usual implementation approach. Users created logins and invited their team on TIP's URL. Organizations use TIP independently and provide verbal and questionnaire-based system usability scale feedback every three months. TIP software collects user data on use patterns, pathway progression, timelines, and user content.
Outcomes: Preliminary findings with early users [N=2 orgs] show satisfactory usability at three months [SUS avg 71.63]. Users report TIP clarifies the implementation pathway, improves understanding of EBI core components, and increases confidence in identifying implementation barriers. They show moderate readiness for TIP use [ORIC avg 49.7]. Nine additional user organizations in recruitment will contribute to the findings presented, which will be contextualized by a software demonstration.
Next Steps: We are interested in learning if organizations find TIP feasible and usable and how it shapes their implementation. Subsequent revisions and global access will enable a first-hand view of how service-based community organizations implement outside research studies, which could inform improved implementation.
Oops! Are Academic Programs Preparing an Implementation-Capable Workforce?
Authors
Rosalyn Bertram - University of Missouri-Kansas City, School of Education, Social Work and Psychological Sciences
Workforce preparation by academic professional degree programs does not address many factors that contribute to the research-practice gap. Developed by the Child and Family Evidence-Based Practice Consortium to inspire future research and development that helps bridge the research-practice gap, this SLAM presentation playfully and thoughtfully invites the audience to consider these overlooked factors.
Since its 2004 inception, Consortium participants have produced studies, publications, and presentations addressing implementation supports for evidence-based practice (Barwick, 2011; Bertram, Blase, & Fixsen 2015; Bertram, Choi, & Elsen, 2018; Bertram & Kerns, 2019; Kerns, et al., 2023; Rolls-Reutz, et al., 2024). Consortium studies of Social Work and Marriage and Family Therapy academic curricula (Bertram, Charnin, et al., 2015; Marlowe, et al., 2020); identified the small number of practices taught, and that primary constraints to this limited instruction included faculty beliefs and knowledge, as well as requirements for academic program accreditation and professional licensing.
As major funding sources increasingly require effective implementation of evidence-based practice, Consortium participants from behavioral health, child welfare, education, and juvenile justice (https://ebpconsortium.com/history/leadership-group/) are increasingly concerned that implementation is not well-considered nor taught by academic professional degree programs This SLAM engages SIRC conference participants to consider that:
Academic curricula seldom present evidence-based practice as a process informed by implementation science and frameworks.
Students don't learn that program values must be operationalized in clearly described elements and activities that shape desired behavior and diminish factors contributing to behaviors of concern.
Therefore, students don't learn that clarity of program and theory of change is the basis for aligning implementation drivers for effective service delivery with fidelity.
They don't learn training is insufficient without a coaching plan, nor that plan-do-study-act feedback loops are essential for effective implementation.
If academic programs are not producing an implementation-capable workforce, we must engage them in curriculum development.
References
Barwick, M. (2011). Master's level clinician competencies in child and youth behavioral healthcare. Emotional & Behavioral Disorders in Youth, 11 (2), 29-58.
Bertram, R.M., Blase, K.A., & Fixsen, D.L. (2015) Improving programs and outcomes: Implementation frameworks and organization change. Research on Social Work Practice, 25(4), 477-487.
Bertram, R.M., Charnin, L.A., Kerns, S.E., & Long, A.J. (2015). Evidence-based practices in North American MSW curricula. Research on Social Work Practice, 25(6), 737-748.
Bertram, R. M., Choi, S. W., & Elsen, M. (2018). Integrating implementation science and evidence-based practice into academic and field curricula. Journal of Social Work Education, 54(sup1), S20-S30.
Bertram, R.M. & Kerns, S. E. (2019). Selecting and implementing evidence-based practice:
A practical program guide. Springer, New York, NY.
Kerns, S. E., Mitchell, C., Rolls Reutz, J. A., & Sedivy, J. A. (2023). Documenting the Implementation Gap: Pre-implementation Supports. Global Implementation Research and Applications, 3(2), 85-98.
Marlowe, D., Cannata, E., Bertram, R.M., Kerns, S.E., & Choi, S., (2020). Comparing evidence-based practice in MSW and Marriage and Family Therapy curricula. Journal of Family Social Work, 24 (2),133-150.
Rolls Reutz, J. A., Kerns, S. E., Sedivy, J. A., & Mitchell, C. (2020). Documenting the implementation gap, part 1: Use of fidelity supports in programs indexed in the California evidence-based clearinghouse. Journal of Family Social Work, 23(2), 114-132.
From Conversation to Action: Codesign Strategies to Enhance the Uptake of Caregiver Psychosocial Supports in a Faith-Based Community
Authors
Michaela Harris - The Baker Center for Children and Families/Harvard Medical School
Michelle Alto - The Baker Center for Children and Families/Harvard Medical School
Jessica Fitts - The Baker Center for Children and Families/Harvard Medical School
Shaleah McNeil-Tersilas - Boston Public Schools
Meri Viano - Parent/Professional Advocacy League of Massachusetts
Background: Despite parents and caregivers (P/CGs) being the primary support for children experiencing mental health challenges, there's a shortage of resources for their own capacity-building and mental wellbeing (Making Caring Common, 2023). Focused initiatives should prioritize skills that P/CGs determine as most relevant to support their children. In a faith-based, community-led, peer support program for P/CGs, community-intermediary partnership guided codesign activities and prioritized materials that are most relevant to P/CGs in local settings. This implementation strategy may further enhance psychosocial supports offered to build family capacity and promote community sustainability.
Approach: The “Caregiver Conversations” program grew from focus groups held with members of a faith-based congregation, in which participants reported “(…) seeking a trusted community space to feel heard and supported in their efforts to support families” and highlighted the importance of church communities as trusted spaces for receiving and providing support. The Transcreation Framework (Nápoles & Stewart, 2018) guided the codesign process by identifying partners from the beginning of implementation efforts to form a community-intermediary partnership, delegating content development by lived experience and system of work, enhancing content to fit local context, and building off existing relationships to guide recruitment and outreach efforts.
Outcomes: Contextual fit of training content and knowledge to discuss strategies that enhance child-caregiver relationships and P/CG well-being were endorsed following training. Of 9 trained facilitators, 7 have implemented more than 20 in-person and virtual group sessions within their communities. This structured co-design process allowed for a greater sense of ownership and investment in implementation efforts, a “trickle down” effect of the transcreation process.
Next Steps: Future directions of this work will include assessing the acceptability and feasibility of content, delivery, and implementation supports through focus group discussions. Lessons learned will be used to strengthen ways to partner in codesign with community members.
Nápoles, A. M., & Stewart, A. L. (2018). Transcreation: an implementation science framework for community-engaged behavioral interventions to reduce health disparities. BMC health services research, 18(1), 710. https://doi.org/10.1186/s12913-018-3521-z
“Lesser known” Implementation Strategies: Promoting Change at all Costs
Authors
Aaron Lyon - University of Washington
Background: Multiple well-known compilations of implementation strategies have been articulated to support the uptake of evidence-based practices. These strategies generally seek to enlighten, empower, or support individuals and systems with the goal of improving capacity and impact. This perspective ignores a growing array of “lesser known” (LK) implementation strategies that exist primarily to confuse, cajole, force, trick, or otherwise impel implementers to adopt against their wishes or better judgement.
Approach: This “slam” presentation will (quickly) introduce the concept of LK strategies and review an initial compilation of LK strategies including (but not limited to): (1) public shaming, (2) bait-and-switch, (3) threats to personal or professional safety, (4) bias confirmation, (5) false scarcity, and (6) the long con. Because, as implementation scientists, we know best.
A Multi-System Collaboration and Researcher Partnership to Prevent Cascading Negative Outcomes for Families with Parental Substance Use and Criminal Justice Involvement
Authors
Lisa Saldana - Chestnut Health Systems, Lighthouse Institute Oregon
Kelly Atkinson - Lackawanna County Department of Health and Human Services
Samual Emile Rutherford - Chestnut Health Systems
Maria Bybee - Chestnut Health Systems, Lighthouse Institute Oregon
Introduction: Parental opioid and methamphetamine use disorders (OUD/MUD) pose intergenerational consequences and are the most significant driver for child welfare (CW) reports and child entries into out-of-home care. In 2022, the top five reasons for CW involvement in Lackawanna County, Pennsylvania were parental drug use, inability to cope, child neglect, incarceration, and inadequate housing. The County identified need for a trauma-informed preventive intervention that would address these areas, but also be appropriate for a range of parents including those currently incarcerated or recently released. Informed by the literature, the County contacted the developers of Just Care for Families (Just Care) — a continuum of care, integrated program for parents involved with CW with or at-risk for OUD/MUD, backed by research but not yet ever independently implemented.
Methods: Just Care includes evidence-based preventive intervention for substance use, mental health, parenting, community building, assistance with system navigation and basic needs. Given the cross-system approach, a multi-system leadership team was formed to pull resources and efforts across systems in the county. The system-researcher partnership encouraged infusion of evidence-based implementation strategies, previously only supported through Just Care research. Collaboratively, CW, criminal justice, providers, housing, Medicaid, and the treatment development team completed a well-specified pre-implementation process.
Results: Following a defined implementation roadmap, developed as part of a NIDA trial, key implementation strategies were completed demonstrating real-world feasibility: (1) secure braided funding; (2) select provider agency with dual licensure and motivation to provide integrated community-based care to families; (3) engage in multi-day readiness planning between all systems and treatment development team; (4) identify program champion; (5) hire and train clinical team; (6) commence fidelity monitoring —coaching and consultation. Services launched April 2024.
Conclusions: Implementation strategies shown to be effective in rigorous trials were demonstrated to be replicable and successful for program launch outside of a research trial.
Where to Start?: Building a National Implementation Support Service in Canada Using Empirical and Experiential Knowledge
Authors
Shannon Tracey - Atlas Institute for Veterans and Families
Tara McFadden - Atlas Institute for Veterans and Families
Elizabeth Bose - Atlas Institute for Veterans and Families
Courtney Wright - Atlas Institute for Veterans and Families
Ghislain Girard - Atlas Institute for Veterans and Families
Background: The landscape of mental health support for Veterans and their Families in Canada is complex, with community-based service providers playing a vital role. However, existing barriers often hinder timely access to specialized services for treating PTSD and related mental health conditions. Community mental health service providers lack supports that enable their access and use of evidence-based therapies. Since 2021, the Atlas Institute for Veterans and Families has been developing a national implementation support service to enhance capacity in first-line evidence-based therapies. This effort began with the introduction of a comprehensive training program.
Approach: Empirical and experiential knowledge were embedded in the design and delivery of the training program. A needs assessment was conducted with community mental health service providers across Canada (n = 696) along with a review of international treatment guidelines to support the selection of evidence-based therapies. A reference group of service providers and researchers further scoped the training program, including identification of target audience, recruitment activities, and equity considerations.
Outcomes: Prolonged exposure therapy, cognitive processing therapy, and cognitive-behavioural conjoint therapy were prioritized for training. Expert-led workshops and group consultations were conducted annually. A project team oversaw training activities and conducted post-hoc evaluations to assess lessons learned, strengths, and ongoing barriers. By April 2024, 470 providers completed workshops and 150 are in or have completed group consultation. The primary barriers to therapy use included client-related factors and provider confidence.
Next Steps: The training program provided an initial opportunity for community mental health service providers to receive support in building capacity to care for Veterans and their Families. Establishing data systems was crucial for inclusive decision-making with various stakeholders, guiding the co-design of new strategies for the national support service. These strategies include creating a community of practice for providers and launching a webinar series for Veterans and Families.
Research Track
Vertical alignment of EBP implementation climate between leaders and providers: testing a theorized mechanism for organizational implementation strategies
Authors
Vivian Byeon - University of California Los Angeles
Gregory Aarons - University of California San Diego
Marisa Sklar - University of California San Diego
Mark Ehrhart - University of Central Florida
Anna Lau - University of California Los Angeles
Motivational interviewing (MI) is a leading EBP for substance use disorder (SUD), but its use is low in routine care. Improving an organization's implementation climate may promote MI use, which interventions like the Leadership and Organizational Change for Implementation (LOCI) aim to do. Organization leaders often facilitate implementation climate but may perceive the organization differently from front-line providers. Therefore, improving vertical alignment (VA)—leader-provider agreement on perceived conditions for implementation—may be a critical mechanism through which strong implementation climate influences MI implementation.
Within a cluster-randomized trial of LOCI in SUD clinics, this study examined whether VA in leader-provider perceptions of implementation climate changed over time and its associations with MI implementation outcomes.
380 providers and 67 leaders (clinical supervisors) from 60 SUD clinics participated across 4 timepoints (baseline, 4, 8, 12 months). Dyadic leader-provider VA was measured by calculating profile correlations using leader and provider responses to the Implementation Climate Scale at each timepoint. Multilevel growth models examined whether VA changed over time, was influenced by clinic-level factors, and was associated with MI implementation outcomes.
Preliminary results suggest that VA trajectories of the control group declined over time (b=-.08, 95%CI [-.14, -.03]). However, the slope difference between LOCI and control clinics was marginally significant (b =.06, 95%CI [-0.02, .13]), suggesting that LOCI exposure may have mitigated the negative VA trajectory for those clinics. VA trajectories did not predict MI implementation outcomes, perhaps due to the relatively flat VA trajectories.
This study sought to advance our understanding of a potential mechanism of implementation strategies. While VA was relatively high at baseline (M=.58, SD=.32) and decreased over time, LOCI may be protective against the negative effects of time. More work is needed to understand the conditions in which VA may be important to EBP implementation (e.g., team dynamics, diverse EBP perspectives).
Applying The Entrepreneurial Process to Implementation Research and Practice
Authors
Margaret Crane - The Warren Alpert Medical School of Brown University
Alethea Desrosiers - The Warren Alpert Medical School of Brown University
Daniel Warshay - Brown University
Background: The Entrepreneurial Processes can enhance implementation strategies to bridge the research-practice gap for both intervention and implementation science. Entrepreneurship is defined as “a structured process for solving problems without regard to the resources currently controlled.,”(Warshay, 2022). Similarly, implementation is a structured process for improving the translation of evidence-based practices (EBPs) often in underfunded sectors.
Methods: This Slam presentation describes the See, Solve, Scale Entrepreneurial Process, highlighting mindsets and strategies from entrepreneurship that can improve the implementation process.
Results: During the See stage, entrepreneurs first identify an unmet need using bottom-up research to avoid creating an innovation in search of a problem. Implementation science often starts with the innovation (an EBP), with the assumption that improving care is an unmet need; however, this may not reflect partners’ perceptions of the need. Bottom-up research identifies a need, like how implementation science starts by conducting a needs assessment. During the Solve stage, entrepreneurs iteratively develop a product and create their value proposition. Scarce resources are considered a benefit, encouraging entrepreneurs and implementers to use only the critical components of EBPs or implementation strategies during iterative testing. Value propositions can motivate partners to participate in implementation efforts. Value propositions describe who will benefit from the product, what the product is, how it will benefit the customer, why the team can best deliver the benefits, and how the product is better than competitors. During the Scale stage, entrepreneurs create a sustainable business model to scale the product. Financial sustainability is essential for a startup but is not always prioritized in implementation efforts. Marketing and branding are used to help the business/initiative grow.
Conclusion: The Entrepreneurial Process can enhance the implementation process and encourage implementation teams to use problem solving strategies and mindsets from startups to create more sustainable and scalable implementation efforts.
References
1. Warshay, D. (2022). See, Solve, Scale. St. Martin's Publishing Group.
Influence of external pressures on adoption of evidence-based practices in human service organizations
Authors
Jared Martin - Department of Medicine, University of California San Francisco
Alicia Bunger - College of Medicine, The Ohio State University
Natasha Slesnick - Department of Human Services, The Ohio State University)
Keeley Pratt - Department of Human Services, The Ohio State University)
Background: Human service organizations are under increasing pressure to demonstrate their impact, often with little guidance, while also facing competition, policy/regulatory shifts, stagnate funding, and high service demand. Implementation frameworks, such as the CFIR, identify external pressures as a determinant of evidence-based practice (EBP) adoption; however, we know little about these pressures or how organizations respond to them on the ground (e.g., fully embrace or superficial adopt). This study examined external pressures on broad-based EBP adoption behavior/quality within organizations addressing homelessness/housing issues among youth, a highly marginalized and underserved population with multifaceted needs.
Method: A convergent mixed-method study recruited twelve organizations, across six states, to describe (1) external pressures, informed by CFIR and organizational theory, and (2) adoption behavior/quality using a modified Fidelity, Attitudes, and Influence Typology (FAIT). Data collection included 4 focus groups, 10 interviews, 30 surveys, and document review of 26 grants/contracts. Data integration classified FAIT adoption behavior/quality and specified primary pressures.
Results: When classified, Adopting Active Supporters (n=5; moderate to high fidelity, positive EBP attitudes, influence on peers) and Adopting Passive Supporters (n=1; moderate to high fidelity, positive EBP attitudes, low peer influence) both reported multiple funder pressures with monitoring requirements. Non-Adopting Passive Resisters (n=3; unclear to low fidelity, negative EBP attitudes, low peer influence) and Non-Adopting Passive Supporters (n=1; unclear to low fidelity, positive EBP attitudes, and low peer influence) reported funder and peer influences. Lastly, Non-Adopting Active Supporters (n=2; unclear to low fidelity, positive EBP attitudes, influence on peers) described funder and education influences.
Conclusion: Funders apply significant pressure for EBP adoption; however, half of the sample was classified as different types of non-adopters. Strategies that directly target payers’ knowledge and ability to provide guidance for organizational adherence and appropriate adaptations may have potential to improve adoption quality. Study limitations and implications for equitable implementation will be discussed.
Where are the implementers in implementation theories, models, and frameworks?
Authors
Ruben Martinez - The Warren Alpert Medical School of Brown University
This slam presentation poses a simple but critical question: Where are the implementers in implementation theories, models, and frameworks (TMFs)? In the ever-growing list of over 114 TMFs, considerations of implementers’ identities and personal characteristics are notably absent. Implementers are defined here as any member of an implementation research or practice team that interacts with the implementation context. The purpose of this slam is to stimulate critical and creative thinking around our identities and personal characteristics as implementers—and how these factors may influence implementation efforts. The presenter will weave together personal experiences from implementation research and practice with insights from critical race theory, community-based participatory research, and social and developmental psychology to illustrate: (1) that implementers are inextricable from the implementation context, and (2) ways that identity and personal characteristics may influence implementation efforts.
The field of implementation science has continuously embraced a positivist paradigm, which suggests that implementers are objectively and categorically disconnected from the reality of the work that they do. This is clear in reviewing implementation TMFs, which are the anchors of implementation training, research, and practice. TMFs have largely ignored the existence, and therefore the impact of implementers’ identities and characteristics as critical components in the implementation process. Indeed, the updated Consolidated Framework for Implementation Research is the only framework to our knowledge that acknowledges the existence of implementers, but even in this case, implementers’ roles are described without any explicit theory about how who they are may impact implementation. Discourse is needed to move the field toward a postpositivist paradigm and overcome this oversight that has persisted across years, contexts, and implementation groups.
Developing Fundamental Laws for Implementation Science
Authors
Michael Pullmann - Department of Psychiatry & Behavioral Sciences, University of Washington
Background: Implementation Science is awash in frameworks and models which, in an attempt to be generalizable and ecologically comprehensive, do not generally provide direct causal implications. Implementation Science has fewer concrete theories or grand theories, and has made no attempt to articulate natural Laws that can be used to explain mechanisms and provide a mnemonic guide for causal predictions.
Approach: This presentation will review seven physics laws and principles that have relevant corollaries with Implementation Science. Some examples: Newton's first law of motion (inertia) states that in a frictionless context, an object in motion tends to stay in motion and an object at rest tends to stay at rest. The corresponding first law of implementation is that in a frictionless context, an entity engaged in a type of practice tends to keep doing that practice; an entity not doing a type of practice tends to not initiate. Newton's second law of motion is that acceleration is proportional to force applied. The second law of implementation states that implementation success is proportional to effort impressed and proximity of strategy to outcome. This law is supported by statistical reasoning evident in path modeling. The second law of thermodynamics states that entropy tends to increase over time in a closed system. The law of implementation dynamics is that entropy will occur (implementation efforts will fall apart) in a system without structures, supports, and ongoing efforts focused on sustainability. The laws of physics provide a foundation for useful and predictive mnemonics for Implementation Science.
Poster Presentations
Policy/Practitioner Track
Training a Trusted Peer Workforce: Church Community Members as Parent Support Group Facilitators
Authors
Michelle Alto - The Baker Center for Children and Families/Harvard Medical
Shaleah McNeil - Tersilas- Boston Public Schools
Meri Viano - Parent/Professional Advocacy League of Massachusetts
Jessica Fitts - The Baker Center for Children and Families/Harvard Medical School
Michaela Harris - The Baker Center for Children and Families/Harvard Medical School
Background: Increasing rates of youth mental health challenges coupled with escalating provider turnover and hiring shortages have created overwhelming waitlists and worsened barriers to care, particularly for traditionally underserved communities. Because caregivers have often turned to trusted community settings for support caring for the behavioral and emotional needs of their children, these contexts present a valuable solution to workforce shortages. Although research has shown that psychosocial support services delivered by non-clinicians like community leaders and peers are feasible and effective (Barnett et al., 2018), community leaders have reported struggling with a lack of training and resources to provide this support. Therefore, there is a critical opportunity to train this potential workforce to increase access to psychosocial support for families, particularly within trusted community settings like the church (Coombs et al., 2022).
Approach: Using the Transcreation Framework (Nápoles & Stewart, 2018), the Caregiver Conversations program was developed as a faith-based peer-led support group for caregivers of children and teenagers. Nine members of Boston area churches were trained as group facilitators. Training structure was based on principles of adult learning and emphasized experiential activities, modeled skills, and role plays. Facilitators were given worksheets and structured guides on planning and implementing groups. Post-training, facilitators joined regular consultation calls with trainers to discuss successes and challenges of implementation.
Outcomes: Preliminary feedback suggests that Caregiver Conversations is acceptable, feasible, and perceived to have a positive impact on members of the community. Focus groups will be held with group facilitators and group members to further assess program acceptability and feasibility and inform next steps.
Next steps: Community members can serve as a valuable resource for increasing psychosocial support for caregivers in community settings. Lessons learned will highlight specific challenges, successes, and recommendations for training this workforce and supporting the sustainable implementation of similar peer-led support programs.
References
Barnett, M. L., Gonzalez, A., Miranda, J., Chavira, D. A., & Lau, A. S. (2018). Mobilizing community health workers to address mental health disparities for underserved populations: a systematic review. Administration and Policy in Mental Health and Mental Health Services Research, 45(2), 195-211. https://doi.org/10.1007/s10488-017-0815-0
Coombs, A., Joshua, A., Flowers, M., Wisdom, J., Crayton, L. S. S., Frazier, K., & Hankerson, S. H. (2022). Mental health perspectives among Black Americans receiving services from a church-affiliated mental health clinic. Psychiatric Services, 73(1), 77-82. https://doi.org/10.1176/appi.ps.202000766
Nápoles, A. M., & Stewart, A. L. (2018). Transcreation: an implementation science framework for community-engaged behavioral interventions to reduce health disparities. BMC Health Services Research, 18(1), 710-725. https://doi.org/10.1186/s12913-018-3521-z
Evolution of the Colorado MOUD Expansion Program: From Pilot to Scale-up and Sustainability
Authors
Claudia Amura - University of Colorado, Anschutz
Medical Campus
Rosario Medina - University of Colorado, Anschutz Medical
Jennifer Place - University of Colorado, Anschutz Medical
Aimme Techau - University of Colorado, Anschutz
Medical
Jose Esquibel - University of Colorado, Anschutz
Medical
Sarah Stalder - University of Colorado, Anschutz Medical
Paul Cool - University of Colorado, Anschutz Medical
Background: Opioid use disorders (OUD) persist as a critical issue. Colorado ranks highest in non-medical opioid use, with alarming overdose rates in rural underserved areas,, predominantly Hispanic, and facing additional challenges due to healthcare shortages. In response, the Colorado Senate sanctioned the Medication for OUD (MOUD) pilot program expansion in July 2019 to serve counties facing the highest overdose fatalities. This study highlights program implementation in rural populations, addressing Social Determinants of Health (SDoH) through multidisciplinary approaches, adaptation strategies, and patient-centric service coordination to complex needs.
Methods: The CU College of Nursing and the Center for Prescription Drug Abuse Prevention facilitated program implementation in 49 clinics. . Guided by PRISM/RE-AIM frameworks, a health systems platform was designed to comprehensively capture SDOH, patient-centric data, services, and treatment impact in the complex Hub-and-Spoke model across 22 rural Colorado counties.
Results: Since its inception in 2019, the program served over 5,000 individuals with OUD. Predominantly White (79%) and Hispanic (49%), with a mean age of 26, patients mostly rely on Medicaid or are uninsured (80%), often facing unstable housing and employment. The MOUD program expansion, enhanced by behavioral health services, case management, care coordination, and peer support, significantly contributed to treatment success. Approximately 40% of patients undergoing MOUD treatment remained engaged at three months, despite challenges like incarceration and hospitalization. Along with ongoing case management, care coordination and connection to resources, contingency management protocols implementation enhanced patient retention in treatment.
Conclusions: The MOUD Program Expansion showcases successes and ongoing efforts to enhance recovery and engagement through a network coordinated care for opioid use. The study underscores the importance of addressing SDoH and tailoring strategies to diverse populations to optimize patient outcomes and combat the opioid crisis in rural Colorado, aligning while advancing equity research, practice, and policy.
Conceptualizing the role of program champions in implementation efforts
Authors
Noemie Bechu - The Baker Center for Children and Families/Harvard Medical School
Chelimer Rivera - Pediatric Associates
Background: The identification of a program champion is a well-known implementation strategy, and one that is integral to successful implementation efforts (Powell et al., 2015). Despite its popularity, a gap remains in the literature about the ways in which program champions are valuable to these initiatives (Santos et al., 2022). Research has focused heavily on the characteristics of programs champions, but little is known about the activities they do to drive implementation (Demes et al., 2020). Operationalizing implementation activities is crucial to cultivating program champions in future efforts.
Approach: A case study will illustrate how to operationalize the implementation activities of a program champion. A program champion from a previous mental health implementation project was identified. This program champion led the implementation of an integrated behavioral health pilot program in a primary care setting and has started exploratory work for implementation in different sites. We conducted content analysis of implementation initiative notes from one-on-one calls with this champion and team implementation meetings. We then mapped findings onto core competencies of implementation support practitioners as defined by Metz and colleagues (2021). Reoccurring themes emerged from the content analysis and illustrated concrete ways in which champions lead and support efforts.
Outcomes: Champions embody implementation support practitioners core competencies such as brokering, co-design and developing relationships. Concrete activities that demonstrate the competencies included coordinating meetings between funders and organizational leadership, sending out monthly data reports to highlight progress, and initiating partnerships with external partners to fill a gap in implementation.
Next Steps: Future directions for this work involve identifying additional program champions from past initiatives, mapping their involvement with Metz et al.'s core competencies, and comparing to other case studies of champions. Creating a repertoire of skills can help better prepare and guide program champions throughout implementation.
The National Race Equity Implementation Center (NREIC)
Authors
Matthew Billings - National Race Equity Implementation Center (NREIC)
Background: Conceived, convened, and incubated by the Children and Youth Cabinet (CYC Rhode Island) the National Race Equity Implementation Center (NREIC) is an intermediary that is reimagining implementation science by installing race, equity, and culture as the primary driver in its theory of change. NREIC develops, disseminates and finances a customized portfolio of evidence-based programs, policies, practices, and strategies specifically designed to improve the quality of evidence-based implementation by prioritizing participant engagement and content relevance. NREIC knows that those closest to the problem are closest to the solution, but they are often the ones farthest from resources and power. What we do is interrupt the traditional flow of investment to historical expertise and power, and redirect it exclusively to resident priorities, outcomes, and strategies.
Approach: NREIC is a nine member intermediary (501(c)(3)) led by Tri-Chairs Lisa Gary PhD (epidemiologist and public health researcher); Cindy Myers, LCSW (clinical outpatient/social expertise with populations to include: Native Americans, Hispanics, Immigrants); and Matthew Billings, (Deputy Director, cofounder Children and Youth Cabinet (CYC) and founder NREIC). NREIC members include: Tracy Anderson, PhD (University of Georgia), Leon Aragon, LCSW (Kil Tani Consulting), Margaret Flynn-Khan, MSW (Mainspring Consulting); Antonio Polo, PhD (DePaul University); Laurie Tochiki, JD, PhD (Executive Director of EPIC ʻOhana); and Maya Boustani, PhD (Loma Linda University).The poster will deliver a detailed visualization/overview of NREIC (mission, vision, purpose, and practice), compendium of published tools (implementation drivers, content relevance/participant engagement measure and protocols, coaching modules, race/equity fidelity tool, and technical assistance framework). The NREIC technical assistance framework will be highlighted as a call out visualization. This TA framework is how NREIC engages with implementation teams from communities across the country to build meaningful, measurable, pragmatic roadmaps that address a variety of systemic inequities that lead to widespread disparities.
Developing guidance to promote language inclusivity and multilingual research in qualitative translational health research
Authors
Julia Reedy - Adult and Child Center for Outcomes Research and Delivery Science (ACCORDS)
Andrea Jimenez-Zambrano - Adult and Child Center for Outcomes Research and Delivery Science (ACCORDS)
Sandra Garcia-Hernandez - Adult and Child Center for Outcomes Research and Delivery Science (ACCORDS)
Ashley Dafoe - Adult and Child Center for Outcomes Research and Delivery Science (ACCORDS)
Carly Ritger - Adult and Child Center for Outcomes Research and Delivery Science (ACCORDS)
Chloe Glaros - Adult and Child Center for Outcomes Research and Delivery Science (ACCORDS)
Brooke Dorsey Holliman - Adult and Child Center for Outcomes Research and Delivery Science (ACCORDS); Department of Family Medicine; University of Colorado Anschutz Medical Campus
Sarah Brewer - Adult and Child Center for Outcomes Research and Delivery Science (ACCORDS)
Background: Within the US, ∼350 different languages are spoken in homes and ∼9% of residents have limited English proficiency. However, individuals with a language preference other than English (LOE) are often excluded from research. This exclusion of LOE participants may widen health disparities. Our work aims to create pragmatic guidance for multilingual and language inclusive qualitative research that is tailored to the context of the Anschutz Medical Campus.
Setting/Population: A workgroup of monolingual and bilingual researchers with relevant experience and training developed pragmatic guidance for multilingual and language inclusive qualitative research.
Methods:We first conducted a literature review to explore barriers, possible solutions, and methodological guidance around the conduct of multilingual and language inclusive qualitative research. Next, we identified context-specific barriers, needs, and solutions to promote language inclusivity. Drawing on the team's experiences, we collaboratively developed guidance documents to support investigators seeking to conduct multilingual and language inclusive research. Drafted guidance documents were piloted with investigators and refined.
Results: Literature and experience concurred that lack of funds and resources are often used as justification to exclude LOE participants. Several key principles emerged from our process: (1) Language inclusivity should be considered during initial study design to ensure adequate resources; (2) Guidance should address the importance of language inclusivity and outline approaches for designing rigorous multilingual research across research stages that align with available resources; (3) It is important to account for variability across studies. Our guidance can be tailored to fit the design, complexity, sample, and resources of myriad projects.
Conclusion: Current practices of LOE exclusion and barriers to more inclusive approaches are prevalent across the United States. Our guidance will improve awareness and knowledge of rigorous approaches to multilingual and language inclusive research and act as a step towards equitable language inclusion in translational health research.
An EPIS-informed conceptual framework for scaling EBPs for children in New York State
Authors
Andrew Cleek - McSilver Institute
Meg Baier - McSilver Institute
Ashley Fuss - McSilver Institute
Rebecca Lengnick-Hall - Washington University in St. Louis
Sarah Horowitz - New York University
Kimberly E. Hoagwood - New York University
Meredith Ray-La-Batt - New York State Office of Mental Health
Shannon Fortran - New York State Office of Mental Health
Myla Harrison - New York State Department of Health
Matthew Perkins - New York State Office of Mental Health
Sarah Kuriakose - New York State Office of Mental Health
Background: In June 2023, New York State partnered with NYU's McSilver Institute for Poverty Policy and Research and NYU's Department of Child Psychiatry to launch a new initiative to scale EBPs for children, youth and families. This initiative offers technical support and expertise for scaling EBP training to providers serving youth and families. The overall goal is to support a continuum of evidence-based services, from assessments to therapies, using state of the art trainings, learning communities, and financial supports, including enhanced rates and CEUs. When fully implemented, the initiative will manage yearly cohorts of EBP trainings, each serving multiple programs throughout the state. Additionally, it will manage ongoing implementation from the previous year's cohorts to improve sustainability.
Approach: We will present a conceptual model for this initiative co-created by implementation researchers, practitioners, and state policy-makers. This model applies the EPIS framework to: (1) illustrate how technical assistance provides a bridging function between NYS and state-supported or licensed programs and agencies, (2) identify key inner and outer context features and their relevance to specific implementation phases, (3) identify multilevel determinants that could be the target of future implementation strategies, (4) identify innovation factors that are common to the diverse set of EBPs offered.
Outcomes: This conceptual model documents measurable and testable multilevel EPIS constructs, positions agencies to select and tailor implementation strategies, and lays the groundwork for assessing determinant-strategy-outcome linkages. It also meaningfully advances the EPIS construct of bridging factors by articulating key forms and functions of this initiative during early and active implementation phases. To make the framework concrete, we will discuss practical insights, challenges and solutions.
Next steps: We will use this conceptual model to guide measurement of key implementation processes and outcomes, document lessons learned, and communicate with a scientific advisory board and policy partners.
Building Capacity to Reduce Military Medical Treatment Facility (MTF) Plan All-Cause Readmissions
Authors
Angela Ilmanen - United States - Defense Health Agency
Ginnean Quisenberry - United States - Defense Health Agency
Faye Curran - United States - Defense Health Agency
Inbal Eshel - United States - Defense Health Agency
Richard Caldwell - United States - Defense Health Agency
Bridget Erickson - United States - Defense Health Agency
Lynn Hallard - United States - Defense Health Agency
Background: Prevention of Plan All-Cause Readmissions (PCR) is a Defense Health Agency (DHA) priority. Ineffective and poorly coordinated care transitions, discharge planning, and follow-up care often result in avoidable readmissions. DHA's PCR initiative incorporates Agency for Healthcare Research and Quality (AHRQ) Re-Engineered Discharge (RED) Toolkit interventions.
Approach: The DHA Medical Affairs Medical Management Policy Section partnered with DHA's Research & Engineering Implementation Science Branch (ISB) to develop an implementation infrastructure for PCR interventions including: (1) Post-Discharge Follow-up Phone Call; (2) Discharge Medication Reconciliation; (3) Multidisciplinary Discharge Rounding; and (4) Post-Discharge Follow-Up Appointing. Stakeholder informational interviews uncovered PCR barriers and facilitators which were addressed through project management, dissemination, implementation, and assessment activities: (1) Project Management: conducted informational interviews with stakeholders at multiple facilities, (2) Dissemination: created materials cultivating leadership buy-in, and enhanced provider and patient/caregiver PCR awareness, (3) Implementation: developed implementation resources, including detailed workflows across interventions, and (4) Assessment: developed a logic model and comprehensive metrics table, supporting an organized and systematic approach to measures prioritization.
Outcomes: Detailed process maps reflected intersections and gaps between AHRQ RED interventions and DHA policy, supporting a targeted implementation effort. At an organizational level, implementation materials were aligned by clarifying required electronic health record workflows. Efforts to improve DHA compliance with the four selected AHRQ RED interventions, benchmarked against AHRQ's PCR HEDIS metrics, are expected to reduce unnecessary hospital readmissions by improving and standardizing DHA discharge processes
Next Steps: Current enhancements are underway within the new electronic health record system to improve and standardize inpatient to outpatient transitions. Future steps include establishing a consistent feedback loop between PCR Champions and DHA. Plans are in place to improve scale-up efforts by providing standardized and comprehensive resources to healthcare personnel who implement PCR interventions and establishing a unified data plan which provides consistent data tracking of compliance targets.
Translating Ethnographic Insights into a Multilevel & Multicomponent Intervention: Implementing the RITE-Size Intervention to Reduce Unnecessary Testing Before Low-Risk Surgeries
Authors
Dana Greene Jr. - University of Michigan, Ann Arbor
Anthony Cuttitta - University of Michigan, Ann Arbor
Valerie Gavrila - University of Michigan, Ann Arbor
Shawna Smith - University of Michigan, Ann Arbor
Jana Stewart - University of Michigan, Ann Arbor
Michael Mathis - University of Michigan, Ann Arbor
Anthony Edelman - University of Michigan, Ann Arbor
Eve Kerr - University of Michigan, Ann Arbor
Mousumi Banerjee - University of Michigan, Ann Arbor
Hari Nathan - University of Michigan, Ann Arbor
Lesly Dossett - University of Michigan, Ann Arbor
Background: Multiple national societies recommend against routine preoperative testing before low-risk surgery. Despite these recommendations, the prevalence of inappropriate, unnecessary testing remains high. Effective, generalizable de-implementation strategies have not been established. A large academic institution, which was noted to be a high utilizer of these tests, conducted a pilot study to understand and address this problem.
Approach: A three-step process was pursued to identify barriers and specify scalable de-implementation strategies. First, an ethnographic field study was conducted to identify determinants (from the Tailored Implementation of Chronic Disease [TICD] framework) associated with preoperative testing before low-risk surgery. Second, these determinants informed an implementation mapping process to select, package, and deploy a multicomponent de-implementation intervention to reduce low-value testing in the local context. Third, learnings and results of the local pilot were used to develop strategies for the multilevel, statewide RITE-Size (Right-Sizing Testing Before Elective Surgery) intervention.
Outcomes: The ethnographic study identified key determinants aligning with three themes: 1) shared values of patient safety and utilizing evidence-based medicine (TICD Social, Political, and Legal Factors), 2) gaps in knowledge (TICD Individual Health Professional Factors, Guideline Factors), and 3) communication breakdown (TICD Professional Interactions, Capacity for Organizational Change). Implementation mapping informed an intervention that combined 4 strategies: provider education, developing stakeholder interrelationships, testing decision aids to train stakeholders, and EMR prompt updates. The local pilot study reduced overall testing rates by 24%, and significantly decreased the proportion of inappropriate tests from 36.8% (N=96) in the baseline period to 14.0% (N=52, p<0.001 on chi square test) after the 6-month intervention period.
Next Steps: The RITE-Size intervention package is currently being deployed in three hospitals to test its feasibility, refine approaches, and scale up with additional sites. Effectiveness of the final implementation intervention package for decreasing low-value testing will be evaluated in a 16-hospital implementation trial.
Implementing EBPs Across the Treatment Team: Integrating Bachelor-Level Providers
Authors
Jessica Fitts - The Baker Center for Children and Families/Harvard Medical School
Noemie Bechu - The Baker Center for Children and Families/Harvard Medical School
Michelle Alto - The Baker Center for Children and Families/Harvard Medical School
Rachel Kim - The Baker Center for Children and Families/Harvard Medical School
Background: Evidence-based practice (EBP) rollouts in children's mental health have largely focused on specialist mental health providers at the master's level or above. In the face of significant workforce shortages, bachelor-level staff play an increasingly important role in community mental health treatment teams. Providing training and guidance on how bachelor-level providers can support the clinical implementation of EBPs has the potential to enable more cohesive, coordinated, and higher-dosage care while maintaining fidelity.
Approach: In three implementation initiatives of a modular CBT intervention for youth (MATCH-ADTC), contextual assessment of the inner settings indicated that bachelor-level providers play an important role on treatment teams and often interact more frequently with families, and in more naturalistic settings, than clinical team leads. Across thesites, 131 bachelor-level providers received training to coach families in skills from the EBP after they are taught to families by the primary clinician. Using the Framework for Reporting Adaptations and Modifications-Expanded (FRAME), modifications to the content, context, and training will be discussed.
Outcomes: Post-training evaluation findings suggest that bachelor providers found the training content to be very useful and appropriate to their roles and needs. Qualitatively, some agencies identified challenges in coordinating care across the treatment team. Successful implementation of an EBP across a treatment team requires a careful assessment of inner setting (e.g., workflow and task division; case coordination and consultation processes) and individual adopter characteristics (e.g., level of training, role definition) to inform modifications to training and implementation support.
Next steps: Implementing EBPs cohesively across a treatment team with different types of mental health providers has the potential to improve quality and dosage of care. Guidance from treatment developers and intermediary organizations, informed by contextual assessment of the implementation context, is needed to support agencies in making thoughtful modifications to EBPs initially targeted only at lead clinicians.
Implementation Science at the Wake Forest University School of Medicine (WFUSM): Then, Now, and 2030
Authors
Kristie Foley - Wake Forest University School of Medicine
Gary Rosenthal - Wake Forest University School of Medicine
Kevin High - Wake Forest University School of Medicine
David Miller - Wake Forest University School of Medicine
Justin Moore - Wake Forest University School of Medicine
Background: The WFUSM Department of Implementation Science (IS) was founded in 2017 as the first US academic department of its kind. The mission is to accelerate the science of implementing and disseminating evidence-based practices to promote effectiveness, efficiency, and equity. Strategies that facilitated growth of the department can assist other academic learning healthcare systems (aLHS).
Approach: IS was 1 of 5 translational catalysts to drive WFUSM research in support of our aLHS with an initial investment of $5M. In 2019, as part of a strategic combination with Atrium Health and a research-focused strategic plan, $8M was infused into the WFUSM to expand IS expertise. In 2022 Atrium and Advocate/Aurora combined to form the 3rd largest not-for profit health system with WFUSM as its academic core; IS plays a key role in amplifying the translation of research into practice. Growth focused on methodological and theoretical expertise, alignment with health system priorities, partnership with the Clinical and Translational Science Institute, integration of clinician implementers into the department, embedded IS expertise in other departments, and a focus on innovation and health equity IS research.
Outcomes: In four years, the department has grown from 3 to 17 primary faculty members. There are also 18 secondary faculty members, and > 30 administrators, staff members, fellows, and interns. Between 2019-2023, there was a 293% increase in grants submitted by department faculty and a 103% growth in extramural funding. The vanguard department hosts 1 of 7 NCI-funded Centers in Implementation Science & Cancer Control and a PCORI Health Systems Implementation Initiative capacity building grant. Department faculty provide mentorship on over 10 current career development awards.
Next Steps: We must balance the need for departmental cohesion and our commitment to rigorous, robust, scalable science in partnership with a complex, multi-state aLHS that desires speed and efficiency in translation.
Co-Design in VA Rural Healthcare: Lessons from the Rural Native Veteran Healthcare Navigator Program
Authors
Jennifer Freytag - Department of Veterans Affairs, Salt Lake City Rural Health Resource Center
Patricia Valverde - University of Colorado, Anschutz Medical Campus
Betsy Risendal - University of Colorado, Anschutz Medical Campus
James Shore - Department of Veterans Affairs
Background: Native American Veterans serve in the military at one of the highest rates of all racial and ethnic groups. The most rural of Veteran groups, their significant disparities are aggravated by barriers to accessing care and fragmented care as users of multiple systems, including the Veterans Health Administration, Indian Health Service, and Tribal Health systems.
One way to improve care for Native American Veterans is to implement a system of patient navigation. In this program, navigators work with Native American Veterans to coordinate care among staff of multiple healthcare systems, cultivate positive community relationships, and develop Veteran trust. Because of the complexity of implementing this multi-faceted program, we used a co-design approach, working with three VA hospitals to create a new staff position for rural Native Veteran healthcare navigators. Using this approach, we created a “menu-based” implementation plan drawn from three important design principles.
Approach: Implementation facilitators from a core team from the Office of Rural Health's Salt Lake City Resource Center worked with sites. The team relied on these principles: (1) Partner sites own the program and manage their implementation process with support from facilitators. (2) Facilitators cultivate an environment of design thinking, fostering collaborative relationships, identifying opportunities for redesign, and emphasizing the value of multiple perspectives. 3) Core program elements (“menu items”), such as navigator workflow and quality improvement measures, have maximum flexibility to adapt the program locally.
Outcomes: Using co-design principles, the program team has worked with partner sites to develop navigator programs shaped by critical local issues. These have included the availability of dedicated rural and Tribal outreach staff, sites’ unique HR and staffing concerns, cooperation between rural IHS and VA clinics, and regional geographic factors.
Next Steps: The program team continues to refine the list of “menu items” as we prepare for wider dissemination.
Co-design Strategies for Scaling Modular Evidence Based Practices for High Acuity Youth
Authors
Kyrill Gurtovenko - University of Washington
Paul Davis - Washington State Health Care Authority
Winslow Lewis - CoLab for Community and Behavioral Health Policy, Department of Psychiatry and Behavioral Sciences, University of Washington
Sherry Wylie - Washington State Health Care Authority
Kari Samuel - Washington State Health Care Authority
Sonya Wohletz - Washington State Health Care Authority
Noah Gubner - University of Washington
Sarah Walker - University of Washington
Background: The Washington state (WA) Healthcare Authority (HCA) recently began an initiative for systems reform implementing Mobile Response and Stabilization Services (MRSS) for youth, intended to reduce emergency services utilization and other costly intensive services. As part of this initiative, CoLab for Community and Behavioral Health Policy are collaborating with community and systems partners to co-design and evaluate a clinical training model based on Dialectical Behavior Therapy and other evidence based practices. We describe our integration of practice, policy, and research to increase access to quality care for youth. We discuss lessons learned, outcomes, next steps, and how findings may inform similar efforts.
Approach: Engagement strategies included meetings with key systems leaders, including administrators and supervisors of service agencies and the HCA. Co-design followed a model recognized by implementation scientists (Walker et al., 2023) and included needs assessment and partner feedback. MRSS staff participated in live polling and discussions of strengths and challenges. Staff completed the Intervention Usability Scale (IUS; Lyon et al., 2021) and Feasibility of Intervention Measure (FIM; Weiner et al., 2017) post-training.
Outcomes: Co-design led to development of a consultation training model that achieved successful buy-in from service agencies. Needs assessment revealed higher priorities in training vs. consultation, and higher needs for stabilization vs. crisis response phase. Training topics received high interest ratings (M = 8.00, scale of 0 = “Not very interested” 10 = “Very interested”). We began delivering pilot trainings, which participants rate as useful (M = 82.16, SD = 10.80) and feasible (M = 4.36, SD = 0.28). Lessons learned include differences between leadership and staff priorities, identifying variables that promote buy-in, and utility of mixed methods assessment and engagement strategies.
Next Steps: Next stage includes completing pilot delivery of training, reviewing feedback/evaluation data, refining for second phase of delivery, and developing an asynchronous adaptation.
References
Lyon, A. R., Pullmann, M. D., Jacobson, J., Osterhage, K., Al Achkar, M., Renn, B. N., Munson, S. A., & Areán, P. A. (2021). Assessing the usability of complex psychosocial interventions: The Intervention Usability Scale. Implementation Research and Practice, 2. doi: 10.1177/2633489520987828
Walker, S.C., Baquero, B., Bekemeier, B., Parnes, M., & Arora, K. (2023). Strategies for enacting health policy codesign: A scoping review and direction for research. Implementation Science, 18, 44. Doi: 10.1186/s13012-023-01295-y
Weiner, B. J., Lewis, C. C., Stanick, C., Powell, B. J., Dorsey, C. N., Clary, A. S., … & Halko, H. (2017). Psychometric assessment of three newly developed implementation outcome measures. Implementation Science, 12(1), 108. doi: 10.1186/s13012-017-0635-3
Achieving Program Success: The Crucial Role of Stakeholder Involvement, Communication, and Collaboration
Authors
Jess Indresano - US Department of Veterans Affairs, Puget Sound Health Care System
Melissa Echevarria Baez - US Department of Veterans Affairs, VA Central Office
Carmen Ripley - US Department of Veterans Affairs, Puget Sound Health Care System
Jessica Chen - US Department of Veterans Affairs, Puget Sound Health Care System
Background: The Comprehensive Addiction and Recovery Act (2016) mandated interdisciplinary Pain Management Teams (PMTs) at all Veterans Health Administration (VHA) facilities. Only 40% of VHA Medical Centers complied by 2022. A 2019 pilot study of hub-and-spoke telehealth specialty pain care demonstrated that implementing telehealth was associated with a 5% increase in reach of pain services, solidifying telehealth as a viable option for ensuring all Veterans have access to mandated PMTs.
Approach: Clinical Resource Hub TelePain Teams are interdisciplinary PMTs utilizing a hub-and-spoke telehealth delivery model. External facilitation was the primary implementation strategy utilized to support 13 sites throughout the United States from 2021-2024. The primary goal was to help sites move from pre-implementation to implementation (defined as launching clinical services) and then from implementation to sustainment (defined as delivering standardized, evidence-based services) over the course of 3 years. A Facilitator and Project Coordinator met with each site bi-/monthly. The Senior Facilitator, Project Coordinator, and Program Office Champion would meet biweekly to develop action plans addressing identified barriers.
Outcomes: Within the first year, 54.5% of sites entered implementation. By the end of year 2, 91.7% of sites entered implementation, serving 3,338 unique Veterans over 20,219 appointments. To help programs move towards sustainment, four large-scale adaptations were identified as needed across multiple sites: 1) Referral pathway for Veterans with co-occurring suicide risk, 2) Protocol for virtual pain exams, 3) Coding mechanism to track interdisciplinary appointments, and 4) Community of Practice for support and collaboration with staff and Subject Matter Experts.
Next Steps: Dynamic, bidirectional communication between the program office sponsor and frontline staff via the Facilitation Team allowed for a global perspective on implementation barriers and streamlined impact of standardization efforts. Including a liaison from the project sponsor's office on the Facilitation Leadership Team was essential to expediting communication on standardization needs.
Facilitators and Barriers to Implementation of a Statewide Early Childhood Curriculum
Authors
Navneet Kaur - Virginia Commonwealth University
Alisandra Macias - Virginia Commonwealth University
Erica Ross - Virginia Commonwealth University
Rosalie Corona - Virginia Commonwealth University
Meghan Reichel - Virginia Commonwealth University
Michael Broda - Virginia Commonwealth University
Bryce McLeod - Department of Psychology, Virginia Commonwealth University
Kevin Sutherland - School of Education, Virginia Commonwealth University
Background: In response to a state policy requiring early childhood settings to use evidence-based school readiness programs, this study examined factors promoting and hindering the implementation of one statewide program: STREAMin3. STREAMin3 is an infant-through-five curriculum designed to promote school readiness skills (academic, behavioral) by helping teachers set up the classroom environment, establish classroom routines, and incorporate intentional teaching practices.
Methods: The sample included 60 early childhood teachers in low-resource centers who received one-on-one coaching to implement STREAMin3. Teachers’ perceptions of STREAMin3 were measured through the Acceptability of Intervention Measure, Intervention Appropriateness Measure, and Feasibility of Intervention Measure at five-time points throughout an academic year. Using purposeful sampling, semi-structured interviews informed by the Consolidated Framework for Implementation Research were conducted with 18 teachers who differed in their perception of STREAMin3 over the academic year. A six-step thematic analysis extracted factors promoting and hindering curriculum implementation.
Results: Four teachers perceived STREAMin3 as acceptable, appropriate, and feasible (AAF) throughout the year, three continued to report low AAF scores, six improved in their AAF, four decreased in their AAF, and one did not fit those groups. Teachers identified coaching as an implementation facilitator, helping them deliver the program in their classrooms and increasing their self-efficacy. For example, one teacher noted that "She's… ready to help you research and find materials for you that you can use to get you going," while another said, "They reassure you… she said, 'there's no right way, there's no wrong way.' So…that helped me a whole lot." Teachers identified barriers to implementation as insufficient training, limited accessibility to coaching, and mixed-age groups in the classroom.
Conclusion: This study indicates that low-resource centers may have difficulties meeting policy requirements without additional implementation supports, such as sufficient training and coaching that helps address classroom challenges.
Scaling a Cross-Program Case Management Model Statewide in Colorado
Authors
Jonathan McCay - Mathematica
Meghan Dawkins - Colorado Department of Human Services
In 2021, CDHS developed a vision for a consistent, high-quality, and equitable approach to case management across all Division of Economic and Workforce Supports (DEWS) programs statewide. As a county-administered state, each of Colorado's 64 counties has considerable discretion in the design and implementation of these programs. CDHS recognized that implementing a consistent practice statewide, across multiple programs, would be a challenge. CDHS has defined the core components of its case management model for DEWS programs and built a Case Management Training (CMT) for all staff. Recognizing the critical role leaders play in implementation, CDHS also developed a Case Management Training for Leaders (CMT-L). The training centers on equipping local leaders to be visionary and strategic, effective change managers, and developmental supervisors, grounded in emotional intelligence and cultivating psychological safety among their teams. This training is designed to equip local leaders to play an active role in the implementation and sustainability of the case management practice.
Recognizing that training alone is insufficient to bring about change, CDHS developed the County Planning Guide – a 6-step process for counties to plan integration of the case management model. It embeds implementation science and practice strategies in accessible ways for practitioners, anchored by the Consolidated Framework for Implementation Research (CFIR) and an explicit focus on mapping implementation barriers and facilitators at the local level in order to create customized implementation plans (integration strategies).
This session will provide a introduction to the DEWS case management model and training curricula, and share an overview of the implementation approach at the state and county levels. The presenters will describe how and why the County Planning Guide was developed to facilitate local adoption and implementation, and how this approach incorporates evidence from implementation science. The presenters will discuss implementation strategies being used, challenges encountered, and how the team has pivoted.
Evaluating the Implementation of Clinical Pharmacist Practitioners to Deliver Medications for Opioid Use Disorder at the Veterans Health Administration
Authors
Elle Pope - VA Boston Healthcare System
Zenith Rai - VA Bedford Healthcare System
Sarah McDannold - VA Bedford Healthcare System
Bo Kim - VA Boston Healthcare System
Tony Pomales - Iowa City VA Medical Center
Megan McCullough - VA Bedford Healthcare System
Background: Clinical pharmacist practitioners (CPPs), with extensive knowledge on pharmacologic treatment using medications for opioid use disorder (MOUD) and harm reduction strategies, have the potential to fill workforce shortages in opioid use disorder (OUD) care. This evaluation examined the implementation of CPPs as rural MOUD treatment providers in the Veterans Health Administration (VHA). We focus on the processes and activities through which CPPs provide rural Veteran OUD care in interprofessional care teams.
Methods: Semi-structured interviews were conducted with CPPs (n=15) and associated interprofessional clinical team members (n=44) in the evaluation of the Clinical Pharmacy Specialist Rural Veterans Access in Substance Use Disorder (CRVA-SUD). The goal was to understand the impact of implementing CPPs in delivering anti-stigmatizing and harm reduction approaches to MOUD. Interviews were recorded, transcribed verbatim, and entered in Excel for coding and analysis. A codebook was iteratively developed using interview debriefs and reviews of transcripts. A thematic analysis identified themes related to medication-assisted treatment, harm reduction, stigma, and the formal and informal ways CPPs collaborated with and mentored members of their interprofessional teams.
Results: Two main themes emerged from the interviews that demonstrate how embedding CPPs in rural Veteran OUD care aids interprofessional team members’ approach to care and improves care for patients with MOUD: (1) mentoring interprofessional team members on mitigating stigma and promoting harm reduction (e.g., introducing de-stigmatizing terminology or re-framing patients who relapse as patients who are in a long-term struggle with addiction); and (2) improving access to care for patients with OUD (e.g., by facilitating immediate medication access, providing comprehensive medication management, and in coordinating care with other health professionals).
Conclusion: Integrating CPPs into rural Veteran MOUD care has the potential to lead to better outcomes for patient with OUD. This study found that CPPs can deliver anti-stigmatizing, harm reduction care to Veterans and provide informal leadership and education on current OUD best practices to interprofessional care team members.
Using Technology to Connect Virginia Communities to Effective Behavioral Health Services
Authors
Ashley Robinson - Virginia Commonwealth University
Brianna Boggs - Virginia Commonwealth University
Teddy Gray - Virginia Commonwealth University
Anastasia Perrius - Virginia Commonwealth University
Rafaella Sale - Virginia Commonwealth University
Michael Southam-Gerow - Virginia Commonwealth University
Background: Improving access to evidence-based programs (EBPs) in behavioral health faces a number of barriers in public settings. One significant barrier is the ability to find local, trained providers who practice the EBPs to fidelity. Accordingly, key components of Virginia's implementation of the Family First Prevention Services Act (“Family First”) included training new practitioners in an array of EBPs, verifying and keeping records of the statewide workforce (those already trained, and those who would be trained using Family First funds), and ongoing fidelity monitoring of each program. Subsequently, the Center for Evidence-based Partnerships, together with state agency partners, was tasked with developing a tool to track this workforce at scale.
Approach: The EBP Registry is an authoritative source for all individuals trained in the EBPs listed in the state's Family First plan: Brief Strategic Family Therapy, Family Check-Up, Functional Family Therapy, High Fidelity Wraparound, Multisystemic Therapy, and Parent-Child Interaction Therapy. The credentialing information of more than 500 practitioners across nearly 50 office locations is tracked and verified by CEP-Va on a quarterly basis, using data from purveyors, the state licensing entity, and provider companies.
Outcomes: In developing the EBP Registry, stakeholders regularly surfaced the need for a publicly accessible way to find providers that wouldn't exacerbate existing workforce retention difficulties, i.e. display personal information to be searched for and used by other companies recruiting practitioners trained in specific EBPs. The EBP Finder was then designed and developed to connect referral sources to team-level information about service availability in their areas.
Next Steps: Ongoing use of this tool, launched in late 2023, will be enhanced by additional EBPs and other features (e.g., funding sources available, fidelity metrics). Lessons learned, next steps, and statewide metrics will be discussed.
Implementation of Social and Emotional Learning in Rural Colorado Schools
Authors
Natalie Rodriguez-Quintana - TRAILS
Kristin Klopfenstein - Colorado Evaluation and Action Lab, University of Denver
Michelle Rozenman - University of Denver
Sarah Jordan - Molina Healthcare of Illinois
Background: Teaching social and emotional learning (SEL) skills in schools improves youth social, emotional, and academic outcomes. While there is some literature on SEL implementation, the majority is focused on a single classroom or school, which limits implications for district-wide implementation. Given that additional challenges are introduced when multi-level system implementation support is needed, wide-scale SEL implementation guidance is warranted.
Approach: Four takeaways will be provided from the first year of SEL implementation across three rural school districts in Colorado receiving external implementation support from a local policy lab. The districts formed implementation teams that worked to understand and address implementation challenges during the first year of the project and identified key lessons learned to support the sustainment of SEL in future years. The takeaways were informed by insights from the district implementation teams, interviews with leadership at each site, and results from a survey distributed to teachers in the first year.
Outcomes: Key adaptive challenges arose during the first year of district-wide SEL implementation at all three districts. The four main considerations to address challenges included: 1) district leadership needs to focus on supporting system alignment, 2) building leadership should use professional development opportunities to respond to challenges with SEL implementation, 3) teachers need protected time to plan and implement SEL, and 4) district and building leadership should create and execute plans for family engagement.
Next steps: While technical challenges in the initial learning and implementation of SEL are crucial, adaptive challenges also arise when implementation is ongoing and should be preemptively addressed. District- and school-wide shifts in systems and culture are needed to facilitate initial implementation and sustainment. The second year of SEL implementation will address the adaptive challenges identified, and continue to provide additional learning opportunities for years to come.
Hierarchy of Qualities in Global Health Partnerships: A Path Towards Equity and Sustainability
Authors
Simone Schriger - University of Pennsylvania
Agnes Binagwaho - University of Global Health Equity
Moses Keetile - Botswana Ministry of Health & Wellness
Vanessa Kerry - Harvard University
Joel Mubiligi - University of Global Health Equity
Doreen Ramogola-Masire - University of Botswana
Michelle Roland - Sutter Medical Group
Frances Barg - University of Pennsylvania
Corrado Cancedda - University of Pennsylvania
Despite two decades of exponential growth in global health partnerships, evidence for their effectiveness remains limited. Furthermore, many partnerships are dysfunctional as a result of inequitable partnership benefits, low trust and accountability, and poor evaluation and quality improvement practices. This poster depicts a theoretical model for partnerships developed through interviews with seven global health experts. Through semi-structured interviews and an open-coding approach to data analysis, and inspired by Maslow's hierarchy of needs, we identify 12 global health partnership pillars spanning three interconnected partnership levels. The six transactional pillars, which are the most foundational of the pillars, are governance, resources and expertise, power management, transparency and accountability, data and evidence and respect and curiosity. The four collaborative pillars (which build upon the transactional pillars) are shared vision, relationship building, deep understanding, and trust. The two transformational pillars (which build upon the collaborative pillars and allow partnerships to achieve their full potential) are equity and sustainability. The theoretical model depicted in this poster will be discussed in the context of real-life examples, which demonstrate both the cost incurred when global health partnerships fail to adhere to these pillars and the benefits gained when global health partnerships uphold them. I will also discuss key lessons learned from these interviews as well as best practices that global health partnerships should adopt to further increase their strength and improve their future effectiveness. To continue improving health outcomes, including mental health, and reducing health inequities globally, we need global health partnerships that are transformational, not just rhetorically but de facto. Such partnerships should serve as a catalyst for greater societal good and not simply as a platform to accrue and exchange organizational benefits.
Research-Policy-Practice Synergy to Support Louisiana's Behavioral Health Workforce – Feasibility and Lessons Learned from the Capacity and Sustainability Funding Pilot
Authors
Lisa Staples - Louisiana State University Health Sciences Center School of Public Health - Center for Evidence to Practice
Stephen Phillippi - Louisiana State University
Brian Bumbarger - Science, Systems & Communities Consulting, LLC
Ronnie Rubin - Impact Reach Consulting
Gabrielle Gonzalez - Louisiana State University Health Sciences Center School of Public Health - Center for Evidence to Practice
Background: Implementation research has identified financial factors as a key barrier to evidence-based practice (EBP) implementation in community settings. In 2022, the LSU Center for Evidence to Practice, and Louisiana's Office of Behavioral Health (OBH) undertook a project to translate this prior implementation research finding into policy and practice.
Approach: OBH allocated one-time pilot funding to support licensed outpatient Medicaid mental health practitioners and agencies delivering seven Evidence-Based Practices (EBPs). The pilot aimed to test how a one-time incentive could be used to better support the uptake and sustainment of EBPs, increase EBP utilization by clinicians and enhance behavioral health services access for Medicaid-insured children and families.
Outcomes: 25 applicants received incentives and provided one or more EBPs after being trained and certified. Seven EBPs were delivered to 93 youth and their families, through 361 research-informed community-based psychotherapy sessions. The incentive increased motivation for achieving EBP qualification; time and costs were perceived as supported. Providers reported feeling an enhanced capacity to grow their practice, see more clients, and pursue advanced training. Providers and Managed Care Organizations (MCOs) needed clarification on EBP qualification requirements and documentation. Billing procedures posed challenges due to compatibility issues with provider systems and credentialing processes. This pilot encouraged the adoption and delivery of EBPs in a Medicaid provider network and insight for future incentive programs.
Next Steps: This project underscores the need for adequate compensation to improve the capacity and sustainability of EBP services and should inform future practice development. Challenges included communication and technical barriers with MCOs, the need to simplify incentive distribution processes, ensure effective communication of technical aspects, align funding qualifications with service delivery, and provide continued support for EBP providers. Through partnerships, this pilot will inform future work supporting EBP-qualified providers within Louisiana's behavioral health workforce, with lessons generalizable to other states.
Effective, Adaptable and Sustainable for Your Community-Operationalizing Program Sustainability (EASY-OPS); pragmatic process to integrate patient feedback into program implementation.
Authors
Nicole Wagner - University of Colorado School of Medicine
Meagan Bean - University of Colorado, Anschutz Medical Campus
Amy Wineland - Summit County Public Health Department
Jude Solano - Southern Colorado Harm Reduction Association
Danielle Harwell - Southern Colorado Harm Reduction Association
Sheila Covarrubias - Denver Health and Hospital Authority
Joshua Blum - Denver Health and Hospital Authority
Background: To address growing rates of opioid overdose deaths, communities have begun implementing naloxone vending machines. However, programs are often implemented in locations and with features people who use opioids report as a deterrent for use (e.g. near police and high foot traffic). To address this concern we used an iterative, community engaged implementation process called Effective, Adaptable and Sustainable for Your Community-Operationalizing Program Sustainability (EASY-OPS) to integrate community feedback into naloxone vending machine program development.
Approach: EASY-OPS was used for naloxone vending machine implementation in an urban federally qualified healthcare center, an urban harm reduction agency serving a predominantly Hispanic/Latinx population, and a public health department in a rural community in Colorado. EASY-OPS integrates ongoing data collection with citizen scientists (community members who used opioids for nonmedical use in the last year) for the implementation team (organization leaders overseeing and staff implementing the program) to maximize program impact and sustainability. We used case study methodology, guided by PRISM, to examine contextual factors contributing to the implementation process and integration of community perspective into each organization's naloxone vending machine program.
Outcomes: Using EASY-OPS, all organizations integrated desired locations and features preferred by citizen scientists into the naloxone vending machine program. PRISM domains of external factors (e.g. pharmacy regulations and politics), implementation and sustainability infrastructure (e.g. machine costs, ongoing funding sources for supplies), and organizational characteristics (e.g. leadership approval processes) played key roles in implementation across organizations though differed in specific factors impacting the implementation process (e.g. pharmacy regulations vs local politics vs contracting). Iterative engagement provided rapid feedback to implementation challenges supporting integration of citizen scientists’ preferences (e.g. neighborhood preference when specific location was not feasible).
Next Steps: Assessment of program development costs, reach and program health outcomes will inform the value of EASY-OPS compared to current program implementation strategies.
Using observational coding to assess quality of asthma education delivery: Partnering with community experts to develop a quality assessment rubric
Authors
Emily Wheat - University of Colorado School of Medicine, Department of Pediatrics, Center for Cancer and Blood Disorders, Hemophilia and Thrombosis Center
Julia Reedy - Adult and Child Center for Outcomes Research and Delivery Science (ACCORDS)
Amy Huebschmann - Adult and Child Center for Outcomes Research and Delivery Science (ACCORDS); University of Colorado School of Medicine, Division of General Internal Medicine; Ludeman Family Center for Women's Health Research
Sarah Brewer - Adult and Child Center for Outcomes Research and Delivery Science (ACCORDS)
Background: The Better Asthma Control for Kids (BACK) program is a school-based asthma program utilizing asthma navigators to support the education, care coordination, and management of students with uncontrolled asthma. This abstract describes the development and implementation of a measure evaluating the quality of asthma navigation delivery. The quality measure is designed to assess (a) competence in asthma education and navigation components and (b) interpersonal and cultural competence with which intervention components are delivered.
Approach: We iteratively developed a quality measure evaluating asthma navigators. Key components of the quality measure were collected from existing measures of patient navigator competence. Expert asthma navigator feedback was utilized to further refine the quality measure and to identify initial exemplars for high, medium, and low competence for each component. Asthma navigators then reviewed and revised the measure further. A 5-point Likert type scale was selected. Ratings were made based on observations of asthma navigator interactions during audio-recorded sessions with students and caregivers of students with uncontrolled asthma.
Outcomes: A brief measure designed to assess navigation quality to be utilized as part of multilevel quality assessments for the BACK program. We will also present preliminary quality data in the poster currently being collected.
Next Steps: This work extends the existing and limited literature on patient navigation quality. The utilization of asthma navigator expertise and review in refining this quality measure offers a unique and innovative perspective to the development of fidelity assessment, something that may be particularly important when considering interpersonal and cultural competency in asthma navigation with a rural population. Future efforts to assess quality as part of intervention delivery may benefit from the current model of adapting existing measures using a participatory approach.
Obstacles to and Facilitators of Accessing and Implementing Services for Child Welfare in Virginia in the Context of the Family First Prevention Services Act
Authors
Juliet Wu - Virginia Commonwealth University
Brianna Boggs - Virginia Commonwealth University
Ashley Robinson - Virginia Commonwealth University
Jasper Oliver - Virginia Commonwealth University
Alana Riso - Virginia Commonwealth University
Michael Southam-Gerow - Virginia Commonwealth University
Rafaella Sale - Virginia Commonwealth University
Background: Prior to the Family First Prevention Services Act (FFPSA), federal child welfare funding primarily assisted families with children already in the foster care system and focused on a) child wellbeing and protection in the context of abuse/neglect allegations, b) caregiver child rearing support, and c) finding a stable, permanent home for children. The FFPSA aimed to expand availability, accessibility, and funding of high quality evidence-based programs (EBPs) to support families with children at risk of being removed from their homes. Virginia's Family First Plan was approved in 2021, amended in 2023, and includes eight EBPs.
Approach: FFPSA implementation has encountered challenges and successes in Virginia. This project examined factors influencing implementation with the goal of providing recommendations designed to meet the FFPSA goals (e.g., quality service access, prevention orientation). The study used a quantitative/qualitative approach. Quantitative data from families involved with Virginia Department of Social Services (VDSS) were examined to predict movement through planned service pathway checkpoints based on race/ethnicity, age, and rurality. The qualitative study involved focus groups with key stakeholders (i.e., service brokers, provider agency staff, and state agency staff) who were asked to describe barriers and facilitators throughout the FFPSA service pathway.
Outcomes: Although there were notable limitations with the data available, the quantitative results demonstrated a slow rate of referral to EBPs via Family First. Results from the qualitative study (n= 4 groups, n= 9 participants) revealed several themes: a) infrastructure factors, b) workforce factors, c) funding logistics, d) family-specific factors, and e) service-specific factors.
Next steps: Lessons learned from this study may be applicable to other states in FFPSA implementation.
Research Track
A Primer on Foundational Principles of Effectiveness or Implementation Randomized Trials
Authors
Daniel Almirall - University of Michigan, Ann Arbor
Background: The intended goal of any effectiveness or implementation randomized trial is to make inferences about the causal effect(s) of an intervention or implementation strategy (or its components) just as they would be used in actual practice settings. In these settings, interventions and implementation strategies are defined as a set of instructions guiding an action, to be taken by an actor, toward a recipient from a specified population, and is intended to improve one or more health or implementation outcome. This stands in stark contrast with the goals of most efficacy randomized trials, which are important to the intervention science knowledge base, but are not designed to make inferences about the effect of interventions offered by clinicians. Rather, efficacy trials more typically focus on the causal effect(s) of actions taken by the intervention recipient.
Approach: The purpose of this presentation is to review the (i) important differences in how interventions are defined among efficacy scientists versus how they are defined among effectiveness-implementation scientists, and (ii) foundational design principles that are characteristic of high-quality effectiveness or implementation randomized trials, including hybrid effectiveness-implementation trials and optimization randomized trials. All intervention or implementation scientists are invited to attend.
Implementation and Evaluation of Mental Health Interventions for Transgender and Nonbinary Individuals and Providers: A Scoping Review
Authors
Hadley Ankrum - Yale School of Public Health
October Mohr - Yale University
Benjamin Eisenstadt - Stony Brook University
Danielle Chiaramonte - Yale School of Public Health
Background: Transgender, nonbinary, and gender diverse (TGNB) individuals disproportionately experience poor mental health outcomes such as depression, anxiety, and suicidality. Evidence-based mental health interventions targeting specific populations have been found to be particularly beneficial in reducing these outcomes. The purpose of this scoping review was to systematically collect and synthesize the literature on mental health interventions for TGNB individuals and/or TGNB-serving providers.
Methods: This review was conducted using the Joanna Briggs Institute methodology for scoping reviews and the PRISMA Extension for Scoping Reviews Checklist and Explanation. An experienced medical librarian was consulted on methodology. Two screeners independently reviewed the titles, abstracts and full text of the eligible articles that met inclusion criteria. Articles were included if they: (1) focused on TGNB individuals or LGBTQ+ individuals with TGNB-specific content; (2) Focused primarily on TGNB individuals over the age of 18; (3) Implemented or evaluated an intervention focused on TGNB mental health for individuals, providers, or systems.
Results: A total of 3,337 articles were imported for screening. 39 articles were identified as duplicates and removed. 3298 articles were screened by two reviewers; 55 were selected for full-text review, and 15 studies were retained for extraction. Of those extracted, three were deemed ineligible. Approximately half of the studies had implementation-focused primary outcomes (e.g., acceptability, feasibility, fidelity, knowledge). Most of these interventions involved training providers, while four studies provided client-level efficacy data. The majority of interventions were at the individual level.
Conclusions: The results of this scoping review provide a bleak yet important picture of evidence-based mental health interventions for TGNB individuals. Given the political landscape in the U.S., it is important that researchers continue to develop and implement interventions and programming that support the mental health of TGNB individuals, particularly in states that lack community, city, and governmental support.
Strategic Synergy for Aid Worker Security: Examining the Diffusion Dynamics of Violence Against Aid Workers Using a Multidimensional Density-Based Spatiotemporal Clustering Algorithm (MDST-DBSCAN)
Authors
Cara Antonaccio - The Warren Alpert Medical School of Brown University
Alethea Desrosiers - The Warren Alpert Medical School of Brown University
Introduction: This study examines the spatial diffusion dynamics of violence and attacks against humanitarian assistance, focusing on security as a key implementation outcome. Aligned with The SIRC 2024 theme of "Strategic Synergy," our research explores the intricate interplay between research, practice, and policy in high-risk conflict zones. Our poster presentation aims to foster a rich and inclusive discussion on the value of spatial analysis techniques for understanding security processes in humanitarian intervention. We believe this approach can inform the development of more effective security protocols, policy strategies, and ultimately, improve aid delivery in humanitarian emergencies.
Methods: Using georeferenced event data from the Aid Worker Security Dataset, this study uses an innovative multi-dimensional spatiotemporal density-based clustering algorithm (MDST-DBSCAN) to analyze the spatial diffusion dynamics of violent attacks against humanitarian aid workers in Gaza and the West Bank between October 7, 2023 and April 27, 2024. MDST-DBSCAN is used to identify "hotspots" of violence and allows us to track their evolution as well as to characterize how they "spread" across time and space.
Results: The results of this study show a pattern of attacks against aid workers, with a mix of incidents involving shelling, shootings, and airstrikes. Results indicate both hotspots of attacks against aid workers as well as isolated attacks—those that are spatially and temporally isolated with varying casualty numbers. Most fatalities in the sample of events observed involve aerial attacks, and the severity of attacks seems to be correlated with the density of attacks within each cluster. The nature and severity of attacks against aid workers varied across the spatiotemporal hotspots—with some low density hotspots primarily involving shelling or shooting of individual aid workers from the Red Crescent Society (RCS) and NGOs; and higher density hotspots emerging later in the conflict and demonstrating a shift towards targeted attacks on ambulances with RCS and UN workers, along with civilian casualties. Here, airstrikes become the dominant form of attack, with a notable increase in fatalities in hotspots with a high density of attacks (41 UN workers killed in November). Overall, the report suggests a systematic increase in the lethality of attacks as the density of incidents within each cluster grows. This indicates a possible targeting of areas with higher concentrations of aid workers. Maps will demonstrate hotspots and their evolution, including increases in attack density over time and directional growth patterns to demonstrate how the identified hotspots spread across geographic space.
Conclusions: The findings from this study contribute to strategic synergy by promoting a crucial link between research and practice. By directly informing the development of targeted security protocols based on spatial risk patterns, this study fosters the effective implementation of evidence-based security measures for humanitarian aid workers. This research also highlights the dynamic nature of risk in conflict zones and emphasizes the need for flexible policy frameworks that can adapt to shifting spatial patterns of violence. This adaptability is crucial for mitigating violence, improving aid delivery, and facilitating effective and safe assistance in high-risk contexts. This study's findings have the potential to inform several policy and practice areas. First, regarding security protocol development, the spatial analysis of violence hotspots can inform targeted security protocols for aid workers, focusing on the areas with the highest risk. With respect to resource allocation, by identifying high-risk areas, resources for security measures and risk mitigation strategies can be strategically allocated. Policy adaptation may also be optimized through a nuanced understanding of the spatial dynamics of violence can inform the development of adaptable policy frameworks that can respond effectively to changing risk patterns. Moreover, this research can contribute to improved risk communication strategies for both aid workers and the communities they serve.
Insights From Multi-sector Social Network Analysis to Inform Collaborative Policymaking Efforts Toward Culturally Responsive Behavioral Health Care for Youth and Families in Washington State
Authors
Alya Azman - University of Washington
Mariam Haris - University of Washington
Merih Mehari - University of Washington
Noah Gubner - University of Washington
Maggie Fenwood Hughes - University of Washington
Sarah Walker - University of Washington
Background: Addressing workforce shortages in behavioral health care is crucial in mitigating disparities and improving outcomes. Policy is an outer context factor that exerts a significant influence on workforce through rate setting, licensure requirements, and accreditation of graduate programs. Aligning policy ideas among diverse sectors is critical for successful policy design and implementation. We present a multi-stage approach to measure alignment in policy ideas that is feasible to replicate for other studies.
Methods: The Culturally Affirming & Responsive Mental Health (CARE) project aims to improve access to culturally relevant mental health services through state workforce policies. This project includes a codesign team collaborating with a community coalition and advisory team. We aim to examine the policy recommendations provided by advisory members and the evolution of themes throughout the project using thematic coding and social network analysis. Data comes from surveys distributed at baseline and 6-month follow-up assessing three key areas: 1) Policy changes to increase access to culturally responsive care in Washington state; 2) Pathways for expanding the mental health workforce; and 3) Previous efforts to expand the workforce.
Results: Data analysis is underway. Our sample encompasses various sectors, including advocacy (N=1), community board (N=1), government (N=2), insurance (N=1), nonprofit (N=1), policy and training non-governmental organization (N=1) and university (N=4), totaling N=11. We will conduct a thematic analysis to identify emerging themes in responses across sectors. Additionally, we aim to examine how responses differ from baseline at T2, to identify a shift in themes and determine which sectors demonstrate the most significant changes.
Conclusions: We anticipate gaining a deeper understanding of policy priorities across sectors and policy recommendations deemed necessary to increase culturally responsive care in Washington state. This study can provide insight into how the participatory codesign approach can influence shifting themes, enriching our understanding of the collaborative process.
Electronic Flashcards: An Untapped Implementation Strategy for Accelerating Knowledge-to-Practice?
Authors
Emily Balczewski - University of Michigan, Ann Arbor
Philip Barrison - University of Michigan, Ann Arbor
Alexandra Vinson - University of Michigan, Ann Arbor
Zach Landis-Lewis - University of Michigan, Ann Arbor
Background: Healthcare professionals learn continuously and must do so at an increasing rate, as the production of biomedical knowledge accelerates. Electronic flashcards (EFs) and EF platforms, such as Anki and Quizlet, are an increasingly widespread e-learning resource in health professions education. There is mounting evidence that EFs improve both the retention and application of many types of medical knowledge. However, to our understanding, EFs have not been directly studied as an implementation strategy to encourage evidence-based practices.
EFs extend the behavior of traditional paper flashcards to easily a) facilitate spaced repetition review of content, b) incorporate multimedia elements, c) link to existing repositories of medical information (e.g., UpToDate), and d) enable crowd-sourcing, versioning, and customization of decks. These qualities make EFs a flexible, powerful, and portable resource to 1) develop and 2) distribute educational materials–two implementation strategies recognized by the Expert Recommendations for Implementing Change (ERIC) study. If used as targeted interventions, could EFs aid in closing the gap between knowledge and practice?
Approach: In this presentation, we aim to explore EFs as a viable and promising implementation strategy, by (1) Providing a brief introduction to EFs and how to use them, including how they differ from other spaced learning interventions, (2)Presenting key findings from our scoping review of EFs for health professions education, (3)Highlighting studies from this scoping review which use EFs as an intervention to close the knowledge-to-practice gap, and, (4) Discussing critical design decisions when using EFs as an educational intervention, informed by our qualitative framework that describes the knowledge representation in Anki cards for undergraduate medical education.
We hope this presentation will encourage open discussion from the implementation science community about the value and direction of future research into EFs as an implementation strategy to close the knowledge-to-practice gap.
Do Quality Improvement Strategies Help Implement Evidence-Based Practices? Evidence From a National Sample of Substance Use Disorder and Mental Health Specialty Treatment Facilities
Authors
Jure Baloh - University of Arkansas for Medical Sciences
Geoffrey Curran - University of Arkansas for Medical Sciences
Background: Implementation of evidence-based practices (EBPs) for substance use disorders (SUDs) in specialty treatment facilities is highly variable. Quality improvement (QI) strategies can support EBP implementation, but the evidence of their impact is limited. In this study, we used a nationally representative dataset of SUD and mental health treatment facilities in the US to assess the relationship between QI strategies and implementation of EBPs for SUD treatment.
Methods: The study sample included 3799 specialty treatment facilities that provide a combination of SUD and mental health treatment and are included in the 2022 National Substance Use and Mental Health Services Survey (N-SUMHSS) dataset from SAMHSA. We constructed an ‘EBP Index,’ a count of behavioral (non-medical) SUD treatment EBPs that facilities frequently use (e.g., motivational interviewing, contingency management), and a ‘QI Index,’ a count of routinely used QI strategies (e.g., supervision, root cause analysis, client outcome tracking). Using a linear regression model, we examined the relationship between the two indices, and adjusted for facility characteristics (ownership type, residential/outpatient services, licensure/accreditation, payment types accepted, client volume, and state). We also examined these relationships for each EBP individually using similar logistic regression models.
Results: Facilities on average used 7 EBPs (SD: 2.4) and 6 QI practices (SD: 1.5). The QI Index was significantly and positively associated with the EBP Index, with each additional QI strategy increasing the EBP Index by 0.2 EBPs (p<.0001). The QI Index was also significantly associated with the use of each individual EBP, with odds ratios ranging from 1.093-1.277 (p<.01 to <.0001).
Conclusions: Our study provides additional evidence that QI strategies can support implementation of EBPs in specialty treatment settings. QI strategies are routinely used by many facilities; however, their use is variable and supporting their use could help enhance implementation of EBPs and other QI initiatives in these settings.
Ongoing Tailoring Approaches and their Monitoring in the Implementation of Evidence-based Practices, Interventions, and Services in Healthcare Settings - A Protocol for a Systematic Review and Modified Delphi Study
Authors
Kathrin Blum - Institute for Implementation Science in Health Care, University of Zurich
Nora Braathu - Norwegian Centre for Violence and Traumatic Stress Studies (NKVTS)
Emanuela Nyantakyi - Institute for Implementation Science in Health Care, University of Zurich
Khic-Houy Prang - Centre for Health Policy, Melbourne School of Population and Global Health, The University of Melbourne
Marie-Therese Schultes - Institute for Implementation Science in Health Care, University of Zurich
Ebony Verbunt - Centre for Health Policy, Melbourne School of Population and Global Health, The University of Melbourne
Bianca Albers - Institute for Implementation Science in Health Care, University of Zurich
Lauren Clack - Institute for Implementation Science in Health Care, University of Zurich
Background: Tailoring involves intentionally selecting implementation strategies to address specific determinants within context and cater to local needs. While often studied as a one-off approach—where strategies are selected once before their eventual application—implementation realities typically demand agility and responsiveness to contextual changes. This is where ongoing tailoring (OT) is needed. OT involves iteratively reconsidering strategies throughout implementation based on continual monitoring to enable informed decisions about required refinements. Deepening our understanding of OT is crucial to elucidate its potential in guiding implementation. This study aims to provide an overview of how OT has been conducted and to identify core elements and pragmatic indicators that can facilitate effective OT in practice.
Methods: A systematic review will inform a modified Delphi study. Articles from 2005 to the present, targeting OT approaches employed in healthcare settings, were searched across five databases. Key outcomes include models and theories utilized for OT, processes and individuals involved, monitoring indicators and measures, and decision-making approaches. The findings will be incorporated into three online Delphi rounds, with questionnaires and group discussions, aimed at converging on core elements and pragmatic indicators. The panels will comprise OT experts (researchers and practitioners) and healthcare workers participating in OT projects.
Results: A total of 5746 identified articles are currently undergoing title and abstract screening. The presentation will include preliminary systematic review findings, methodological considerations for incorporating these findings into the Delphi study, and an outline of the methodology for the foreseen Delphi study.
Conclusion: This study will deepen our understanding of OT, an agile and structured implementation approach that accommodates contextual changes and necessary adaptations. By integrating diverse perspectives, it will provide insights into how OT can be effective in real-world application. Ultimately, the outcomes can serve as a basis for applying and evaluating OT efforts in implementation research.
Good Enough? A Scoping Review Characterizing Use of the Intervention Appropriateness Measure (IAM), Acceptability of Intervention Measure (AIM), and Feasibility of Intervention Measure (FIM)
Authors
Meredith Boyd - Center for Dissemination and Implementation Science, Northwestern University Feinberg School of Medicine
Elizabeth Casline - Institute for Sexual and Gender Minority Health and Wellbeing, Northwestern University, Feinberg School of Medicine
Kaitlin Piper - Center for Dissemination and Implementation Science, Northwestern University, Feinberg School of Medicine
Zabin Patel Center for Dissemination and Implementation Science, Northwestern University, Psychiatry and Behavioral Sciences
Kira DiClemente-Bosco - Center for Dissemination and Implementation Science, Northwestern University, Feinberg School of Medicine
Sara Becker Center for Dissemination and Implementation Science, Northwestern University, Feinberg School of Medicine
Background: Appropriateness, acceptability, and feasibility have been identified as key implementation outcomes thought to be relevant to an organization's adoption, implementation, and sustainment of an innovation (Proctor et al., 2023). The Intervention Appropriateness Measure (IAM), Acceptability of Intervention Measure (AIM), and Feasibility of Intervention Measure (FIM) are quantitative measures designed for ease of administration and generalizability across settings (Weiner et al., 2017). As these measures proliferate the field, there is substantial variability in how they are used. To inform future implementation outcomes research, the aims of the present scoping review are to 1) characterize use of the IAM/AIM/FIM and 2) identify strengths and limitations of measurement.
Method: A scoping review methodology was selected to characterize the breadth of measure use across study designs (Arksey & O’Malley, 2003). An a-priori protocol for the screening and data extraction was developed in collaboration with a research librarian. Articles were included if they were peer reviewed, published in English, used at least one of the three measures, and did not alter the measures beyond customization recommended by measure developers.
Results: The initial search yielded 1,597 articles, of which 232 were included for data extraction. Data have been extracted for 10% of included articles as of April 2024 and will be complete by September 2024. Observed mean scores were consistently high (>3.7) across measures, indicating ceiling effects. In 39% of extracted articles, the IAM/AIM/FIM were the only measures of appropriateness, acceptability, and feasibility. In most cases, authors concluded the innovation was appropriate, acceptable, and/or feasible (70%) or did not draw conclusions based on these measures (17%).
Conclusions: The present review will characterize use of the IAM/AIM/FIM. Special attention will be given to how researchers interpret and act upon measure scores given the critical need for measures that can inform decision making across phases of implementation.
References
Arksey, H., & O'Malley, L. (2005). Scoping studies: towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19-32.
Proctor, E. K., Bunger, A. C., Lengnick-Hall, R., Gerke, D. R., Martin, J. K., Phillips, R. J., & Swanson, J. C. (2023). Ten years of implementation outcomes research: A scoping review. Implementation Science, 18(1), 31.
Weiner, B. J., Lewis, C. C., Stanick, C., Powell, B. J., Dorsey, C. N., Clary, A. S., Boynton, M. H., & Halko, H. (2017). Psychometric assessment of three newly developed implementation outcome measures. Implementation Science, 12, 1-12.
Refugee Review Board: Members’ Experiences in the First Two Years 2022-2023
Authors
Sarah Brewer - Adult and Child Center for Outcomes Research and Delivery Science (ACCORDS)
Taryn Bogdewiecz - University of Colorado, Anschutz Medical Campus
Background: The Refugee Review Board (RRB) was created as a committee of the Society of Refugee Healthcare Providers to center the lived experiences of being a refugee in the work of the society. We sought to understand the experiences of the RRB members during the creation and initial activities of the RRB.
Methods: Every RRB member (N=12) who attended at least one meeting between September 2022 and December 2023 was invited to participate in 30-minute semi-structured interviews about their experiences with the RRB, including operations, meetings, accomplishments, and about their ideas for improvements and future activities of the RRB. Interviews were recorded and transcribed using Zoom. Data were coded in ATLAS.ti and thematic content analysis was conducted.
Results: Ten RRB members completed interviews in February-March 2024. Members reported overwhelmingly positive experiences with the RRB, including that it was a meaningful way to spend their time, build leadership capacity, and be part of an authentic and diverse community. Second, they shared pride in their accomplishments, including developing board operating documents, a consultation process for Society members, and a guide to research partnership for refugees. Third, ideas for future directions focused on increasing the RRB's involvement in research study teams, engaging refugees in narrative storytelling, and training/mentoring other refugees to engage in research.
Conclusion: The RRB provided a positive experience and accomplishments for members in its first two years. Future RRB work has potential for additional benefit to members and impact in the field.
Partnership-Building Considerations for Embedding Implementation Science in Health Systems
Authors
Stephanie Brooks - University of Alberta
Cody Alba - Alberta SPOR SUPPORT Unit
Denise Thomson - Alberta SPOR SUPPORT Unit
Sara Davison - University of Alberta
Kate Storey - University of Alberta
Introduction: Implementation of health innovations is inherently collaborative, requiring trans-sectoral partnerships between implementation researchers, innovation teams, and implementation practitioners. Implementation science has been shown to improve implementation successes; however, challenges that hinder partnerships to advance implementation science continue to persist.
Methods: We conducted a case study of Alberta, Canada's learning health system, using semi-structured group and individual interviews to create a nuanced understanding of the considerations required for implementation research collaborations. We interviewed 53 participants representing 21 offices in the health system, academia, professional associations, and government who regularly plan, evaluate, and/or study health system implementation initiatives in Alberta. Using the Partnership Model for Research Capacity Building, we identified current facilitators and challenges for partnerships for conducting and using implementation science, at different levels of Alberta's health-research ecosystem.
Results: Alberta's healthcare system is well set up to readily embed intervention effectiveness and efficacy research. Infrastructure was also in place to strengthen implementation practice. However, weaknesses around exchanging knowledge and skills, providing feedback and mentoring, and accommodating diversity affected the ability of both individuals and teams to build implementation science capacity. Without this capacity, teams could not participate in embedded implementation research collaborations. We report the response of the Alberta Strategy for Patient-Oriented Research SUPPORT Unit to these barriers to provide practical guidance on various program options to strengthen individual- and organization-level implementation science capacity.
Conclusions: This study applied a whole-system approach to assess factors across Alberta's health-research ecosystem, which affect partnerships to advance implementation science. Our findings illustrated that partnership considerations go beyond interpersonal factors and include system-wide considerations. With the results, health organization leaders have (1) a method for assessing organizational capability to readily embed implementation research and (2) a catalog of potential responses to create conditions to readily engage with implementation science in their day-to-day implementation processes.
Tailoring Implementation Strategies in Multi-Site Quality Collaboratives: Insights from Mapping Determinants for Enhanced Pulmonary Embolism Care Pathway Adoption
Authors
Jessica Burns - University of Michigan, Ann Arbor
Background: Implementation mapping has emerged as a key process framework for ensuring that implementation support is designed to address key barriers. However, the use of implementation mapping within existing multi-site quality collaboratives—and in particular, optimal ways to use mapping to accommodate differences in determinants across sites—has not been fully explored.
Method: Implementation mapping was initiated to specify strategies to evaluate in a stepped wedge trial implementing a new evidence-based care pathway for low-risk pulmonary embolism in emergency departments. Stakeholders from six health systems, including academic and non-academic from urban, suburban, and rural locations, were engaged through local site champions. Semi-structured interviews were conducted to identify determinants and evaluate the feasibility and perceived impact of different implementation strategies. Interviews were coded by two trained reviewers using the CFIR framework to pinpoint the most prominent barriers and facilitators within and across sites.
Results: Across six sites, N=25 interviews were conducted with organization and department leaders, physicians, and pharmacy, social work, and administrative stakeholders. Key barriers consistently endorsed included limited follow-up access, issues with medication access at local pharmacies, and concerns about reliable insurance coverage of medications. There was also staff hesitancy towards practice change and limited exposure to the pathway's evidence. Conversely, key facilitators included the current use or endorsement of relevant risk assessment tools, general clinician adherence to standardized protocols, and active involvement of multidisciplinary teams to support high-fidelity implementation. Another set of determinants was identified as important but operating differently as barriers/facilitators depending on the site. These included quality improvement resource availability, strength of interdepartmental relationships, and leadership engagement.
Conclusion: Implementation mapping holds promise as a framework for informing strategy specification for multi-site efforts. Generalized and site-specific barriers will be used to specify implementation strategies and adaptations that can be deployed as necessary to address site-specific barriers.
Utilization and Adaptation of Quality Dashboards as a Strategy to Facilitate Evidence-Based Preventative Care Implementation among Women Veterans
Authors
Catherine Chanfreau - VA Informatics and Computing Infrastructure (VINCI)
Bevanne Bean-Mayberry - Center for the Study of Healthcare Innovation, Implementation, and Policy (CSHIIP), VA Greater Los Angeles
Erin Finley - Center for the Study of Healthcare Innovation, Implementation & Policy (CSHIIP), VA Greater Los Angeles Healthcare System
Kimberly Clair - Center for the Study of Healthcare Innovation, Implementation, and Policy (CSHIIP), VA Greater Los Angeles
Cody Knight - Center for the Study of Healthcare Innovation, Implementation, and Policy (CSHIIP), VA Greater Los Angeles
Rebecca Oberman - Center for the Study of Healthcare Innovation, Implementation, and Policy (CSHIIP), VA Greater Los Angeles
Rachel Lesser - Center for the Study of Healthcare Innovation, Implementation, and Policy (CSHIIP), VA Greater Los Angeles
Alison Hamilton - Center for the Study of Healthcare Innovation, Implementation, and Policy (CSHIIP), VA Greater Los Angeles
Tannaz Moin - Center for the Study of Healthcare Innovation, Implementation, and Policy (CSHIIP), VA Greater Los Angeles
Melissa Farmer - Center for the Study of Healthcare Innovation, Implementation, and Policy (CSHIIP), VA Greater Los Angeles
Background: Quality dashboards are well-established tools for healthcare improvement, but their utility in supporting implementation is underexplored. We developed quality dashboards for an implementation trial comparing Replicating Effective Programs (REP) versus Evidence-Based Quality Improvement (EBQI) to support preventive care evidence-based practices (EBPs) for women at 20 Veterans Health Administration sites. We report on ongoing dashboard utilization and adaptations requested by implementing sites.
Methods: We developed two types of dashboards integrating data from electronic health records and program participation: aggregated-data reports focused on population health and implementation progress, and a patient-level Sharepoint to facilitate recruitment for one EBP. Dashboards provided site-specific data to limit contamination between sites receiving EBQI versus REP. During pre-implementation, sites received dashboard demonstrations and were asked for lists of users. During implementation, facilitators used the dashboards for monthly calls with EBQI sites.
Results: Over 21 months, dashboards were deployed at 8 EBQI and 11 REP sites (total 188 staff registered). Dashboards were accessed by 14 staff at 6 EBQI sites (346 views), 21 staff at 6 REP sites (341 views), and 8 implementation team members (1351 views). The Sharepoint was shared with 5 EBQI and 7 REP sites (total 59 staff) and accessed at 5 EBQI and 5 REP sites. User number was higher at EBQI sites (25) versus REP (11), but usage volume was similar (7,474 views versus 7,298). EBQI sites requested several Sharepoint modifications to improve workflow and patient tracking, and those adaptations were deployed at all sites. One site chose to offer an EBP to all patients leading to additional tools documenting enrollment across genders.
Conclusion: Dashboards facilitating workflow were more widely adopted than those providing information on population health and implementation progress. User feedback helped improve usability. Communication with sites and responsiveness to their requests may increase the likelihood of sustained use of these tools over time.
Lessons Learned While Adapting an Evidence-Based Preventative Intervention for Transcultural Families
Authors
Miriam Clark - Oregon Social Learning Center
Katie Bennett - Oregon Social Learning Center
LaShaun Brooks - Oregon Social Learning Center
Rohanna Buchanan - Oregon Social Learning Center
Jessica Hughitt - Oregon Social Learning Center
Jayme Rosado - Oregon Social Learning Center
Carol Warren - Oregon Social Learning Center
Inga Wilson - Oregon Social Learning Center
Background: In 2022, the US Department of Health and Human Services’ Administration for Children and Families released a National Call to Action to address the racial disparities within the child welfare system. Soon the State of Oregon Department of Human Services decided to invest in the adaptation of KEEP [an evidence-based program (EBP) for parents caring for children/youth in the foster care system shown to improve child behavior, decrease parental stress, and improve placement stability] to be implemented specifically for transcultural families in the child welfare system. Creating an adaptation that was well received by families proved difficult during a time of polarized tension. Through the extensive adaptation process, we learned important lessons for adapting and implementing an EBP.
Methods: The adaptation process involved two years of collaborating with experts in the area, running the EBP, conducting focus groups with EBP participants, soliciting feedback from group leaders and continually implementing changes to the curriculum that were both in line with the EBP and would support transcultural families.
Results: Lessons learned include 1) The adaptation team must check personal biases and recognize need for changes; 2) Pre-teaching participants how to have discussions about race is critical to foster open communication and initial guidelines should be set; 3) Conversations about race must be interwoven within the curriculum; 4)Though difficult, it is important to recognize that the adaptation of an EBP will not fix all the problems regarding racial/ethnic marginalization among youth/children in the child welfare system, but can serve as just one important step; 5) We need to recognize that the adaptation will be a process that will take continual adjustment over time.
Conclusions: Our lessons learned from adapting/implementing this EBP can provide a framework for researchers and practitioners aiming to adapt other EBPs for resource parents in transcultural families.
Suggestions from facilitators on how to increase cultural relevancy of an evidence-based intervention
Authors
Katie Combs - University of Colorado Boulder
Amanda Ladika - University of Colorado Boulder
Veronica Goldberg - University of Colorado Boulder
Christine Steeger - University of Colorado Boulder
Brittany Hubler - University of Colorado Boulder
Lore Stacey - University of Colorado Boulder
Pamela Buckley - University of Colorado Boulder
Introduction: There is debate about cultural relevance of evidence-based interventions (EBI) in prevention and implementation science. However, few studies consider insights from facilitators of EBIs, who are on the ground delivering EBIs in diverse contexts with diverse students. Our study examines suggestions for cultural relevance from facilitators of a universal prevention program delivered in schools.
Method: This study used process evaluation data from a national dissemination project of Botvin LifeSkills Training (LST) middle school program over three academic years (2019-2022). At the end of each academic year, LST facilitators (primarily teachers) completed a survey on fidelity of implementation that included the open-ended item, “Please list any suggestions you may have for improving the cultural relevance of LST.” Of the 1,950 surveys completed over the three years, 566 included open-ended responses. On average, this sample had 14.8 years of teaching experience, and was 72% White, 18% Black, 5% Latinx, and 68% female. The Framework Method, a qualitative content analysis approach, was used to analyze the data.
Results: Five broad themes were identified from the 758 coded suggestions: social or technological relevance (n=393), teacher or school related considerations (e.g., teacher autonomy, placement of program, n=100), relevance to diverse groups (e.g., youth who have experienced trauma, n=69), mechanics of the curriculum (e.g., quantity of material, user-friendliness, n=32), and engagement of students (e.g., more interactive activities, n=75). Social or technological relevance was the predominant theme including suggestions for integrating technology, updating curriculum to fit current trends in drug use and other risks to youth (e.g., social media), and modernizing visuals and scenarios.
Conclusion: Teachers in this sample highlight an overlooked component of cultural relevance, that is social and technological relevance. To sustain implementation of EBIs decades beyond its initial development, a feasible way of updating EBIs to the evolving world is essential.
Comparing two approaches for tailoring implementation strategies: case study of a structured type 1 diabetes education programme (DAFNE Ireland)
Authors
Ana Contreras Navarro - Health Implementation Research Hub, School of Public Health, University College Cork
Aoife O'Mahony - Health Implementation Research Hub, School of Public Health, University College Cork
Fiona Riordan - Office of Vice President for Research & Innovation, University College Cork, Ireland
Sheena McHugh - Health Implementation Research Hub, School of Public Health, University College Cork
Laura-Jane McCarthy - Health Implementation Research Hub, School of Public Health, University College Cork, Ireland
Claire Kerins - Health Promotion Research Centre, School of Health Sciences, University of Galway, Ireland
Geoffrey Curran - University of Arkansas for Medical Sciences
Cara C. Lewis - National Institutes of Health / National Heart, Lung, and Blood Institute / Center for Translation Research and Implementation Science
Jane Murphy - Health Research Board, Ireland
Background: Tailoring implementation strategies is an effective method to facilitate improvement in healthcare services. The tailoring process follows three key steps: identification of determinants, prioritisation of determinants, and selection of strategies by addressing determinants. These steps are performed prospectively and are informed by theory, evidence, and stakeholder's perspectives. However, it is not known whether adopting a specific participatory design method over another can produce better tailoring outcomes. This study aimed to identify advantages/disadvantages of alternative tailoring approaches (TA) designed to support the implementation of Dose Adjustment for Normal Eating (DAFNE), a group patient education programme for adults with type 1 diabetes in Ireland.
Methods: We followed two TA. TA1: initial determinant identification by conducting a survey and three discussion groups focused on all tailoring steps. TA2: three surveys to identify and rank determinants and one discussion group to select strategies. Stakeholders included 62 healthcare professionals (HCPs) involved in DAFNE delivery (dieticians, nurses, consultants, and administrators) in 16 hospitals. The analysis of outcomes was primarily inductive, led by pre-specified determinants as per the Consolidated Implementation Research Framework.
Results: TA1 generated twice as many determinants (n=73) compared to TA2 (n=36) and, respectively, more strategies were selected (n=45 vs. n=36) by engaging HCPs in sequential, researcher-led discussions. In TA1, 86% of the strategies related to knowledge and beliefs about the intervention (n=16), engaging (n=14) and available resources (n=9). Similarly, in TA2, 72% of the strategies focused on engaging (n=18) and available resources (n=8).
Conclusions: The study offers insights into persistent determinants that can be best addressed at system-level (e.g., use of mass media to improve recruitment). In addition, the use of a repeated participatory research method in tailoring resulted in the identification of local attitudes that were not measured by self-administered data collection tools, highlighting the importance of well-chosen methods for stakeholder participation.
The Role of Stigma in Cervical Cancer Screening, Diagnosis, and Treatment in Mexico City Clinics: A Qualitative Implementation Study
Authors
Emilie Egger - Yale University
Jorge Salmerón - Universidad Nacional Autónoma de México
Julia Hernandez - Universidad Nacional Autónoma de México
Sonia Hernandez - Universidad Nacional Autónoma de México
Leith Soledad Leon-Maldonado - INSP
Sonia Pérez-Matus - Universidad Nacional Autónoma de México
Aarón Osvaldo Rodríguez-Ojeda - Universidad Nacional Autónoma de México
Sangini Sheth - Yale School of Medicine
Leticia Torres-Ibarra - Universidad Nacional Autónoma de México
Donna Spiegelman - Yale School of Public Health
Raúl U. Hernández-Ramírez
Cervical cancer is nearly entirely preventable. However, 4,000 women in Mexico die of cervical cancer each year disease and it is the second-leading cause of cancer death among Mexican women. Most deaths are due to delays in screening and treatment. Screening and diagnostic care is subsidized by the Mexican government and free to users, yet adherence remains low. We are conducting an ongoing study to identify barriers and facilitators to follow-up diagnostic care after abnormal screenings in Mexico City clinics in the Tlalpan District. Here, we focus on the findings regarding the role of stigma in cervical cancer screening, diagnosis, and treatment.
We conducted semi-structured interviews (n=13) with patients and focus group discussions (n=1) with implementers of the program for cervical cancer screening, diagnosis, and treatment providers at two clinics. We used purposive sampling to identify women at two stages: 1) who followed up on abnormal screening and 2) who did not follow up on abnormal screening. After achieving theoretical saturation, we analyzed the data using the Consolidated Framework for Implementation Research codebook and in vivo codes from our data.
Our results reflect that, in addition to hypothesized barriers to diagnostic screening such as difficulty of securing transportation and distance to colposcopy clinic, women expressed fear of experiencing intense social stigma that manifested as domestic and intimate partner violence. This experience often prevented them from following up with recommended diagnostic care after abnormal screening.
These findings could shift influence priorities in cervical cancer care to include a focus on stigmas as they manifest in other research locations in Latin America and more broadly in research on cervical cancer and other cancers that are stigmatized. This change in research could shift implementation processes and clinical recommendations for women who experience abnormal screening who may need tailored support for pursuing recommended care.
Adapting the Stages of Implementation Completion® (SIC) to Assess Fidelity and Time to Competence in the Veterans Affairs (VA) Healthcare System: Development and Test Case in a Hybrid Type 3 Effectiveness-Implementation Trial
Authors
Erin Finley - Center for the Study of Healthcare Innovation, Implementation, and Policy (CSHIIP), VA Greater Los Angeles
Lisa Saldana - Chestnut Health Systems - Lighthouse Institute Oregon
Holle Schaper - Chestnut Health Systems - Lighthouse Institute Oregon
La Shawnta Jackson - Center for the Study of Healthcare Innovation, Implementation, and Policy, VA Greater Los Angeles
Karissa Fenwick - Center for the Study of Healthcare Innovation, Implementation, and Policy, VA Greater Los Angeles
Tannaz Moin - Center for the Study of Healthcare Innovation, Implementation, and Policy, VA Greater Los Angeles
Melissa Farmer - Center for the Study of Healthcare Innovation, Implementation, and Policy, VA Greater Los Angeles
Alison Hamilton - Center for the Study of Healthcare Innovation, Implementation, and Policy, VA Greater Los Angeles
Bevanne Bean-Mayberry - Center for the Study of Healthcare Innovation, Implementation, and Policy, VA Greater Los Angeles
Background: Although implementation fidelity and progress over time are foundational outcomes in implementation science, only the Stages of Implementation Completion® (SIC) has reliably evaluated these outcomes across research assessing diverse evidence-based practices (EBPs), settings, and implementation strategies. The SIC process model spans eight standardized stages, populated by activities tailored for the intervention being implemented, measuring activities completed and time to complete them. As part of EMPOWER 2.0, a hybrid type 3 effectiveness-implementation trial of three virtual EBPs for women Veterans in Veterans Affairs (VA) healthcare, we tailored the SIC for use in VA accounting for three EBPs and two implementation strategies (Replicating Effective Programs [REP] and Evidence-Based Quality Improvement [EBQI]). We report on this adaptation and findings from one of our EBPs, Telephone Lifestyle Coaching (TLC), a lifestyle coaching intervention, delivered in four VA sites.
Methods: Following a standard SIC operationalization process, the EMPOWER and SIC teams met over approximately nine months to adapt the SIC to allow: (1) use in VA, accounting for multiple levels of leadership engagement and diverse local organizational structures; (2) assessment of progression toward competence for distinct EBPs; and (3) assessment of fidelity to two implementation strategies. The EMPOWER-SIC was then used to assess TLC implementation in four sites randomly assigned to REP or EBQI.
Results: Across TLC sites, the EMPOWER-SIC successfully distinguished between REP and EBQI sites while assessing fidelity to pre-implementation activities (100% in all sites). Three sites (2 REP and 1 EBQI) achieved competence during the study period; mean time to competence was 191 days (range 138-266).
Conclusions: The EMPOWER-SIC allowed assessment of fidelity and time to competence despite our study's complexity and diversity of sites. Validated tools like the SIC are essential as trials increasingly seek to compare the effectiveness of implementation strategies across diverse settings and EBPs, toward a more generalizable implementation science.
Which Learning Collaborative Features Predict Successful Implementation of a Patient Decision Aid for Early-stage Breast Cancer? Data from the SHAIR Collaborative
Authors
Rachel Forcino - University of Kansas School of Medicine
Renata Yen - Geisel School of Medicine at Dartmouth
Hannah Leavitt - Geisel School of Medicine at Dartmouth
Danielle Schubbe - Geisel School of Medicine at Dartmouth
Christopher Jacobs - Geisel School of Medicine at Dartmouth
Jaclyn Engel - Geisel School of Medicine at Dartmouth
Marie-Anne Durand - Geisel School of Medicine at Dartmouth
Glyn Elwyn - Geisel School of Medicine at Dartmouth
Introduction: There is limited research using learning collaboratives to implement conversation aids in clinical practice. We aimed to evaluate the independent association of learning collaborative components with the overall patient reach of a conversation aid for early-stage breast cancer treatment decisions.
Methods: For each clinical site participating in the learning collaborative, we collected the following data on a monthly or quarterly basis over two years: extent of engagement and communication with learning collaborative leadership; number of patient evaluation surveys collected; use of the learning collaborative website; social interconnectedness with other learning collaborative members; and integration of the conversation aid into the electronic health record. We conducted multiple linear regression analysis to evaluate the role of each of these factors in predicting site-level patient reach (i.e. proportion of eligible patients receiving the conversation aid).
Results: Full results available in June 2024. To date, patient reach varies substantially across the 15 sites, ranging from 47% (84/178) to 100% (174/174) of eligible patients receiving the conversation aid. Observed trends suggest that clinician engagement and communication with SHAIR Collaborative leadership promotes adoption and active use of the conversation aids.
Discussion: A learning collaborative demonstrates promise in promoting adoption and use of conversation aids among breast cancer surgeons and clinical teams, leading to substantial patient reach. However, the interactivity needed in learning collaboratives, including communication with busy clinicians, requires thoughtful attention to maintaining participant engagement in key areas.
Conclusion: Given varied patient reach and learning collaborative engagement across sites, certain learning collaborative activities are associated with higher levels of implementation.
Implementation Process Fidelity of a Longitudinal, Pragmatic Study of the Collaborative Care Model
Authors
Emily Fu - The Warren Alpert Medical School of Brown University
Allison Carroll - Northwestern University Feinberg School of Medicine
Lisa Rosenthal - Northwestern University Feinberg School of Medicine
Jeffrey Rado - Northwestern University Feinberg School of Medicine
C. Hendricks Brown - Northwestern University Feinberg School of Medicine
Neil Jordan - Northwestern University Feinberg School of Medicine
Inger Burnett-Zeigler - Northwestern University Feinberg School of Medicine
Andrew Carlo - Northwestern University Feinberg School of Medicine
Sarah Philbin - Northwestern University Feinberg School of Medicine
J.D. Smith - University of Utah
Background: The Collaborative Care Model (CoCM) for anxiety and depression can improve mental health treatment access and outcomes for patients, including racial/ethnic minority populations. There is a dearth of research in real-world primary care settings concerning effective planning, implementation, and sustainment of CoCM.
Methods: This is a randomized roll-out type 2 effectiveness-implementation study of CoCM in 11 primary care clinics within a single academic health system. Each clinic had a 3-month Implementation Preparation period, a 12-month Implementation Period, and a 12-month sustainment period. We used the PRECIS-2 to assess and describe study pragmatism, and a study-tailored Universal SIC to measure implementation process fidelity across pre-implementation, implementation, and sustainability phases.
Results: The PRECIS-2 scored between 4 (rather pragmatic) and 5 (very pragmatic) for all categories. Across the 11 clinics, the pre-implementation phase had a mean(M)=268 days (SD=206), the implementation phase had M=427 days (SD=211), and the total duration had a M=536 (SD=219). All clinics had overlap between the pre-implementation and implementation phases, often due to “Stage 4: Staff hired and trained,” as several social workers provided services in multiple clinics. The total proportion of completed SIC activities ranged from 13 (41%) to 29 activities (91%; M=19.18, SD=4.94) and 7 clinics reached a final stage of Adherence (Stage 7) and 4 clinics achieved Competency (Stage 8).
Conclusion: This study was a rigorously designed randomized roll-out implementation trial that was rather to very pragmatic on the domains of the PRECIS-2. Evaluation using the SIC allowed us to track implementation processes across the 11 primary care clinics and examine activity and stage completion over the duration of the implementation period of the trial. Consistent with other studies, most clinics reached a final stage of Adherence and few achieved Competency. Competency was challenging given the required caseloads and graduation rates specified a priori.
Effectiveness of a Holistic Mental Health & Wellbeing Model for Educators: Timely Practice Improvement, Continued Dissemination, and Setting Policy Discussions
Authors
Jacob Gustaveson - American University
Elizabeth Demeusy - MedStar Georgetown University Hospital Center for Wellbeing in School Environments (WISE)
Megan McCormick - MedStar Georgetown University Hospital Center for Wellbeing in School Environments (WISE)
Background: High rates of teacher burnout have inspired a variety of novel and adapted mental health interventions (Agyapong et al., 2023). This intervention research is limited to the efficacy study of individual interventions and has not evaluated the implementation of more comprehensive mental health models that provide a broader range and combination of services to meet the diversity of educator mental health needs. We aim to bridge this large gap through our continuous evaluation of an educator wellbeing multi-tiered system of support (MTSS) which provides evidence-based services across different “intervention tiers” that match type and intensity of services to an individual's needs and preferences.
Methods: We used a quasi-experimental design to study the effectiveness of the MTSS program in meeting educator mental wellbeing needs. Two schools received the full implementation of the MTSS and two comparison schools received services from its lightest-touch intervention tier. We will assess means differences of burnout and its demonstrated correlates, including perception of supportive leadership, self-efficacy, and sense of belonging (Fleming et al., 2023). We also will be gathering service engagement and acceptability information within intervention schools to better understand avenues for continued implementation improvement.
Results: We will be gathering our end-of-year assessment in May 2024 for inferential tests. Similar to our process with the October 2023 baseline assessment (N=139), we will provide programmatic takeaways and engage in consultative discussions with stakeholders at the individual schools.
Conclusions: Our rapid presentation of results to school stakeholders will improve understanding and enable timely adjustments for the next school year. In the research front, this study will provide the very first effectiveness evaluation of a holistic co-located integrated mental health care model for educator wellbeing. We hope to insert newfound evidence into broader policy discussions of how we all collectively make more effective human capital investments to address educator burnout.
References
Agyapong, B., Brett-MacLean, P., Burback, L., Agyapong, V. I. O., & Wei, Y. (2023). Interventions to Reduce Stress and Burnout among Teachers: A Scoping Review. International Journal of Environmental Research and Public Health, 20(9), Article 9. https://doi.org/10.3390/ijerph20095625
Implementing an Evidence-based Intervention for Homeless-experienced Veterans: Compatibility and Adoption
Authors
Lauren Hoffmann - VA HSR Center for the Study of Healthcare Innovation, Implementation & Policy (CSHIIP), VA Greater Los Angeles Healthcare System
Matthew McCoy - VA HSR Center for the Study of Healthcare Innovation, Implementation & Policy (CSHIIP), VA Greater Los Angeles Healthcare System
Agatha Palma - Center for the Study of Healthcare Innovation, Implementation, and Policy, VA Greater Los Angeles
Eleni Skaperdas - VA HSR Center for the Study of Healthcare Innovation, Implementation & Policy (CSHIIP), VA Greater Los Angeles Healthcare System
Erica Fletcher - Center for the Study of Healthcare Innovation, Implementation, and Policy, VA Greater Los Angeles
Maharshi Rawal - VA HSR Center for the Study of Healthcare Innovation, Implementation & Policy (CSHIIP), VA Greater Los Angeles Healthcare System
Sonya Gabrielian - VA HSR Center for the Study of Healthcare Innovation, Implementation & Policy (CSHIIP), VA Greater Los Angeles Healthcare System
Kristina Cordasco - VA HSR Center for the Study of Healthcare Innovation, Implementation & Policy (CSHIIP), VA Greater Los Angeles Healthcare System
Erin Finley - VA HSR Center for the Study of Healthcare Innovation, Implementation & Policy (CSHIIP), VA Greater Los Angeles Healthcare System
Background: Implementation science frameworks frequently posit that successful implementation of an evidence-based intervention (EBI) requires perceived compatibility or “fit” between the EBI, setting(s), and those delivering and/or receiving the intervention. Critical Time Intervention (CTI) is a time-limited case management EBI for vulnerable populations experiencing care or housing transitions. The Department of Veterans Affairs (VA) is supporting implementation of CTI within partnering community agencies that provide six months of case management for homeless-experienced Veterans (HEVs) transitioning to permanent housing. To examine the interplay between adoption and compatibility in this initiative, we examined case manager and supervisor perspectives on CTI implementation one year after program launch.
Methods: We conducted semi-structured interviews with case managers (n=11) and supervisors (n=5) at 11 community sites nationwide. Using rapid qualitative analysis via structured interview summaries and a domain matrix, we identified changes made in case management and supervision practices, including elements of CTI adopted by sites, and considered how perceived compatibility of CTI emerged as both a challenge to and facilitator of implementation.
Results: Community partners most frequently reported adopting elements of CTI that included enhanced documentation practices, improved connection of HEVs to key resources (e.g., healthcare, housing, and finances), and a phased approach to case management, noted to allow for “more structure” in assisting HEVs toward independence. Where elements of CTI were partially or not adopted, sites reported a lack of perceived compatibility between the needs of higher-acuity HEVs served, goals and values of case managers and/or their agencies, and the recommended time-limited nature of CTI activities.
Conclusions: Some elements of CTI were more readily adopted than others by partnering sites, illustrating the power of ensuring perceived compatibility between EBIs and their intended settings, partners, and populations served. Future initiatives should consider investing in engagement and pre-implementation activities to ensure high compatibility for planned initiatives.
Initiating A Community-Engaged Approach to Mitigating Maternity Care Deserts in West Alabama
Authors
Holly Horan - University of Alabama at Birmingham
Abby Emerson - University of Alabama at Birmingham
Isabel Scarinci - University of Alabama at Birmingham
Larry Hearld - University of Alabama at Birmingham
Akila Subramaniam - University of Alabama at Birmingham
Martha Wingate - University of Alabama at Birmingham
Sharon Phelan - University of Alabama at Birmingham
Background: An estimated 36% of counties in the United States are designated as maternity care deserts (MCDs), affecting more than 2 million females of reproductive age and >146,000 infants, mostly in rural counties with high rates of racial and ethnic disparities in outcomes. MCDs are increasing and they are characterized by a lack of accessibility and availability of maternity care. MCDs are also associated with severe maternal morbidity and mortality. Researchers and federal authorities have highlighted the need for local, community-engaged partnerships to implement evidence-based strategies to mitigate health inequities in MCDs.
Methods: Community-engaged research (CEnR) is the philosophical framework of this investigation, and the Consolidated Framework for Implementation Research (CFIR) is the guiding conceptual framework. This research includes two participatory coalitions from two counties that are MCDs in West Alabama (West AL). The coalitions collaborated with the research team to co-design an interactive map as the template for additional community-based research that will lead to community-engaged intervention design. Community collaboration was systematically documented using the qualitative method, continuous procedural memoing.
Results: Capacity building and pre-implementation planning activities with the coalitions led to the development of a preliminary interactive map; a visual of the services, resources, barriers, and facilitators to accessing maternity care in each MCD and in West AL. A preliminary framework for community engagement within the context of implementation science (IS) research was co-created and operationalized.
Conclusion: Community-engagement is a key activity of IS research, however, there are no rigorous frameworks that guide their approach to measurement and evaluation. The CFIR can be used to ground the community-engaged process within the context of the intervention development. This study systematically, visually, and qualitatively documents the contribution of local coalitions and the efficacy of this approach in identifying contextually relevant solutions to improving the accessibility and availability of maternity care.
Apples and Oranges? Comparing De-Implementation in School-wide Interventions and Healthcare
Authors
Maria Hugh - Department of Special Education, University of Kansas
Catherine Corbin - University of Florida
Kelsey Smith - Department of Special Education, University of Kansas
Jessica Kidd - University of Florida
Background: Implementation science can only successfully promote the adoption and implementation of evidence-based practices (EBPs) if there is room to adopt these new practices. However, schools and educators are overwhelmed by increasing demands, citing low indicators for well-being, increased paperwork and behavioral support needs of students, and innovation fatigue. Educators are increasingly asked by implementation researchers, administrators, districts, and policy-makers to adopt new school-wide innovations, with little attention toward what responsibilities and practices can be removed to make room fosr these. The de-implementation of ineffective or low-value practices (LVPs) is becoming an increasing focus in healthcare and medicine. A translation of de-implementation outcomes, frameworks, strategies, and processes into the education context to decrease the use of LVPs and increase the use of EBPs is needed.
Methods: This project aims to create a conceptual model to guide school-based de-implementation research. We identified and reviewed published papers on de-implementation, de-adoption, or abandonment until saturation of concepts/content was reached (n = 46 papers). Through a narrative review, we summarize de-implementation outcomes, methods, frameworks, stages, and strategies used in de-implementation research in healthcare and compare them to the needs of de-implementation in schools when adopting a school-wide program.
Results: Healthcare de-implementation research primarily focuses on individual-level determinants, such as those from the Theoretical Domains Framework, and strategies that rely on multilevel contextual features (e.g., individuals, systems) that differ from those of educational settings. As a result, we propose a de-implementation conceptual model to apply when conducting implementation research for school-wide programs.
Conclusion: We propose that implementation researchers and practitioners attend to what they can support de-implementing when supporting the implementation of a new practice and consider potential de-implementation outcomes, methods, stages, and strategies alongside those of implementation when conducting research in educational contexts.
Multilevel Predictors of Engagement with Feedback-Based Learning Tools for Motivational Interviewing Skill Development in Behavioral Healthcare Settings
Authors
Manon Ironside - University of California, San Diego
Marisa Sklar - University of California San Diego
Cathleen Willging - Pacific Institute for Research and Evaluation
Mark Ehrhart - University of Central Florida
Gregory Aarons - University of California San Diego
Background: Motivational interviewing (MI) is an empirically supported treatment across diverse clinical contexts. MI skill development is often reinforced through trainings; however, participation in trainings does not guarantee the acquisition and maintenance of MI skills over time. Leveraging survey data from two trials designed to improve implementation climate for MI, we investigate provider- and supervisor-level predictors of provider engagement with two feedback-based learning tools to support quality MI in behavioral healthcare settings.
Methods: In study 1 (n=380 providers across n=60 clinics), providers could submit monthly MI session recordings for human expert review and feedback; in study 2 (n=121 providers across n=27 clinics), providers had access to an AI-based MI learning tool. Multilevel logistic regressions were used to estimate initial and ongoing engagement with the MI learning tools based on provider- and clinic supervisor-reported attitudes toward MI, provider-reported intentions to use MI, and supervisor reports of implementation climate within their clinic.
Results: In study 1, more supportive clinic-level implementation climate for MI increased odds that providers engaged with the expert rating learning tool at least once (OR=2.6, CI95=1.5-5.4). Provider and supervisor attitudes toward MI were not significantly associated with initial or ongoing engagement with the expert rating tool after accounting for supervisor-reported clinic implementation climate. In study 2, more positive provider and supervisor attitudes towards MI increased odds that providers engaged at least once with the AI-based learning tool (provider OR=3.3, CI95=1.0-10.7; supervisor OR=3.0, CI95=1.1-8.6). Across studies, none of the provider- or supervisor-level variables emerged as significant predictors of ongoing engagement with either learning tool.
Conclusions: Organizational-level support for MI implementation may have a greater influence on provider engagement with a human expert-based learning tool, while individual attitudes toward MI influence engagement with an AI-based learning tool. Research is needed to better understand factors that drive ongoing engagement with learning tools.
The Moderating Role of Change Fatigue on the Relationship Between Implementation Leadership and Implementation Attitudes
Authors
Alexandra Kandah - University of Central Florida
Mark Ehrhart - University of Central Florida
Gregory Aarons - University of California San Diego
Marisa Sklar - University of California San Diego
Background: Implementation leadership (i.e., the extent to which leaders are proactive, knowledgeable, supportive, and perseverant during implementation efforts) can be particularly impactful when seeking to improve subordinates’ positive attitudes toward implementation. The present study sought to establish support for this relationship within the context of implementation of evidence-based practices (EBP) in mental healthcare settings. Further, we investigated the potential negative implications of the experience of provider change fatigue, conceptualized as an overwhelming feeling of stress, exhaustion, and burnout associated with rapid and continuous organizational change. We posited that implementation leadership would be less influential in improving employees’ attitudes towards EBPs when employees are high in change fatigue.
Method: This quantitative study used archival data from a cluster randomized trial that tested the effectiveness of a leadership intervention (Leadership and Organizational Change for Implementation, LOCI) designed to promote evidence-based practice (EBP) Implementation. For the purpose of this study, data were utilized from treatment staff across three cohorts (N=120). Data were collected electronically via web surveys, administered using Qualtrics. Hypotheses were tested in SPSS using moderated multiple regression analysis.
Results: Analyses indicated that implementation leadership was significantly related to EBP attitudes (B=.459, SE=.150, p<.05). Further, we found that change fatigue moderated the relationship between implementation leadership and EBP attitudes (B=-.062, SE=.028, p<.05). More specifically, the relationship between implementation leadership and EBP attitudes was significant and positive at low (B=.332, SE=.099, p<.005), and medium (B=.212, SE=.062, p<.005) levels of change fatigue, but at high levels, the relationship was weaker and non-significant (B=.109, SE=.058, p=.069).
Conclusion: This study uncovered a boundary condition in which the positive effect of implementation leadership on EBP attitudes is weakened. Thus, we encourage organizations to take steps to reduce change fatigue.
Evaluation of the Effectiveness and Implementation of Community Health Worker-based Hypertension Screening and Referral in Eastern Uganda: A RE-AIM Framework Assessment
Authors
Andrew Marvin Kanyike - Mengo Hospital
Raymond Bernard Kihumuro - Way Forward Youth Africa Limited
Timothy Mwanje Kintu - Mulago Hospital
Lee Seungwon - University of Pennsylvania
Ashley Winfred Nakawuki - Busitema University
Kevin Apio - Way Forward Youth Africa Limited
Richard Katuramu - Busitema University
Background: Hypertension affects about 26.4% of Ugandans with suboptimal control due to limited qualified health workers, screening, and management resources. Shifting tasks to community health workers (CHWs) has improved maternal and child health outcomes in Uganda. We studied the effectiveness and implementation of a CHW-based screening and referral of hypertensive patients in Eastern Uganda.
Methods: This mixed-method study was conducted at Bugembe Town Council in Eastern Uganda. We trained twelve CHWs on hypertension and blood pressure measurement techniques using automatic machines and deployed them to screen community members. We determined the ability of trained CHWs to correctly identify high blood pressure and increased diagnosis of hypertension at the referral facility. The RE-AIM framework was used to evaluate the implementation outcomes after six months of intervention. We conducted in-depth interviews (IDIs) with CHWs (n=12) and community members (n=30) post-intervention. Quantitative data was analyzed using STATA 15.0 and IDIs thematically.
Results: Reach: The CHWs screened 5215 (96.6%) of the targeted 5400 participants with a mean age of 34 (SD: 12.3). Most (54.2%) had never had their blood pressure measured before. Effectiveness: Of those screened, 1167 (22.4%) had high blood pressure and were referred. CHWs and qualified health workers had a 95% agreement on blood pressure readings for referred patients (95% CI: 90.72% - 100%). The mean number of new hypertension diagnoses at the facility pre- and post-intervention was significantly higher (4.6 vs 12.7, p=0.0014). Adoption: All 12 (100%) CHWs were trained and adopted the intervention. Implementation: The intervention was feasible; however, a minority of referred patients reached the facility. Financial constraints and concerns about service delivery were the primary reasons cited. Maintenance: Training and integrating CHWs into the primary healthcare system will promote sustainability.
Conclusions: With training and appropriate resource allocation, CHWs in Uganda can contribute to hypertension screening and early detection.
Kwiis-hen-niip (Change): Improving Emergency Care in Remote Indigenous Communities
Authors
Alex Kent - University of British Columbia- Okanagan
Jim Christenson - University of British Columbia
Megan Muller da Silva - University of British Columbia
Nicole Malcomson - Nuu-chah-nulth Tribal Council
Jeannette Watts - Nuu-chah-nulth Tribal Council
Background: There are persisting inequities in access to emergency medicine and patient outcomes in remote Indigenous communities, where healthcare facilities are often poorly resourced and patient transfer can be time-intensive and dangerous. First responders can be an essential first point of contact and emergency care.
Kwiis-hen-niip (Change) is an implementation study in partnership with Nuu-chah-nulth Tribal Council, First Nations Health Authority, Island Health Authority, BC Emergency Health Services, among others. Through our collaborative efforts, we seek to enhance first responder care in four remote Nuu-chah-nulth Nations: Hesquiaht, Ahousaht, Tla-o-qui-aht and Kyuquot-Checleseht.
Methods: Kwiis-hen-niip conducted a needs assessment that involved an in-depth consultation process with Nuu-chah-nulth community members, local first responders, community leaders, nurses, physicians, Elders, traditional healers, and external agencies. We used participatory methods to co-define priority areas for emergency care improvement. In response to these priorities, partners are now working together to co-design, implement, and evaluate a comprehensive package of interventions to improve first responder care. The EPIS Framework (Aarons, Hurlburt, & McCue Horwitz, 2010) and the CFIR 2.0 (Damschroder, Reardon, Opra Widerquist, & Lowery, 2022) help inform our understanding of implementation process and the mediating determinants shaping implementation, with consideration for equity-centered implementation (Stanton, Ali, & the SUSTAIN Center Team, 2022).
Results: Community-level solutions are being co-created with first responder teams to promote local ownership, ensure cultural relevance, and contribute to long-term sustainment. The Kwiis hen niip team and its partners are making progress in providing the four Nuu-chah-nulth Nations with a more effective first response system, with access to remote help and rapid transportation when needed, delivered with cultural safety and community bystander support.
Conclusions: Kwiis-hen-niip is contributing to building the evidence base and strengthening capacity to continue advancing innovative change in first responder care in Nuu-chah-nulth Nations and among other rural, remote, and Indigenous settings.
References
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and policy in mental health, 38(1), 4–23. https://doi.org/10.1007/s10488-010-0327-7
a.*note: there is a typo in the abstract, citing this as 2010
Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022). The updated consolidated framework for implementation research based on user feedback. Implementation Science, 17(1), 75. https://doi.org/10.1186/s13012-022-01245-0
Stanton, M. C., Ali, S. B., & the SUSTAIN Center Team. (2022). A typology of power in implementation: Building on the exploration, preparation, implementation, sustainment (EPIS) framework to advance mental health and HIV health equity. Implementation Research and Practice, 3. https://doi.org/10.1177/26334895211064250
Adapting Concepts and Methods from Other Fields to Strengthen Implementation
Author
Bo Kim - VA Boston Healthcare System
Background: Implementation science adapts concepts and methods from other disciplines to better understand contexts, make decisions, and examine phenomena. Although accounts of such adapted concepts/methods are increasing, there is limited effort to explicitly delineate impetuses and considerations for adapting concepts/methods, as well as making clear which aspects of the originating disciplines’ concepts/methods are tried, abandoned, or modified for implementation science. This work aimed to devise a structured roadmap for adapting concepts/methods from other disciplines, so that implementation studies can optimally build on other studies’ prior cross-disciplinary adaptation successes and shortcomings.
Methods: By engaging an existing university-based colloquium of implementation scholars and practitioners, a five-phase roadmap (“adaptation journey”) was created for adapting concepts/methods from other disciplines. Matrixed Multiple Case Study (MMCS; Kim et al., 2020) for multi-site implementation evaluation was used as an example of an adapted method to demonstrate the application of each roadmap phase.
Results: Phase 1 (Identifying a need) noted the importance of traceably documenting cross-site heterogeneities for systematic comparison. Phase 2 (Selecting concepts and methods) drew on case study research's investigation of contemporary events, process analysis’ notions of standardization versus variation, and visualization's focus on data curation and presentation. Phase 3 (Synthesizing, adapting, and integrating) combined these concepts/methods into MMCS steps that utilize a sortable matrix to triangulate data, conduct pattern-matching, and address rival explanations. Phase 4 (Pilot testing and refining) learned from MMCS-applied studies to develop sharable tools for MMCS’ analytical steps. Phase 5 (Collaborating and obtaining feedback) occurred throughout Phases 1-4, incorporating input from MMCS’ co-developers and external implementation experts.
Conclusions: The roadmap serves as a framework for methodically documenting the journey of concepts/methods that get adapted from other disciplines. Future work is warranted to further assess the utility of this roadmap in enhancing the clarity and efficiency with which future cross-disciplinary adaptations are approached.
References
Kim, B., Sullivan, J. L., Ritchie, M. J., Connolly, S. L., Drummond, K. L., Miller, C. J., Greenan, M. A., & Bauer, M. S. (2020). Comparing variations in implementation processes and influences across multiple sites: What works, for whom, and how?. Psychiatry research, 283, 112520. https://doi.org/10.1016/j.psychres.2019.112520
Rural-specific Determinants of Evidence-based Miscarriage Care
Authors
Jamie Krashin - University of New Mexico Health Sciences Center
Victoria Trujillo - University of New Mexico Health Sciences Center
Jessica Rodriguez-Hernandez - University of New Mexico Health Sciences Center
Michelle Diaz - Cassie Health Center for Women
Analisa Villarreal - Cibola General Hospital
Ambroshia Mandagaran - Cibola General Hospital
Larissa Myaskovsky - University of New Mexico Health Sciences Center
Cathleen Willging - Pacific Institute for Research and Evaluation
Background: Pregnant women in rural New Mexican communities have less access to evidence-based interventions (EBIs) for miscarriage management than urban dwellers. We lack knowledge about the barriers and facilitators that rural healthcare organizations experience providing such EBIs, including mifepristone pretreatment before misoprostol and uterine aspiration. This qualitative investigation aims to understand determinants affecting the adoption of these EBIs in rural healthcare organizations to develop a rural-specific multifaceted implementation strategy.
Methods: We conducted in-person and virtual qualitative interviews with clinicians (n=8), clinical staff (n=6), and administrators (n=7) at two rural healthcare organizations (total n=21) between September 2023 and April 2024. We incorporated key constructs from the Exploration, Preparation, Implementation, Sustainment framework (EPIS) into our semi-structured interview guides. We deductively analyzed transcribed interview data by applying key EPIS constructs hypothesized to influence adoption of innovations in healthcare organizations.
Results: Facilitators in the inner context included substantial interest in offering the EBIs, leadership's prioritization of high-quality and accessible care, and the ease of scheduling operating rooms. However, abortion-related stigma shaped perceptions of innovation fit. Outer-context barriers included transportation and state and national policy-related challenges to recruiting and retaining clinical staff. Ultrasound availability emerged as a barrier and facilitator spanning inner and outer contexts. Bridging factors to overcome barriers included partnerships with insurers and academic programs. Additional solutions centered on stocking medications on-site, community education, job aides, and addressing state and national polices affecting retention of rural healthcare providers.
Conclusions: Despite barriers to EBIs, healthcare employees in rural-serving communities valued them and identified many facilitators and solutions to challenges. Their suggestions will be integrated with questionnaire and chart review data. We will bring the integrated results to a panel of healthcare and patient partners from the study locations and use implementation mapping to create a rural-specific multifaceted implementation strategy for EBIs.
But…How Does it Work Exactly? Plausible Middle-range Theories for Learning Collaboratives, Practice Facilitation and PDSA Cycles
Authors
Rebecca Lengnick-Hall - Washington University in St. Louis
Bryan Weiner - University of Washington
Callie Walsh-Bailey - Washington University in St. Louis
Rosemary Meza - Kaiser Permanente Washington Health Research Institute
Byron Powell - Washington University in St. Louis
Predrag Klasnja - University of Michigan, Ann Arbor
Gretchen Buchanan - Hennepin Healthcare Research Institute
Aaron Lyon - University of Washington
Cara Lewis - Kaiser Permanente Washington Health Research Institute
Background: Three common implementation strategies with roots in quality improvement (QI) are learning collaboratives, practice facilitation, and conducting cyclical small tests of change (‘PDSA cycles’). Although widely used, we still cannot often articulate exactly how these strategies work or what each of these strategies’ unique theory of change is when part of a bundled strategy package. To address this, we developed plausible middle-range theories using causal pathway diagramming.
Methods: Causal pathway diagrams (CPD) visually map out how a strategy addresses determinants and leads to changes in proximal, intermediate and distal outcomes. CPDs also allow users to explain mechanisms, and moderators that contextualize these determinant- strategy-outcome linkages. Our CPDs were iteratively developed and refined using research literature, research team meetings and presentations, and expert panels who rated the plausibility of each CPD.
Findings: Learning collaboratives address insufficient capacity to integrate the EBP into the organization (barrier) by creating a way to learn about the EBP and how to identify and address barriers alongside developers, experts and other implementers in real time (mechanisms). This, through a cascade of proximal and intermediate outcomes, leads to EBP sustainment. Practice facilitation addresses low internal practice change capability (barrier) and increases adherence to evidence-based guideline adherence (outcome) by engaging practice members in facilitated QI (mechanism). PDSA cycles address uncertainty around the feasibility or effectiveness of an implementation effort (barrier) through collective sequential evidence generation (mechanism), which leads to a shared articulation of a change model, adoption decision, and ultimately adoption (outcomes).
Discussion: Learning collaboratives, practice facilitation, and PDSA cycles are burdensome for implementers. Implementers and researchers should have a clear understanding of how they work, and why they should be chosen over other strategy options. These CPDs present plausible propositions that can inform study design decisions, encourage debate and refinement, and support deeper mechanistic theorizing in the field.
Planning-focused Learning Collaboratives: A Strategy to Adapt and Promote Adoption of System-level Interventions in New Settings and Contexts
Authors
Dennis Li- Northwestern University Feinberg School of Medicine
Lori DeLorenzo - Organizational Ideas
Adam Thompson - Organizational Ideas
Aaron O'Brien - Organizational Ideas
Jorge Cestou - Chicago Department of Public Health
David Kern - Chicago Department of Public Health
Nanette Benbow - Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine
Background: People with HIV who have complex needs (e.g., housing instability, mental health) face significant barriers to sustaining viral suppression within the US HIV care system. Low-barrier care (LBC) is a promising, efficacious care model designed for this population, comprising walk-in services, intensive support, high-value incentives, and coordination with public health systems. However, scale-out of this multilevel, multicomponent intervention can be challenging due to complexity and other factors.
Methods: We partnered with the Chicago Department of Public Health (CDPH) to select LBC for implementation among its funded clinics. Between 1/2023 and 3/2024, we conducted a planning-focused learning collaborative (cf. ones typically used for quality improvement) to co-design an adaptation of the LBC model with agencies and prepare for future implementation. Leveraging structural facilitators (e.g., funding timelines, external funding), we invited the 12 CDPH-funded agencies to participate.
Results: Eleven agencies joined the collaborative, which included four day-long learning sessions, consisting of expert-led and cross-agency information sharing around LBC components, and action periods, where agencies worked with coaches on potential instantiations. Agencies’ models were compiled to describe an overall LBC model for CDPH to include in its next funding announcement. Acceptability and feasibility of the collaborative and LBC were high and increased over time. Facilitators of successful engagement included promoting it as a CDPH-sponsored initiative; having CDPH engaged in the process; maintaining structured but flexible interactions; and focusing on the agencies’ priorities and user experience.
Conclusions: The learning collaborative provided a framework for agencies to exchange ideas, pilot approaches, and collectively shape an LBC model that is tailored to their contexts and responsive to their clients’ needs. This strategy offers unique time and space to intentionally plan for adaptation and promote adoption of a complex intervention and can be applied to other contexts and emerging innovations in HIV prevention and care.
Contextual Factors Related to School Providers’ Attitudes about Implementing EBPs
Authors
Ainsley Losh - University of Colorado, Anschutz Medical Campus
Katherine Pickard - Emory University
Nailah Islam - Emory University
Selena Valladares Ortiz - Emory University
Emma Chatson- Emory University
Judy Reaven - University of Colorado, Anschutz Medical Campus
Background: Schools are well positioned to increase access to evidence-based mental health services (EBPs). However, school providers often face barriers to implementing EBPs (e.g., training support, time) that likely impact their attitudes towards and subsequent adoption of EBPs. This study examined contextual (e.g., leadership support, implementation climate) and individual (e.g., burnout) factors related to school providers’ attitudes towards delivering EBPs.
Methods: Elementary and middle school mental health and non-mental health providers (e.g., counselors, special educators; N=65) were enrolled in an ongoing randomized trial comparing anxiety interventions for autistic students. Providers completed the Evidence-Based Practice Attitudes Scale (Aarons, 2004) along with measures of burnout (Emotional Exhaustion subscale of the Organizational Social Context Questionnaire; Glisson et al., 2008), leadership support for EBP use (Implementation Leadership Scale; Aarons et al., 2014), and organizational focus on, educational support for, recognition of, and rewards for EBPs (Implementation Climate Scale; Ehrhart et al., 2014). Bivariate Pearson correlations and stepwise multiple regression were conducted to identify predictors of school providers’ attitudes towards EBPs.
Results: Provider attitudes towards EBPs were significantly correlated with burnout (r=-.27, p<.05), leadership support of EBPs (r=.43, p<.001), organization focus on EBPs (r=.48, p<.001), and organization recognition of EBPs (r=.35, p<.01). In a stepwise multiple regression (R2=.28, F(2,56)=10.71, p<.001), only organization focus on EBPs (β =.35, p<.05) and leadership support for EBPs (β=.26, p<.05) predicted provider attitudes about EBPs.
Conclusions: Providers had more positive attitudes towards EBPs when they had more leadership support of EBPs and when there was a positive school culture surrounding EBP use. Provider burnout and other contextual factors (e.g., rewards/recognition for EBPs) did not predict attitudes about EBPs. Findings suggest the importance of promoting a school climate and leadership structure that emphasizes and supports EBP use to enhance school providers’ positive attitudes about EBPs and subsequent implementation of these practices.
References
Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research, 6, 61-74. https://doi.org/10.1023/B:MHSR.0000024351.12294.65
Aarons, G. A., Ehrhart, M. G., & Farahnak, L. R. (2014). The Implementation Leadership Scale (ILS): Development of a brief measure of unit level implementation leadership. Implementation Science, 9, 1-10. https://doi.org/10.1186/1748-5908-9-45
Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2014). Assessing the organizational context for EBP implementation: The development and validity testing of the Implementation Climate Scale (ICS). Implementation Science, 9(1), 1-11. https://doi.org/10.1186/s13012-014-0157-1
Glisson, C., Landsverk, J., Schoenwald, S., Kelleher, K., Hoagwood, K. E., Mayberg, S., Green, P., & Research Network on Youth Mental Health. (2008). Assessing the organizational social context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health and Mental Health Services Research, 35, 98-113. https://doi.org/10.1007/s10488-007-0148-5
A Scoping Review of Theory Use in Peer Support Interventions in Behavioral Health
Authors
Rosemary Meza - Kaiser Permanente Washington Health Research Institute
Camilo Estrada - Kaiser Permanente Washington Health Research Institute
Rasheed AlRasheed - University of Washington
Noah Triplett - Yale University
Clara Johnson - University of Washington
Background: Adjunctive interventions target intervention recipients to increase motivation, self-efficacy, engagement, and adherence to clinical interventions. This, in turn, can improve the implementation of evidence-based treatments. Peer support (PS) is an adjunctive intervention in which individuals with lived experience of mental health recovery to provide non-clinical, strengths-based support to recipients of mental health services. The availability of PS in public mental health programs is associated with a 6-19% increase in outpatient visits and reductions in service use disparities. However, reviews of PS interventions show mixed effectiveness and call for a theory-driven understanding of how PS works to improve intervention development and effectiveness.
Method: We conducted a scoping review of RCTs of peer support interventions in behavioral health. The purpose was to identify 1) what constitutes PS in behavioral health, 2) what theories have informed the development of PS interventions, and 3) to what degree do PS components align with the theories proposed?
Results: Among 2490 articles identified, 57 met criteria for inclusion. Most (95%) described PS components, but these were often poorly operationalized. For instance, “providing support” was the most frequent component (52%). Twenty studies (35%) reported a theoretical rationale for how PS works. Of those, 9 (16%) drew on an existing theory or model, while eleven (19%) provided a general rationale. Most studies demonstrated low alignment between the stated theory and description of PS components (55%). This was due to poorly articulated theories, poorly described PS components, or components that did not align with stated theories. The remaining studies demonstrated moderate (30%) or high (15%) alignment between theory and components.
Conclusion: Despite their potential for improving equitable implementation, PS interventions would benefit from clearer operationalization of their intervention components and greater alignment between those components and their theory of change. Relevant theories and future directions are discussed.
Stakeholder Insights Improve Digital Intervention Acceptability and Engagement
Authors
Vijaya Nandiwada-Hofer - Arizona State University
Sydni Basha - Arizona State University
Joanna Kim - Arizona State University
Introduction: Digital health interventions have been heralded as the solution to reaching and providing services to underserved populations (e.g., rural, impoverished, non-English speaking). However, for digital tools to be effective, they must be useable and acceptable for the target user. User experience research (UXR) is a crucial component of developing high quality and persuasive digital interventions. The current study describes the process of involving multiple stakeholders in UXR to conduct rapid-cycle refinements of a digital intervention to support caregiver home practice of evidence-based parenting skills. Study aims included (Aim 1) gathering UX feedback on the digital intervention's user interface, including visual presentation, interaction usability, and task flow and (Aim 2) evaluating users’ task completion rates to measure usability and identify areas for improvement in completion efficiency.
Method: We conducted in-person usability testing with five participants. All participants were parents who had previously participated in an evidence-based parenting intervention or interventionists themselves. Sessions consisted of an introduction (e.g., study overview, app introduction), usability session (e.g., guided tasks and scenarios with success tracking), and freeform phase (e.g., impressions/feedback). Study staff conducted debriefing sessions following UX session during which they discussed key findings and summarized action items to increase usability for the development team. Developers made rapid changes to the digital intervention to ensure iterative design improvement.
Results: UX feedback determined action items for improvement (e.g., icon and text changes, reorganization) and app aspects that users enjoyed (e.g., visual presentation, interaction usability) that were implemented by the app developers.
Discussion: The usability testing allowed for direct feedback from the users for the team and app developers to build a user-friendly app that was persuasively designed to promote task completion and usability for a future RCT. Plans to conduct UX with low-literacy populations are discussed.
Variability in Implementation Needs by Setting and Format (In Person vs. Digital) of a Relationship Education and HIV Prevention Program for Young Male Couples
Authors
Michael Newcomb - Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine, Chicago, IL
Ricky Hill - Northwestern University Feinberg School of Medicine
Dennis Li - Northwestern University Feinberg School of Medicine
Brian Mustanski - Northwestern University Feinberg School of Medicine
Background: Couple-based health-promotion programs have demonstrated effects on myriad mental and physical health outcomes. Such programs have been adapted for online delivery to enhance reach, adding complexity to community-based implementation. 2GETHER is a relationship education and HIV prevention program for young male couples with efficacious online, in-person, and hybrid formats. In the context of a hybrid type 1 effectiveness–implementation trial, we sought to understand how to prepare 2GETHER for future implementation in diverse settings.
Methods: We conducted 40 in-depth interviews with potential deliverers involved in selection of programming and/or service delivery from 5 types of organizations: LGBTQ community centers, AIDS-serving organizations, university/college LGBTQ resource centers, LGBTQ-affirming psychotherapy practices, and public health departments. We used the Consolidated Framework for Implementation Research to develop the interview guide and codebook around determinants of future implementation but with room for emergent themes. Two staff coded transcripts with consensus >.80.
Results: High perceived cost was a barrier across all settings (most frequently psychotherapy practices), but community-based and AIDS-serving organizations noted that grant-based funding opportunities could facilitate implementation. Staff capability around implementing digital programming was a barrier for some organizations (e.g., university centers) but a facilitator in others (e.g., community-based organizations). Most interviewees believed a fully automated, online intervention would be easiest to implement in their setting–except for university centers who preferred in-person. Psychotherapy practices, in particular, described enthusiasm for digital, if it could be delivered as an ancillary component to therapy or to couples on their waitlist.
Conclusions: Stakeholders across settings felt that 2GETHER would benefit their clients, but preferred format and potential determinants varied greatly by context. The digital version was not universally desired. As we consider scale-up, implementation strategies will need to be tailored to format and setting, which may have upstream implications for how these approaches can be funded.
Healthcare Professionals’ Perceptions of the Acceptability, Feasibility and Sustainability of Tailoring to Select Implementation Strategies
Authors
Aoife O'Mahony - Health Implementation Research Hub, School of Public Health, University College Cork, Ireland
Ana Contreras Navarro - Health Implementation Research Hub, School of Public Health, University College Cork, Ireland
Fiona Riordan - Office of Vice President for Research & Innovation, University College Cork, Ireland
Claire Kerins - Health Promotion Research Centre, School of Health Sciences, University of Galway, Ireland
Jane Murphy - Health Research Board, Ireland
Laura-Jane McCarthy - Health Implementation Research Hub, School of Public Health, University College Cork, Ireland
Geoffrey Curran - Department of Psychiatry, University of Arkansas for Medical Sciences, USA
Cara C. Lewis - Kaiser Permanente Washington Health Research Institute, USA
Sheena McHugh - Health Implementation Research Hub, School of Public Health, University College Cork, Ireland
Background: Tailored implementation strategies can be effective for improving professional practice. However, evaluation of tailoring tends to focus on the effectiveness of tailored strategies, rather than evaluating perceptions of the tailoring process itself, and different tailoring approaches are rarely compared to each other, hindering efforts to rioriti this approach. This study involved a mixed method evaluation of two tailoring processes to develop strategies to support the implementation of Dose Adjustment for Normal Eating (DAFNE), a patient education programme for type 1 diabetes. We explored healthcare professionals’ perceptions of acceptability, feasibility, and sustainability of the tailoring process.
Methods: Healthcare professionals (HCPs; nurses, dieticians and doctors) from 16 DAFNE centres participated in the tailoring exercises, identifying and rioritizing barriers to implementation and selecting strategies to address these. Two different tailoring approaches were used: a participatory approach which relied more on group discussion and a pragmatic approach that used surveys with limited group discussion. We conducted semi-structured evaluation interviews with participants after the tailoring exercise.
Findings: 62 HCPs engaged in tailoring, 42 of whom were interviewed (68% response rate). Most considered tailoring useful, as they valued the deliberate approach to service planning and the opportunity to reflect on practice. Group discussion, consensus-building and multidisciplinary involvement were deemed important, due to the differing perspectives involved. However, both approaches were considered time-consuming and frustrations were expressed with the repetitiveness of the questions, and the complex terminology used, particularly in surveys. Tailoring was considered to have potential to be sustainable, though possible challenges and refinements were highlighted.
Conclusions: Initial findings support the acceptability, feasibility and sustainability of tailoring to develop implementation strategies and suggest features of this process that HCPs value, and those that could be improved. These insights into how tailoring works in practice will be valuable in informing researchers and practitioners selecting tailoring approaches.
Barriers and Facilitators Influencing the Development, Implementation, and Sustainability of Patient Registries: A Scoping Review
Authors
Elizabeth O. Obekpa - Institute of Implementation Science, The University of Texas Health Science Center at Houston, School of Public Health
Jan A. Catindig - University of Texas Health Science Center at Houston, School of Public Health
Hugo B. Sanchez - Institute of Implementation Science, The University of Texas Health Science Center at Houston, School of Public Health
Joe R. Padilla - Institute of Implementation Science, The University of Texas Health Science Center at Houston, School of Public Health, Houston, TX
Ogochukwu R. Abasilim - University of Texas Health Science Center at Houston, School of Public Health
Stephanie L. Silveira - Institute of Implementation Science, The University of Texas Health Science Center at Houston, School of Public Health
Bijal A. Balasubramanian - Institute of Implementation Science, The University of Texas Health Science Center, School of Public Health, Dallas
Maria E. Fernandez - Institute of Implementation Science, The University of Texas Health Science Center at Houston, School of Public Health
Background: Patient registries are crucial for collecting disease-specific data and evaluating patient outcomes, offering significant real-world insights. However, their development and implementation encounter numerous barriers and facilitators influencing care quality and long-term sustainability. We aimed to present an overview of these barriers and facilitators.
Methods: The scoping review research followed the Arksey and O'Malley framework. A systematic search of English peer-reviewed articles published between 2000 and 2022 was conducted in PubMed, Medline (Ovid), the Cochrane Library, Web of Science, and Google Scholar. Five authors independently reviewed the articles and charted key information. Results were collated separately for rare disease (RD) and non-rare disease (non-RD) registries and compared.
Results: Ninety-five articles (52 RD and 43 non-RD) met the inclusion criteria and encompassed various diseases, including neuromuscular disorders and cancers. The geographical coverage of most registries was national, international, or multicontinental. The review identified that bias was frequently introduced by recruitment methods and registry design, especially in RD registries. Interoperability issues were more prevalent in RD registries, primarily stemming from the absence/lack of standards. Other barriers included a lack of ethical considerations that affected implementation, participation, and data collection and sharing, low motivation and interest, technological challenges, registry usability, lack of capacity (human resources, time, and effort), and absence or lack of funding, training, and standardized diagnostic tools. The most frequently reported facilitators included high motivation, interest, and participation of patients and practitioners, collaborative approach, data quality management, user-friendly design, participation in a learning network, training of registry users, incentives (financial or non-financial), and long-term funding.
Conclusions: Patient registries can be invaluable for observing disease progression, treatment and health outcomes, and assessing care quality. We identified modifiable factors influencing the development and implementation of patient registries to support successful and sustainable RD and non-RD registries.
The Help Your Keiki Website: Developing and Sustaining Effective Website Dissemination Tools for Caregivers
Authors
Tessa Palafu - The Baker Center for Children and Families/Harvard Medical School
Catherine Waye - The Baker Center for Children and Families/Harvard Medical School
Lucy Khaner - The Baker Center for Children and Families/Harvard Medical School
Zoe Primack - The Baker Center for Children and Families/Harvard Medical School
Casey Park - State of Hawaiʻi Department of Health Child and Adolescent Mental Health Division
Summer Pascual - The Baker Center for Children and Families/Harvard Medical School
Dorian Higashi - University of Hawaiʻi at Mānoa Department of Psychology
Tristan Maesaka - University of Hawaiʻi at Mānoa Department of Psychology, Evidence Based Services Committee, Center for Cognitive Behavioral Therapy
Trina Orimoto - University of Hawaiʻi at Mānoa Department of Psychology
Kelsie Okamura - The Baker Center for Children and Families/Harvard Medical School, University of Hawaiʻi at Mānoa John A. Burns School of Medicine Department of Native Hawaiian Health and Department of Psychology
Background: The mental health treatment gap is well documented, with lack of knowledge on where to access services being listed as a primary reason for not engaging in care (Child and Adolescent Measurement Initiative, 2021; US Department of Health and Human Services, 2021). Direct-to-consumer (DTC) marketing promotes disseminating evidence-based practice information directly to consumers to increase help-seeking behaviors (Becker, 2015). To create effective dissemination tools, it is imperative to consider the who, how, and what (Baker et al., 2021). The Evidence-Based Services Committee within the State of Hawaiʻi Department of Health Child and Adolescent Mental Health Division (CAMHD) developed the Help Your Keiki (HYK) website in 2009 to provide trustworthy mental health information to caregivers (Okamura et al., 2018). Aligned with DTC marketing aims, this study collected caregiver input on HYK to inform site updates and improvements. Method: Forty-one caregivers participated in a 35-minute semi-structured interview and think-aloud exercise, where participants verbalized their thoughts while trying to find the answer to a preselected mental health related question on HYK (Ericsson & Simon, 1998). Two independent coders (ICC = .61) deductively coded the think-aloud exercises and interviews into five domains: common pages, content, design and aesthetics, navigation, and where their answer was found. Further refinement to increase reliability is ongoing. Results: Preliminary analyses reveal caregiver preference towards simple and efficient navigation features, such as the incorporation of a strong search engine and common mental health topics listed on the home page. Participants also shared they would likely return to the site if presented with understandable and actionable information. Conclusion: Caregivers’ response suggests a propensity toward simple, efficient, and effective web designs that provide next steps in the help-seeking process. This poster will provide insight into caregiver perspectives that can inform the creation of guidelines for the successful development of website dissemination tools.
References
Baker, E. A., Brewer, S. K., Owens, J. S., Cook, C. R., & Lyon, A. R. (2021). Dissemination science in school mental health: A framework for future research. School Mental Health, 13. https://doi.org/10.1007/s12310-021-09446-6
Becker, S. J. (2015). Direct-to-Consumer Marketing: A Complementary Approach to Traditional Dissemination and implementation Efforts for Mental Health and Substance Abuse Interventions. Clinical Psychology : A Publication of the Division of Clinical Psychology of the American Psychological Association, 22(1), 85–100. Https://doi.org/10.1111/cpsp.12086
Child and Adolescent Health Measurement Initiative. (2021). 2021 National Survey of Children's Health (NSCH) data query. Data Resource Center for Child and Adolescent Health supported by the U.S. Department of Health and Human Services, Health Resources and Services Administration, Maternal and Child Health Bureau.
Ericsson, K. A., & Simon, H. A. (1998). How to Study Thinking in Everyday Life: Contrasting Think-Aloud Protocols With Descriptions and Explanations of Thinking. Mind, Culture, and Activity, 5(3), 178–186. https://doi.org/10.1207/s15327884mca0503_3
Okamura K.H., O, T.E., Mah A.C., Slavin, L.A., Rocco, S., Shimabukuro S.K., Michels M.S., & Nakamura B.J. (2018). Insights in Public Health: The Help Your Keiki Website: Increasing Youth and Caregiver Awareness of Youth Psychosocial Mental Health Treatment. PubMed, 77(8), 203–207.
U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration. (2021). Key substance use and mental health indicators in the United States: Results from the 2020 National Survey on Drug Use and Health (HHS Publication No. PEP21-07-01-003, NSDUH Series H-56). HTTPS://www.samhsa.gov/data/report/2020-nsduh-annual-national-report
Factors influencing implementation of an evidence-based secure firearm storage program: Results from qualitative inquiry
Authors
Mallika Pandey - Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine
Claire Waller - Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine
Jennifer Boggs - Institute for Health Research, Kaiser Permanente Colorado
Katelin Hoskins - Department of Biobehavioral Health Sciences, University of Pennsylvania School of Nursing
Shari Jager-Hyman - Department of Psychiatry, University of Pennsylvania Perelman School of Medicine
Christina Johnson - Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine
Melissa Maye - Center for Health Policy and Health Services Research, Henry Ford Health
Leeann Quintana - Institute for Health Research, Kaiser Permanente Colorado
Courtney Benjamin Wolk - Department of Psychiatry, University of Pennsylvania Perelman School of Medicine
Leslie Wright - Institute for Health Research, Kaiser Permanente Colorado
Celeste Pappas - Center for Health Policy and Health Services Research, Henry Ford Health
Arne Beck - Institute for Health Research, Kaiser Permanente Colorado
Alison Buttenheim - Department of Family and Community Health, University of Pennsylvania School of Nursing
Matthew Daley - Institute for Health Research, Kaiser Permanente Colorado
Debra Ritzwoller - Institute for Health Research, Kaiser Permanente Colorado
Nathaniel Williams - Boise State University School of Social Work
Brian Ahmedani - Center for Health Policy and Health Services Research, Henry Ford Health
Rinad Beidas - Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine
Background: Firearm injuries are the leading cause of death for young people. While the American Academy of Pediatrics recommends that pediatricians provide secure firearm storage counseling, this practice is not routinely done. Our team conducted a cluster randomized type III hybrid effectiveness-implementation trial (the "ASPIRE" trial; R01MH123491) comparing two implementation strategies to promote clinician delivery of an evidence-based secure firearm storage program (S.A.F.E. Firearm) during pediatric well-visits at two large health systems. The present analysis focuses on findings from qualitative interviews with constituents involved in the implementation of S.A.F.E. Firearm to elucidate factors affecting likelihood of program delivery.
Methods: We conducted semi-structured qualitative interviews with clinicians, clinic change agents (i.e., individuals who served as the study team's point of contact at the clinic), and health system leaders after the trial had ended. Interviews (N=37) sought to assess factors that affected participants’ likelihood of implementing S.A.F.E. Firearm. Interviews were professionally transcribed. Analysis used an integrated approach that included inductive and deductive coding.
Results: Preliminary findings suggest that several factors across levels of context increased likelihood of S.A.F.E. Firearm delivery, including cable lock visibility in clinics, constituent buy-in, tangible help in the form of intervention resources, and patient perspectives on the voluntariness of the intervention as a whole. At the clinic level, consistently visible cable locks served as reminders for clinicians to deliver the intervention and allowed families to voluntarily take a cable lock home. At the intervention level, parent handouts stressed the importance and benefits of secure firearm storage and provided more information to help families understand the intervention, impacting buy-in at the individual level.
Conclusion: These qualitative findings can inform future implementation of S.A.F.E. Firearm and other evidence-based practices that focus on potentially sensitive topics.
Measuring Mental Health Treatment Adherence Within and Across Sessions
Authors
Alayna Park - University of Oregon
Anna Bartuska - University of Oregon
Rachel Kim - The Baker Center for Children and Families/Harvard Medical School
Daniel Cheron - The Baker Center for Children and Families/Harvard Medical School
Background: Treatment adherence, or the extent to which clinicians deliver practices prescribed by an evidence-based treatment (EBT), is one of the most reported implementation outcomes and often serves as an indicator of mental healthcare quality. Treatment adherence is commonly measured using session checklists that capture whether a clinician delivered a prescribed practice in a given session. While measuring treatment adherence within sessions provides a snapshot of what is happening in the psychotherapy room (which is often useful for guiding supervision and consultation), measuring treatment adherence within and across sessions may offer a more comprehensive picture of mental healthcare quality. This study explored different ways of measuring adherence to the Modular Approach to Therapy for Children (MATCH), an EBT for youths with anxiety, depression, trauma, and conduct problems.
Methods: Community-based clinicians (n=218) reported on MATCH practices covered with 525 youth clients (65.0% White, 57.7% female) with anxiety across 6605 sessions (1-61 sessions per client, M=12.58, SD=9.21), as part of quality improvement initiatives led by a nationally recognized center facilitating implementation of youth EBTs.
Results: Assessment of treatment adherence within sessions found that MATCH practices were covered in 96.5% of sessions and core MATCH practices for anxiety (i.e., practices conceptualized to be essential for producing treatment effects) were covered in 26.5% of sessions. Assessment of treatment adherence within and across sessions found that 100.0% of clients received at least one MATCH practice, and these clients received MATCH practices in an average of 12.18 (SD=8.99) sessions over a treatment episode; 64.4% of clients received at least one core MATCH practice for anxiety, and this subsample of clients received core MATCH practices for anxiety in an average of 5.19 (SD=5.18) sessions over a treatment episode.
Conclusions: Findings highlight how measuring treatment adherence within and across sessions can produce complementary pictures of mental healthcare quality.
Syringe Service Programs’ Response to Xylazine Adulteration of Unregulated Drug Supplies: Insights from People Who Use Drugs and Syringe Services Program Staff
Authors
Marina Plesons - University of Miami Miller School of Medicine
William Eger - University of California San Diego
Erika Crable - University of California San Diego
Angela Bazzi - University of California San Diego
Maia Hauschild - University of Miami Miller School of Medicine
Corbin McElrath - University of Miami Miller School of Medicine
Cyrus Owens - University of Miami Miller School of Medicine
David Forrest - University of Miami Miller School of Medicine
Hansel Tookes - University of Miami Miller School of Medicine
Tyler Bartholomew - University of Miami Miller School of Medicine
Background: Xylazine, a veterinary tranquilizer, is increasingly contaminating the unregulated opioid supply across the United States. Due to its sedative effects when combined with opioids and its association with severe tissue necrosis, local emergence of xylazine carries important public health implications. We sought to explore how syringe services programs (SSPs) adapted their services to meet the needs of people who use drugs (PWUD) in response to the emergence of xylazine adulteration.
Methods: In June and July 2023, we conducted semi-structured interviews with PWUD and staff members recruited from the IDEA Miami SSP. Four researchers used deductive and inductive memos to develop codes, which two researchers then condensed into salient themes.
Results: From interviews with 17 SSP participants and eight staff, three themes emerged. First, the SSP reinforced its foundational harm reduction principles through information provision and low-threshold prevention strategies (e.g., by expanding wound care services according to evolving evidence and offering xylazine test strips (XTS)). Notably, though, different types of drug checking services (DCS; e.g., XTS, Fourier transform infrared) had variable acceptability and feasibility. Second, participants and staff attributed unmet service needs to the legal ambiguity of xylazine DCS, lack of information about xylazine, and limited supply of XTS. Finally, implementation considerations focused on confirming CS legality, considering the financial resources and staff capacity to expand SSP services, selecting the type of DCS based on its use-case (e.g., point-of-care decision-making vs. ongoing surveillance of the drug supply), and combining DCS with other services to provide comprehensive co-located care.
Conclusions: The emergence of xylazine prompted numerous SSP service adaptations, including the implementation of XTS. Findings from this study can inform efforts within other SSPs to equip PWUD with information about reducing health harms from novel adulterants in the unregulated opioid supply.
Clinic Key Players’ Evaluation of a Team-Based Telemedicine Intervention for Hypertension Management in Patients at High Social Risk
Authors
Sunit Chhetri - Wake Forest University School of Medicine
Srista Manandhar - Wake Forest University School of Medicine
Justin Kramer - Wake Forest University School of Medicine
Animita C Saha - Wake Forest University School of Medicine
Kimberly Wiseman - Wake Forest University School of Medicine
Alexandra Peluso - Wake Forest University School of Medicine
Jeff Williamson - Wake Forest University School of Medicine
Hayden Basworth - Department of Population Health Sciences, Duke University
Justin Moore - Wake Forest University School of Medicine
Yhenneko J Taylor - Wake Forest University School of Medicine
Yashashwi Pokharel - Wake Forest University School of Medicine
Background: Although clinically more effective than traditional clinic-based hypertension care, telemedicine interventions are currently less effective in patients at high social risk. Creating interventions responsive to clinics serving these patients is important for adoption.
Methods: We implemented a 12-week, team-based telemedicine intervention among 20 patients from two clinics in NC. The intervention involved home blood pressure monitoring (HBPM) by patients, HBPM-based pharmacotherapy, telemedicine-based self-management support by nurses, and social support by community health workers (CHWs) and social workers (SWs). The Exploration, Preparation, Implementation, and Sustainment and Health Equity Implementation Frameworks guided our study. At the study conclusion, we interviewed clinic key players using semi-structured interview guide and delivered Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), Feasibility of Intervention Measure (FIM), and Organizational Readiness for Implementing Change (ORIC) measures. We used inductive and deductive coding for thematic analysis. We merged themes with survey results during analysis/results interpretation using concurrent embedded mixed methods (Qual + quant) for convergence/complementarity.
Results: Two physicians, 3 advanced practice providers, 1 pharmacist, 4 nurses, 1 nursing leader, 1 clinic manager, 2 CHWs, and 2 SWs participated. They found the program, particularly its emphasis on self-management and social support, very appealing, while identifying demand for clinical staffing as an important barrier. Key players found all program elements (HBMP, telemonitoring use, pharmacotherapy, self-management and social support protocols, communication tools to providers) acceptable, appropriate, and feasible, and suggested restructuring nurses’ calls and its content. All expressed interest in continued participation if there was a choice. Survey showed AIM 4.56 ± 0.59, IAM 4.61 ± 0.50, FIM 4.44 ± 0.53 and ORIC 4.11 ± 0.78 (higher score is better, range 1-5).
Conclusions: Despite staffing barriers, clinic key players found that the team-based hypertension telemedicine intervention emphasizing self-management and social support acceptable, appropriate, and feasible in high-risk patients.
Exploring Potential Mechanisms Through Which Implementation Strategies Focused on Designing, Tailoring, and Planning Implementation Work
Authors
Byron Powell - Washington University in St. Louis
Rebecca Lengnick-Hall - Washington University in St. Louis
Rosemary Meza - Kaiser Permanente Washington Health Research Institute
Bryan Weiner - University of Washington
Predrag Klasnja - University of Michigan, Ann Arbor
Aaron Lyon - University of Washington
Gretchen Buchanan - Hennepin Healthcare Research Institute
Callie Walsh-Bailey - Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine
Cara Lewis - Kaiser Permanente Washington Health Research Institute
Background: Implementation strategies are often poorly matched to determinants, leading to calls for more systematic approaches to developing and tailoring strategies. This study demonstrates how group model building (GMB), tailoring strategies, and developing an implementation blueprint could work to improve the precision and efficiency of implementation efforts.
Methods: A causal pathway diagramming (CPD) approach was used to depict plausible mechanisms through which the strategies work to address key determinants and achieve proximal and distal outcomes. CPDs include preconditions and moderators that influence strategy effectiveness. The CPDs were developed and iteratively refined using research literature, research team meetings and presentations, and expert panels that rated the plausibility of each CPD.
Findings: GMB addresses incomplete knowledge of the system(s) and uncertainty about interventions’ and/or implementation strategies’ effects (determinants) by developing a shared mental model of the system, supporting learning and decision-making, and generating collective commitment (mechanisms). This leads to changes in proximal (e.g., assessment of feasibility) and distal outcomes (adoption decision). Tailoring implementation strategies addresses determinants related to poor alignment between determinants and strategies, heterogeneous determinants across implementers, and emergent and dynamic determinants by increasing the fit between implementation strategies and evolving site-specific needs, strengths, and preferences. This improves proximal (e.g., efficient use of resources) and distal outcomes (e.g., successful implementation). Finally, an implementation blueprint addresses disorganization and lack of clarity about key roles, activities, and resource needs, and lack of accountability (determinants) by structuring and prioritizing activities, enhancing role clarity, developing a shared mental model, and aligning expectations for follow-through and outcome attainment (mechanisms). This improves the efficiency of implementation (distal outcome).
Discussion: This study informs decisions about when to use these strategies, presents plausible hypotheses about how they work, and demonstrates the value of articulating mechanisms to ensure strategies are well-aligned with the needs and preferences of implementers and communities.
Sustainability Strategies for Tobacco Treatment Programs: Multi-Stakeholder Co-Design and Prioritization from the Cancer Center Cessation Initiative (C3I)
Authors
Edmond Ramly - Indiana University
Aisha Khan - Indiana University
Reid Parks - Indiana University
Mara Minion - University of Wisconsin-Madison
Magda Montague - University of Florida
Jennifer LeLaurin - University of Florida
Jennifer Bird - University of Wisconsin-Madison
Donna Shelley - New York University
Jamie Ostroff - Memorial Sloan Kettering Cancer Center
Graham Warren - Medical University of South Carolina
Ramzi Salloum - University of Florida
Background: Sustaining evidence-based tobacco treatment programs (TTP) can improve care for cancer patients. However, the sustainability of these programs over time is not well-studied, leaving concerns about long-term impact. We sought to develop a stakeholder-driven compilation of strategies targeting sustainability determinants and assess the perceived importance and feasibility of these strategies.
Methods: The National Cancer Institute (NCI) Cancer Center Cessation Initiative (C3I) funded TTPs in 52 cancer centers, with the expectation they would be sustained post-funding. The C3I Sustainability Working Group (22 representatives from 16 C3I sites) engaged in user-centered design including modified Delphi activities to develop, adapt, and agree on definitions, importance (10-pt scale), and feasibility (10-pt scale) of sustainability strategies. Co-design activities included 1) problem-framing, 2) strategy-generation to address Clinical Sustainability Assessment Tool (CSAT) determinants, 3) convergence by importance and feasibility, 4) prototyping, and 5) a strategy evaluation survey of all C3I programs contrasted individual and group assessments using descriptive statistics.
Results: Problem-framing and strategy-generation produced 83 strategies addressing the 7 CSAT domains (mean=11.8). Convergence produced an initial group prioritization of 10 strategies as high-, 21 as medium-, and 50 as low-priority. Prototyping refined the list to 66 strategies. The evaluation survey had 44 respondents representing 32 sites (61%) and prioritized 26 strategies as highly important (top 25%, mean>7.76) and/or highly feasible (top 25%, mean>6.40). The 9 strategies that were both highly important and feasible address 4 of the 7 CSAT domains, “engaged staff and leadership”, “engaged stakeholder”, “organizational readiness”, and “outcomes and effectiveness”, and do not address the “workflow integration”, “implementation and training”, and “monitoring and evaluation” domains.
Conclusions: A nationwide network of NCI-funded cancer centers co-designed strategies for sustaining tobacco treatment programs, prioritized by importance and feasibility. These strategies and their mechanisms for addressing sustainability determinants and improving sustainment outcomes will be evaluated in future work.
Characterizing Implementation Leadership and Climate Plans for Autism Interventions in Children's Mental Health Services and School Districts
Authors
Elizabeth Rangel - SDSU/UC San Diego Joint Doctoral Program in Clinical Psychology
Aubyn Stahmer - University of California Davis
Laurel Benjamin - SDSU/UC San Diego Joint Doctoral Program in Clinical Psychology
Devynne Diaz - University of California, San Diego
Allison Jobin- CSU San Marcos
Anna Lau - University of California Los Angeles
Lauren Brookman-Frazee - University of California San Diego
Background: The TEAMS trials consisted of two linked randomized Hybrid Type 3 implementation trials that tested implementation strategies paired with autism evidence-based interventions (EBI) in children's mental health agencies and schools (Brookman-Frazee & Stahmer, 2018). The TEAMS Leadership Institute (TLI) is a leader-focused implementation strategy targeting implementation leadership and climate to facilitate autism EBI implementation. TLI was effective in improving implementation and clinical outcomes. This study characterizes the forms and functions of implementation support strategies included in the TLI implementation plans for autism EBIs.
Methods: Data were drawn from 36 of 65 programs/districts randomized to the TLI condition. TLI included a 360-degree assessment of implementation leadership and climate, a pre-implementation workshop, data-informed development of implementation climate and leadership goals, and ongoing coaching. Qualitative Content Analysis was used to identify the forms (goals and actions) and functions (implementation climate or leadership subdomains) of the plans.
Results: Across the 36 programs, there were 53 plans (some programs participated in multiple cohorts). Most plans (98%, n=52) included at least one goal focused on functions within the implementation climate domain. The most common climate functions were “Recognition” (n=52), “Rewards” (n=36), and “Focus” (n=33). Common forms included 1) staff-informed and tangible rewards, 2) public recognition of staff efforts, and 3) public endorsement of EBI. Plans also included leadership goals (58%, n=31). The most common leadership functions were “Proactive” (n=28) and “Knowledgeable” (n=21). Commons forms included 1) proactively problem solving and 2) gaining hands-on experience in EBI.
Conclusion: Leaders developed implementation plans that prioritized rewards, recognition, and focus. They also concentrated on goals and actions that centered on proactive and knowledgeable leadership. The next phase of this study will analyze audio-recorded coaching calls by TLI leaders. These calls will be compared to the implementation plans to assess how goals and actions were employed.
References
1. Brookman-Frazee, L., & Stahmer, A. C. (2018). Effectiveness of a multi-level implementation strategy for ASD interventions: study protocol for two linked cluster randomized trials. Implementation Science, 13(1), 66. https://doi.org/10.1186/s13012-018-0757-2
Applying User-Centered Design Methods to Promote Sustainability, Scalability, and Reach of Digital Mental Health Tools for Families After Pediatric Traumatic Injury
Authors
Leigh Ridings - Medical University of South Carolina
Mia Rogers - University of Georgia
Caitlyn Hood - University of Kentucky
Hannah Espeleta - Medical University of South Carolina
Tatiana Davidson - Medical University of South Carolina
Ken Ruggiero - Medical University of South Carolina
Background: Digital health tools have potential to expand reach and scalability of mental health services for children and families after traumatic events. However, even the most novel interventions are limited if both the end user and contexts for implementation are not considered early in the development phases. User-centered design (UCD) promotes early and ongoing input from key constituents that can influence intervention implementation in real-world settings. This approach is particularly beneficial for intervention delivery in healthcare settings with higher resource constraints, such as pediatric trauma centers whose reach is often limited to the acute care phase of inpatient treatment. The objective of this presentation is to apply UCD methods to inform sustainability and reach of a technology-enhanced mental health intervention for caregivers and children after pediatric traumatic injury (PTI).
Methods: Guided by the Accelerated Creation-to-Sustainment Model, we used UCD techniques (interviews, questionnaires, personas, storyboards, think aloud, and usability testing) with caregivers of children hospitalized for PTI (aged 0-11 years) to evaluate and redesign (1) a daily SMS-facilitated symptom self-monitoring service (N=20) and (2) an mHealth app to promote education and coping ∼1-12 weeks post-PTI (N=10). Caregivers were asked about implementation strategies to enhance digital health tool sustainability.
Results: Caregivers were enthusiastic about the SMS service's support of parent-child mental health communication post-injury and the app's potential to facilitate community mental health connections. Suggestions to enhance sustainability included options to tele-connect with trauma providers and improving digital health integration.
Conclusions: UCD techniques are key to iteratively evaluate and design interventions to address the unique needs of target populations in a way that is acceptable and feasible to the settings that serve them. These findings have high applicability to other settings and highlight the importance of integrating key constituents throughout intervention design and implementation to maintain synergy between research, practice, and policy.
The Use of FRAME to Guide and Document the Process of Proactively Adapting an Integrated Total Worker Health® Program for Sign Language Interpreters
Authors
Gretchen Roman - University of Rochester
Cristina Demian - University of Rochester
Tanzy Love - University of Rochester
Reza Yousefi Nooraie - University of Rochester
Background: Communication access is a social determinant of health in Deaf communities; thus, there is heightened concern regarding the variables that impact the occupational well-being of sign language interpreters. Total Worker Health® (TWH) is a strategy that protects and promotes the health of workers. We aimed to adapt an integrated TWH program for sign language interpreters.
Methods: Eight 90-minute listening sessions were conducted virtually with sign language interpreters. Components of the previous TWH program, including 1) information on stress, coping, health and nutrition, 2) physical exercise and 3) worksite examination were explained. Participants shared work-related experiences that impacted their occupational well-being and were encouraged to add, remove, extend or condense components of the previous program. Transcripts were analyzed using rapid qualitative assessment and portions of the Framework for Reporting Adaptations and Modifications-Enhanced (FRAME) were used to guide and document the process of proactively adapting the integrated TWH program.
Results: Twenty-seven interpreters (aged 53.74±10.69 years, 81% female, 81% hearing, 85% white) participated. Content modifications were fidelity consistent with slight changes to worksite examination secondary to the variability across interpreting settings. In addition to maintaining the core components of the previous program, other components like establishing a community of practice, job selectivity and debriefing were suggested to impact the target intervention group with the goal of increasing occupational well-being, reach, appropriateness, acceptability and fit. Intervention delivery via brief pre-recorded asynchronous lectures, then synchronous one-hour active learning sessions virtually for eight weeks with some engagement in between for accountability were acceptable contextual modifications. Deaf interpreters preferred communication directly in sign language to provide primary access to the content.
Conclusions: The use of FRAME successfully guided and documented the process of adapting the content and context of an evidence-based intervention pre-implementation. The adapted TWH program will be used in a subsequent behavioral clinical trial.
Understanding Barriers, Facilitators, and Priorities for Mental Health Service Provision at Community Colleges: Findings from a Rapid Ethnographic Assessment and Barrier Prioritization at a Michigan Community College
Authors
Amy Rusch - University of Michigan, Ann Arbor
Alex Ammann - University of Michigan, Ann Arbor
Mira Wang - University of Michigan, Ann Arbor
Sara Abelson - Temple University
Shawna Smith - University of Michigan, Ann Arbor
Background: Half of community college (CC) students report clinically significant symptoms of anxiety or depression, yet only about 5% use campus-provided services, let alone evidence-based supports (EBPs). Treatment gaps are especially prevalent for minority and low-income students. To understand opportunities for improving CC student access to mental health care, this initiative used two novel methods to engage CC partners in identifying and prioritizing barriers and facilitators to CC student access to evidence-based mental health practices.
Methods: A Rapid Ethnographic Assessment (REA) was conducted at an urban, midsize Michigan CC. REA components included interviews with providers; focus groups with administrators, faculty, staff, and students; and observations of CC workflows and processes. Following REA, our team conducted a facilitated barrier prioritization to have CC members strategically rate and discuss a subset of identified barriers using three pre-established criteria of importance, equity impact, and addressability.
Results: N=23 students, 22 faculty/staff, six administrators, and four providers participated in REA. We focused our initial barrier prioritization on the six biggest barriers to students accessing mental health services: transportation, stigma, availability, knowledge, language, and insurance. N=16 administrators, staff and faculty rated these barriers from 1 (low importance/equity impact/addressability) to 4 (high importance/equity impact/addressability). While availability (3.34) and knowledge (3.35) had the highest average ratings, all six barriers had averages around 3.
Conclusions: Community-partnered implementation science efforts often require use of new methods for optimally collecting data around implementation determinants and their prioritization, particularly in resource constrained settings. Our use of two new methods, REA and barrier prioritization, offered mixed results when deployed at a CC. REA offered an opportunity for our team to quickly assess key determinants of EBP access across a broad range of decision-makers, but barrier prioritization provided less actionable insight as decision-makers rated identified barriers similarly.
The Impact of Centering Health Literacy and Equity in the Co-creation of a Patient Education Tool: Findings from the RePeAT Pilot RCT
Authors
Mechelle Sanders - University of Rochester
Mary Barnes - Community Health Worker Association of Rochester
Ronald Carthen - Catholic Charities Family and Community Services
Delories Griffin - Action for a Better Community
Rufina Sanchez - Action for a Better Community
Jacqueline Sweeney - Community Health Worker Association of Rochester
Michael McKee - University of Michigan, Ann Arbor
Kevin Fiscella - University of Rochester
Background: Healthy People 2030 emphasizes the need to improve health literacy at both personal and organizational levels. Approximately 80 million U.S. adults have limited or low health literacy (LHL). Studies have reported associations between LHL and uncontrolled blood pressure, suggesting LHL may affect adherence to evidence-based interventions (EBI). A scoping review found only 3 studies that rigorously reported on the role of health literacy in the implementation of EBIs for chronic diseases. Implementation science would benefit from practical, equity-focused strategies for addressing health literacy throughout the implementation process. This study describes the impact of centering health literacy and equity in the co-creation of a heart health decision-aid for patients with LHL in a pilot RCT.
Methods: Researchers (n=2) partnered with community health workers (CHWs) (n=5) to co-create a decision-aid about the ABCS of heart health (Aspirin, Blood Pressure, Cholesterol, Smoking Cessation). The PI sourced content from public domains. We met 4 times (90 minutes each) to revise the decision-aid. We used the validated Patient Education Materials Assessment Tool (PEMAT) and Flesch-Kincaid Grade Level (FKGL) tools to assess pre-post changes in understandability, actionability, clarity, and readability. We also calculated interrater agreement on PEMAT. Tenets of Equity-based Co-Creation (EqCC) guided the co-creation process.
Results: The baseline interrater agreement on the PEMAT was 0.69 (95% CI = 0.53–0.84), indicating adequate agreement. The mean PEMAT rating (81% to 100%) and FKGL (9th to 7th-grade level) improved. The total word count (51 to 106) and number of sentences (11 to 17) increased. Examples of changes included active voice, more infographics, and adapting the process for reviewing the decision-aid with participants. Adherence to EqCC was feasible and acceptable. For example, CHWs’ input was weighted more heavily to address perceived power imbalances.
Conclusion: Co-creation improved readability. Addressing health literacy and equity during co-creation is feasible.
Nudging HPV Vaccination Rates: Partnering with Software Developers and Pharmacies to Improve Vaccine Clinical Decision Support Systems
Authors
Jessie Schwartz - Fred Hutch Cancer Center
Jennifer Bacci - UW School of Pharmacy
William Calo - Penn State College of Medicine
Morgan Glascock - Fred Hutch Cancer Center
Kate Watabayashi - Fred Hutch Cancer Center
Parth Shah - Fred Hutch Cancer Center
Background: Community pharmacies are accessible, convenient, and acceptable to patients, holding promise to improve suboptimal HPV vaccination coverage. However, current vaccination workflows and software systems are not optimized for pharmacy staff to provide HPV and other adolescent vaccines to patients. We systematically identify and describe implementation strategies in the form of clinical decision support systems (CDSS) to facilitate proactive assessment and recommendation of HPV vaccines in pharmacies.
Methods: Guided by implementation frameworks (CFIR, TDF, IOF) and in partnership with two national software developers (n=5) and pharmacy staff (n=4), we conducted a series of eight focus groups to examine current practices, barriers, facilitators, and proposed modifications to optimize vaccination workflow. Data were analyzed using FRAME.
Results: Analysis identified nine key behaviors to aid pharmacy staff in providing HPV vaccination, resulting in the recommendation of 19 CDSS that could be built within the two software systems to optimize vaccination workflow. Proposed CDSS included: vaccine coverage assessment and setting vaccine uptake goals, predictive inventory management, tailored prompts and reminders for providers and patients, documentation of vaccine refusal or delays, and integrated scheduling systems. The CDSS were methodically characterized using FRAME and directly linked to behavioral and implementation outcomes. For example, CDSS in the form of PioneerRx Care Action templates can be programmed to notify staff of a due vaccine, supporting desired behavioral outcomes (i.e., assessing vaccination coverage and proactively soliciting HPV vaccines) and implementation outcomes (i.e., fidelity and adoption) within the pharmacy.
Conclusions: The framework-driven approach to inquiry and analysis resulted in clearly defined CDSS and their behavioral targets. This formative work operationalized implementation outcomes that should be measured in subsequent studies to effectively understand the impact of CDSS on vaccine uptake. The developed or modified CDSS hold promise in shifting vaccination workflows to a proactive process, reducing missed opportunities for HPV vaccination.
Using Rapid Assessment Procedure-Informed Clinical Ethnography (RAPICE) in School-Based Health Centers: Reducing Burden and Enhancing Data in Community-Based Implementation Science Research
Authors
Daniel Shattuck - Pacific Institute for Research and Evaluation
Cathleen Willging - Pacific Institute for Research and Evaluation
Laura Steele - Pacific Institute for Research and Evaluation
Elizabeth Dickson - University of New Mexico
Mary Ramos - University of New Mexico
Background: While critical to overcoming healthcare access barriers impacting LGBTQ+ youth, school-based health centers (SBHCs) are also commonly understaffed and under-resourced. In a four-year implementation trial to facilitate LGBTQ+ inclusive practices in SBHCs, we sought to balance scientific rigor and the burdens of participation using Rapid Assessment Procedure-Informed Clinical Ethnography (RAPICE). This approach shifted our reliance on qualitative methods with higher burden (e.g., scheduled formal interviews) to contextualize and expand on quantitative measures of implementation readiness.
Methods: Three trained anthropologists conducted iterative ethnographic observations and informal, unstructured interviews in SBHCs (n=8) assigned to the first of three implementation cohorts. This formative work was punctuated by RAPICE team meetings to discuss data collection and generate real-time recommendations for subsequent RAPICE activities. We used rapid qualitative analysis to establish baseline readiness conditions at the SBHCs and inform implementation action planning.
Results: RAPICE was effective in collecting in-depth qualitative data on readiness without unduly burdening participants, despite requiring substantial time and energy investments from researchers compared to other methods. Data revealed the nuances of applying LGBTQ+ inclusive practices not captured by other methods, including 1) how SBHC personnel navigate politically and socially conservative school communities where SBHCs are viewed negatively for providing confidential services and 2) the limitations of electronic medical records for documenting LGBTQ+ identity. They also underscore disjuncture between high, self-reported confidence among personnel in providing inclusive care and their disclosed lack of training. Importantly, interactions afforded through RAPICE fostered buy-in and engagement among SBHC personnel in other research (e.g., surveys) and implementation (e.g., action planning) activities.
Conclusions: RAPICE can help jumpstart implementation efforts in medical settings by identifying readiness issues affecting uptake of new practices for stigmatized populations. In addition to enriching the analysis of quantitative measures while reducing participant burden, this approach fosters relationships required for successful implementation.
Addressing the Evolving Opioid Crisis in Health Care Systems and Primary Care: A Synthesis of Findings from Implementation Studies Over the Past Decade
Authors
Sarah Shoemaker-Hunt - Abt Global
Ellen Childs - Abt Global
Background: Between 1999 and 2021, nearly 645,000 people died from an overdose involving an opioid (CDC). To support clinicians in addressing the opioid crisis, CDC and VA released (and recently updated) guidelines. Health systems pursued efforts to improve pain management, mitigate opioid harms, and improve OUD treatment the past 10 years. Implementation research has played a key role in understanding the most effective approaches to implement the evidence. This review examines implementation research on strategies to address pain, opioids, and OUD in health systems, and the determinants of success.
Methods: We conducted a review of the literature. We searched PubMed for implementation research conducted between Jan 2014 and March 2023 in health systems or primary care implementing pain, opioid prescribing, or OUD assessment or treatment. We reviewed, coded and analyzed the literature on implementation strategies and determinants and synthesized findings for each.
Results: We reviewed 49 peer-reviewed articles; most articles had been published since 2020. The most common settings were primary care (46.9%), emergency department (24.5%), and health systems (18.4%). Forty-two percent of studies reported on multi-component interventions and included more than one implementation strategy. The most common strategies included education/training (36.7%), updating/implementing a policy or guideline (36.7%), and audit and feedback (24.5%). Seventy-three percent of studies reported positive outcomes. Each ED implemented guidelines and improved opioid prescribing (decreased opioid dosages). We will update our review with literature published through April 2024, and present a synthesis of the findings on determinants.
Conclusions: Health care is expanding its strategies used to respond to the opioid crisis. Implementation research examining the effectiveness of implementation strategies is also advancing. Most interventions are multicomponent. While they may be thought to be comprehensive and likely more effective, it is also hard to understand which, if any, specific components are more effective and in which contexts.
Providers and Caregiver's Preferences for Trauma-informed Intervention: Results from a Best Worst Scaling Exercise
Authors
Sean Snyder - The Baker Center for Children and Families/Harvard Medical School
Stevie Grassetti - West Chester University of Pennsylvania
Brittany Rudd - University of Illinois Chicago Department of Psychiatry
Background: Evidence-based practices (EBP) for responding to traumatic stress in adolescents exist; however, the implementation of these practices within the juvenile-legal system (JLS) system is limited. This is problematic given the high rates of trauma experiences among youth in the JLS. To inform future implementation efforts, we sought to understand preferences for trauma-informed interventions among JLS service providers and service recipients (i.e., caregivers) through a choice experiment known as best-worst scaling.
Approach: A total of 95 participants (65 JLS service providers and 30 JLS service recipients) completed a best-worst scaling choice experiment. Participants evaluated 19 components of trauma-informed intervention developed through consultation with experts at the National Child Traumatic Stress Network. Count analysis and probability of choice scores identified the most preferred components of intervention and analysis of variance tested whether service provider and caregiver preferences differed.
Outcomes: Participants expressed a strong preference for programs that prevent youth from experiencing trauma, provide general trauma-informed intervention, prevent emotional problems after the experience, and are provided 30 days after the traumatic event. In comparison to service recipients, service providers expressed stronger preferences for the trauma programs to be 1:1 with the service provider, delivered in schools, delivered in residential settings, and low cost. In comparison to service providers, service recipients expressed stronger preferences that the program be delivered in the youth's home, in a traditional mental health clinic, or via self-help.
Next Steps: Knowledge of these preferences can be instrumental in planning and designing trauma interventions for JLS involved adolescents. These results helped to inform key policies recommendations for Pennsylvania's “Trauma-Informed State” initiative, and insights from the study can further guide implementation within this statewide initiative
Patterns and Predictors of Fidelity Trajectories for Therapists Trained in An Individualized Mental Health Intervention for Autism (AIM HI): Findings from a Hybrid Type III Implementation Trial
Authors
Kameron Stout - California State University Long Beach
Barbara Caplan - California State University Long Beach
Teresa Lind - San Diego State University
Lauren Brookman-Frazee - University of California San Diego
Background: An Individualized Mental Health Intervention for ASD (AIM HI) has been shown to reduce interfering behavior in children with ASD, an important treatment target for this population. Higher 6-month average AIM HI provider fidelity mediated these decreases in child-interfering behavior1, highlighting the importance of understanding fidelity predictors. Most implementation research focuses on predictors of EBI implementation outcomes (i.e., overall fidelity), rather than process (i.e., fidelity growth during training). This study aims to: 1) examine patterns of provider AIM HI fidelity trajectories over the 6 months training process and 2) investigate therapist-level (e.g., discipline, licensure, EBI attitudes), and child-level (child interfering behavior) as predictors of 6-month fidelity trajectories.
Methods: This study is a secondary analysis of Translating EBIs for ASD: Multi-Level Implementation Strategy (TEAMS) trial. Participants include therapists (N = 187; 43% Hispanic) and their clients with autism (N = 187; mean age: 9.7). Fidelity was measured with the AIM HI Observational Coding System3; a total adherence score was calculated at three training intervals (training months 0-2, 2-4, 5-6). Therapist demographics were collected at baseline. EBI attitudes were measured using the Evidence-Based Practice Attitude Scale at baseline4. Child-interfering behavior severity was measured at baseline using the Eyberg Child Behavior Inventory5.
Results: Multi-level growth mixture modeling revealed that provider AIM HI fidelity significantly improved over time (B= .220, p < .001). Therapist discipline was associated with fidelity trajectories (psychology/psychiatry > marriage and family therapy; B =. 162, p = .033). No other therapist-child factors predicted EBI fidelity trajectories.
Conclusion: This study highlights that provider fidelity increases over the course of AIM HI training, which includes ongoing consultation and performance feedback over six months1. Findings suggest that providers from certain disciplines (psychology/psychiatry) differ in rates of fidelity growth, suggesting that some AIM HI training may need to be emphasized differently per discipline.
Untapping the potential of Multilevel Modeling of Single-case Designs to Identify Active Ingredients of Implementation Strategies
Authors
Diondra Straiton - Center for Mental Health, University of Pennsylvania Perelman School of Medicine
Mariola Moeyaert - SUNY Albany
Jessie Greatorex - Michigan State University
Brooke Ingersoll - Michigan State University
Background: Little is known about the mechanisms by which implementation strategies like group consultation function. The first step in understanding strategy-mechanism linkages is to identify which discrete components can be considered “active ingredients” because they are related to distal implementation outcomes of interest (e.g., fidelity). Research designs like Multiphase Optimization Strategy (MOST) are resource-intensive and costly. Component analysis is a feasible option to identify potential active ingredients of implementation strategies.
Methods: Using a component analysis design, this study demonstrated the extent to which 3 group consultation components (i.e., case support, skill rehearsal, and feedback on videotaped sessions) were associated with improvements in clinician fidelity (i.e., manual adherence and competence) to an evidence-based parent coaching intervention for autism called Project ImPACT. Twenty clinicians across 6 agencies submitted weekly videotapes of sessions with 21 families of Medicaid-enrolled autistic children during a baseline period of 3-9 weeks, and while receiving 12 weeks of group consultation in groups of 3-5 clinicians per agency. Consultation components were delivered in isolation in 3, 4-week blocks. Two-level hierarchical linear modeling was used to estimate the effect of the intervention on fidelity, and to estimate whether this was dependent on consultation components.
Results: Multilevel models included 154 videotaped sessions. Average baseline manual adherence was 62.67%, which was well below the fidelity threshold (80%). Manual adherence did not change significantly from baseline. Competency scores for reflective coaching practices in the Feedback consultation phase were 0.32 points higher than those in the Baseline phase (5-point Likert scale); this was a statistically significant increase, t(62.12) = 2.65, p = .01. No other consultation components were associated with changes in competency domains.
Conclusions: Component analysis is a promising methodology to identify active ingredients of implementation strategies. Results suggest that feedback may be an active ingredient that activates a skill-building mechanism of consultation.
Assessing Readiness in Early Care and Education: Scale Properties of Organizational Readiness for Change and Organizational Readiness for Implementing Change
Authors
Taren Swindle - University of Arkansas for Medical Sciences
Dong Zhang - University of Arkansas for Medical Sciences
Julie Rutledge - Louisiana Tech University
Background: Organizational readiness describes the resolve, commitment, and support that members of an organization have for undertaking and sustaining an implementation effort.1 The current study examined the scale properties of two prominent measures of organizational readiness 2,3 in early care and education (ECE).
Methods: As part of a larger enhanced non-responder trial, ECE teachers (N=179) completed baseline surveys including: (1) an adapted Context Subscale of the Organizational Readiness for Change Assessment (ORCA, 1-5 scale)2 and (2) the Organizational Readiness for Implementing Change (ORIC, 1-5 scale)3 measure. Data analyses examined means, standard deviations (SD), internal consistency, and factor structure via confirmatory factor analyses (CFA).
Results: Careless responders (18%, N=32) were excluded from analyses. The ORCA subscales were slightly higher than those of the original (0.20–0.43 units) except the Resources scale which was more than 1 SD higher. SD were slightly smaller for 3 subscales and the overall scale (0.24 units or less); SD were slightly higher for Opinion Leaders and Resources. All Cronbach's alpha levels in the ECE sample were ≥ 0.76 and similar to the original sample (within 0.05) except the Leader Culture scale, which had a notably weaker internal consistency. CFA for the ORCA indicated adequate fit (RMSEA = 0.08, CFI = 0.95, SRMR = 0.05). For ORIC, standard deviations and internal consistency values were very similar (within 0.06 and 0.05 and units, respectively). However, means were higher in the ECE sample (1.5 SD higher). Fit indices did not reflect proper fit (RMSEA = 0.28, CFI = 0.77, SRMR = 0.09).
Conclusions: ORCA scales of Leader Culture and Resources may have different properties in the ECE sample than the original. ORIC means were also notably higher in the ECE sample than the comparative hospital sample, and fit indices did not reflect fit of the original factor structure.
2. Helfrich, C. D., Li, Y. F., Sharp, N. D., & Sales, A. E. (2009). Organizational readiness to change assessment (ORCA): development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) framework. Implementation science, 4, 1-13. https://doi.org/10.1186/1748-5908-4-38
3. Shea, C. M., Jacobs, S. R., Esserman, D. A., Bruce, K., & Weiner, B. J. (2014). Organizational readiness for implementing change: a psychometric assessment of a new measure. Implementation science, 9, 1-15. https://doi.org/10.1186/1748-5908-9-7
Systematic and AI-assisted Curation of Knowledge Ontology to Support Rapid Learning in Implementation Science
Authors
Jeffery Chan - University of New South Wales
Frank Lin - Garvan Institute of Medical Research
Minh Tran - University of New South Wales
Skye McKay - University of New South Wales
Shuang Liang - University of New South Wales
Natalie Taylor - University of New South Wales
Background: Rapid advance in medical discoveries has potential to improve patient outcomes. However, evidence-based innovations often fail to be integrated into routine practice by unidentified barriers and strategies. Even with evidence currently available in implementation trial publications, manual screening and extraction remain labour-intensive.
Methods: To facilitate the process of gathering evidence and identifying barrier-matched strategies for successful implementation, we applied the Technology Acceptance Model (TAM) to develop a novel computational framework for curating an implementation science (IS) ontology, guided by implementation models, theories, and frameworks. The TAM was applied in collaboration with computer and implementation scientists, and an oncology physician, based on the perceived ease of use and usefulness of the ontology. An artificial intelligence (AI)-enabled computational pipeline was designed to systematically structure IS-related concepts extracted from published literatures.
Results: This ontology-building pipeline involves interaction between algorithmic extraction and IS experts, collaboratively analysing published implementation studies from MEDLINE using a four-step process: (1) fine-turning a BERT-based transformer natural language processing (NLP) model using a minimum of 50 pre-annotated publications, (2) automating annotation by NLP named-entity recognition, (3) conducting manual review of annotations by implementation scientists, and (4) organising an ontology through consensus expert opinion. This yields IS-related entities, facilitating implementation information retrieval by aligning with annotations. The ontology is related to Theoretical Domains Framework (TDF) and Consolidated Framework for Implementation Research (CFIR) associated with cancer screening, diagnostics, treatment, and survivorship management, and the best-known barrier-matched strategies can be retrieved for evaluation. The multi-disciplinary approach fosters ongoing improvement of qualitative research accuracy as the literature grows, laying the foundations to predict success through automated extraction of implementation outcomes and enabling a continuous learning process.
Conclusions: Our novel ontology-building approach has a potential to systematically and efficiently index IS literature, aiding in the identification of effective implementation strategies to improve cancer care.
Relations Between TF-CBT Fidelity and Clinical Outcomes in Community Mental Health Clients
Authors
Noah Triplett - University of Washington
Rasheed AlRasheed - University of Washington
Celine Lu - University of Washington
Priya Dahiya - University of Washington
Lucy Berliner - University of Washington
Michael Pullmann - Department of Psychiatry & Behavioral Sciences, University of Washington
Shannon Dorsey - University of Washington
Background: Evidence-based treatment (EBT) fidelity has been identified in theory as an indicator of quality of care and a mechanism for achieving desired clinical outcomes; however, studies have yielded mixed findings on its relation to clinical improvements, often due to methodological limitations. Research is needed that investigates fidelity using rigorous, validated measures in usual care settings.
Methods: This study presents outcomes of a randomized trial of supervision strategies to support trauma-focused cognitive behavioral therapy (TF-CBT) implementation in community mental health centers. The trial compared the effects of two supervision techniques (i.e., symptom and fidelity monitoring [SFM] v. SFM + behavioral rehearsals) with supervision-as-usual. One aspect of fidelity—extensiveness—was examined using the Therapeutic Process Observational Coding System for Child Psychotherapy-Strategies. Youth and their guardians reported on clinical outcomes via the UCLA Post Traumatic Stress Disorder (PTSD) Reaction Index. Associations between adherence to each TF-CBT element and changes in symptoms were examined using two-level hierarchical generalized linear models. Benjamin-Hochberg corrections were used to adjust for multiple testing.
Results: Across all study conditions, youth (ß = -1.31, 95% CI: -1.82, -0.79, p < .001) and guardians (ß = -1.75, 95% CI: -2.26, -1.24, p < .001) reported significant reductions in PTSD symptoms. As the extensiveness with which clinicians delivered the trauma narrative increased, youth-reported PTSD symptoms decreased (ß = -0.33, p<0.05). No associations between other TF-CBT elements and clinical outcomes were found (all p>0.05).
Conclusions: Findings revealed a significant relationship between fidelity to TF-CBT's exposure-based active ingredient and improved clinical outcomes. Findings invite further research on the connections between adherence to EBT elements and clinical outcomes as well as additional research on the conceptualization and measurement of treatment fidelity to capture the mechanisms through which interventions may operate to influence clinical outcomes.
Leveraging Rapid Randomized Controlled Trials to Improve Pediatric Immunization Rates in a Federally Qualified Health Center
Authors
Sarah Tsuruo - NYU Langone
Leora Horwitz - NYU Langone
Holly Krelle - NYU Langone
William King - NYU Langone
Nate Klapheke - NYU Langone
Kyra Rosen - NYU Langone
Jeremy Lu - NYU Langone
Simon Jones - NYU Langone
Background: Health systems have long sought to improve their care and outcomes through QI programs. But often these programs rely on weak evidence or are weighed down by slow randomized controlled trials (RCT). Our team, the Rapid RCT lab, has combined the best of RCT and QI approaches to implement rapid randomization within NYU Langone. We have run over 20 short tests across 16 departments in our hospitals and clinics. Collaborating with our front-line staff, we’ve identified key needs and have worked iteratively to improve these pain points. One of three examples we’d like to present is improving pediatric immunization rates via text message reminders in our federally qualified health centers due to Covid-19 related drop-offs.
Methods: We tested multiple iterative rounds of this intervention and randomized in a 1:1 ratio (intervention to control). In the third and most successful round of this intervention, we sent two text messages 42 hours apart (6pm/noon). Texts were sent to the parents of 1,034 children (517 in each arm) 0-2 years old who were due or overdue for at least 1 out of 10 childhood vaccines.
Results: We found that adjusting the timing (after work hours) and frequency (sending two texts) of text messages can triple the number of vaccination appointments. Most appointments were made after the second text (p=0.01), indicating that it was the second text that prompted action. The intervention group also received significantly more vaccines (per child & overall) than the control.
Conclusion: Our lab aims to transform NYU Langone into a learning healthcare system by using rapid-cycle randomized controlled trials to test simple, pragmatic ideas. We've identified successful approaches in cheaply and quickly improving our care, as seen in our pediatric immunization intervention, and demonstrate how to leverage methodologies within and outside of implementation science to do so.
Partnering with Senior Centers to Guide Implementation Strategies for Lay-delivered Depression Care: Determinants, Implementation Outcomes, and Recommended Adaptations from an Ongoing Randomized Clinical Trial
Authors
Margaret Wang - University of Washington
Leslie Steinman - University of Washington
Jo Anne Sirey - Weill Cornell Medicine
Amber Gum - University of South Florida
Isabel Rollani - Weill Cornell Medicine
Nicole Crawford- University of South Florida
Brittany Blanchard - University of Washington
Alex Dillabough - University of Washington
Patrick Raue - University of Washington
Background: Approximately 10-25% of older adults in senior centers experience clinical depression but face barriers in accessing depression care. “Do More, Feel Better” (DMFB), a streamlined version of Behavioral Activation (BA), is an evidence-based program designed to be delivered by lay health providers and be integrated within senior centers. To prepare for broader program implementation outside the research context, we partnered with senior centers from an ongoing effectiveness multi-site trial to examine: 1) determinants of DMFB implementation using a train-the-trainer approach; 2) recommended adaptations to implementation strategies; and 3) pre/early implementation outcomes.
Methods: Stratified, purposive sampling identified eight senior center directors and four clinical social workers from 12 partnering sites. We used a mixed-method design with individual interviews and surveys. Using CFIR, our guide explored barriers, facilitators, and adaptations needed to implement DMFB using a train-the-trainer approach, and surveys included the Feasibility of Intervention (FIM), Acceptability of Intervention (AIM), and Intervention Appropriateness measures (IAM). We conducted thematic analysis of transcripts using rapid qualitative analysis and implementation science frameworks (CFIR, FRAME, IOF) for deductive coding of determinants, adaptations, and pre/early implementation outcomes. Survey data was descriptively analyzed.
Results: Preliminary analysis (n=3) indicated facilitators to DMFB implementation include fitting organizational mission and recruiting “the right fit” personnel. Barriers include lead trainer funding, stigma hindering engagement, and volunteer burnout. Incorporating cultural identity, simplifying lead trainer role, increasing clinical mental health support, and fostering community for DMFB lay providers were recommended adaptations for DMFB train-the-trainer strategies. FIM, AIM, and IAM scores show high feasibility, acceptability, and appropriateness of ongoing DMFB implementation without research support using a train-the-trainer approach. Data analysis (n=12) will be completed by September 2024.
Conclusions: This study will guide future research and practice of DMFB implementation strategies in senior centers and other aging service organizations.
Applying the Exploration, Preparation, Implementation, Sustainment Model to Assess Adoption of an Artificial Intelligence Platform for Fidelity Monitoring in Behavioral Healthcare
Authors
Cathleen Willging - Pacific Institute for Research and Evaluation
Daniel Shattuck- Pacific Institute for Research and Evaluation
Marisa Sklar - University of California San Diego
Erika Crable - University of California San Diego
Manon Ironside- University of California San Diego
Mark Ehrhart - University of Central Florida
Gregory Aarons - University of California San Diego
Background: Artificial Intelligence (AI) language models hold promise for monitoring fidelity to evidence-based practice (EBP) in behavioral healthcare. This qualitative investigation employs the Exploration, Preparation, Implementation, Sustainment (EPIS) model to assess inner-context, outer-context, bridging, and innovation-fit factors affecting adoption of a novel AI platform that rates provider-patient interactions to produce real-time fidelity and quality metrics for two widely used EBPs.
Methods: We conducted qualitative interviews and focus groups with frontline providers, supervisors, and administrators in 27 behavioral healthcare clinics tasked with using the AI platform over one year in a single state. We posed open-ended questions to elicit perceptions and experiences with implementation. The research team also participated in periodic reflections where they documented uptake-related issues. We deductively analyzed these data by applying key EPIS constructs hypothesized to influence the adoption of innovations in behavioral healthcare settings.
Findings: Inner-context factors affecting the platform's uptake centered on peer influence, with enthusiastic providers and supervisors championing and modeling its use among their colleagues. However, storytelling conveying negative experiences with the platform, insufficient incentivization, and fears about being evaluated diminished their interest in adoption. Cost concerns and worries about inadequate financial support from outer-context sources (e.g., government agencies/payors) raised doubt among clinic administrators about their ability to fully integrate the platform into quality assurance operations. Bridging factors included platform developer technical assistance, with researchers often troubleshooting emergent challenges. Innovation-fit factors presenting barriers to uptake included perceived incompatibility with provider work responsibilities and patient populations and the platform's limitations in decoding linguistic nuances and assuring the accuracy of AI-generated metrics.
Conclusions: The EPIS constructs reveal facilitators and barriers to adopting an AI platform intended to increase the quality of EBP delivery. Understanding these facilitators and barriers is essential to crafting implementation strategies and enhancing the platform to enable effective uptake and EBP provision.
Implementing Evidence for Real World Impact Requires Practical Guidance Informed by End-users
Authors
Gabrielle Zimmermann - Alberta SPOR SUPPORT Unit, University of Alberta
Background: Applying implementation science concepts in practice is challenging, given the complexity and sheer number of implementation science theories, models and frameworks. Additionally, health-system people tasked with leading implementation initiatives in complex health systems typically have high time pressures and limited capacity. In order to effectively move evidence into practice and create lasting impact, practical guidance is needed.
Methods: Our team synthesized implementation guidance from the field of implementation science to create a practical, functional guide informed by our team's experience working as consultants on implementation projects in Alberta. To further tailor the guide, focus groups were conducted with individuals leading implementation initiatives in Alberta's health system. In summer 2024, selected members of health-system implementation or improvement teams will be engaged to trial the guide. Feedback on user experience will be gathered and will inform improvements/updates to the guide.
Results: Comparing and contrasting existing implementation guidance highlighted common elements and identified gaps in each. Building on what we found in the literature and heard from focus groups, we created a comprehensive Implementation Guidebook adapted to fit the Alberta health-system context. Engaging with potential end-users increased the relevance and usability of the Guidebook. Based on user feedback, we incorporated commonly used methods and terminology from other overlapping disciplines often used in Alberta (change management and quality improvement). Additional findings on the acceptability, appropriateness, and functionality collected during focus groups were used to revise the Guidebook substantially. For example, there are now three levels tailored to different audience needs. Feedback from those who trial the guide in real time will be used to optimize and sustain usability of the Guidebook.
Conclusions: Adopting a user-focused approach has been instrumental in shaping the Guidebook for practical implementation support in Alberta. Additional feedback from people doing the work will add to its value/utility in practice.
