Proceedings of the Sixth Conference of the Society for Implementation Research Collaboration (SIRC) 2022: From Implementation Foundations to New Frontiers
Elizabeth H. Connors1, Jared K. Martin2, Gregory A. Aarons3, Melanie Barwick4, Alicia C. Bunger5, Tatiana E. Bustos6, Katherine Anne Comtois7, Margaret E. Crane8, Hannah E. Frank9, Tyler L. Frank10, Andrea K. Graham11, Clara Johnson12, Madeline F. Larson13, Bo Kim14,15, Sheena M. McHugh16, James L. Merle17, Kayne Mettert18, Sapana R. Patel19, Sheila V. Patel20, Taren Swindle21, Lisa Saldana22, Nicole A. Stadnick23,24,25, Clare Viglione26 & Byron J. Powell27,28,29
Author details:
1Division of Prevention and Community Research, Department of Psychiatry, Yale University, New Haven,
CT
2Department of Human Sciences, The Ohio State University, Columbus, OH 43210
3UC San Diego, Department of Psychiatry, UC San Diego ACTRI Dissemination and Implementation Science Center, La Jolla, CA 92039
4Research Institute, The Hospital for Sick Children; Department of Psychiatry, Faculty of Medicine, The University of Toronto
5College of Social Work, The Ohio State University, Columbus OH 43210
6Department of Psychology, Michigan State University
7Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA 98195
8School of Psychology, Temple University, Philadelphia, PA 19122
9The Warren Alpert Medical School of Brown University, Department of Psychiatry and Human Behavior, Providence, RI 02903
10Public Health Sciences, Washington University in St. Louis, 1 Brookings Dr., St. Louis, MO, 63130
11Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine, Chicago, IL
12Department of Psychology, University of Washington, Seattle, Guthrie Hall (GTH), 119A 98195-1525, Seattle, WA 98105
13CharacterStrong, Puyallup, WA 98373
14Center for Healthcare Organization and Implementation Research, VA Boston Healthcare System, Boston, MA 02130
15Department of Psychiatry, Harvard Medical School, Boston, MA 02115
16School of Public Health, University College Cork, Cork, Ireland
17Department of Population Health Sciences, Spencer Fox Eccles School of Medicine, University of Utah, Salt Lake City, UT 84108
18University of Washington; Kaiser Permanente Washington Health Research Institute (KPWHRI)
19The New York State Psychiatric Institute, New York, NY 10032
20Department of Psychiatry Columbia University Vagelos College of Physicians and Surgeons, New York, NY 10032
21Department of Family and Preventive Medicine, University of Arkansas for Medical Sciences; Little Rock, AR 72212
22Oregon Social Learning Center; Eugene, OR 97401
23University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center, La Jolla, CA, United States
24University of California San Diego, Department of Psychiatry, La Jolla, CA, United States
25Child and Adolescent Services Research Center, San Diego, CA, United States
26University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center, La Jolla, CA, United States; University of California San Diego Herbert Wertheim School of Public Health and Human Longevity Science
27Center for Mental Health Services Research, Brown School, Washington University in St. Louis, St. Louis, MO 63130
28Center for Dissemination and Implementation, Institute for Public Health, Washington University in St. Louis, St. Louis, MO 63110
29Division of Infectious Diseases, John T. Milliken Department of Medicine, School of Medicine, Washington University in St. Louis, St. Louis, MO 63110
Corresponding author:
Elizabeth H. Connors, Division of Prevention and Community Research, Department of Psychiatry, Yale University 389 Whitney Avenue, New Haven, CT, 06511 Email: elizabeth.connors@yale.edu
Acknowledgements:
We are deeply appreciative of the 22 on-site conference volunteers, 75 volunteer reviewers, 10 volunteer planning committee members, 18 volunteer board members, the awards committee’s 10 core members and 9 additional reviewers, several partner organizations, and numerous sponsors who supported SIRC 2022 and made this conference possible.
Funding:
Components of the 2022 SIRC conference were supported by sponsorships, but funding is primarily derived from registration fees. Conference organizers, leaders, and the planning committee supported SIRC as a volunteer service. Some authors have funding that supported their role, as follows:
• Dr. Connors was supported by NIMH K08MH116119
• Dr. Powell was supported by the National Institutes of Health (R25MH080916; R01CA262325) and the Agency for Healthcare Research and Quality (R13HS025632)
• The projects presented under the Center for Community Health described were partially supported by the National Institutes of Health, Grant UL1TR001442.
The content is solely the responsibility of the authors and does not necessarily represent the official views of AHRQ or the NIH.
Conflicts of Interest:
Byron Powell is Co-Editor-in-Chief and Bo Kim is Associate Editor of Implementation Research and Practice. They were not involved in any editorial decisions about this supplement.
Introduction
The Society for Implementation Research Collaboration (SIRC) is an international, non-profit professional society dedicated to facilitating partnerships among researchers, practitioners, policymakers and community partners to advance the public health impact of evidence-based practices. SIRC began in 2010 as the Seattle Implementation Research Conference series via conference grant awarded to disseminate training models and methods via interdisciplinary collaboration to advance the dissemination and implementation of evidence-based behavioral health treatments (NIMH 1R13MH086159-01A1; PI: Comtois). The first conference was in 2011, followed by conferences in 2013, 2015, 2017, and 2019. SIRC transitioned to a registered non-profit membership society in the United States in 2019. These proceedings detail the 6th SIRC Conference, “From Implementation Foundations to New Frontiers,” which was held in 2022 and celebrated the 10-year anniversary of SIRC as a community (a year late due to a COVID-19 delay).
This conference marked an important point in time to consider synergies in implementation science and practice to lead, support, and transform opportunities to accelerate the use of evidence domestically and internationally. Thus, the conference theme, “From Implementation Foundations to New Frontiers,” was designed to promote reflection on the past decade of what has been learned and established in implementation research collaboration and to inform future thinking about achieving more equitable, generalizable, and global impact. A primary goal of the 2022 SIRC Conference was to reconvene the implementation community in-person for the first time since the start of the COVID-19 pandemic to facilitate collaboration, shared learning, and inspiration. This conference provided an opportunity to learn and share about the cutting-edge advancements in implementation research and real-world application of implementation practice that endured and, in fact, proliferated during the pandemic.
The COVID-19 pandemic, the reckoning with racial injustices, and other global events since the last SIRC conference in 2019 illuminated the complex intersections across policies, organizations, and practices within behavioral health, health care, and human service systems. Although SIRC was initially established with the implementation of behavioral health interventions and settings in mind, the 2022 SIRC Conference intentionally welcomed presentations from all settings and contexts in which implementation science and research-to-practice collaborations and partnerships can be used to close the evidence-to-practice gap. We also sought to feature a full range of implementation research, practice, and policy. This includes community-engaged partnerships, evidence-based implementation practices, rigorous methods and innovations in capacity building and infrastructure development, dissemination and implementation strategies, designs, approaches to pragmatic measurement, and theories and frameworks.
We would like to highlight three prominent themes represented at the 2022 SIRC Conference that we regard as central to current and future directions in implementation research collaboration. These are 1) partnerships and building implementation infrastructure, 2) equity-explicit implementation, and 3) adaptive methods to optimize impact.
Partnerships and Building Implementation
Infrastructure
Strong and meaningful partnerships are the foundation of a successful, equitable, sustainable implementation infrastructure. Indeed, partnership and collaboration are inherent in SIRC’s mission and goals. The 2022 SIRC Conference program comprised a multitude of presentations and activities that emphasized the essential role of implementation support professionals, community-based organizations, front-line professionals doing the implementation, healthcare recipients and their families, policymakers, and other key stakeholders. Conference presentations highlighted the importance of partnerships for all stages of implementation and beyond to transform and change daily practices within a system, community, clinic, or other units of interest. This spirit was embodied by a team-based plenary presentation, “Beyond the Evidence: Community-Powered Transformation.” Experts at the UC San Diego Altman Clinical and Translational Research Institute Center for Community Health (Blanca Meléndrez, Amina Sheik Mohamed, Elle Mari, Shana Wright, Joe Prickitt, and Eric Hekler) presented a framework for centering authentic community partnership based on goals of inclusivity and building capacity to improve health equity. Blanca Meléndrez encouraged implementation researchers to leverage their privilege of institutional position to advocate for community priorities and “act with care…seek to understand…lead with love.” This plenary panel highlighted how strong community-academic partnerships that center local needs and strengths can lead to new knowledge and practical, large-scale impact. Community-led partnerships are consistent with values, methods, and best practices commonly represented in social work, community psychology, and public health that are highly applicable yet less often leveraged by implementation science.
Equity-explicit Implementation
Fostering authentic community partnerships is one type of equity-explicit implementation effort, because when our implementation approach is guided by local knowledge, experience, and collaboration, we are more likely to develop, discover and promote equitable processes and outcomes. Scholars and practitioners at the 2022 SIRC conference presented new insights about equity-explicit implementation strategies designed to improve outcomes for individuals and groups who disproportionately experience health disparities, extending the reach of interventions by tailoring implementation to unique communities and groups and expanding intervention targets to assess and address social determinants of health and other contextual factors that influence implementation processes and outcomes. This research illustrated the challenge of advancing inclusion and responsiveness within slow-moving institutional systems and contexts. Utibe Essien delivered a plenary presentation titled “Implementing Equity: A Path Toward Justice in Health,” in which he shared culturally relevant strategies for equitable implementation and a call to action to ensure we consider social determinants of health in our multi-level contextual frameworks. This equity-explicit implementation work has the potential to “push the envelope” in implementation and generate new insights by scaling out evidence-based interventions into new settings and addressing policies and structures that reinforce inequities. Invited panelists at the closing plenary, “Foundations and Frontiers of Implementation Research Collaboration,” emphasized numerous growth areas with respect to equity-explicit implementation for conference attendees to consider. These included ensuring authentic representation at implementation conferences and among SIRC leadership by community members, partners, and scholars who are from and live in the diverse array of communities and settings in which implementation work is conducted. Panelists offered calls to action to embed research in communities to inform study designs that stand to make the most impact. Moreover, it was noted that intentional dissemination about community-led work could shine a light on the excellent but often underrepresented efforts occurring in the community, that are often funded by non-traditional mechanisms and iteratively sustained despite shifting team members and local collaborations.
Innovative Methods to Optimize Impact
Featuring innovative, pragmatic methods in dissemination and implementation science has long been a hallmark of SIRC. Bridging the research-to-practice gap inherently gives way to new designs and methodological innovations as we seek to answer questions needed for real-world decision-making. This necessitates methods such as multiple case studies, adaptive and pragmatic designs, comparative methods, mixed methods, and hybrid studies that can handle tensions of timing demands across research and practice, align efforts across multiple systems, and account for contextual factors, costs, and baseline conditions. Several presentations featured work designed to explore these methods. A prominent theme since the 2017 SIRC Conference has been on developing a better understanding of mechanisms of implementation (Lewis, Stanick, et al., 2018). Understanding how strategies work to exert their effects, or the processes through which they achieve change will facilitate efforts to better design, tailor, and optimize implementation strategies for different populations and contexts (Geng et al., 2022; Lewis, Klasnja, et al., 2018; Lewis et al., 2021; Lewis et al., 2022). The 2022 SIRC Conference featured a plenary update on the study and importance of mechanisms, as well as an invited symposium and numerous submitted presentations representing work on implementation mechanisms. Much of this work has stemmed from an Agency for Healthcare Research and Quality-funded conference series in partnership with SIRC that is leading to a research agenda to advance research on implementation mechanisms (Lewis et al., 2021). Many of the products of that work will be published in a special collection of articles in Implementation Science and Implementation Science Communications (https://www.biomedcentral.com/collections/mechanisms). This body of work is undoubtedly a future frontier of implementation research collaboration.
Finally, the 2022 SIRC Conference included the debut of Multilevel, Adaptive Implementation Strategies (MAISYs) by plenary speaker Daniel Almirall. A MAISY offers a sequence of decision rules used to guide how best to adapt the provision of implementation strategies and offers a pragmatic method to optimize how we intentionally integrate speed of adoption, strategy sequencing, parsimonious strategy selection and resource constraints in our work. MAISYs also invite researchers to design studies without themselves in mind; that is, to develop a decision-making guide or framework to be used by the implementers (i.e., implementation practitioners, community service providers, policymakers, and/or clinical leaders). Innovative approaches that account for real-world factors affecting implementation decisions have the potential to build self-sustaining implementation capacity in systems. Dr. Almirall also acknowledged a growing concern in our field that implementation research may be inadvertently, and ironically, replicating the research to practice gap when strategies become too complex or costly (for more on this topic, see Beidas et al., 2022 and Westerlund, Nilsen and Sundberg, 2019). We regard this concern as an important topic to monitor and proactively address in current and future implementation research collaboration efforts.
Conference Summary
The 2022 SIRC Conference was attended by nearly 500 participants representing 14 countries (Australia, Canada, Denmark, Finland, Ireland, Netherlands, Norway, Poland, South Africa, Spain, Sweden, Ukraine, United States, and Zambia). Most (approximately 90%) attendees were based in the United States, representing 35 states and Washington, DC. However, 24% of accepted presentations reflected work conducted outside of the United States, regardless of where the implementation team lead(s) resided. We requested that authors self-report several other characteristics of their work at the time of submission to establish indicator baselines on identified areas of growth for SIRC. For instance, 67% of accepted presentations represented a research-practice/policy partnership, consistent with our emphasis on collaboration. We also wanted to balance the representation of research, practice, and policy; many accepted presentations represented multiple domains of research (84%), practice (52%), and policy (15%). As mentioned, we intended to broaden our programming beyond behavioral health contexts and interventions; accordingly, 44% of presentations were prevention-oriented, as the importance of prevention for improved health outcomes and lower healthcare costs is being increasingly recognized. Student-led presentations represented 17% of the accepted presentations, a metric important for SIRC to continue tracking as we seek to support and elevate the work of trainees who are the future of the field.
Preconference Open Workshops
SIRC 2022 held four pre-conference workshops on Thursday, September 8th, that offered attendees engaging, skills-based, interactive learning opportunities on a variety of topics. The workshop “Accelerate Your Academic Writing Success in Dissemination and Implementation Science” focused on concrete ways to improve writing structure and style for implementation science journals and how to effectively write with others on large teams, community partners, and interdisciplinary groups. The UC San Diego Altman Clinical and Translational Research Institute Center for Community Health and Editors from Implementation Research and Practice (the journal that SIRC publishes in partnership with Sage; https://journals.sagepub.com/home/irp) and Implementation Science (https://implementationscience.biomedcentral.com) facilitated the workshop.
The second workshop, “Advancing Implementation Science in HIV and Global Contexts,” engaged participants in a Strategies Café, which highlighted tested implementation strategies in global settings, shared methodological challenges, and ways to leverage implementation science and promote equitable implementation in global contexts. The workshop was facilitated by an international group of scholars associated with the ViiV Healthcare and NIH-funded HIV, Infectious Disease and Global Health Implementation Research Institute (HIGH IRI; https://sites.wustl.edu/highiri/).
A third workshop, “Developing Comprehensive Infrastructure to Support Evidence-Based Practice Implementation and Sustainability,” used interactive facilitation to share a series of guidance documents and resources for providers, intermediaries, and policy/funders to organize key concepts and strategies for the development of implementation infrastructure across provider, intermediary, and policy/funder levels. Presenters included experts from the United States Department of Health and Human Services, University of Washington, University of North Carolina-Chapel Hill, Washington State University, Columbia University, Public Health Seattle & King County, and the Annie E. Casey Foundation.
The fourth workshop, “Using Causal Pathway Diagramming to Understand Implementation Mechanisms,” provided information and mentoring on using causal pathway diagrams to select, operationalize, optimize, and prioritize implementation strategies (Lewis, Klasnja, et al., 2018, 2022). Presenters included experts at Kaiser Permanente, University of Michigan, and the University of Washington.
Invited Implementation Development Workshops
The Implementation Development Workshop (IDW) is a half-day event where researchers and practitioners from the Networks of Expertise are invited to present early works-in-progress and receive feedback from expert colleagues. This invited meeting focuses on maximizing the potential implementation science impact and methodology of implementation projects and research studies and building collaborations between participants. Feedback can have a helpful and substantial impact on the developing project and lead to a useful discussion of cutting-edge issues in implementation. The IDW at SIRC 2022 featured 12 invited presentations by practitioners and researchers that focused on implementation projects across a range of populations, including behavioral health, school-based mental health, cardiovascular disease, cancer prevention, obesity, and suicide prevention. Evaluation of the IDW revealed that most participants learned something new about the principles and methods of implementation science, what they had learned can be applied to their work, and the feedback and materials were helpful for advancing their work.
Main Conference Program
The main conference program occurred over two days and included four plenary sessions, 36 breakout sessions (symposia and conference workshops), and 120 posters. We received a record number of nearly 400 submissions. Accordingly, acceptance rates were 55% for the overall program (33% for workshops, 42% for symposia, 23% for oral presentations, 35% of oral presentation submissions accepted as posters, and 56% for poster submissions). Presentation submissions were submitted to a masked review process from a minimum of two reviewers, followed by final decisions by consensus of two program leaders (EC, BP) guided by reviewer ratings. Four invited plenary sessions complemented accepted presentations submitted (see themes discussed above).
The conference also hosted several special interest group meetings, including the Student and Trainee Meet and Greet, the SIRC Diversity Equity and Inclusion (DEI) Meeting, and the Practitioner Network of Expertise Luncheon. The Student and Trainee Meet and Greet provided a space for students and trainees to ask questions, network, and learn from 10 implementation science experts from various industries and regions. At the SIRC DEI Meeting, the SIRC DEI workgroup shared results from the SIRC Community Survey and hosted an open forum with over thirty SIRC practitioners and researchers. Topics discussed included amplifying the work of underrepresented scholars and those engaged in community-engaged work, increasing transparency and communication around action steps SIRC has taken to support DEI as an organization, creating a SIRC DEI officer board member role and ways to support mentorship of students and trainees from underrepresented minority backgrounds. We also hosted a luncheon for the Practitioner Network of Expertise to update existing and new members about the activities of each subnetwork: the intermediary, policy/funder, and provider subnetworks. The luncheon discussion focused on ways to foster growth and collaboration across subnetworks and a collaborative learning forum to discuss practitioner career advancement and professional topics. As an organization, SIRC has several investigator and practitioner Networks of Expertise that offer mentorship, career development opportunities, and a forum for discussing current issues in implementation research collaboration efforts within these affinity groups by role. A Meet the Editors luncheon was also held for conference participants to engage with editorial board members of the field’s top journals, including Global Implementation Research and Applications, Implementation Research and Practice, and Implementation Science.
Awards
We presented several awards to celebrate the implementation research- and practice-related efforts represented at the conference. First, sponsored by The Hospital for Sick Children (SickKids) in Toronto, Canada, we granted one student (Diondra Straiton) and one practitioner (Ahmad Firas Khalid) presenter a Conference Registration Award to defray some of the costs associated with attending SIRC 2022. Second, we awarded Conference Presentation Awards to acknowledge an outstanding student presentation (Noah Triplett); early career practitioner (Amy Doyle), and early career researcher (Katherine Hirchak) presentations; practice-focused (Ahmad Firas Khalid et al.) and research-focused (Shawna Smith et al.) oral presentations; and practice-focused (Chair: Rebecca Roppolo; Presenters: Will Aldridge, Jacquie Brown, Brian Bumbarger, & Renee Boothroyd; Discussant: Rohit Ramaswamy) and research-focused (Chair: Sara Becker; Presenters: Kelli Scott, Katherine Hirchak, Sara Parent, & Michael McDonell; Discussant: Rick Rawson) symposia. Third, we recognized one winner (Stepping Up Initiative & I.M. Stepping Up Study) and two finalists (Center for Public Health Systems Science at Washington University in St. Louis; UC San Diego Altman Clinical and Translational Research Institute Center for Community Health) of the SIRC Mission Award, each being awarded to a collaborative team advancing implementation research and practice consistent with the spirit of SIRC’s mission. All awardees were selected by a committee of implementation researchers and practitioners from the SIRC Networks of Expertise, rated based on established criteria for the collaboration between research and practice collaboration, equity focus, and potential impact.
Conclusion and Future Directions for
Implementation Research Collaboration
We believe the work and themes represented at SIRC 2022 signal impactful contributions to implementation research and practice on the horizon of our field. For example, almost all plenary speakers represented community-engaged implementation research and practice partnerships focusing on equity in implementation and/or innovative methods to close the knowledge-to-practice gap and drive true public health impact.
We also noticed increased emphasis on the implementation professional workforce that extends the impact of implementation strategies (such as facilitators, technical assistance providers, and effective organizational leaders) to include the important roles and functions these professionals have in the broader ecosystem of adopting, implementing, and sustaining evidence-informed innovations at scale. This emerging workforce is at the nexus of translating science to practice, and it is our hope that the training, networking, and resource opportunities of SIRC and other professional societies are widely accessible to and responsive to workforce needs and opportunities.
Relatedly, SIRC has had a longstanding commitment to supporting emerging and early career scholars as they navigate implementation research collaborations that often exist as a special interest field representing a wide array of disciplines. As a result, not all early career professionals who are implementation enthusiasts have natural environments and mentoring relationships to support their ongoing development in implementation. Therefore, we see SIRC and related professional organizations as having a critical role in nurturing, supporting, and mentoring trainees and professionals across the career span to retain their passion and talent in the field. The future of implementation research and practice relies on the next generation's hope, enthusiasm, and creativity. As emphasized by invited panelists in our closing plenary, “Foundations and Frontiers of Implementation Research Collaboration,” trainees and early career professionals in implementation research, policy, and practice are passionate about solving urgent social and health issues and they benefit from mentorship and support that encourage them toward creative solutions and authentic partnerships to advance the field. Inadvertently, our systems and structures in the implementation field, such as funding climates, research publications behind paywalls, traditional metrics for high-quality scholarship, and silos across institutional, human-serving, and community systems may be disempowering to early career professionals, silencing opportunities for innovation and real-world impact (see Maddox et al., 2022 for an example of how traditional promotion metrics are not well-aligned with successful implementation). It is the responsibility of SIRC and other related professional organizations to protect, support, and pave the way for early career professionals who will shape the future of the field.
The presentations at SIRC and feedback we received about the conference indicate a hunger for more work, methods, mentoring, and strategies for implementing policy. To advance our knowledge in this area, we might tap into the substantial literature and lessons learned in the field of public administration, where scholars have a long history of investigating policy implementation (Nilsen et al., 2013).
As an organization, SIRC is committed to advancing the field in these areas and promoting a community that fosters implementation research collaboration. This supplement includes the abstracts of the 2022 SIRC Conference presentations to disseminate the most up-to-date work occurring in our field. Consistent with the 10-year history of SIRC and our conference theme, these presentations represent the strong foundation of implementation research collaboration that has been developed to date and the future frontiers of what we might expect in the upcoming decade.
References
Beidas, R. S., Dorsey, S., Lewis, C. C., Lyon, A. R., Powell, B. J., Purtle, J., … & Lane-Fall, M. B. (2022). Promises and pitfalls in implementation science from the perspective of US-based researchers: learning from a pre-mortem. Implementation Science, 17(1), 1-15.
Geng, E. H., Baumann, A. A., & Powell, B. J. (2022). Mechanism mapping to advance research on implementation strategies. PLOS Medicine, 19(2), e1003918. https://doi.org/10.1371/journal.pmed.1003918
Landes, S. J., Kerns, S. E., Pilar, M. R., Walsh-Bailey, C., Yu, S. H., Byeon, Y. V., … & Franks, R. P. (2020). Proceedings of the Fifth Biennial Conference of the Society for Implementation Research Collaboration (SIRC) 2019: where the rubber meets the road: the intersection of research, policy, and practice-part 1. Implementation Science, 15(3), 1-5.
Lewis, C., Darnell, D., Kerns, S., Monroe-DeVita, M., Landes, S. J., Lyon, A. R., … & Karuntzos, G. T. (2016). Proceedings of the 3rd Biennial Conference of the Society for Implementation Research Collaboration (SIRC) 2015: Advancing efficient methodologies through community partnerships and team science. Implementation Science, 11(1), 1-38.
Lewis, C. C., Klasnja, P., Lyon, A. R., Powell, B. J., Lengnick-Hall, R., Buchanan, G., Meza, R. D., Chan, M. C., Boynton, M. H., & Weiner, B. J. (2022). The mechanics of implementation strategies and measures: Advancing the study of implementation mechanisms. Implementation Science Communications, 3(1), 114. https://doi.org/10.1186/s43058-022-00358-3
Lewis, C. C., Klasnja, P., Powell, B. J., Lyon, A. R., Tuzzio, L., Jones, S., & Walsh-Bailey, C. (2018). From classification to causality: Advancing Understanding of Mechanisms of change in implementation science. Frontiers in Public Health, 6(136), 1–6. https://doi.org/10.3389/fpubh.2018.00136
Lewis, C. C., Powell, B. J., Brewer, S. K., Nguyen, A. M., Schriger, S. H., Vejnoska, S. F., Walsh-Bailey, C., Aarons, G. A., Beidas, R. S., Lyon, A. R., Weiner, B., Williams, N., & Mittman, B. (2021). Advancing mechanisms of implementation to accelerate sustainable evidence-based practice integration: Protocol for generating a research agenda. BMJ Open, 11(10), e053474. https://doi.org/10.1136/bmjopen-2021-053474
Lewis, C. C., Stanick, C., Lyon, A., Darnell, D., Locke, J., Puspitasari, A., … & Landes, S. J. (2018). Proceedings of the fourth biennial conference of the Society for Implementation Research Collaboration (SIRC) 2017: implementation mechanisms: what makes implementation work and why? Part 1. Implementation Science, 13(2), 1-5.
Maddox, B. B., Phan, M. L., Byeon, Y. V., Wolk, C. B., Stewart, R. E., Powell, B. J., … & Beidas, R. S. (2022). Metrics to evaluate implementation scientists in the USA: what matters most? Implementation Science Communications, 3(1), 1-10.
Nilsen, P., Stahl, C., Roback, K., & Cairney, P. (2013). Never the twain shall meet?: A comparison of implementation science and policy implementation research. Implementation Science, 8(63), 1–12. https://doi.org/10.1186/1748-5908-8-63
Westerlund, A., Nilsen, P., & Sundberg, L. (2019). Implementation of implementation science knowledge: the research-practice gap paradox. Worldviews on Evidence-based Nursing, 16(5), 332.
PRE-CONFERENCE WORKSHOPS
Developing comprehensive infrastructure to support evidence-based practice implementation and sustainability: Preparing guidance for providers, intermediaries, policymakers, and funders
Authors
Dr. Caryn Blitz - Administration for Children and Families, US DHHS
Dr. Kate Comtois - University of Washington
Dr. Will Aldridge - University of North Carolina- Chapel Hill
Dr. Brittany Cooper - Washington State University
Dr. Sapana Patel - Columbia University
Dr. Tatiana Bustos - Public Health Seattle & King County
Dr. Maria Monroe-DeVita - University of Washington
Dr. Doyanne Darnell - University of Washington
Dr. Cynthia Weaver - Annie E. Casey Foundation
The Practitioner Network of Expertise (PNoE) Implementation Infrastructure Initiative (I3), with the support of the SIRC Board, is addressing the lack of infrastructure for the implementation of prevention and treatment interventions. Implementation infrastructure is an issue that cuts across the SIRC membership. Moreover, the issue serves as a focal point within each of the three PNoE sub-networks (provider, intermediary, and policy/funder) for advocacy and fundraising, practice, and scholarship. The I3 aims to develop a series of guidance documents and resources for the development of implementation infrastructure across provider, intermediary, and policy/funder levels. Within the past year, I3 kicked off with a series of three webinars to summarize what has already been documented about these issues and successful exemplars at state and national levels (for more information and recordings, visit https://societyforimplementationresearchcollaboration.org/sirc-projects/i3/). During summer 2022, the I3 steering committee will develop and administer Delphi surveys across the PNoE to generate ideas about infrastructure guidance related to each of the three system levels of interest. This preconference workshop will bring together PNoE members and interested conference participants for a “deep dive” into the Delphi survey results. Using nominal group or similar strategies, participants will begin to refine concepts that will guide initial drafts of I3 guidance documents. The preconference workshop will also build social connections among I3 participants and momentum for the next phase of the initiative. The I3 preconference workshop will be highly interactive, prioritizing facilitation to engage participants with Delphi survey results, the I3 initiative, and with each other. Participants will use their conceptual skills and applied experiences during facilitated group processes using the Delphi survey results and to begin to plan guidance document development. All PNoE members, interested conference participants, and similarly interested SIRC members are invited to participate in the workshop and to join future I3 activities.
References
1. Blitz, C. (Co-Chair), Brooke-Weiss, B., Bullock, H., Comeau, C., Comtois, K., Jaouich, A., Monroe-DeVita, M. (Co-Chair), Shelafoe, G., Sundar, P., Venti, A. (2019). Developing Comprehensive Infrastructure to support evidence-based practice implementation and sustainability: Where does SIRC go from here? Preconference session at the 2019 Society for Implementation Research Collaboration (SIRC) Conference. Seattle, WA.
2. Fagan, A. A., Bumbarger, B. K., Barth, R. P., Bradshaw, C. P., Cooper, B. R., Supplee, L. H., Walker, D. K. (2019). Scaling up evidence-based interventions in US public systems to prevent behavioral health problems: Challenges and opportunities. Prevention Science, 20(8), 1147-1168. doi: https://doi.org/10.1007/s11121-019-01048-8
3. National Academies of Sciences, Engineering, and Medicine. (2019). Fostering healthy mental, emotional, and behavioral development in children and youth: A national agenda. The National Academies Press. https://doi.org/10.17226/25201
Disclosures of Interest: None declared
Accelerate your academic writing success in D&I Science: from fledgling idea to publication
Authors
Dr. Bonnie Kaiser - Department of Anthropology and Global Health Program, University of California San Diego, La Jolla, CA
Dr. Elizabeth McGuier - University of Pittsburgh
Dr. Sonja Schoenwald - Oregon Social Learning Center
Ms. Clare Viglione - University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
Dr. Nicole Stadnick - University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
Dr. Borsika Rabin - University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center &
Dr. Lauren Brookman-Frazee - UC San Diego Dissemination and Implementation Science Center & UC San Diego Department of Psychiatry & Child and Adolescent Services Research Center
Dr. Gregory Aarons - University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
In our work at UC San Diego Altman Clinical and Translational Research Institute’s Dissemination and Implementation Science Center, there is high demand for guided D&I science writing sessions and workshops. This workshop will support attendees to: 1) Build a strategic writing agenda to strengthen D&I writing skills, 2) Develop structured writing plans to advance current manuscripts-in-progress, and 3) Optimize writing structure and style for leading D&I journals like Implementation Research and Practice. This 3.5-hour pre-conference writing workshop will include interactive lectures, guided individual and group activities, and ample opportunities for discussion. The first part of the workshop will focus on a review of D&I writing products (e.g., grants, journal articles, reports for community partners) and writing with others (e.g., large research teams, community partners, interdisciplinary groups). Through guided activities, participants will develop a ‘pipeline’ of their current writing projects, articulate priorities and writing goals for the next 1-2 years and define next steps for writing projects. The second part of the workshop will focus on strategies for increasing structure and accountability to enhance writing productivity (e.g., protected writing time, accountability groups, tracking, rewards). We will discuss resources and supports that participants can use after the conference to identify what strategies work best for them. Finally, Editors from Implementation Research and Practice and
Implementation Science will share concrete ways to align purpose, content, and style of manuscript writing style for leading publication outlets in the field of D&I. The workshop will close with a 30-minute writing block during which participants can edit a draft abstract or manuscript section, work on a strategic plan for their writing, take steps to establishing new practices like writing accountability, or other tasks that will allow them to apply concepts from the workshop.
References
1. Cassese, E. C., & Holman, M. R. (2018). Writing Groups as Models for Peer Mentorship among Female Faculty in Political Science. PS: Political Science & Politics, 51(02), 401–405. https://doi.org/10.1017/S1049096517002049
2. Galipeau, J., Moher, D., Campbell, C., Hendry, P., Cameron, D. W., Palepu, A., & Hébert, P. C. (2015). A systematic review highlights a knowledge gap regarding the effectiveness of health-related training programs in journalology. Journal of Clinical Epidemiology, 68(3), 257–265. https://doi.org/10.1016/j.jclinepi.2014.09.024
3. Kozar, O., & Lum, J. (2013). Factors likely to impact the effectiveness of research writing groups for off-campus doctoral students. Journal of Academic Language and Learning, 7(2), A132–A149.
4. Noone, J., & Young, H. M. (2019). Creating a Community of Writers: Participant Perception of the Impact of a Writing Retreat on Scholarly Productivity. Journal of Professional Nursing, 35(1), 65–69. https://doi.org/10.1016/j.profnurs.2018.07.006
5. Penney, S., Young, G., Badenhorst, C., Goodnough, K., Hesson, J., Joy, R., McLeod, H., Pickett, S., Stordy, M., Vaandering, D., & Pelech, S. (2015). Faculty Writing Groups: A Support for Women Balancing Family and Career on the Academic Tightrope. 45(4), 23.
6. Schick, K., Hunter, C., Gray, L., Poe, N., & Santos, K. (2011). Writing in Action: Scholarly Writing Groups as Faculty Development. Journal on Centers for Teaching and Learning, 3, 21.
7. Silvia, P. J. (2007). How to write a lot: A practical guide to productive academic writing (1 edition). Amer Psychological Assn.
Disclosure of Interest: Sonja Schoenwald is an Editor-in-Chief of Implementation Research and Practice.
Using Causal Pathway Diagrams to Identify Mechanisms of Implementation Strategies: Applications for Selecting, Operationalizing, Optimizing, and Prioritizing Strategies
Authors
Dr. Rosemary Meza - Kaiser Permanente Washington Health Research Institute
Dr. Predrag Klasnja - University of Michigan
Dr. Bryan Weiner - University of Washington
Dr. Michael Pullmann - University of Washington
Dr. Byron Powell - Brown School, Washington University in St. Louis, St. Louis, MO, United States
Dr. Cara Lewis - Kaiser Permanente Washington Health Research Institute
Implementation efforts often struggle to achieve the desired impact on implementation outcomes. Nearly two-thirds of efforts fail to achieve the intended change, and almost half have no effect on desired outcomes. A major challenge to implementation success is the lack of understanding of how and why implementation strategies work, that is, implementation mechanisms. Selecting a strategy appropriate for a specific barrier arising in a specific context remains one of the most vexing problems in both implementation practice and research. Methods are needed to advance precision in selecting and designing implementation strategies to achieve their desired impact on critical outcomes. This workshop introduces causal pathway diagramming – a method to (1) select and match strategies to barriers, (2) inform the operationalization of promising implementation strategies, (3) optimize the impact of existing strategies, and (4) prioritize strategies based on context. Causal pathway diagrams are graphical representations of the processes that connect an implementation strategy with the desired outcomes, in a given setting. This tool aims to help researchers and practitioners unpack ideas about how an implementation initiative is supposed to work, so the likely threats to effectiveness of implementation can be identified and addressed before and throughout the implementation process. Workshop activities will include an introduction to the causal pathway diagramming method, a live demonstration of the use of the causal pathway diagrams to match implementation strategies to prioritized barriers of implementation, and facilitated interactive small group exercises. Small groups will use implementation vignettes to practice developing causal pathway diagrams for three use cases: selecting implementation strategies to match identified barriers, optimizing multi-component strategies, and conducting formative evaluations of ongoing implementation projects. Large group discussion will address lessons learned, challenges, and future applications of the causal pathway diagramming method.
References
1. Hekler, E. B., Klasnja, P., Riley, W. T., Buman, M. P., Huberty, J., Rivera, D. E., & Martin, C. A. (2016). Agile science: creating useful products for behavior change in the real world. Translational behavioral medicine, 6(2), 317–328. https://doi.org/10.1007/s13142-016-0395-7
2. Lewis, C. C., Klasnja, P., Powell, B. J., Lyon, A. R., Tuzzio, L., Jones, S., … & Weiner, B. (2018). From classification to causality: advancing understanding of mechanisms of change in Implementation Science. Frontiers in public health, 6, 136.
3. Lewis, C. C., Hannon, P. A., Klasnja, P., Baldwin, L. M., Hawkes, R., Blackmer, J., & Johnson, A. (2021). Optimizing Implementation in Cancer Control (OPTICC): protocol for an implementation science center. Implementation Science Communications, 2(1), 1-16.
Disclosures of Interest: None declared
Advancing implementation science and practice in HIV and global contexts
Authors
Dr. Byron Powell - Brown School, Washington University in St. Louis, St. Louis, MO
Dr. Elvin Geng - Washington University School of Medicine
Dr. Rachel Sturke - Fogarty International Center, U.S. National Institutes of Health, USA
Dr. Nadia Sam-Agudu - Institute of Human Virology Nigeria and Institute of Human Virology, University of Maryland School of Medicine, USA
Dr. Hikabasa Halwiindi - University of Zambia
Dr. Kate Lovero - Columbia University
Dr. Radhika Sundararajan - Weill Cornell Medical College
Dr. Dan Wu - Sichuan University
Dr. Christopher Kemp - John Hopkins University
Dr. Whitney Irie - Boston College
Dr. Jennifer Velloza - University of California San Francisco
Dr. Melissa Mugambi - University of Washington
Dr. Maria Fernandez - University of Texas Health Science
This workshop will focus on emerging directions in implementation science in HIV, infectious diseases, and global contexts. Participants will be engaged in 1) a set of mini-keynote lectures from international leaders in the field; 2) an “Implementation Strategies Café,” which will offer insight into innovative implementation strategies deployed and tested in global settings (e.g., Sundararajan et al., 2021; Tang et al., 2021); and 3) a session focusing on methodological challenges in global contexts (e.g., Mugambi et al., 2021). Practical approaches to promote equitable implementation in global contexts will be emphasized throughout.
References
1. Mugambi, M. L., Baeten, J. M., Kinuthia, J., Hauber, B., Weiner, B. J., John-Stewart, G., & Barnabas, R. V. (2021). Design and evaluation of strategies to implement HIV prevention interventions for pregnant women in community pharmacy settings in western Kenya: A mixed-methods study protocol. BMJ Open, 11(12), e052311. https://doi.org/10.1136/bmjopen-2021-052311
2. Sundararajan, R., Ponticiello, M., Lee, M. H., Strathdee, S. A., Muyindike, W., Nansera, D., King, R., Fitzgerald, D., & Mwanga-Amumpaire, J. (2021). Traditional healer-delivered point-of-care HIV testing versus referral to clinical facilities for adults of unknown serostatus in rural Uganda: A mixed-methods, cluster-randomised trial. The Lancet Global Health, 9(11), e1579–e1588. https://doi.org/10.1016/S2214-109X(21)00366-1
3. Tang, W., Wu, D., Yang, F., Wang, C., Gong, W., Gray, K., & Tucker, J. D. (2021). How kindness can be contagious in healthcare. Nature Medicine, 27(7), 1142–1144. https://doi.org/10.1038/s41591-021-01401-x
Disclosures of Interest: None declared
SYMPOSIA
Bridging the evidence-to-practice gap for contingency management: One of the most effective, but least implemented interventions for persons with substance use disorders
Authors
Dr. Sara Becker - Northwestern University Feinberg School of Medicine
Dr. Kelli Scott - Northwestern University Feinberg School of Medicine
Dr. Katherine Hirchak - Washington State University
Dr. Sara Parent - Washington State University
Dr. Michael McDonell - Washington State University
Abstract
Contingency management (CM) is one of the most effective behavioral interventions for persons with substance use disorder, yet one of the least available interventions in community settings. The CM evidence-to-practice gap is so dire that in Fall of 2020, the New York Times ran a featured story called, “This addiction treatment works. Why is it so underutilized?” In recent years, the rise of overdoses associated with opioid and stimulant use has heightened interest in the implementation and scale-out of CM in usual care settings. This symposium will feature a series of contemporary, large-scale initiatives designed to promote the implementation of CM across systems of care. First, Dr. Kelli Scott will discuss the vital role of formative research to prepare for and sustain CM implementation. Specifically, she will share results of qualitative interviews with 43 opioid treatment program staff (representing 11 programs) that informed pre-implementation decisions as well as interviews with 18 staff (representing 18 programs) that informed CM sustainment strategies. Second, Dr. Sara Becker will present on Project MIMIC, a cluster randomized trial testing two distinct strategies to implement and sustain CM across 30 opioid treatment programs in New England. She will share key design decisions and present preliminary outcomes from 131 staff and 378 patients across 18 programs. Third, Dr. Kait Hirchak will share how two randomized trials with 272 adults from Alaska Native/American Indian communities informed development of a suite of dissemination tools. Fourth, Drs. Mike McDonnell and Sara Parent will discuss the implementation strategy used to roll-out CM across 30 programs in partnership with state authorities in Montana and Washington. Finally, Dr. Richard Rawson, a nationally renowned CM expert, will discuss how lessons learned from these initiatives have informed the ongoing $56 M investment in the implementation of CM as a reimbursable service across the state of California.
Title: Formative and Sustainment Evaluation to Inform Community Opioid Treatment Program Contingency Management Implementation
Background: Contingency management (CM) is an evidence-based intervention that involves incentivizing patients for achieving treatment goals. New initiatives have emerged throughout the United States to scale up CM in community opioid treatment programs (OTPs). The current study describes a state-wide formative evaluation with all OTPs in Rhode Island that assessed OTP context and guided changes to the CM intervention and implementation strategies. We also report upon sustainment evaluations with 18 OTPs that identified barriers and facilitators to CM sustainment.
Methods: The state-wide formative evaluation involved conducting semi-structured interviews with 43 leaders and treatment providers across 11 of the 13 OTPs in Rhode Island. These interviews inquired about pReferences regarding CM intervention components, CM training, structure and format of post-training support (e.g. supervision), and CM performance feedback. A subsequent round of sustainment evaluations with 18 OTP leaders (from 18 unique programs) who had implemented CM across a wider geographic area inquired about CM barriers, facilitators, and adaptations. All interviews were transcribed and coded using a directed content analysis approach.
Results: Formative evaluation results revealed key preferences that informed both the CM intervention and implementation strategy used in a large-scale cluster randomized trial. OTP staff preferred organization-specific and flexible CM targets focused on treatment engagement. Staff also desired multi-level implementation support including ongoing performance feedback and coaching focused on CM delivery and workflow integration. Sustainment evaluation results indicated wide variability in CM adaptation across OTPs, and revealed key adaptations that had been made as well as concrete recommendations to refine the implementation strategy.
Conclusion: These findings highlight the value of engaging in formative evaluation to modify both the intervention and implementation strategies prior to CM scale up. Results also highlight the importance of systematically evaluating CM sustainment to document adaptations and identify needs for additional CM implementation support for OTPs.
Title: Project MIMIC (Maximizing the Implementation of Motivational Incentives in Clinics): A Type 3 Hybrid Trial Implementing Contingency Management in Opioid Treatment Programs
Background: CM is one of the most effective adjunctive treatments to medication for opioid use disorders, but its implementation in opioid treatment programs (OTPs) remains low. Project MIMIC is a cluster-randomized, type 3 hybrid effectiveness-implementation trial comparing two strategies to implement CM in the OTP setting. We describe Project MIMIC’s design and share preliminary results from the first 18 OTPs, including 131 staff (counselors/leaders) and 378 patient participants.
Methods: Eighteen OTPs were cluster-randomized to receive either the Addiction Technology Transfer Center (ATTC) strategy (workshop + feedback + coaching) or the Enhanced ATTC (E-ATTC) strategy, which layered in two additional strategies: Pay-For-Performance and Implementation Sustainment Facilitation. Consistent with the exploration, preparation, implementation, and sustainment (EPIS) framework, OTPs engaged in 5 months of preparation, 10 months of implementation, and 6 months of sustainment monitoring.
Results: During the preparation phase, 105 counselors (55 E-ATTC, 50 ATTC) enrolled in Project MIMIC, of which 99 (100% EATTC, 94% ATTC) completed the didactic CM workshop and 64 (67% EATTC, 54% ATTC) submitted a role play for performance feedback. During the implementation phase, rates of patient recruitment, providers adopting CM, and providers meeting the CM exposure benchmark all favored E-ATTC relative to ATTC (recruitment: 87% vs 77%; adoption: 60% vs. 44%; exposure: 38% vs. 16%, all p-values < .0001). Four E-ATTC sites and three ATTC sites sustained CM with fidelity following removal of external support: another four OTPs (1 E-ATTC, 3 ATTC) sustained CM incentives with low fidelity to the model.
Conclusion: Preliminary data indicate that CM training engagement, recruitment, adoption, and exposure rates were greater in the E-ATTC condition, relative to the ATTC condition. Next steps include examining effects on patient outcomes, and refining ongoing fidelity monitoring. These data have informed design decisions for the implementation strategies used in the planned rollout of CM throughout the state of California.
Title: From Research to Action: Translating Evidence for a Culturally Adapted Contingency Management Program into Practice in Partnership with American Indian Communities
Background: We conducted two community engaged randomized controlled trials of a culturally adapted contingency management (CM) intervention for alcohol and illicit drug use among American Indian and Alaska Native (AI/AN) adults residing in rural and urban areas. Across 272 AI/AN participants, CM was associated with reduced alcohol, stimulant, and cannabis use. Responding to interest from multiple distinct Tribal communities, our team pivoted to translate lessons learned from these studies to develop dissemination and implementation resources for AI/AN communities.
Methods: Tenets of community-based participatory research were applied. Guided by the Quality Improvement Framework, four members of the university team partnered with AI/AN community members to develop a suite of dissemination and implementation support tools.
Results: We developed a modified CM manual for Indigenous communities to aid dissemination. Adaptations included integrating cultural values and alignment of CM with Indigenous worldviews; strategies for successfully identifying staff and settings for successful implementation; client narratives (i.e., case studies); infographics to illustrate CM fidelity and opportunities for further cultural or local adaptations. Tools developed also included an incentive tracker that assures Medicaid-compliance, a modified practice facilitation guide (i.e., a clinic asset and readiness assessment), pre- and post- knowledge surveys and resources for coaching consultations to support ongoing, high-fidelity implementation among new Tribal partners on the West Coast.
Conclusion: A suite of tools for implementation, dissemination and scaling-up was developed to quickly meet the interest in CM training by AI/AN communities. Results indicate that rapid movement from community-engaged Phase III clinical trials to dissemination may be feasible and can strengthen the capacity among AI/AN communities in their efforts to provide culturally and clinically meaningful treatment to Tribal members. Resources have subsequently been used in state-wide initiatives in Montana, Washington, and California.
Title: State Level Implementation of Contingency Management for Stimulant Use Disorders: It’s about Time
Background: Until 2020, the primary barrier to CM implementation was the Office of Inspector General’s prohibition on the use of incentives with Medicaid and Medicare enrollees. Since then, increased regulatory flexibility and the ongoing methamphetamine epidemic are driving widespread interest in CM. This presentation describes tools and strategies to support state-wide implementation of CM in Montana and Washington State.
Methods: In partnership with Montana and Washington state healthcare authorities and behavioral health training experts, we co-designed CM interventions for stimulant use disorders that were feasible, based on research evidence, and financially sustainable. The implementation strategy included didactic materials, performance feedback, coaching calls, and a suite of implementation resources (e.g. treatment manuals, digital incentive trackers, policy templates). An iterative process allowed us to refine the implementation strategy and provide dynamic and responsive training supports.
Results: Over 30 clinical sites have completed our didactic CM training and engaged in our on-going CM coaching focused on implementation. Partnering sites were primary care, outpatient addiction clinics, outreach and social service agencies, emergency departments, jails, and Tribal housing and substance use providers. Geographic locations were urban, rural and frontier. Implementation challenges included incentive management with sufficient fraud-protection, high staff turnover, sustainability of frequent twice a week in-person visits, and appropriate use and payment for point of care urine testing. Other barriers were related to the desire to deliver individualized, patient-centered care versus the need to deliver a standardized, research-based model, and unique barriers to novel treatment settings, such as jails or emergency departments.
Conclusion: Consistent with CM implementation research, providing ongoing training and feedback, as well as opportunities to see successful CM in action (Oluwoye, 2020), have been keys to addressing state-level implementation challenges. The refined implementation strategy is informing the California CM initiative that will provide training and implementation support up to 200 clinics.
References
1. Scott, K., Jarman, S., Moul, S., Murphy, C. M., Yap, K., Garner, B. R., & Becker, S. J. (2021). Implementation support for contingency management: preferences of opioid treatment program leaders and staff. Implementation science communications, 2(1), 1-10.
2. Becker, S. J., Murphy, C. M., Hartzler, B., Rash, C. J., Janssen, T., Roosa, M., … & Garner, B. R. (2021). Project MIMIC (Maximizing Implementation of Motivational Incentives in Clinics): A cluster-randomized type 3 hybrid effectiveness-implementation trial. Addiction science & clinical practice, 16(1), 1-16.
3. McDonell, M. G., Skalisky, J., Burduli, E., Foote Sr, A., Granbois, A., Smoker, K., Hirchak, K., Herron, J., Ries, R. K., Echo-Hawk, A., Barbosa-Leiker, C., Buchwald, D., Roll, J. M., & McPherson, S. M. (2020). The Rewarding Recovery Study: A randomized controlled trial of incentives for alcohol and drug abstinence with a rural American Indian community. Addiction. https://doi.org/10.1111/add.1534
4. Hirchak, K. A., Lyons, A. J., Herron, J. L., Kordas, G., Shaw, J. L., Jansen, K., … & HONOR Study Team. (2021). Contingency management for alcohol use disorder reduces cannabis use among American Indian and Alaska Native adults. Journal of Substance Abuse Treatment, 108693.
Building a bridge between the practice and science of implementation support: Three examples from prevention science
Authors
Ms. Kathryn Bruzios - Washington State University
Dr. Sarah Chilenski - Pennsylvania State University
Dr. Louis Brown - University of Texas Health Science Center, Houston
Dr. Jochebed Gayles - Pennsylvania State University
Dr. Brittany Cooper - Washington State University
Dr. Meg Small - Pennsylvania State University
Ms. Rebecca Roppolo - The Impact Center at FPG Child Development Institute, University of North Carolina at Chapel Hill
Dr. Will Aldridge - The Impact Center at FPG Child Development Institute, University of North Carolina at Chapel Hill
Ms. Nataly Barragan - Pennsylvania State University
Dr. Janet Welsh - Pennsylvania State University
Abstract
The adoption, high quality delivery, and sustained implementation of evidence-based preventive programs (EBPs) requires not only funding and initial training, but also ongoing support to the organizations and individuals responsible for delivering interventions to youth and families (Albers et al., 2021; Fagan et al., 2019). The Interactive Systems Framework posits that three systems, working in concert, are needed to support the translation of EBPs into community settings, ultimately achieving public health impact. The Prevention Support System is particularly important in this process as it bridges the other two systems and is tasked with building general- and intervention-specific capacity in the organizations delivering these interventions (Wandersman et al., 2008).
In this symposium, we spotlight three studies aimed at advancing the practice and science of implementation support. The first presentation describes an innovative process used to develop and pilot a quantitative measure of the collaborative relationship between technical assistance (TA) providers and coalition leaders in a sample of prevention coalitions before initiation of an implementation support system called the Coalition Check-up. The second presentation reports a tracking system to document and refine external implementation support provided over five years to 13 regions across two states scaling an evidence-based family intervention. The third presentation describes a study analyzing over 10,000 instances of TA provided to state-funded grantees implementing various EBPs in one state from 2011-2017. Results of regression analyses indicate that organizational experience, program infrastructure, and program characteristics are related to amount of TA delivered depending on the type of TA examined. Findings from all three studies use data to improve our understanding of the supports needed to scale EBPs in real-world settings.
The discussant, the Director of a workforce development center aimed at accelerating the application of implementation science to improve outcomes, will synthesize common themes and outline implications for implementation practice.
Title: Measurement and Implications of Collaborative Technical Assistance for Community Coalitions
Background: Over 5,000 coalitions serve as a cornerstone of substance use prevention in the US. These coalitions, however, have only demonstrated efficacy when they use technical assistance (TA) and implement evidence-based programs. Furthermore, research has demonstrated the importance of creating a collaborative relationship between coalition leaders and TA providers. Consequently, this presentation (1) will describe the process used to create a measure of this collaborative relationship for coalition TA, and (2) will share baseline data that characterizes it in a sample of coalitions prior to initiation of an implementation support system called the Coalition Check-Up (CCU).
Methods: To create the measure, a human-centered design technique was used to collect and map common themes across the TA experience for both TA providers (n = 4) and coalition leads (n = 5).
The results from the heat mapping were then used to develop items for a holistic TA Quality measure. The survey was refined through cognitive interviewing and then piloted with the 68 community coalitions participating in the CCU.
Results: Results from the human-centered design technique revealed 11 themes describing dimensions of TA provider-coalition leader relationship. Sixty items were developed based on these themes. After cognitive interviewing, seven domains were identified as most theoretically relevant and were consequently used in pilot data collection.
Preliminary analyses showed that more than half of the coalition leaders reported receiving support and assistance from more than 10 different sources in the year prior to participating in the CCU. Additionally, results suggest that, on average, the quality of relationships is high, and that the measure can be further refined to better distinguish each construct.
Conclusion: These results have the potential to empirically identify and outline effective TA strategies in the community coalition context, thus improving the effectiveness of community coalitions and the health of the communities they serve.
Title: Use and Feasibility of External Implementation Support Activities in Practice: Descriptive Data from the Implementation Capacity for Triple P (ICTP) Projects
Background: Reporting on practice strategies to advance implementation outcomes is imperative (Bunger et al., 2017). The presentation reports descriptive information about external implementation support (EIS) provided over 5 years to 13 regions in North Carolina and South Carolina scaling an evidence-based system of parenting and family supports. Regional support teams employed core practice components (CPCs) for EIS as proposed by Aldridge and colleagues (2022) and local leaders and implementation teams (LITs) rated the feasibility of participating in EIS.
Methods: Practice activities associated with CPCs were developed and iteratively refined across the study period. Regional support teams providing EIS systematically tracked their use of CPCs following each substantive support interaction, including duration of time and use of specific activities. LITs submitted evaluations of EIS, rating implementation outcomes (e.g., feasibility, appropriateness, acceptability, likelihood and actual use of strategies and resources, accessibility of support, and institutionalization) and practice outcomes (e.g., working alliance). Regional support teams rated the working alliance, collaboration and contact with LITs.
Results: From November 2016 through December 2021, ICTP support teams tracked 749 support interactions. Monthly support decreased year over year, though dose varied considerably. Patterns of CPC use indicated a high dose of “foundational” and “explorative” CPCs early, followed by a blended and more diverse use thereafter, with some notable trends. Initial results of feasibility data indicate that participating in EIS is well received by LITs. Further analyses of implementation and practice are in progress and notable trends will be discussed.
Conclusion: This descriptive study offers a case study for how EIS might be operationalized, tracked, and employed. Initial data from participants in EIS support the feasibility of this practice model to build the capacity and resources of local leaders and teams scaling an evidence-based program. Findings suggest several interpretations that might refine our understanding and use of EIS.
Title: The Implication of Evidence-based Program Characteristics in Determining Types and Amount of Technical Assistance Support to Community Organizations
Background: This study investigates how EBPs (evidence-based programs) and provider organization characteristics contribute to variation in type and amount of technical assistance (TA). First, we empirically classify types of TA strategies provided to individuals and agencies implementing EBPs. Second, we explicate how provider EBP experience, EBP infrastructure (developer supports and resources), setting (prevention continuum), and program delivery (school-based, family-focused, group) characteristics predict variation in amount of TA delivered across types of TA strategies.
Methods: The current study analyzes over 10,000 coaching/TA interactions between implementation specialists (n = 14) and community provider organizations (n = 114) delivering various EBPs (n = 13) from 2011 through 2017. Four separate hierarchical regressions were executed for the following TA types: consultation, coordination-logistics, resource distribution, and monitoring. Predictor variables included provider implementation capacity (block 1), EBP infrastructure and rating (block 2), and EBP setting and delivery characteristics (block 3).
Results: Results indicated that treatment programs, family-focused vs. other, and group delivery vs. single, all required greater consultation, regardless of the EBP being delivered. Treatment and group delivery also predicted greater resource distribution; whereas exemplar vs. promising programs and greater implementation capacity predicted less resource distribution. Treatment, family-focused, group delivery and greater program infrastructure predicted more coordination, whereas exemplar programs required less coordination. Finally, family-focused programs required greater monitoring, and greater implementation capacity predicted less monitoring.
Conclusion: Results inform research and practical understanding of TA support for EBP implementation in real-world settings, in three ways: (1) informing the developers in considering resource and infrastructure needs to support wide-scale dissemination of EBPs; (2) providing guidance to funders and policymakers on how best to invest limited dollars for TA support of EBPs; and (3) informing the growing cadre of TA provider organizations on how best to focus operations by deploying more efficient TA models to match EBP and provider contexts.
References
1. Albers, B., Metz, A., Burke, K., Bührmann, L., Bartley, L., Driessen, P., & Varsi, C. (2021). Implementation support skills: Findings from a systematic integrative review. Research on Social Work Practice, 31(2), 147-170.
2. Bunger, A. C., Powell, B. J., Robertson, H. A., MacDowell, H., Birken, S. A., & Shea, C. (2017). Tracking implementation strategies: a description of a practical approach and early findings. Health Research Policy and Systems, 15(1), 1-12.
3. Fagan, A. A., Bumbarger, B. K., Barth, R. P., Bradshaw, C. P., Cooper, B. R., Supplee, L. H., & Walker, D. K. (2019). Scaling up evidence-based interventions in US public systems to prevent behavioral health problems: Challenges and opportunities. Prevention Science, 20(8), 1147-1168.
4. Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., … & Saul, J. (2008). Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. American Journal of Community Psychology, 41(3), 171-181.
Advances and best practices in configurational comparative methods: Taking implementation science to the next frontier
Authors
Dr. Brittany Cooper - Washington State University
Dr. Heather Kane - RTI International
Dr. Dallas Elgin - RTI International
Dr. Leila Kahwati - RTI International
Ms. Meredith Crews - RTI International
Dr. Sapna Mendon-Plasek - Division of Behavioral Health Services and Policy Research, Columbia University – New York State Psychiatric Institute, New York, NY
Dr. Meghan Lane-Fall - Perelman School of Medicine, University of Pennsylvania
Dr. Saira Bashir - Barnard College
Dr. Lindsey Zimmerman - Office of Mental Health and Suicide Prevention, National Center for PTSD, Veterans Affairs, Palo Alto, CA
Ms. Erica Doering - Washington State University
Dr. Tessa Matson - Kaiser Permanente Washington Health Research Institute
Ms. Claire Gwayi-Chore - Department of Global Health, University of Washington
Abstract
Implementation models emphasize the complexity involved in understanding how and under what conditions implementation is successful and when it fails (Nilsen, 2020). To add to this complexity, the interventions being studied are themselves often multi-faceted and are being delivered across a variety of diverse contexts. Innovative methods aimed at acknowledging and understanding this complexity are critical to move the field forward. One exciting direction is the increasing use of Configurational Comparative Methods (CCM) in implementation-related health research. CCM are a group of methods based in regularity theories of causation that aim to identify combinations of conditions that jointly produce an outcome (Whitaker et al., 2020). Although CMM have been used for decades in other fields (e.g., political science) to understand causal complexity, their application is still relatively new to implementation science (Kane et al., 2014). Therefore, this symposium aims to 1) introduce researchers to CCM advances in implementation science and 2) enhance understanding of CMM best practices.
To this end, the first presentation describes a review of large-N applications of Qualitative Comparative Analysis in implementation-related fields to identify the use of methodological best practices related to reporting case numbers, truth tables, robustness tests, solution types and parameters of fit. The second presentation describes results from a scoping review of CCM used to study implementation in healthcare services. The third presentation describes results from a coincidence analysis (CNA) as part of a hybrid type 1 cluster randomized controlled trial of community-wide drug administration intervention in Benin, India, and Malawi. The results identify necessary and sufficient combinations of activities that were key to producing positive outcomes.
The discussant will end the session by synthesizing common themes and sharing recommendations for how CMM can advance implementation science.
Title: Building Best Practices in Large-N Qualitative Comparative Analysis
Background: In recent years, large-N applications of Qualitative Comparative Analysis (QCA) have grown in multiple fields, including implementation science-informed program evaluations. Large-N applications involve anywhere from 50 to several thousand cases. Because the method is relatively young, reporting conventions for journals have not been standardized, nor has a set of conventions been informally agreed upon in spite of Schneider and Wagemann’s 2010 monograph outlining best practices for conducting and reporting QCA results. This presentation will share the results of a scoping literature review of large-N applications of QCA in social science studies.
Methods: Five academic databases were searched using 24 key terms to identify QCA articles published in peer-reviewed political science/public policy, social science, and public health journals over the past 15 years. This search process identified over 2,200 articles that met initial search criteria. Abstracts were subsequently screened to identify studies where QCA was employed on 50 or more cases, and full-text reviews identified 222 large-N QCA studies that met this study’s inclusion criteria. Hybrid inductive-deductive coding of the 222 included articles was conducted to identify the use of methodological best practices.
Results: Current reporting practices across journals are inconsistent, with many authors not adopting QCA best practices. Inconsistencies in reporting findings included: 1) reporting the number of cases, 2) displaying the truth table, 3) conducting robustness tests, 4) reporting solution types (intermediate, parsimonious, conservative), and 5) reporting of “parameters of fit.” Notable differences were also found in the approaches to interpreting solution terms (i.e., “going back to the cases”).
Conclusion: To ensure better standardization in reporting of QCA, scholars reporting on QCA results and journals should adopt minimal set of best practices for reporting findings.
Title: Identifying Pathways for Promoting Implementation Success: A scoping review on the use of configurational comparative methods in implementation science
Background: Since first introduced to Implementation Science nearly a decade ago, Configurational Comparative Methods (CCM) are increasingly used to study implementation in complex systems. Configurational methods, including Qualitative Comparative Analysis (QCA) and Coincidence Analysis (CNA), help implementation researchers assess combinations of conditions necessary and/or sufficient for determining implementation outcomes. Consistent with other mixed methods approaches to implementation research, configurational methods integrate theoretical and substantive case or contextual knowledge throughout the iterative phases of data analysis and interpretation of findings. This scoping review will evaluate how studies describe what informs their decisions throughout analysis, including condition-selection, operationalization, data calibration, truth table analysis, and standard analyses. This review in progress is characterizing application of CCM in implementation research, including the research questions addressed, the use of theoretical models or frameworks in data analysis, and the use of CCM along with other qualitative or quantitative statistical inference methods.
Methods: This scoping review sought to identify published pieces using 12 Boolean operators across three electronic databases (PubMed, ScienceDirect, APA PsycInfo). This search yielded 1782 total results, including 231 duplicates. Abstracts were subsequently screened for peer-reviewed papers published since 2014 which applied configurational methods to study implementation in healthcare services.
Results: Of 37 papers examined in full-text reviews, QCA was used in the majority of studies. Most studies focused on implementation outcomes examining the configurations of determinants and strategies used to facilitate successful implementation. However, many described implementation success broadly and did not specify outcomes established by the field. Of the studies specifying at least one implementation outcome, adoption was most frequently examined. Few studies used theories and frameworks to inform condition selection and analysis. Further, identification of norms in qualitatively linking configurational findings to contextual knowledge of cases will be reviewed.
Conclusion: The review paper concludes with a discussion of key themes concerning factors that inform decision-making throughout analysis when using CCM.
Title: Defining Optimal Implementation Packages for Delivering Community-Wide Mass Drug Administration for Soil-Transmitted Helminths with High Coverage
Background: Recent evidence suggests that community-wide mass drug administration (MDA) may interrupt the transmission of soil-transmitted helminths (STH), a group of intestinal worms that infect 1.5 billion individuals globally. Although current operational guidelines provide best practices for effective MDA delivery, they do not describe which activities are most essential for achieving high coverage or how they work together to produce effective intervention delivery. We aimed to identify the various packages of influential intervention delivery activities that result in high coverage of community-wide MDA for STH.
Methods: We applied coincidence analysis to process mapping data as part of the implementation science research of the DeWorm3 Project, a Hybrid Type 1 cluster randomized controlled trial assessing the feasibility of interrupting transmission of STH using bi-annual community-wide MDA in Benin, India, and Malawi. Our analysis aimed to identify any necessary and/or sufficient combinations of intervention delivery activities (i.e., implementation pathways) that resulted in high MDA coverage. Activities were related to drug supply chain, implementer training, community sensitization strategy, intervention duration, and implementation context. We used pooled implementation data from three sites and six intervention rounds, with study clusters serving as analytical cases (N = 360).
Results: Across all sites and intervention rounds, efficient duration of MDA delivery (within 10 days) singularly emerged as a common and fundamental component for achieving high MDA coverage when combined with other particular activities, including a conducive implementation context, early arrival of albendazole before the planned start of MDA, or a flexible community sensitization strategy. No individual activity proved sufficient by itself for producing high MDA coverage. We observed four possible overall models that could explain effective MDA delivery strategies, all which included efficient MDA duration as an integral component.
Conclusion: Effective MDA delivery can be achieved with flexible implementation strategies that include various combinations of influential intervention components.
References
1. Kane, H., Lewis, M. A., Williams, P. A., Kahwati, L. C. (2014). Using qualitative comparative analysis to understand and quantify translation and implementation. Translational Behavioral Medicine, 4(2), 201-208. https://doi.org/10.1007/s13142-014-0251-6
2. Nilsen, P. (2020). Making sense of implementation theories, models, and frameworks. In B. Albers, A. Shlonsky, & R. Mildon (Eds.), Implementation Science 3.0 (pp. 53-79). Springer, Cham.
3. Whitaker, R. G., Sperber, N., Baumgartner, M., Thiem, A., Cragun, D., Damschroder, L., Miech, E. J., Slade, A., Birken, S. (2020). Coincidence analysis: A new method for causal inference in implementation science. Implementation Science, 15(1), 108. https://doi.org/10.1186/s13012-020-01070-3.
Facilitating implementation of evidence-based innovations for substance use disorders
Authors
Dr. Bryan Garner - RTI International
Dr. Michele Staton - University of Kentucky
Dr. James Ford - University of Wisconsin-Madison
Dr. Andrew Quanbeck - University of Wisconsin
Abstract
According to Powell and colleagues (2015), facilitation is defined as a process of interactive problem solving and support that occurs in a context of a recognized need for improvement and a supportive interpersonal relationship. This symposium is focused on facilitation as a strategy that is being used as part of four distinct implementation research projects, each focused on improving the implementation of evidence-based innovations for substance use disorders. The first three implementation research projects are focused on improving the implementation of medications for opioid use disorder (MOUD). In terms of context, the first is a type 1 implementation-effectiveness hybrid trial conducted within jail settings; the second is a sequential multiple assignment randomized trial (SMART) conducted within specialty addiction clinics; and the third is using a hybrid type 3 clustered, sequential, SMART design in primary care clinics. The fourth project is a type 3 implementation-effectiveness hybrid trial focused on improving the implementation of a motivational interviewing-based brief intervention (MIBI) for substance use disorders within HIV service organizations.
Title: Facilitating the implementation of MOUD pretreatment telehealth for justice-involved women during community re-entry from jail
Background: Women are disproportionately affected by the on-going opioid crisis due to increased pain sensitivity, physician prescribing practices, and a faster trajectory from opioid exposure to opioid use disorder. However, research on implementation of approaches to increase initiation of medications for opioid use disorder (MOUD) among women is limited.
Methods: Through the NIH/NIDA-funded Justice Community Opioid Innovation Network (JCOIN), this hybrid type 1 effectiveness implementation trial is the first to examine innovative telehealth assessments to increase MOUD initiation among justice-involved women transitioning to the community from jail. Facilitation of the implementation of telehealth and related services is guided by the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework. The EPIS framework examines the multilevel nature of service systems, organizations within systems, service providers, and service consumers in the implementation of evidence-based innovations. The aim of this presentation is to describe adaptations by the research team and study partners to facilitate implementation of MOUD PreTreatment Telehealth in light of COVID19 restrictions in local jail sites using the EPIS framework.
Results: Preliminary findings will be discussed in the context of individual jail-based case studies and facilitation adaptations made due to COVID19. Adaptations were made in each EPIS stage including Exploration (e.g., initial partner planning discussions); Preparation (e.g., facilitating meetings with stakeholders to establish care linkages); and Implementation (e.g., facilitating telehealth linkages to MOUD providers and peer navigators).
Conclusion: Facilitation includes the activities needed to implement an innovation in a particular setting with community partners. COVID19 led to a number of protocol adaptations during implementation, which required significant facilitation effort between the research team and community jail and treatment partners. These efforts underscore the need for strong partnerships in increasing the capacity for the justice system to respond to the opioid crisis through increased access to treatment to reduce opioid relapse and overdose.
Title: Facilitating access to medications for opioid use disorders in specialty addiction clinics using a Stepped Implementation-to-Target Approach
Background: Significant efforts have been implemented to expand access to medications for opioid use disorder (MOUD). While growth has occurred in general medical care settings, specialty addiction clinics have lagged in reach and adoption of MOUD. Advanced implementation scientific approaches are needed to address the challenges of implementing and sustaining MOUD in specialty addiction clinics.
Methods: In partnership with the Washington State Health Care Authority, this project seeks to recruit 64 specialty addiction clinics and employs a measurement-based stepped implementation-to-target approach within an adaptive trial design to improve access to MOUD. Agencies will be exposed to a sequence of implementation strategies: enhanced monitoring and feedback (EMF), a two-day academy focused on MOUD and NIATx process improvement strategies, NIATx internal facilitation, and NIATx external facilitation. The study has three aims: (1) evaluate the relative impact of implementation strategies on target outcomes; (2) examine contextual moderators and mediators of performance; and (3) document and model costs per implementation strategy to achieve target outcomes. Outcomes organized by the RE-AIM taxonomy and the Addiction Care Cascade include MOUD capability measured by the IMAT, proportion of patients on an MOUD, number of integrated prescribers, proportion of patients receiving MOUD within 72 hours of diagnosis, and 6-month retention.
Results: The study launches in June 2022. We are currently recruiting addiction clinics, developing online data collection tools for primary outcomes and contextual determinants, creating EMF feedback reports, adapting the Stages of Implementation Completion for this study, and establishing systems to support our economic evaluation.
Conclusion: This implementation project has elements of a sequential multiple assignment randomized trial (SMART) design and a criterion-based design. The innovation is that programs on receive the implementation strategies needed to achieve implementation outcomes. Findings have the potential to advance drug abuse treatment research by identifying an optimization of strategies to implement MOUD.
Title: An Adaptive Implementation Strategy Design to Facilitate Access to Medications for Opioid Use Disorders
Background: Rates of opioid prescribing tripled in the USA between 1999 and 2015 and were associated with significant increases in opioid misuse and overdose death. Although clinical guidelines describe recommended opioid prescribing practices, implementing these guidelines in a way that balances safety and effectiveness vs. risk remains a challenge. Systems consultation consists of (1) educational/engagement meetings with audit and feedback reports, (2) practice facilitation, and (3) prescriber peer consulting. The study is designed to discover the most cost-effective sequence and combination of strategies for improving opioid prescribing practices in diverse primary care clinics.
Methods: The study is a hybrid type 3 clustered, sequential, multiple-assignment randomized trial (SMART) that randomizes clinics from two health systems at two points, months 3 and 9, of a 21-month intervention. Clinics are provided one of four sequences of implementation strategies: a condition consisting of educational/engagement meetings and audit and feedback alone (EM/AF), EM/AF plus practice facilitation (PF), EM/AF + prescriber peer consulting (PPC), and EM/AF + PF + PPC. The study’s primary outcome is morphine-milligram equivalent (MME) dose by prescribing clinicians within clinics. The study’s primary aim is the comparison of EM/AF + PF + PPC versus EM/AF alone on change in MME from month 3 to month 21. The secondary aim is to derive cost estimates for each of the four sequences and compare them.
Results: The study ends in October 2022. We are currently generating opioid evaluation metrics from participating health systems’ electronic medical records and beginning primary quantitative, qualitative, and economic outcomes.
Conclusion: Systems consultation is a practical blend of implementation strategies used in this case to improve opioid prescribing practices in primary care. The results of this study promise to help us understand how to cost effectively improve the implementation of evidence-based practices.
Title: Improving implementation of a motivational interviewing-based brief intervention for substance use disorders in HIV service settings: Enhancing team-focused facilitation with staff-focused financial incentives
Background: As part of a dual-randomized type 2 implementation-effectiveness hybrid trial, which included 39 HIV service organizations (HSOs) across the U.S., 78 HSO staff, and 824 client participants with HIV and a substance use disorder (SUD), a motivational interviewing-based brief intervention (MIBI) was found to be effective. However, the MIBI was only effective when implemented in the organization-level implementation condition that provided HSOs with the Implementation and Sustainment Facilitation (ISF) strategy (i.e., monthly 30-60 minute team-focused facilitation meetings via Zoom) as an adjunct to the multifaceted control strategy (i.e., staff-focused training, feedback, and consultation regarding the MIBI), referred to as TFC. This presentation highlights results from the subsequent Substance Abuse Treatment to HIV Care II (SAT2HIV-II Project) – a cluster-randomized type 3 implementation-effectiveness hybrid trial testing the effectiveness of pay-for-performance (P4P; TFC + ISF + P4P) to improve MIBI implementation beyond the TFC + ISF strategy.
Methods: As part of the SAT2HIV-II Project, 25 HSOs as well as participating staff and clients, were cluster randomized to either the control strategy (TFC + ISF) or the experimental strategy (TFC + ISF + P4P). MIBI staff working at HSOs randomized to the experimental strategy had the opportunity to receive $10 USD per MIBI implemented, as well as $10 USD per MIBI implemented at or above the 80th percentile level of fidelity. Guided by the Theory of Implementation Effectiveness, the primary implementation outcome measure was implementation effectiveness (i.e., the consistency and quality of MIBI implementation), a staff-level measure representing the standardized sum of the total number of MIBIs implemented and the total quality/fidelity scores.
Results: The P4P strategy had a medium-sized impact (d = .47) that significantly (p = .001) on implementation effectiveness during the project’s implementation phase.
Conclusion: Consistent with prior research, the current research supports P4P as a highly-promising strategy to improve the implementation of brief motivational interventions.
References
1. Garner, B. R., Godley, S. H., Dennis, M. L., Hunter, B. D., Bair, C. M., & Godley, M. D. (2012). Using pay for performance to improve treatment implementation for adolescent substance use disorders: results from a cluster randomized trial. Archives of pediatrics & adolescent medicine, 166(10), 938-944.
2. Garner, B. R., Gotham, H. J., Chaple, M., Martino, S., Ford, J. H., Roosa, M. R., … & Tueller, S. J. (2020). The implementation and sustainment facilitation strategy improved implementation effectiveness and intervention effectiveness: results from a cluster-randomized, type 2 hybrid trial. Implementation Research and Practice, 1, 2633489520948073.
3. Garner, B. R., Lwin, A. K., Strickler, G. K., Hunter, B. D., & Shepard, D. S. (2018). Pay-for-performance as a cost-effective implementation strategy: results from a cluster randomized trial. Implementation Science, 13(1), 1-11.
4. Hinde, J. M., Garner, B. R., Watson, C. J., Ramanan, R., Ball, E. L., & Tueller, S. J. (2022). The implementation & sustainment facilitation (ISF) strategy: cost and cost-effectiveness results from a 39-site cluster randomized trial integrating substance use services in community-based HIV service organizations. Implementation Research and Practice, 3, 26334895221089266.
5. Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., … & Kirchner, J. E. (2015). A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10(1), 1-14.
6. Staton, M., Webster, J. M., Leukefeld, C., Tillson, M., Marks, K., Oser, C., … & Winston, E. (2021). Kentucky Women's Justice Community Opioid Innovation Network (JCOIN): A type 1 effectiveness-implementation hybrid trial to increase utilization of medications for opioid use disorder among justice-involved women. Journal of substance abuse treatment, 128, 108284.
7. Quanbeck, A., Almirall, D., Jacobson, N., Brown, R. T., Landeck, J. K., Madden, L., … & Schumacher, N. (2020). The balanced opioid initiative: protocol for a clustered, sequential, multiple-assignment randomized trial to construct an adaptive implementation strategy to improve guideline-concordant opioid prescribing in primary care. Implementation Science, 15(1), 1-13.
Financing strategies to support the implementation and sustainment of evidence-based practices in community behavioral health
Authors
Ms. Marylou Gilbert - RAND Corporation
Dr. Alex Dopp - RAND Corporation
Ms. Maddison North - University of Oklahoma Health Sciences Center
Dr. Jeanne Ringle - RAND Corporation
Dr. Jane Silovsky - University of Oklahoma Health Sciences Center
Dr. Susan Schmidt - University of Oklahoma Health Sciences Center
Dr. Beverly Funderburk - University of Oklahoma Health Sciences Center
Dr. Byron Powell - Brown School, Washington University in St. Louis, St. Louis, MO, United States
Dr. David Mandell - University of Pennsylvania
Dr. Daniel Edwards - Evidence-Based Associates
Dr. Douglas Luke - Washington University in St. Louis
Ms. Mellicent O'Brien Blythe - University of North Carolina- Chapel Hill
Dr. Dana Hagele - University of North Carolina at Chapel Hill
Dr. Heather Pane Seifert - NC Child Treatment Program
Dr. Donna Potter - NC Child Treatment Program
Dr. Jerica Knox - North Carolina State University
Dr. Vanesa Mora Ringle - University of Pennsylvania
Ms. Amy Miller - University of Pennsylvania
Ms. Amberlee Venti - Community Behavioral Health, Philadelphia Department of Behavioral Health and Intellectual disAbility Services
Ms. Carrier Comeau - Community Behavioral Health, Philadelphia Department of Behavioral Health and Intellectual disAbility Services
Ms. Tamra Williams - Department of Behavioral Health and Intellectual disAbility Services
Dr. Torrey Creed - University of Pennsylvania
Abstract
Limited and fragmented funding is often cited as one of the biggest impediments to the use of evidence-based practices (EBPs) in community mental health. Nevertheless, it is rare for publicly funded behavioral health systems to support EBP use through financing strategies, which provide funds for critical implementation and sustainment activities involved in using EBPs.
This symposium will present a series of studies illustrating the use of financing strategies to support EBP adoption and sustainment. Presentation #1 will present on the development of the Fiscal Mapping Process tool that guides mental health service agencies in the selection and coordination of financing strategies for EBP sustainment. Presentation #2 will describe the Clinical Service Delivery Time Model Series, developed by an intermediary organization to determine costs and delivery requirements of an EBP within specific provider contexts. Finally, Presentation #3 will share results from a study evaluating implementation outcomes (acceptability, feasibility, satisfaction, effectiveness) of an enhanced reimbursement rate strategy used by a public behavioral health system to support the ongoing use of EBPs.
Following these presentations, our discussant, will summarize and contextualize findings from the three studies within the broader literature on implementation financing strategies and policies. This discussion will provide perspectives on how implementation researchers can best incorporate the strategies covered in this symposium, within the context of strong collaborative relationships with community providers and behavioral health systems, to promote the financial sustainability of EBPs adopted through implementation research or practice initiatives.
Title: Stakeholder-Partnered Development of the Fiscal Mapping Process for Sustainable Financing of Evidence-Based Practices: “It’s No Longer Beyond Our Realm”
Background: Behavioral health service agencies experience significant cost-related barriers to sustaining evidence-based practices (EBPs). We are developing and evaluating a multi-step strategic planning tool – the Fiscal Mapping Process – that guides selection and coordination of financing strategies for behavioral health EBP sustainment.
Methods: We adapted an established implementation mapping process (Fernandez et al., 2019) into the Fiscal Mapping Process prototype, incorporating existing resources (e.g., a compilation of financing strategies; Dopp et al., 2020). We recruited 12 service agencies to pilot-test the prototype with either Parent-Child Interaction Therapy (PCIT) or Trauma-Focused Cognitive-Behavioral Therapy (TF-CBT), two EBPs for youth. Across the year-long pilot, we provide training and monthly coaching for the Fiscal Mapping Process. We also collect mixed-method data – e.g., surveys, focus groups – from service agencies and their stakeholder partners (EBP trainers, funding agencies; N = 48 participants) to document impacts on sustainment. Throughout, we are incorporating stakeholder feedback to improve the tool.
Results: We have completed one round each of surveys and focus groups. The survey established consensus for the five steps of the Fiscal Mapping Process. Rapid analysis of open-ended survey items and focus group notes led to numerous updates, such as more guidance and structure around financing strategy selection, additional resources (e.g., descriptions of PCIT and TF-CBT for engaging stakeholders), and space to document action items. Focus group analyses also revealed how stakeholders viewed the Fiscal Mapping Process as improving certain sustainment capacities – strategic planning and financial stability – but constrained by others (e.g., organizational capacity for the EBP). We anticipate completing all data collection prior to this presentation and will incorporate additional findings as appropriate.
Conclusion: This pilot-test will produce a Fiscal Mapping Process that builds behavioral health service agencies’ capacities to sustain funding for EBPs in coordination with stakeholders. The results will inform future large-scale tests of the tool.
Title: Clinical Service Time Modeling to Support Financial Sustainability of Evidence Based Treatments
Background: Across implementation frameworks, a key determinant is the extent to which the external environment incentivizes the Evidence-Based Practice (EBP) being implemented (e.g., Damschroder et al, 2009; Moullin et al, 2019). Many community-based providers do not have the organizational capacity to independently pursue financial incentives for their EBPs, given the complex and fragmented nature of the financing system (Dopp et al., 2021). Clinical service time modeling can provide a framework for determining the true cost and delivery requirements of an EBP within a specific provider context.
Methods: The presenter’s intermediary organization provides a statewide platform for dissemination of child mental health EBPs. To support sustainability, the organization has developed a Clinical Service Delivery Time Model Series. The Time Model for each EBP, developed in collaboration with model developers, includes required in-session clinical activities with estimates of the typical range of time for each; out-of-session clinical support activities with estimates of the typical range of time for each; additional agency-level resource requirements; clinical inclusion, assessment, and fidelity requirements; and a summary of the research base, including outcomes from in-state Learning Collaboratives. The Time Models are being disseminated to providers, policy makers, and Medicaid payors. The Time Models support two financial sustainability strategies: 1) increasing payment rates to cover the true cost of each EBP, and 2) reducing administrative burden through consistent and clinically appropriate requirements for contracting, service authorization, and billing strategies across payors.
Results: To date, two state Medicaid payors are using the Time Models to develop higher rates for at least one EBP, and others have expressed interest in doing so. Several payors have supported a task force to develop consistent administrative requirements. Outreach and technical assistance continue.
Conclusion: Clinical service time modeling can provide a helpful structure for specifying the resources required for specific EBPs to support sustainability.
Title: Community Provider Perceptions of an Enhanced Reimbursement Rate Strategy to Support the Use of Evidence-Based Practices
Background: To address the costs of EBP implementation, some large mental health systems offer enhanced reimbursement rates to agencies to better cover EBP costs (Dopp et al., 2020). The Philadelphia Department of Behavioral Health and Intellectual disAbility Services (DBHIDS) has employed this financing strategy to support wide-scale EBP implementation, offering an opportunity to examine the real-time experiences of community providers (Powell et al., 2016). In partnership with DBHIDS, we are conducting a mixed-methods investigation of community providers’ perceptions of the enhanced reimbursement rate and its impact on implementation outcomes.
Methods: All community mental health organizations in the DBHIDS network were invited to first complete quantitative rating scales assessing the acceptability, feasibility, utilization, and perceived effectiveness of the enhanced reimbursement rate (Weiner et al., 2017). Recruitment is restricted to one therapist, one supervisor, and one administrative/clinical leader per agency, with a goal of 60 total participants. As of May 2022, 9 therapists, 8 supervisors, and 3 administrative/clinical leaders from 15 mental health agencies have completed the survey. Data collection will be complete by July 2022.
Results: In preliminary analyses, 65% (n = 13) of providers reported being familiar with the enhanced reimbursement rate, and 55% (n = 11) reported working at an agency that receives the enhanced rate. Providers endorsed moderate enhanced reimbursement rate acceptability (M = 3.6) and feasibility (M = 3.9). Most providers (82%; n = 9) endorsed low-ambivalent satisfaction with the enhanced reimbursement rate. Average effectiveness scores (M = 2.9) indicate overall low perceived effectiveness. However, 64% providers still agree that their agency relies on the enhanced rate to implement EBPs.
Conclusion: Findings from this investigation will reveal real-time outcomes of the financing strategy as reported by key community providers. The feedback data we obtain will inform plans for the enhanced reimbursement rate, and may have implications for the efforts of other similar under-resourced publicly funded behavioral health systems.
References
1. Damschroder, L.J., Aron, D.C., Keith, R.E., Kirsch, S.R., Alexander, J.A., & Lowery, J.C. (2009). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science 4, 50. https://doi.org/10.1186/1748-5908-4-50
2. Dopp, A. R., Narcisse, M. R., Mundey, P., Silovsky, J. F., Smith, A. B., Mandell, D., Funderburk, B. W., Powell, B. J., Schmidt, S., Edwards, D., Luke, D., & Mendel, P. (2020). A scoping review of strategies for financing the implementation of evidence-based practices in behavioral health systems: State of the literature and future directions. Implementation Research and Practice. https://doi.org/10.1177/2633489520939980
3. Dopp, A. R., Gilbert, M., Silovsky, J., Ringel, J. S., Schmidt, S., Funderburk, B., Jorgensen, A., Powell, B. J., Luke, D. A., Mandell, D., Edwards, D., Blythe, M., & Hagele, D. (2022). Coordination of sustainable financing for evidence-based youth mental health treatments: Protocol for development and evaluation of the Fiscal Mapping Process. Implementation Science Communications, 3:1. https://doi.org/10.1186/s43058-021-00234-6
4. Dopp, A.R., Kerns, S.E.U., Panattoni, L., Ringel, J.S., Eisenberg, D., Powell, B.J., Low, R., & Raghavan, R. (2021).Translating economic evaluations into financing strategies for implementing evidence-based practices. Implementation Science 16, 66. https://doi.org/10.1186/s13012-021-01137-9
5. Fernandez, M. E., ten Hoor, G. A., van Lieshout, S., Rodriguez, S. A., Beidas, R. S., Parcel, G., Ruiter, R. A. C., Markham, C. M., & Kok, G. (2019). Implementation mapping: Using intervention mapping to develop implementation strategies. Frontiers in Public Health, 7:158. https://doi.org/10.3389/fpubh.2019.00158
6. Hagele D, Pane-Siefert H, Potter, D, Blythe M., and Knox, J (2020). Clinical Service Delivery Time Models. North Carolina Child Treatment Program Implementation Support. https://www.ncchildtreatmentprogram.org/implementation-support/
7. Mettrick, J., Harburger, D.S., Kanary, P.J., Lieman, R.B., & Zabel, M. (2015). Building Cross-System Implementation Centers: A Roadmap for State and Local Child Serving Agencies in developing Centers of Excellence (COE). Baltimore, MD: The Institute for Innovation & Implementation, University of Maryland. https://assets.aecf.org/m/resourcedoc/aecf-BuildingCrossSystemCenters-2015.pdf
8. Moullin, J.C., Dickson, K.S., Stadnick, N.A., Rabin, B. & Aarons, G.A. (2019). Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implementation Science 14, 1. https://doi.org/10.1186/s13012-018-0842-6
9. Powell, B. J., Beidas, R. S., Rubin, R. M., Stewart, R. E., Wolk, C. B., Matlin, S. L., … & Mandell, D. S. (2016). Applying the policy ecology framework to Philadelphia’s behavioral health transformation efforts. Administration and Policy in Mental Health and Mental Health Services Research, 43(6), 909-926.
10. Weiner, B. J., Lewis, C. C., Stanick, C., Powell, B. J., Dorsey, C. N., Clary, A. S., . . . Halko, H. (2017). Psychometric assessment of three newly developed implementation outcome measures. Implementation Science, 12(1), 1-12.
Optimization and implementation science: The new frontier for behavioral interventions
Authors
Dr. Kate Guastaferro - New York University
Dr. Emily Feinberg - Boston University School of Medicine
Dr. R. Christopher Sheldrick - Boston University School of Public Health
Dr. Lindsey Knowles - University of Washington; VA Puget Sound Health Care System
Dr. Karey O'Hara - Arizona State University
Dr. Cara Lewis - Kaiser Permanente Washington Health Research Institute
Abstract
The past decade has seen an increase in the call for and investment in the development of optimized multicomponent behavioral interventions. An optimized intervention is one in which effectiveness is strategically balanced with implementation constraints imposed by the need for affordability, scalability, and efficiency. One approach used to identify optimized interventions is the principled multiphase optimization strategy (MOST). An engineering-inspired framework, MOST has been applied to a number of public health priorities including smoking cessation, obesity prevention, and reduction of sexually transmitted infections. However, an effective behavioral intervention is only as good as its implementation and dissemination strategy. This symposium explores the intersection of optimization and implementation science – what we identify as the new frontier of behavioral interventions by presentation highlighting the ways in which implementation science can provide a methods-rich roadmap for achieving important goals of MOST. Dr.Guastaferro, will provide an overview of MOST and describe three ways in which MOST can benefit implementation science. Then two presentations will highlight ways in which MOST may be used to optimize aspects of implementation science. Dr. Feinberg will provide an applied example of using MOST to identify the most effective implementation strategies for an intervention in a pediatric behavioral health clinic. Dr. Knowles will present a strategy for integrating human-centered design and implementation science methods into the development of an intervention using MOST. Attendees of this symposium will understand how MOST can be used to advance implementation science and a foundational understanding of how MOST may be applied to their areas of study. The overarching goal is to inspire the optimization of interventions as well as the implementation and dissemination of those interventions.
Title: How MOST can be used to advance implementation science objectives (Kate Guastaferro)
Abstract: When integrated with implementation science, the multiphase optimization strategy (MOST) has the potential to achieve maximal public health impact by disseminating effective interventions to those in need. This presentation will first provide a brief overview of MOST, a principled framework for the development, optimization, and evaluation of multicomponent behavioral interventions. Then, building off of a recently published editorial, the presentation will review three ways in which MOST may advance implementation science: (1) development of an immediately scalable intervention; (2) adaption of interventions; and (3) optimization the implementation of intervention. This introduction to MOST and the proposed integration of implementation science with optimization sets up the other two presentations.
Title: Optimizing the implementation of Family Navigation strategies in a pediatric primary care clinic (Emily Feinberg & Chris Sheldrick)
Abstract: The current study consists of the Optimization and Evaluation phases. Children aged 3-to-12 years old who are detected as “at-risk” for behavioral health disorders (n = 304) at a large, urban federally qualified community health center will be referred to a Family Partner—a bicultural, bilingual member of the community with training in behavioral health and systems navigation—who will perform FN. Families will then be randomized to one of 16 possible combinations of FN delivery strategies (2 × 2 × 2× 2 factorial design). The primary outcome measure will be achieving a family-centered goal related to behavioral health services within 90 days of randomization. Implementation data on the fidelity, acceptability, feasibility, and cost of each strategy will also be collected. Results from the primary and secondary outcomes will be reviewed by our team of stakeholders to optimize FN delivery for implementation and dissemination based on effectiveness, efficiency, and cost. We will also explore how one might integrate decisions related to equity into the optimization process.
Title: Human-centered design within the MOST framework for intervention development: Designing with implementation in mind (Lindsey Knowles & Karey O’Hara)
Abstract: The Multiphase Optimization Strategy (MOST) consists of three phases - Preparation, Optimization, and Evaluation – and each phase has explicit goals and a variety of research methods to achieve them. Methods for achieving Optimization and Evaluation phase goals are well-developed. However, methods used to achieve Preparation phase goals are often highly researcher specific and concrete ways to achieve preparation phase goals is a priority area for further development (Collins et al., 2021). We propose that the Discover, Design, Build, and Test (DDBT; Lyon et al., 2019) framework provides a theory-driven and methods-rich roadmap for achieving the goals of the Preparation phase of MOST, including specifying the conceptual model, identifying, designing, and testing candidate components, and defining the optimization objective (i.e., operational definition of the best-expected outcome under key constraints of the implementation setting). The DDBT framework utilizes strategies from the fields of human-centered design and implementation science to study and improve the usability of candidate intervention components as a key determinant of intervention implementation (Lyon et al., 2019). The MOST and DDBT frameworks share many conceptual features, including an explicit focus on implementation determinants, being iterative and flexible in nature, and designing interventions for the greatest public health impact. In this presentation, we propose and describe combining the MOST and DDBT frameworks to enhance the intervention optimization process. We will begin with a brief overview of DDBT and MOST (focused on the Preparation phase) and discuss the overlap in core principles shared by MOST and DDBT approaches to intervention development. Then, we will discuss practical applications of this synthesized approach by describing how the human-centered design and implementation science strategies leveraged in DDBT can be used to address each MOST Preparation phase goal and present an applied example.
References
1. Guastaferro, K. & Collins, L.M. (2021). Optimization methods and implementation science: An opportunity for behavioral and biobehavioral interventions. Implementation Research and Practice, 2, 1-5.
2. Broder-Fingert, S., Kuhn, J., Sheldrick, R.C., Chu, A., Fortuna, L., Jordan, M., Rubin, D., & Feinberg, E. (2019). Using the multiphase optimization strategy (MOST) framework to test intervention delivery strategies: A study protocol. Trials, 20, 728.
3. Lyon, A. R., Munson, S. A., Renn, B. N., Atkins, D. C., Pullmann, M. D., Friedman, E., & Areán, P. A. (2019). Use of Human-Centered Design to Improve Implementation of Evidence-Based Psychotherapies in Low-Resource Communities: Protocol for Studies Applying a Framework to Assess Usability. JMIR Research Protocols, 8(10), e14990. https://doi.org/10.2196/14990
4. Collins, L. M., Strayhorn, J. C., & Vanness, D. J. (2021). One view of the next decade of research on behavioral and biobehavioral approaches to cancer prevention and control: Intervention optimization. Translational Behavioral Medicine, 11(11), 1998–2008. https://doi.org/10.1093/tbm/ibab087
The process of designing for the implementation of digital mental health interventions in diverse settings
Authors
Dr. Ashley Knapp - Northwestern University Feinberg School of Medicine
Dr. Kaylee Kruzan - Northwestern University Feinberg School of Medicine
Dr. Emily Lattie - Northwestern
Dr. Andrea Graham - Northwestern University Feinberg School of Medicine
Abstract
Despite significant progress in the design and development of digital mental health interventions, most are not widely accessible or available to the public. Reducing the burden of mental health conditions through providing broad access to evidence-based care is a promise of digital intervention, yet examples of the successful implementation of these interventions in practice remains relatively limited. Instead, the design of digital mental health interventions must include attention to how they will be implemented within the workflows and processes of the settings in which they are used. The Accelerated Creation-to-Sustainment (ACTS) Model has been proposed as a framework for designing technologies, service protocols, and implementation plans for digitally-delivered mental health services. In this symposium, we present four examples of applying the ACTS model to design for the implementation of digital mental health interventions in diverse settings. The first presentation will describe the process of partnering with a community-based teen program within a public library through advisory boards and needs assessment interviews to design for the implementation of technology-enabled mental health services for teens within the library. The second presentation will focus on a collaboration with the largest mental health advocacy organization in the US to create digital screening and intervention options for non-suicidal self-injury. The third presentation will be of findings from co-designing a clinical service plan for implementing a technology-enabled service with care coordinators in a large urban healthcare system. The fourth presentation will present on how service design methods were applied to design for the implementation of a guided self-help digital intervention for binge eating and weight management in a specialty clinic. The session will close with a discussion of critical future directions for the sustainable implementation of digital mental health interventions.
Title: Implementation of Teen Technology-Enabled Mental Health Services into Public Libraries
Background: Developing digital mental health (MH) services has high potential to extend reach and maximize impact; however, it is critical to understand the needs and priorities of the organizations and individuals who will implement and use these services prior to implementation. Our research team partnered with a community-based teen program within a public library, which is a key social service system for vulnerable populations, to inform implementation of digital MH services for teens.
Methods: Using an updated Accelerated Creation-to-Sustainment (ACTS) Model as a guiding framework, we worked with two Teen Investigators to facilitate teen and adult community advisory boards to understand the needs, priorities, and strengths of the surrounding communities. Next, using the “Create” ACTS phase, we conducted needs assessment interviews with teen patrons (n = 16) and library staff (n = 17) to understand the preferred digital MH services as well as the organizational and individual determinants of implementing these services. The majority of teens in the advisory board and those involved with interviews identified as BIPOC.
Results: First, we will discuss updates to the ACTS Model that includes pre-“Create” activities focused on partnership building. Then, we will exemplify these updates by going over the themes from the advisory board discussions (e.g., insufficiency of current teen MH resources, especially for BIPOC youth) and the preferred MH services indicated by teens (i.e., digital skill-building apps for teens) and library staff (i.e., training curriculum on supporting teens MH needs) from needs assessment interviews.
Conclusion: Using the ACTS Model, we better understand the values, priorities, and needs of the organization that will deploy digital MH services and those served by the tools to ensure uptake and sustainability of designed services. Next steps are to co-create strategies to best implement a staff MH training curriculum and protocol for teens to “check-out” commercial digital MH apps for free.
Title: Designing a publicly accessible digital intervention for non-suicidal self-injury: Lessons from early design and implementation planning
Background: Nonsuicidal self-injury (NSSI) is a high-risk behavior characterized by low rates of treatment-engagement. I describe research activities aimed at developing a digital intervention for young adults with NSSI in parallel with implementation planning in partnership with [NAME], the nation’s oldest and largest mental health advocacy group. Over 5 million people take online screeners through [NAME's] website per year, providing access to a diverse population at risk of worsening symptoms.
Methods: This research involved meetings with our community partner to understand their needs, their constituents, and the best way to integrate the intervention within their existing workflows and qualitative interviews with young adults with lived experience of NSSI. Prototype feedback sessions are underway and will be followed with usability testing and a randomized-controlled trial.
Results: In early conversations, our partners at [NAME] revealed their desire to expand resources for NSSI. Though [NAME] has several information sheets for NSSI it does not currently have a screener to identify visitors with NSSI. Therefore, a direct line to provide individuals with our digital intervention, and route people to existing services, is needed. We convened a set of experts to develop a NSSI Severity Screener, which will be hosted on [NAME], and validated in a sample of [NAME] constituents. In parallel with implementation planning, design activities have revealed three priorities for the intervention: (1) customization to account for varying motivation and goals, (2) features enabling pattern recognition and self-knowledge, and (3) highly personalized suggestions for self-management.
Conclusion: To realize the potential of digital interventions to address mental health conditions, strategic implementation planning is needed throughout the early design through trialing process. In this talk, we describe research activities involved in designing a digital intervention alongside efforts planning for intervention dissemination and sustainment with our partners at [NAME].
Title: Designing the Service Plan for a Technology-Enabled Service: An Example in Care Coordination
Background: When designing a technology-enabled service (TES), key service issues need to be considered. The service of a TES refers to the clinical goals, behavioral strategies, and expected roles of those delivering the TES. We have worked with a division of care coordination in a large urban healthcare system to identify needs and opportunities for a TES to help care coordinators (CCs) support the mental health of their patients who typically have multiple health conditions.
Methods: Our academic team and the care coordination administrators have developed a deep partnership to design a TES that would fit into the workflow of CCs. Through this partnership, we conducted interviews, questionnaires, workflow observations, and focus groups. Qualitative and quantitative data were analyzed using a mixed methods approach.
Results: Questionnaire data identified that a high proportion of patients (>50%) experience symptoms of depression and/or anxiety. Interviews identified that while CCs had numerous skills in supporting behavioral change (including training in motivational interviewing), they were more comfortable discussing physical health concerns. They reported a lack of mental health resources and a high degree of role ambiguity, in that the degree to which they should address mental health with patients was uncertain. Workflow observations identified that dynamic work processes were far less structured than processes observed with healthcare providers who work with patients during prescheduled appointments.
Conclusion: To maximize use of and comfort with a TES for mental health, the service plan must be designed to be flexible and accessible in the brief bursts of time that CCs are communicating with their patients. To fit into their existing workflow, there must be clear guidance to apply existing skills to support patient behavior change in the context of mental health, and maintain a focus on providing resources (e.g. the technology) and support.
Title: Designing for the Implementation of a Digital Behavioral Intervention in Specialty Clinics
Background: Despite the need for intervention, no treatment effectively addresses both obesity and binge eating, leaving this subpopulation stuck in a clinical gap. [NAME] is a newly designed guided self-help intervention to address obesity and binge eating, delivered by mobile device to increase scalability. The ultimate goal is to broadly implement [NAME] in routine clinical practice. To achieve this goal, it is imperative to understand how to implement the intervention in clinical practice.
Methods: We are applying service design methods to understand how to implement [NAME] in clinics that are typical settings where people with obesity and binge eating present for treatment. Service design focuses on the process of how patients and clinicians engage with each other and with technologies over intervention delivery, from the first point of contact to monitoring outcomes over time.
Results: To date, interviews with stakeholders in an outpatient eating disorders program have yielded a delivery model and implementation plan for that clinic. It has been determined that [NAME] would be a welcome addition, especially for patients on the waitlist or referred to the clinic from the hospital’s bariatric surgery program. Stakeholders indicated coaching services would need to be offered externally (i.e., not by clinicians in the practice) due to billing and time constraints. To complement these clinic-level learnings and calibrate them against market trends, we reviewed the digital health marketplace; commercial companies also are delivering coaching in-house as part of their clinical offering. This suggests that our finding that coaching be supported by individuals external to the clinic is a viable approach.
Conclusion: Attending to service design can ensure [NAME] is maximized for clinical impact when it is delivered in practice. Next steps are to complete data collection with additional clinics and test the implementation of [NAME] in a hybrid trial.
References
1. Graham, A. K., Lattie, E. G., Powell, B. J., Lyon, A. R., Smith, J. D., Schueller, S. M., Stadnick, N. A., Brown, C. H., & Mohr, D. C. (2020). Implementation strategies for digital mental health interventions in health care settings. Am Psychol, 75(8), 1080-1092. https://doi.org/10.1037/amp0000686
2. Mohr, D. C., Lyon, A. R., Lattie, E. G., Reddy, M., & Schueller, S. M. (2017). Accelerating Digital Mental Health Research From Early Design and Creation to Successful Implementation and Sustainment. J Med Internet Res, 19(5), e153. https://doi.org/10.2196/jmir.7725
3. Mohr, D. C., Riper, H., & Schueller, S. M. (2018). A Solution-Focused Research Approach to Achieve an Implementable Revolution in Digital Mental Health. JAMA Psychiatry, 75(2), 113-114. https://doi.org/10.1001/jamapsychiatry.2017.3838
4. Knapp, A. A., Carroll, A.J., Mohanty, N., Fu, E., Powell, B.J., Hamilton, A., Burton, N. D., Coldren, E., Hossain, T., Limaye, D. P., Mendoza, D., Sethi, M., Padilla, R., Price, H. E., Villamar, J. A., Jordan, N., Langman, C. B., & Smith, J. D. (2022). A stakeholder-driven process for selecting implementation strategies: An example of supporting adherence to the pediatric hypertension clinical practice guidelines. Implementation Science Communications, 3, 25. DOI: 10.1186/s43058-022-00276-4. PMCID: PMC8900435
5. Knapp, A.A., Cohen, K., Nicholas, J., Mohr, D.C., Carlo, A.D., Skerl, J.J., & Lattie, E.G. (2021). Integration of digital tools into community mental health care settings that serve young people: Focus group study. Journal of Medical Internet Research in Mental Health, 8(8), e27379. https://doi.org/10.2196/27379. PMCID: PMC8414307
6. Kruzan, K. P., Meyerhoff, J., Reddy, M., Mohr, D. C., & Kornfield, R. (2022). “I wanted to see how bad it was:” Mental health self-screening as a critical transition point among young adults. In Proceedings of the 2022 ACM Conference on Human Factors in Computing Systems (CHI)
7. Kruzan, K. P., Meyerhoff, J., Biernesser, C., Goldstein, T., Reddy, M., & Mohr, D. C. (2021). Centering Lived Experience in Developing Digital Interventions for Suicide and Self-injurious Behaviors: User-Centered Design Approach. JMIR Mental Health, 8(12), e31367. https://doi.org/10.2196/31367
8. Witt, K., Spittal, M. J., Carter, G., Pirkis, J., Hetrick, S., Currier, D., Robinson, J., & Milner, A. (2017). Effectiveness of online and mobile telephone applications (‘apps’) for the self-management of suicidal ideation and self-harm: A systematic review and meta-analysis. BMC Psychiatry, 17(1), 297. https://doi.org/10.1186/s12888-017-1458-0
9. Lattie, E.G., Graham, A.K., Hadjistavropoulous, H.D., Dear, B.F., Titov, N., & Mohr, D.C. (2019). Guidance on defining the scope and development of text-based coaching protocols for digital mental health interventions. Digital Health, DOI: 10.1177/2055207619896145
10. Lattie, E.G., Burgess, E., Reddy, M., & Mohr, D.C. (2021). Care managers and role ambiguity: The challenges of supporting the mental health needs of patients with chronic conditions. Computer Supported Cooperative Work. DOI: 10.1007/s10606-020-09391-z
11. Graham, A. K., Munson, S. A., Reddy, M., Neubert, S. W., Green, E. A., Chang, A., Spring, B., Mohr, D. C., & Wildes, J. E. (2021). Integrating User-Centered Design and Behavioral Science to Design a Mobile Intervention for Obesity and Binge Eating: Mixed Methods Analysis. JMIR Form Res, 5(5), e23809. https://doi.org/10.2196/23809
Tackling mental health inequities among youth through stakeholder-centered partnerships: Lessons learned from the implementation of trauma-informed, evidence-based interventions across sectors and the globe
Authors
Mx. Briana Last - University of Pennsylvania
Ms. Christina Johnson - University of Pennsylvania
Ms. Chynna Mills - University of Pennsylvania
Ms. Natalie Dallard - Community Behavioral Health, Philadelphia Department of Behavioral Health and Intellectual disAbility Services
Ms. Sara Fernandez-Marcote - Community Behavioral Health, Philadelphia Department of Behavioral Health and Intellectual disAbility Services
Dr. Rinad Beidas - University of Pennsylvania
Dr. Rosaura Orengo Aguayo - Medical University of South Carolina
Dr. Andel Nicasio - Albizu University
Dr. Aubrey R. Dueweke - East Tennessee State University
Dr. Michael A. de Arellano - Medical University of South Carolina
Dr. Susana Rivera - Serving Children and Adults in Need (SCAN) Inc.
Dr. Judith A. Cohen - Allegheny Health Network
Dr. Anthony P. Mannarino - Allegheny Health Network
Dr. Regan W. Stewart - Medical University of South Carolina
Mr. Noah Triplett - University of Washington
Mr. Rashed AlRasheed - University of Washington
Ms. Clara Johnson - University of Washington
Dr. Anne Mbwayo - University of Nairobi
Ms. Cyrilla Amanya - Ace Africa
Dr. Shannon Dorsey - University of Washington
Ms. Catalina Ordorica - University of Illinois at Chicago
Dr. Brittany Rudd - University of Illinois at Chicago
Mx. Jax Witzig - University of Illinois at Chicago
Dr. Danielle Stern - University of Illinois at Chicago
Ms. Emily Potter - University of Illinois at Chicago
Ms. Lea Parker - Drexel University
Dr. Joseph Gardella - Drexel University
Ms. Angela Pollard - Drexel University
Ms. Rena Kreimer - Drexel University
Dr. Naomi Goldstein - Drexel University
Abstract
Over half of children and adolescents across the world are exposed to a traumatic event, yet most youth who need mental health services do not receive them (Gunaratnam & Alisic, 2017; Kessler et al., 2017). Social inequities confer greater risk for the onset and persistence of post-traumatic stress disorder, are inversely related to treatment access (Koenen et al., 2017), and prevent providers from offering the highest quality services. Implementation efforts to improve trauma-informed care for youth impacted by social inequities are needed. This requires an understanding of the environments of youth and providers as well as sustained collaboration with stakeholders across sectors. Consistent with the conference’s focus on new frontiers, our symposium presents innovative, multisectoral research that can guide scientists, policymakers, and practitioners committed to improving the well-being of trauma-exposed youth.
First , we will present outcomes from two trauma-focused cognitive behavioral therapy (TF-CBT) implementation efforts: one with community clinicians serving publicly insured youth in the city of Philadelphia; and another in Puerto Rico’s largest managed behavioral health organization. Findings from both efforts reveal significant improvement from treatment and point to the importance of linguistic, cultural, and context-specific tailoring. Next, we will present results from a human-centered design study involving lay counselors implementing TF-CBT in Western Kenya. Findings again highlight the importance of cultural and context-specific tailoring, specifically for implementation strategies. Fourth, we will present mixed-methods findings from an evaluation of the Positive School Safety (PSS) coaching program in the Philadelphia School District. The PSS program uses peer coaches to support the implementation of trauma-informed safety practices. Findings suggest the program is acceptable, feasible, and appropriate and highlighted identified barriers and facilitators to improving positive officer-youth interactions.
The discussant, an expert in implementing trauma-informed mental health treatments, will facilitate a discussion on the implications of these findings for the future of implementation research.
Title: Effectiveness of trauma-focused cognitive behavioral therapy in Philadelphia’s publicly funded mental health clinics
Background: Philadelphia is an economically unequal city, with 68% of its youths insured by Medicaid. To address the high rates of trauma faced by Philadelphia’s youths, in 2012, the city began training clinicians serving publicly insured children and adolescents in an evidence-based intervention for posttraumatic stress disorder (PTSD), trauma-focused cognitive behavioral therapy (TF-CBT). This study seeks to evaluate the effectiveness of TF-CBT delivered to youths (ages 3-21) at Philadelphia’s public mental health clinics from 2013-2021—one of the longest evaluations of TF-CBT implementation in a large metropolitan area.
Methods: Data collection occurred in two waves. Wave 1 (n = 114) occurred between 2013-2016 and Wave 2 (n = 91) occurred from 2016-2021. Youths receiving TF-CBT in Philadelphia’s public clinics were diagnostically evaluated with trauma measures, including the Child PTSD Symptom Scales (CPSS-4 and CPSS-5) at 6-month intervals. Data across waves will be combined and multilevel linear models will assess youth mental health treatment outcomes.
Results: Preliminary analyses of each wave are complete; multilevel model analysis across both waves will be completed by July 2022. The 205 participating youths ranged in age from 4-20 years-old (M = 11.88, SD = 3.97). All 205 participating youth completed a baseline assessment, and 131 (63.9%) completed at least one follow-up assessment. From start of treatment to follow-up, both Wave 1 and Wave 2 participants’ PTSD symptom severity significantly decreased (Cohen’s d = 0.34 and Cohen’s d = 0.75, respectively).
Conclusion: Despite the significant stressors and traumatic experiences faced by many youths seeking treatment at Philadelphia’s publicly funded mental health clinics, TF-CBT was effective at improving their PTSD symptoms. Combining youths’ outcome data across waves facilitated the examination of an 8-year implementation effort undertaken by an academic partnership with the city of Philadelphia, revealing the great promise of these sustained and collaborative efforts.
Title: Trauma focused cognitive behavioral therapy with Puerto Rican youth in a post disaster context: Tailoring, implementation, and program evaluation outcomes
Background: Climate change is disproportionately impacting the most vulnerable communities (Bennett & Friel, 2014). Puerto Rico is a prime example of this phenomenon with back-to-back disasters including devastating hurricanes, earthquakes, and the current COVID-19 pandemic. Trauma-Focused Cognitive Behavioral Therapy (TF-CBT) has not yet been systematically evaluated in the Caribbean context, particularly with Hispanic youth exposed to multiple disasters. The objective of this project was twofold: 1) to train mental health providers in Puerto Rico in TF-CBT as part of a clinical implementation project within the largest managed behavioral health organization (MBHO) on the island, and 2) to conduct a program evaluation to determine the feasibility of implementation and the effectiveness of the treatment.
Method: Fifteen psychologists were trained in TF-CBT. These psychologists then provided TF-CBT to 56 children and adolescents, ages 5-18, in community-based mental health clinics and one primary care clinic with a co-located psychologist in Puerto Rico. The mean number of traumatic events reported by youth referred for TF-CBT was 4.11.
Results: Thirty-six out of 56 children enrolled in the project (64.3%) successfully completed all components of TF-CBT. Most of these cases (89%) transitioned successfully to telehealth by early April 2020 (due to COVID-19). Results demonstrated large effect sizes for reduction in youth-reported posttraumatic stress symptoms (PTSS) (Cohen’s d = 1.32), depressive symptoms (Cohen’s d = 1.32), and anxiety symptoms (Cohen’s d = 1.18). Cultural and linguistic tailoring of materials and training, as well as implementation strategies utilized to fit the local post-disaster Caribbean context will be discussed.
Conclusion: These results suggest that it was feasible to train providers in TF-CBT, that providers were able to deliver TF-CBT in community-based settings both in person and via telehealth (due to the COVID-19 pandemic), and that TF-CBT was an effective treatment option to address trauma-related concerns for youth in Puerto Rico in a post-disaster context. This project is an important first step in the dissemination and implementation of evidence-based trauma-focused treatment for Hispanic youth and disaster-affected youth in the Caribbean.
Title: Implementation facilitation to maximize acceptability, feasibility, and usability of mobile phone supervision in Kenya: Results from a pilot study
Background: Supervision is an important implementation strategy to ensure evidence-based psychotherapies (EBPs) are delivered with fidelity; however, the resources required for in-person supervision may limit scale-up and sustainment. Opportunities exist to leverage mobile phones to replace or supplement in-person supervision. However, contextual variables, such as network connectivity and provider preferences, must be considered and addressed. This study sought to evaluate the effect of an implementation facilitation program on the acceptability, feasibility, and usability of mobile phone supervision.
Methods: Using an iterative and mixed-method approach based on Human-Centered Design, lay counselors and supervisors in Western Kenya co-designed a mobile phone supervision facilitation program to address challenges with mobile phone supervision. Counselors were randomized to receive facilitation (n = 30 in intervention, n = 29 in control). Counselors responded to adapted versions of the Acceptability of Intervention Measure, Feasibility of Intervention Measure, and Intervention Usability Scale. Independent samples t-tests were conducted to assess group differences.
Results: Counselors felt that mobile phone supervision was acceptable (M = 4.2; SD = 0.5), feasible (M = 4.0; SD = 0.8), and usable (M = 3.8; SD = 0.6). There were significant differences in acceptability between counselors who received facilitation (M = 4.4; SD = 0.4) and those who did not (M = 4.1; SD = 0.6; p = 0.04). There was also a significant difference in usability between those who received facilitation (M = 4.0; SD = 0.5) and those who did not (M = 3.6; SD = 0.7; p = 0.02). There were no significant differences in feasibility between the facilitation group (M = 4.2; SD = 0.5) and control group (M = 3.8; SD = 0.9; p = 0.06).
Conclusion: By working alongside counselors and supervisors to understand and anticipate barriers to EBP implementation, co-developed implementation facilitation programs may hold promise for improving EBP implementation.
Title: Implementation process and outcomes of a trauma-informed positive school safety program with school safety officers in Philadelphia
Background: Approximately 50% of U.S. students attend a school with police presence (Diliberti et al., 2019). The Positive School Safety Program (PSSP) is a 16-session, manualized peer-to-peer coaching program that teaches trauma-informed school safety skills (e.g., relationship building, behavior management) to school officers to enhance interactions with students and reduce school-based arrests.
Methods: A convergent, mixed methods longitudinal design was used to investigate implementation process and outcomes of the PSSP among school officer coaches in the Philadelphia school district who were trained in the 2020-2021 school year. Via surveys, coaches (n = 25) provided quantitative data regarding knowledge and attitudes across three time points; these data were analyzed using multilevel modeling. Perceptions of program feasibility, acceptability, and appropriateness (Weiner et al., 2017; scale ranges: 1 = completely disagree and 5 = completely agree) of PSSP were assessed post-training and analyzed descriptively. Qualitative interviews (informed by the Exploration, Preparation, Implementation, and Sustainment framework; Aarons et al., 2011), were conducted with coaches and untrained school officers (n = 17), and analyzed via matrix analysis (Averill, 2002). We merged quantitative and qualitative data via joint displays to evaluate convergence.
Results: Mixed methods analyses regarding coaches’ perceptions of the feasibility, acceptability, and appropriateness of the trauma-informed practices revealed coaches’ agreement with PSSP feasibility (M = 4.71; SD = 0.90), acceptability (M = 4.81, SD = 0.68), and appropriateness (M = 4.71, SD = 0.90). Qualitative results supported and expanded upon the survey results. Coaches noted that the success of program implementation was contingent on administrative buy-in. Coaches highlighted barriers to implementation, including logistical constraints of the training (e.g., technology), and potential for low trainee buy-in. Additional analyses will be complete by July 2022.
Conclusion: Schools interested in reducing school-based arrests by implementing the PSSP should consider how they will target identified determinants to support successful implementation in their school context.
References
1. Gunaratnam, S., Alisic, E. (2017). Epidemiology of Trauma and Trauma-Related Disorders in Children and Adolescents. In: Landolt, M., Cloitre, M., Schnyder, U. (eds) Evidence-Based Treatments for Trauma Related Disorders in Children and Adolescents. Springer, Cham. https://doi.org/10.1007/978-3-319-46138-0_2
2. Kessler, R. C., Aguilar-Gaxiola, S., Alonso, J., Benjet, C., Bromet, E. J., Cardoso, G., Degenhardt, L., de Girolamo, G., Dinolova, R. V., Ferry, F., Florescu, S., Gureje, O., Haro, J. M., Huang, Y., Karam, E. G., Kawakami, N., Lee, S., Lepine, J. P., Levinson, D., Navarro-Mateu, F., Pennell, B., Piazza, M., Posada-Villa, J., Scott, K.M., Stein, D.J., Ten have, M., Torres, Y., Viana, M., Petukhova, M.V., Sampson, N.A., Zaslavsky, A.M., & Koenen, K. C. (2017). Trauma and PTSD in the WHO World Mental Health Surveys. European journal of psychotraumatology, 8(sup5), 1353383. https://doi.org/10.1080/20008198.2017.1353383
3. Koenen, K. C., Ratanatharathorn, A., Ng, L., McLaughlin, K. A., Bromet, E. J., Stein, D. J., Karam, E. G., Meron Ruscio, A., Benjet, C., Scott, K., Atwoli, L., Petukhova, M., Lim, C., Aguilar-Gaxiola, S., Al-Hamzawi, A., Alonso, J., Bunting, B., Ciutan, M., de Girolamo, G., Degenhardt, L., Gureje, O., Haro, J.M., Huang, Y., Kawakami, N., Lee, S., Navarro-Mateu, F., Pennell, B.E., Piazza, M., Sampson, N., ten Have, M., Torres, T., Williams, D., Xavier, M., & Kessler, R. C. (2017). Posttraumatic stress disorder in the World Mental Health Surveys. Psychological medicine, 47(13), 2260–2274. https://doi.org/10.1017/S0033291717000708
Advancing pragmatic implementation measure development
Authors
Dr. Anna Lau - UCLA Department of Psychology
Ms. Elizabeth Rangel - SDSU/ UC San Diego Joint Doctoral Program in Clinical Psychology; Child and Adolescent Services Research Center
Dr. Julia Cox - UCLA Semel Institute
Dr. Joyce H. L. Lui - University of Maryland Department of Psychology
Dr. Teresa Lind - Department of Child and Family Development, San Diego State University; Child & Adolescent Services Research Center
Ms. Elizabeth Lane - UCSD Psychiatry
Dr. Debbie Innes-Gomberg - Los Angeles County Department of Mental Health
Dr. Mojdeh Motamedi - University of California San Diego; Child and Adolescent Services Research Center
Dr. Colby Chlebowski - San Diego State University & UC San Diego Department of Psychiatry & Child and Adolescent Services Research Center
Ms. Devynne Diaz - University of California San Diego; Child and Adolescent Services Research Center
Dr. Scott Roesch - San Diego State University
Dr. Aubyn Stahmer - UC Davis MIND Institute
Dr. Allison Jobin - California State University San Marcos; Child and Adolescent Services Research Center
Ms. Alex Kandah - University of Central Florida Department of Psychology
Dr. Mark Ehrhart - Department of Psychology, University of Central Florida
Dr. Marisa Sklar - Department of Psychiatry, University of California, San Diego; Child and Adolescent Services Research Center; UC San Diego ACTRI Dissemination and Implementation Science Center
Ms. Hannah Samuels - UC San Diego Department of Psychiatry; UC San Diego ACTRI Dissemination and Implementation Science Center; Child and Adolescent Services Research Center
Dr. Gregory Aarons - Department of Psychiatry, University of California, San Diego; Child & Adolescent Services Research Center; University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
Dr. Lauren Brookman-Frazee - UC San Diego Dissemination and Implementation Science Center & UC San Diego Department of Psychiatry & Child and Adolescent Services Research Center
Abstract
Pragmatic and cost-effective tools are needed to assess the impact of Evidence Based Practice (EBP) implementation efforts on purported mechanisms and target outcomes. Presentations will focus on the reliability and validity of scalable and feasible measures including provider-report and administrative claims data. Data are presented on proof-of-concept evidence for novel potential indicators of EBP fidelity and reach, as well as new adaptations and expansions of tools for assessing implementation leadership and climate-embedding mechanisms.
Cox et al. examine whether administrative claims data may be used as indicators of the reach of EBP implementation efforts. Findings reveal that EBP procedure codes claimed by providers predict observed in-session therapist delivery of expected strategies within a system-driven implementation in children’s mental health services.
Lind et al. report on the predictive validity of therapist-reported delivery of EBP strategies for explaining variance in child symptom outcome trajectories. Findings indicate that therapist reports of active teaching strategy use on the revised Evidence-Based Practice Concordant Care Assessment (ECCA; Brookman-Frazee et al., 2020) may have utility as quality indicators in children’s mental health services.
Rangel et al. report on the psychometric properties of the Implementation Support Strategies (ISS) measure (adapted from the Assessment of Climate Embedding Mechanisms; Aarons et al., 2017). Within the TEAMS trial (Brookman-Frazee & Stahmer, 2018), providers rated organizational leader use of implementation strategies to support two EBPs for autism. Analyses will examine organizational predictors of implementation support strategy use at 6 months post training.
Kandah et al. present initial psychometric data on measures of Implementation Leadership, Climate, and Citizenship behavior. These measures were initially validated in health settings, and later adapted and expanded for educational settings. The reliability and validity of the expanded measures will be presented along with evidence for relevance in behavioral health settings, highlighting iterative, interdisciplinary approaches to assessing implementation constructs.
Title: The Utility of Administrative Claims to Index the Delivery of EBP Strategies in Children’s Mental Health Services
Background: Administrative claims are a vast, cost-efficient data source that have been used to characterize mental health service delivery (e.g., Hoagwood et al., 2016). Therapist delivery of evidence-based practice (EBP) strategies is an important outcome of EBP implementation efforts. Administrative data may represent one form of therapist-report of EBP delivery, but concerns exist about the quality and depth of these data (Harron et al., 2017). This study describes the frequency and extensiveness of observed EBP strategies relative to what was expected based on the EBP claimed for reimbursement within a system-driven, multi-EBP implementation effort in children’s mental health services.
Methods: In a two-step matching process, we probabilistically matched administrative claims with session-level observer ratings (Brookman-Frazee et al., 2020), yielding 418 sessions from 180 clients and 85 therapists that represent six claimed EBP families by problem target (i.e., trauma, conduct, anxiety, depression) and approach (i.e., theoretical underpinnings of EBP, caregiver present vs. youth-only). We then compared individual EBP strategy scores and problem target composites across the EBP families.
Results: Descriptive analyses revealed several EBP strategies occurred with moderate to high frequency, with some variability across the EBP families. The extensiveness for many items also fell in the moderate range, with some notable outliers (e.g., when it occurred, exposure was delivered with relatively low extensiveness). Similarly, the problem target composites yielded moderate extensiveness for all but one of the EBP families.
Conclusion: Therapists claiming EBPs were largely observed delivering EBP strategies at a moderate level of extensiveness, but some strategies were favored over others. These findings may improve confidence in using administrative claims to index delivery of EBPs following implementation efforts and highlight a number of measurement challenges.
Title: Validating Pragmatic Therapist-Report of EBP Delivery: Which EBP strategies are Associated with Child Outcome Trajectories?
Background: Pragmatic therapist-report measures of EBP implementation can support and evaluate implementation efforts. As a crucial step in developing a pragmatic assessment, the current study examined the predictive validity of therapist-reported delivery of multiple EBP strategies for client outcomes in children’s mental health services.
Methods: In two large, diverse county mental health systems in Southern California, data were obtained for 1,337 sessions with 239 children delivered by 195 community therapists. Children (Mage = 11.8 years, SD = 3.7) presented with mood (32%), anxiety (22%), conduct (23%), trauma (15%), and other (8%) concerns. Therapists reported their delivery of EBP strategies on the revised Evidence-Based Practice Concordant Care Assessment (ECCA; Brookman-Frazee et al., 2020). Therapists rated the extensiveness with which they delivered 25 content strategies (e.g., relaxation, praise) and 12 technique strategies (e.g., modeling, practice/role-play) in each session. On average, 5.6 session reports (SD = 2.5) were obtained, and caregivers reported symptoms on the Brief Problem Checklist (Chorpita et al., 2010) at baseline, weekly over two months, and at four months. Multilevel models examined whether mean extensiveness of each EBP strategy predicted the trajectory of child outcomes over time.
Results: Overall, more technique (4 of 12) than content strategies (1 of 25) were significantly associated with child outcome trajectories. For techniques, more extensive use of Psychoeducation and Performance Feedback were associated with a greater decline in child total symptoms, and more extensive use of Assigning/Reviewing Homework was associated with a greater decline in externalizing symptoms. More extensive use of Addressing Barriers was associated with a smaller decline in child total symptoms. For content strategies, more extensive use of Cognitive Restructuring was associated with a greater decline in total symptoms.
Conclusion: Therapist-reported active teaching techniques appeared to contribute to symptom improvement over the course of treatment and may constitute quality indicators in community mental health.
Title: Implementation Support Strategies (ISS) Measure: Assessing Factor Structure and Effects of Multi-Level Implementation Strategies
Background: The Implementation Support Strategies (ISS) measure is a 20-item questionnaire adapted from the Assessment of Climate Embedding Mechanisms (ACEM) [1] to measure use of strategies supporting implementation of evidence-based interventions (EBI) in the TEAMS study [2]. This study is a Hybrid Type 3 implementation trial testing two implementation strategies when paired with two EBIs for ASD (AIM Hi, CPRT). The TEAMS Leadership Institute (TLI) is an organizational strategy targeting implementation leadership and climate. TEAMS Individualized Provider Strategies for Training (TIPS) is a provider-level strategy to facilitate provider engagement in EBI training. This study evaluates the initial psychometrics and predictors of the leader-report version of the ISS.
Methods: The ISS was completed by 159 leaders from 62 programs at 6 months following training in the EBI and implementation interventions. A series of exploratory factor analyses using principal axis factoring (PAF) (3) and direct oblimin rotation explored the factor structure of leader reported implementation support strategies. Predictors of the ISS scores were explored using multiple regression analyses.
Results: The PAF indicated that a 1-factor solution best explained the data with 60.17% of the variance explained by the solution. All 20 variables loaded on the one factor (values ranged from .592 to .856). Multiple regression analysis models included leader characteristics (age, years of education) and implementation condition. The overall model was significant, F (7, 76) = 4.70, p < 0.01, = 0.302, with more extensive use of implementation support strategies in the TLI condition relative to TIPS only or standard training. (ß = 24.27, p < 0.001).
Conclusion: Exploratory factor analyses support a unidimensional structure of the ISS and use of total scores in analyses. The results suggest the implementation leadership strategy was effective at increasing leader reported use of implementation strategies within their organization. Future directions will include analyses provider-rated ISS.
Title: Measuring implementation-focused organizational constructs: An update and extension
Abstract: Over the past decade, there has been an increase in measures focused on the organizational context as it relates to evidence-based practice (EBP) implementation. Three such measures target implementation leadership, implementation climate, and implementation citizenship behavior. Implementation leadership captures how much leaders emphasize and support EBP implementation in their units (Aarons et al., 2014). Implementation climate addresses shared perceptions of the extent to which the policies, practices, and procedures in an organization are aligned with the goal of EBP implementation (Ehrhart et al., 2014; Klein et al., 2001; Weiner et al., 2011). Finally, implementation citizenship behavior focuses on providers going above and beyond requirements to support and advocate for EBP implementation (Ehrhart et al., 2015). After initial development and validation in the context of behavioral health, social services, and nursing, these measures were adapted and extended for educational settings (e.g., Lyon et al., 2018). That effort resulted in multiple new dimensions identified for each of the three constructs, including communication and vision/mission for implementation leadership, use of data, existing supports, and integration for implementation climate, and taking initiative and advocacy/boosterism for implementation citizenship behavior. This presentation will address recent efforts to validate the extended version of these measures back into the context of behavioral health. Data will be presented from a study of the implementation of combined motivational enhancement/cognitive-behavioral therapy (MET/CBT) and a study of the implementation of motivational interviewing, both in mental health and substance use treatment. The psychometric functioning of the measures will be reviewed, including confirmatory factor analyses and construct-based validity evidence. Overall, this presentation will demonstrate ongoing efforts to strengthen measure validity over time, and how research across disciplines can mutually influence our understanding of key implementation constructs.
References
1. Aarons, G. A., Ehrhart, M. G., & Farahnak, L. R. (2014). The implementation leadership scale (ILS): Development of a brief measure of unit level implementation leadership. Implementation Science, 9, 45.
2. Brookman-Frazee, L., & Stahmer, A. C. (2018). Effectiveness of a multi-level implementation strategy for ASD interventions: study protocol for two linked cluster randomized trials. Implementation Science, 13(1), 1-14. https://doi-org.libproxy.sdsu.edu/10.1186/s13012-018-0757-2
3. Brookman-Frazee, L., Stadnick, N. A., Lind, T., Roesch, S., Terrones, L., Barnett, M. L., Regan, J., Kennedy, C. A., F. Garland, A., & Lau, A. S. (2020). Therapist-observer concordance in ratings of EBP Strategy Delivery: Challenges and targeted directions in pursuing pragmatic measurement in children’s Mental Health Services. Administration and Policy in Mental Health and Mental Health Services Research, 48(1), 155–170. https://doi.org/10.1007/s10488-020-01054-x
4. Chorpita, B. F., Reise, S., Weisz, J. R., Grubbs, K., Becker, K. D., & Krull, J. L. (2010). Evaluation of the Brief Problem Checklist: child and caregiver interviews to measure clinical progress. Journal of Consulting and Clinical Psychology, 78(4), 526. https://doi.org/10.1037/a0019602
5. Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2014). Assessing the organizational context for EBP implementation: The development and validity testing of the Implementation Climate Scale (ICS). Implementation Science, 9, 157.
6. Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2015). Going above and beyond for implementation: The development and validity testing of the Implementation Citizenship Behavior Scale (ICBS). Implementation Science, 10, 65.
7. Harron, K., Dibben, C., Boyd, J., Hjern, A., Azimaee, M., Barreto, M. L., & Goldstein, H. (2017). Challenges in administrative data linkage for research. Big Data & Society. https://doi.org/10.1177/2053951717745678
8. Hoagwood, K. E., Essock, S., Morrissey, J., Libby, A., Donahue, S., Druss, B., Finnerty, M., Frisman, L., Narasimhan, M., Stein, B. D., Wisdom, J., & Zerzan, J. (2016). Administration and Policy in Mental Health and Mental Health Services Research, 43(1), 67-78. https://dx.doi.org/10.1007%2Fs10488-014-0620-y
9. Klein, K. J., Conn, A. B., & Sorra, J. S. (2001). Implementing computerized technology: An organizational analysis. Journal of Applied Psychology, 86, 811–824.
10. Lyon, A. R., Cook, C. R., Brown, E. C., Locke, J., Davis, C., Ehrhart, M. G., & Aarons, G. A. (2018). Assessing organizational implementation context in the education sector: Confirmatory factor analysis of measures of implementation leadership, climate, and citizenship. Implementation Science, 13, 5.
11. Meyer, L. S., Gamst, G. C., & Guarino, A. J. (2006). Principal components and factor analysis. In L. S. Meyer & G. C. Gamst (Eds.), Guarino Applied multivariate research: Design and interpretation (pp. 465–514). Sage Press.
12. Weiner, B. J., Belden, C. M., Bergmire, D. M., & Johnston, M. (2011). The meaning and measurement of implementation climate. Implementation Science, 6, 78.
A multipronged approach for creating generalizable and actionable implementation knowledge to end the HIV epidemic in the US
Authors
Dr. Dennis Li - Northwestern University Feinberg School of Medicine
Dr. James Merle - University of Utah
Dr. Artur Queiroz - Northwestern University Feinberg School of Medicine
Dr. Virginia Mckay - Brown School, Washington University, St. Louis, MO, United States
Ms. Nanette Benbow - Northwestern University Feinberg School of Medicine
Dr. Lisa Hirschhorn - Northwestern University Feinberg School of Medicine
Mr. Brennan Keiser - Northwestern University Feinberg School of Medicine
Dr. Brian Mustanksi - Northwestern University Feinberg School of Medicine
Ms. Ana Pachicano - Northwestern University Feinberg School of Medicine
Dr. JD Smith - University of Utah
Mr. Juan Villamar - Northwestern University Feinberg School of Medicine
Dr. Alithia Zamantakis - Northwestern University Feinberg School of Medicine
Abstract
The national Ending the HIV Epidemic (EHE) plan sets a goal to virtually eliminate new HIV infections in the United States by 2030, predicated on the fact that effective tools exist for diagnosis, prevention, and treatment. The current scientific challenge is how to implement these tools effectively and with equity. Under EHE, the National Institutes of Health (NIH) funded the Implementation Science Coordination Initiative (ISCI) to support rigorous and actionable implementation research (IR) in HIV. The primary goal of ISCI is to develop generalizable implementation knowledge that can then be efficiently disseminated to inform and support effective implementation of HIV services across the country.
In this symposium, we will briefly describe the formation of ISCI and the services it provides to 147 NIH-funded IR studies and the HIV field. We will then explicate the multiple pathways we have established to create generalizable and actionable HIV implementation knowledge: The first presentation will describe how we develop generalizable knowledge from local findings among ongoing IR studies through coordination, support, and data harmonization. The second presentation will describe how we develop generalizable knowledge through the systematic synthesis of published HIV IR literature using implementation science frameworks. The third presentation will describe how we rapidly disseminate HIV IR knowledge to inform implementation practice and future research through two interactive dashboards. The fourth presentation will describe how we evaluate research on implementation strategies to identify best practices for implementation.
Our ability to reach EHE 2030 goals requires rapid translation of research to practice through a simultaneous, multipronged approach. The methods, tools, and lessons learned from each of these pathways can also inform the how implementation science is applied to other health domains to achieve implementation goals.
Title: Coordination of NIH-funded HIV implementation research to develop generalizable knowledge from local knowledge
Background: In the first three years of the US Ending the HIV Epidemic (EHE) plan, the National Institutes of Health funded 147 planning/pilot projects in 36 high-priority jurisdictions to begin studying implementation of evidence-based HIV interventions in local healthcare and public health systems. The Implementation Science Coordination Initiative (ISCI) was funded to maximize the value of implementation science in these projects through technical assistance and coordination so that local findings could contribute to generalizable knowledge. Although some frameworks exist to facilitate this process, additional guidance and infrastructure were needed to help align the disparate studies and harmonize their data. This presentation will describe both the tools used and preliminary findings from this work.
Methods: We developed surveys in REDCap to periodically collect information on project descriptors, progress/processes, implementation determinants (organized by the Consolidated Framework for Implementation Research [CFIR]), and evaluation of ISCI services. Using a modified Delphi method with HIV implementation research experts, we developed a crosswalk of 71 HIV implementation outcomes organized by stage of implementation research to guide operationalization of and collect data about outcomes from projects.
Results: HIV pre-exposure prophylaxis was the most frequently studied intervention among projects. The most common implementation partners were health departments, community-based organizations, and federally qualified health centers. Most projects used CFIR for determinants and the RE-AIM and Proctor frameworks for outcomes. Across interventions, intervention complexity and cost, patient needs and resources, and external policies were the most frequently identified barriers, but projects also reported numerous facilitators across CFIR domains. Acceptability, appropriateness, and feasibility were the most frequently measured outcomes.
Conclusion: Monitoring the NIH-funded EHE projects facilitates the building up of generalizable knowledge emerging from individual studies at an accelerated pace and helps identify remaining gaps in the portfolio requiring further research to ensure equitable scale-up of interventions to achieve EHE goals.
Title: Systematic review in HIV implementation research: Findings and considerations for reporting
Background: Integrating implementation determinants and strategies that have been explored and evaluated across the continuum of HIV/AIDS prevention, testing, linkage to treatment, and care is needed and could benefit both researchers and practitioners. Of the existing systematic reviews of HIV-related implementation research, all focused on only a small subset of priority populations or settings, none explored implementation strategies, and only two used an implementation framework. Moreover, although reporting standards for implementation strategies exist (Proctor et al., 2013), sufficient details are rarely reported in published studies (Gold et al., 2016). The extent to which strategies are specified in the HIV implementation literature is largely unknown, though likely underspecified.
Methods: Our team completed a systematic review of U.S. studies that identified multilevel determinants of Pre-Exposure Prophylaxis (PrEP). Determinants were coded using the updated CFIR2.0 (Damschroder et al., 2022). A subset of these studies (n=∼100) also included implementation strategies, of which we are in the process of coding using an iteratively developed codebook that incorporates established frameworks and reporting standards (Powell et al., 2015; Proctor et al., 2013; Proctor et al., 2011).
Results: We identified over 1,900 measured determinants from 240 peer-reviewed articles using CFIR2.0. Findings concerning determinants suggested near-saturation of determinants at the patient level (i.e., innovation determinants in CFIR2.0). Initial findings of strategy coding also indicate that implementation strategy specificity is lacking and if present, is often difficult to extract.
Conclusion: Future inquiries should focus on system-level determinants that influence provision of PrEP in existing and new settings as well as using those identified determinants to develop or select strategies. We present a framework, which all studies examining implementation strategies can use, to increase alignment with reporting standards and reduce research burden. Our team is conducting similar ongoing and future systematic reviews around HIV/AIDS testing and linkage to care, and treatment.
Title: Disseminating HIV implementation science: Development of interactive visualization dashboards
Background: Implementation Science is a focus of the US Ending the HIV Epidemic (EHE) initiative, but use of implementation research (IR) findings is still limited due to lack of guidance, complex taxonomies of concepts, and difficulty navigating available evidence. To make IR findings more accessible to practitioners and researchers, the Implementation Science Coordination Initiative developed two data visualization tools: a HIV implementation review dashboard and a dashboard of NIH-funded EHE projects.
Methods: Based on findings from a systematic review of pre-exposure prophylaxis (PrEP) determinants and ongoing data collection from EHE projects, our team (1) created databases in MS Excel; (2) evaluated variables, measures, and data visualizations for inclusion; and (3) developed the dashboards in MS Power BI. Following principles of user-centered design, we engaged personas of potential end users (researcher and practitioner consumers of HIV IR) to inform decisions throughout tool development and testing.
Results: The review dashboard currently allows users to examine more than 1,900 determinants of PrEP implementation, coded by the Consolidated Framework for Implementation Research (CFIR), from 239 peer-reviewed articles. Its interface comprises two primary screens that organize results at the (1) paper level, with filtering enabled by year, priority population, study participants, setting, and US region, and (2) determinant level, with additional filters by CFIR constructs, valence, and data collection method. Users can export a list of articles and/or determinants based on their search. The EHE project dashboard allows users to similarly explore the portfolio of projects, filtering on geographic location, intervention, EHE pillar, implementation partner(s), priority population(s), frameworks, and study findings.
Conclusion: These dashboards make public and facilitate access to information on NIH-funded IR projects and IR findings to inform current implementation practice and build future research to end the HIV epidemic. Results from ongoing reviews of other HIV interventions and additional project data will be added over time.
Title: Developing Criteria to evaluate and identify best practice implementation strategies
Background: A critical mass of evidence is available to synthesize and evaluate tested implementation strategies targeting evidence-based HIV interventions. While there is wide discussion about the type of evidence needed for implementation strategies (Brownson et al., 2022), there are no established criteria to evaluate and recommend these implementation strategies in research and practice. We describe our process for developing these criteria.
Methods: We conducted a literature review of existing criteria and approaches to evaluate strategies and key informant interviews with implementation scientists, HIV service providers, quality improvement practitioners and researchers (N = 10). Participants were asked about existing criteria used to evaluate scientific evidence, what special considerations are needed to evaluate evidence for implementation strategies, and how to best evaluate multi-component implementation strategies. Data were analyzed to identify potential criteria, corresponding levels of evidence, and emergent themes.
Results: While our literature review found evaluation tools and criteria for evidence-based interventions (EBIs; e.g., Guyatt, 2008), we did not find any existing criteria for implementation strategies. Participants identified criteria routinely used to evaluate interventions, like effectiveness and effect size, but also identified implementation-specific criteria. In alignment with implementation-specific outcomes (Proctor, 2011) and evidence needs (Brownson, 2022), this included strategy feasibility, cost, and applicability across contexts. Additionally, our participants overwhelmingly wanted the evaluation tool we develop to be useful for and accessible to implementers. Participants displayed this desire when discussing how to handle multicomponent strategies, tool dissemination, and criteria of importance for assessing implementation strategies. We will discuss how results will be used to inform a Delphi survey to be conducted with a larger set of stakeholders similar to participants. Finally, we will discuss lessons learned, complexities of developing evaluation criteria, and recommendations for improvement.
Conclusion: We describe our process for developing criteria to evaluate implementation strategies and emerging results suggest criteria must balance scientific rigor and effectiveness with practical implementation.
References
1. Queiroz, A., Mongrella, M., Keiser, B., Li, D. H., Benbow, N., & Mustanski, B. (in press). A profile of the portfolio of NIH-funded HIV implementation research projects to inform Ending the HIV Epidemic strategies. Journal of Acquired Immune Deficiency Syndromes.
Mustanski, B., Smith, J. D., Keiser, B., Li, D. H., & Benbow, N. (in press). Supporting the growth of domestic HIV implementation research in the U.S. through coordination, consultation, and collaboration: How we got here and where we are headed. Journal of Acquired Immune Deficiency Syndromes.
2. Li, D. H., Benbow, N., Keiser, B., Mongrella, M., Ortiz, K., Villamar, J., Gallo, C., Deskins, J. S., Xavier Hall, C., Miller, C., Mustanski, B., & Smith, J. D. (in press). Determinants of implementation for HIV pre-exposure prophylaxis based on an updated Consolidated Framework for Implementation Research: A systematic review. Journal of Acquired Immune Deficiency Syndromes.
3. Damschroder, L. J., Reardon, C. M., Opra Widerquist, M. A., & Lowery, J. (2022). Conceptualizing outcomes for use with the Consolidated Framework for Implementation Research (CFIR): the CFIR Outcomes Addendum. Implementation Science, 17(1), 7. https://doi.org/10.1186/s13012-021-01181-5
4. Gold, R., Bunce, A. E., Cohen, D. J., Hollombe, C., Nelson, C. A., Proctor, E. K., Pope, J. A., & DeVoe, J. E. (2016). Reporting on the Strategies Needed to Implement Proven Interventions: An Example From a “Real-World” Cross-Setting Implementation Study. Mayo Clinic Proceedings, 91(8), 1074-1083. https://doi.org/10.1016/j.mayocp.2016.03.014
5. Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015). A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci, 10, 21. https://doi.org/10.1186/s13012-015-0209-1
6. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health, 38(2), 65-76. https://doi.org/10.1007/s10488-010-0319-7
7. Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013). Implementation strategies: recommendations for specifying and reporting. Implementation Science, 8(1), 139. https://doi.org/10.1186/1748-5908-8-139
8. Dixon, B. E., Dearth, S., Duszynski, T. J., & Grannis, S. J. (2022). Dashboards Are Trendy, Visible Components of Data Management in Public Health: Sustaining Their Use After the Pandemic Requires a Broader View. American Journal of Public Health, (0), e1-e4.
9. Park, Y., & Jo, I. H. (2015). Development of the learning analytics dashboard to support students’ learning performance. Journal of Universal Computer Science, 21(1), 110.
10. Brownson, R. C., Shelton, R. C., Geng, E. H., & Glasgow, R. E. (2022). Revisiting concepts of evidence in implementation science. Implementation Science, 17, 26. https://doi.org/10.1186/s13012-022-01201-y
11. Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schunemann, H. J. (2008). GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ, 336(7650), 924-926.
12. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65-76. doi: 10.1007/s10488-010-0319-7.
Equity-centered implementation strategies to promote equitable mental health and educational outcomes for youth in schools
Authors
Dr. Freda Liu - Department of Psychiatry and Behavioral Sciences, University of Washington School of Medicine
Dr. Larissa Gaias - University of Massachusetts Lowell
Dr. Kimberly Arnold - Department of Family Medicine and Community Health, Department of Psychiatry, University of Pennsylvania Perelman School of Medicine
Dr. Michael Pullmann - University of Washington
Ms. Jessica Coifman - Kaiser Permanente Washington Health Research Institute
Ms. Erin McRee - Google
Dr. Jeff Stone - University of Arizona
Ms. Amy Law - University of Washington School of Medicine
Ms. Rosemary Reyes - Department of Psychiatry and Behavioral Sciences, University of Washington School of Medicine
Dr. Calvin Lai - Washington University in St. Louis
Dr. Irene Blair - Department of Psychology and Neuroscience, University of Colorado Boulder
Ms. Heather Cook - University of Washington
Dr. Mylien Duong - OpenMind Platform
Dr. Clayton Cook - Character Strong Organization
Ms. Sharon Kiche - University of Washington
Ms. Casey Chandler - Department of Psychiatry and Behavioral Sciences, University of Washington School of Medicine
Ms. Jodie Buntain-Ricklefs - Department of Psychiatry and Behavioral Sciences, University of Washington School of Medicine
Dr. Catherine Corbin - University of Washington
Prof. Aaron Lyon - University of Washington
Abstract
In recent years, many scholars have recognized the power of implementation science to address persistent inequities in our education and healthcare system (e.g., [1]–[5]); at least 20 peer reviewed articles have been published on this topic within the last 5 years. Despite the many calls to action to apply an equity lens to implementation and dissemination, there are scarce examples of empirical studies or real-world implementation efforts heeding the call. This symposium addresses this gap in the literature and showcases innovative equity-centered implementation strategies by presenting two empirical studies and a conceptual framework for leveraging implementation science to promote equitable mental health and educational outcomes for youth in the K-12 setting. Schools are a critical context in which to address equitable implementation, as they are the most common setting for youth to receive support for their mental health and social-emotional and behavioral well-being, ranging from universal prevention to targeted intervention.
The first presentation will describe the development and feasibility trial of a highly scalable implementation strategy for promoting equitable implementation of any evidence-based intervention (EBI). It is an interactive online training for school-based mental health professionals to address clinician bias—a critical, yet often ignored, determinant of implementation for any EBI.
The second presentation is a mixed methods case study conducted in three schools randomized to the intervention condition of an equity-explicit student-teacher relationship intervention, called Equity-Explicit Establish-Maintain-Restore (E-EMR). This study presents an in-depth examination of implementation determinants across different levels of social-ecological influence that impact the fidelity and likelihood of sustaining E-EMR.
Finally , the last presentation provides an overview of the Adapting Strategies to Promote Implementation Reach and Equity (ASPIRE) framework, which is a generalizable 3-step process for evaluating and adapting implementation strategies to explicitly center the goal of reducing disparities and promoting equitable implementation and service recipient outcomes.
Title: Addressing Clinician Bias to Improve Equitable Implementation of Evidence-Based Interventions
Background: Clinician bias has been identified as a potential contributor to persistent healthcare disparities via negative impacts on the patient-clinician relationship and inequitable delivery of high-quality EBIs.[6] The implementation of any EBI runs the risk of worsening existing health disparities due to inequitable access, delivery, or benefit of the intervention.[7] Clinician bias can be a critical and unaddressed determinant of implementation for any EBI. This presentation will describe an implementation intervention development study and a pilot feasibility trial of the Virtual Implicit Bias Reduction and Neutralization Training (VIBRANT) for school-based mental health clinicians—where most youth in the U.S. access mental healthcare.
Methods: VIBRANT was iteratively developed following an established framework for developing behavioral health interventions leveraging human-centered design principles and methodology [8] with two rounds of lab-based user-testing. Clinicians (N = 12) in the feasibility study—a non-randomized open trial—completed the self-paced online training (45 minutes) along with another Brief Online Training (BOLT) for Measurement-Based Care.
Results: Clinicians rated VIBRANT as highly usable, appropriate, acceptable, and feasible for their school-based practice. Results from this preliminary study suggest that clinicians appeared to demonstrate improvements in implicit bias knowledge, use of bias-management strategies, and implicit biases (as measured by the Implicit Association Test [IAT]) post-training. Moreover, putative mediators (e.g., clinicians’ VIBRANT strategies use, IAT D scores) and outcome variables (e.g., clinician-rated quality of rapport) generally demonstrated correlations in the expected directions.
Conclusion: These pilot results suggest that brief and highly scalable online interventions such as VIBRANT are feasible and promising as an equity-focused implementation strategies that can be paired with any EBI to address an important implementation determinant—clinician bias, with potential downstream impacts on minoritized youth’s experience receiving mental healthcare in schools.
Title: Integrating bias-reduction strategies into practices for improving student-teacher relationships
Background: Equity-Explicit Establish-Maintain-Restore (E-EMR) is an intervention that trains teachers to integrate bias-reduction strategies into practices for improving teacher-student relationships. E-EMR includes multiple equity-explicit implementation supports (e.g., weekly reminder emails, monthly Professional Learning Communities [PLCs]). E-EMR has demonstrated promise for improving outcomes, particularly for students of color and those who start the year with poor-quality teacher relationships.[9] Beyond efficacy, it is also critical to examine factors related to E-EMR implementation to better understand how equity-focused programs can be successfully delivered and sustained over time. The purpose of this mixed methods case study was to examine implementation determinants that facilitate or inhibit adoption and sustainment of E-EMR.
Methods: Data for this study came from three schools that were in the intervention condition of a pilot randomized control trial of E-EMR. Data on implementation determinants were triangulated from teacher and administrator surveys and interviews. Implementation outcomes were captured via teacher-reported fidelity, attendance at monthly PLCs, and administrator-reported plans for sustainment.
Results: All three schools demonstrated moderate-high levels of teacher-reported fidelity, but only one school indicated strong likelihood of sustainment into the next school year. The school with the lowest sustainment intentions also demonstrated the lowest PLC participation. Teachers across schools evidenced favorable individual-level (e.g., self-efficacy, attitudes) and innovation-specific (e.g., design/packaging, evidence-base) determinants. However, high numbers of inner setting barriers emerged in schools with lower likelihood of sustaining E-EMR.
Conclusion: These barriers highlighted the importance of leadership and school climate for promoting implementation outcomes, especially for a program like E-EMR, where teachers are expected to collaborate with one another to promote equitable relationships with students. Given the current focus on integrating anti-bias initiatives into schools, this study has important implications for promoting the successful implementation of such programs.
Title: Adapting Strategies to Promote Implementation Reach and Equity (ASPIRE) [10]
Background: While implementation strategies hold promise for addressing inequitable delivery of mental health or educational services, without explicitly examining implementation strategies through an equity lens, it is unclear the extent to which they will promote equitable implementation or service recipient outcomes. This presentation will describe the Adapting Strategies to Promote Implementation Reach and Equity (ASPIRE) framework, a generalizable process for adapting implementation strategies to explicitly center the goal of reducing inequities in implementation and service recipient outcomes.
Methods: ASPIRE is a three-step process for using an equity lens to adapt implementation strategies. (1) understand the assumptions underlying the implementation strategy. (2) consider the potential sources of disparities if this implementation strategy is used without explicit attention given to equity (e.g., individuals involved in implementation strategy, resources needed, putative mechanisms of the strategy). (3) Identify what needs to change to minimize the potential for the implementation strategy to perpetuate or worsen inequities.
Results: The ASPIRE framework can help implementation researchers and practitioners to critically evaluate and adapt implementation strategies to promote equity in implementation and service recipient outcomes. Using examples from the School Implementation Strategies, Translating ERIC Resources (SISTER) compilation[FFL1], we will demonstrate how ASPIRE can be used to adapt specific implementation strategies to promote racial equity in schools and other real-world settings.
Conclusion: ASPIRE has broad implication for actualizing the potential of implementation science to address longstanding inequities in our healthcare and education systems.
References
1. A. A. Baumann and L. J. Cabassa, “Reframing implementation science to address inequities in healthcare delivery,” BMC Health Serv. Res., vol. 20, no. 1, pp. 1–9, 2020.
2. B. Prusaczyk and A. Baumann, “Eliminating Disparities and Achieving Health Equity Using Implementation Science,” Innov. Aging, vol. 5, no. Suppl 1, pp. 50–50, 2021.
3. R. Y. Nooraie et al., “Advancing health equity through CTSA programs: Opportunities for interaction between health equity, dissemination and implementation, and translational science,” J. Clin. Transl. Sci., vol. 4, no. 3, pp. 168–175, 2020.
4. R. C. Shelton, P. Adsul, A. Oh, N. Moise, and D. M. Griffith, “Application of an antiracism lens in the field of implementation science (IS): Recommendations for reframing implementation research with a focus on justice and racial equity,” Implement. Res. Pract., vol. 2, p. 26334895211049480, 2021.
5. R. C. Shelton, D. A. Chambers, and R. E. Glasgow, “An extension of RE-AIM to enhance sustainability: addressing dynamic context and promoting health equity over time,” Front. Public Health, vol. 8, p. 134, 2020.
6. C. A. Zestcott, I. V. Blair, and J. Stone, “Examining the presence, consequences, and reduction of implicit bias in health care: a narrative review,” Group Process. Intergroup Relat., vol. 19, no. 4, pp. 528–542, 2016.
7. K. C. Lion and J. L. Raphael, “Partnering health disparities research with quality improvement science in pediatrics,” Pediatrics, vol. 135, no. 2, pp. 354–361, 2015.
8. A. R. Lyon et al., “Use of human-centered design to improve implementation of evidence-based psychotherapies in low-resource communities: protocol for studies applying a framework to assess usability,” JMIR Res. Protoc., vol. 8, no. 10, p. e14990, 2019.
9. M. T. Duong et al., “A Cluster Randomized Pilot Trial of the Equity-Explicit Establish-Maintain-Restore Program among High School Teachers and Students,” School Ment. Health, pp. 1–16, 2022.
10. L. M. Gaias, K. T. Arnold, F. F. Liu, M. D. Pullmann, M. T. Duong, and A. R. Lyon, “Adapting strategies to promote implementation reach and equity (ASPIRE) in school mental health services,” Psychol. Sch., 2021.
New frontiers in understanding the multilevel nature of implementation: Recommendations and empirical applications of multilevel theory and research
Authors
Dr. Rosemary Meza - Kaiser Permanente Washington Health Research Institute
Dr. Predrag Klasnja - University of Michigan
Dr. Cara Lewis - Kaiser Permanente Washington Health Research Institute
Dr. Michael Pullmann - University of Washington
Dr. Bryan Weiner - University of Washington
Dr. Mark Ehrhart - Department of Psychology, University of Central Florida
Dr. Rebecca Lengnick-Hall - Brown School, Washington University, St. Louis, MO, United States
Dr. Nate Williams - Institute for the Study of Behavioral Health and Addiction; School of Social Work, Boise State University;
Dr. Cathleen Willging - Center Director, Senior Research Scientist Pacific Institute for Research and Evaluation
Dr. Alicia Bunger - The Ohio State University
Dr. Rinad Beidas - Penn Medicine Nudge Unit, University of Pennsylvania Health System
Dr. Gregory Aarons - Department of Psychiatry, University of California, San Diego; Child & Adolescent Services Research Center; University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
Dr. Emily Becker-Haimes - Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania
Ms. Simone Schriger - Department of Psychology, University of Pennsylvania
Dr. Gwen Lawson - Children’s Hospital of Philadelphia; Perelman School of Medicine, University of Pennsylvania
Dr. David Mandell - Perelman School of Medicine, University of Pennsylvania
Ms. Liza Tomczuk - Perelman School of Medicine, University of Pennsylvania
Dr. Jessica Fishman - Perelman School of Medicine, University of Pennsylvania; Annenberg School, University of Pennsylvania
Dr. Steven Marcus - University of Pennsylvania
Dr. Melanie Pellecchia - University of Pennsylvania
Abstract
Implementation occurs within multilevel systems, requiring a multilevel conceptualization and methodological paradigm for implementation research and practice. Implementation barriers and facilitators operate at multiple levels and these cross-level determinants interact to impede or enable successful implementation. The complex multilevel nature of determinants necessitates implementation strategies that have a multilevel causal influence or that strategically target key determinants that can, in turn, influence subsequent cross-level determinants. However, the use of multilevel implementation strategies in research and practice often do not work better than single level strategies. Implementation scientists and practitioners continue to grapple with understanding the complex multilevel causal pathways through which care delivery outcomes are impacted. Efforts to understand and address the multilevel implementation processes will require multilevel theorizing that explains how determinants at multiple levels interact to produce implementation outcomes and cross-level research approaches that examine the interdependence of determinants at multiple levels.
This symposium aims to stimulate advances in multilevel approaches to implementation research and practice. We provide recommendations for multilevel theorizing and research and showcase empirical studies that demonstrate the application of key recommendations to understand functional relations between determinants and their implications for implementation strategies. The first presentation provides guidelines for advancing multilevel theorizing to advance the fields’ understanding of the relations between determinants and the causal pathways to effectiveness. The next presentation discusses the integration of theory and design decisions and provides guidelines for enhancing multilevel research. The third presentation presents a multilevel theory-driven approach to testing the top-down influence of an organizational determinant on clinician care delivery. The final empirical presentation applies a mixed method approach to understanding multilevel determinants of clinicians intended use of parent coaching. Implications for science and practice are highlighted throughout the presentations.
Title: Principles of multilevel theory development and applications to implementation research and practice
Background: There are calls for IS to advance beyond compilations of implementation determinants to theories that explicate the functional relations between determinants and the causal pathways to effectiveness. Such theorizing must capture the inherent multilevel nature of implementation, but almost none exist. Tools to support development of multilevel implementation theory could guide rigorous multilevel research and practical implementation.
Methods: Drawing on principles of theory development and multilevel organizational theory, this presentation provides conceptual and methodological recommendations for multilevel theory development for IS.
Results: We propose a 6-step process for developing multilevel theory in IS. Steps include defining: 1) what: specify the phenomenon of interest (e.g., care-quality gap) and the constructs theorized to cause or explain the phenomenon; 2) how: specify how constructs are linked, within and across levels including horizontal linkages between constructs, vertical linkages between cross-level constructs, and the form and interaction processes that comprise bottom-up processes, 3) where: specify the hierarchical level(s) at which the causal influence of constructs resides; 4) when: specify the role of time in construct linkages within and across levels; 5) who, where and when: specify the boundaries of the theory; 6) why: clearly articulate assumptions underlying the theory. Implications for multilevel methodological decisions are discussed, such as alignment of hierarchical level, measurement, and analysis, implications of top-down and bottom-up effects on measurement timing, and analytic approaches to model emergence. We round out the overview of this process by illustrating a practical application of multilevel theory to guide implementation efforts that involves prioritizing determinants based their theoretical linkages (i.e., what, how, where, when) and boundary conditions (who, where, when).
Conclusion: Practical implications of multilevel theory for implementation research and practice are illustrated.
Title: Recommendations for conducting rigorous multilevel implementation research with organizations
Background: Many implementation studies incorporate multilevel designs, sampling, and analysis due to the inherently multilevel context within which healthcare is delivered. Generating valid inferences from multilevel research requires careful attention to a range of design issues for which methodologists have developed guidelines to enhance rigor (González-Romá & Hernández, 2022; Klein & Kozlowski, 2000). Awareness and use of these guidelines is variable across implementation studies and reports (Powell et al., 2019). In this presentation, we summarize key design decisions investigators face in multilevel implementation studies and provide guidelines to make these decisions in a methodologically sound way. Our goal is to raise awareness of critical issues, build capacity to address them with rigor, and ultimately, to improve the quality of multilevel implementation research.
Methods: An interdisciplinary team of implementation scientists identified six essential design decisions in multilevel implementation research. For each decision, we explain its scientific and practical importance and describe approaches for enhancing methodological rigor to support valid inferences. Application of qualitative, quantitative, and mixed methods are discussed.
Results: We focus on the following design decisions: operationalizing the multilevel context; aligning levels of theory, measurement and analysis; articulating within and cross-level relationships; specifying temporal scope; accounting for nested structures and measurement dependencies; and avoiding atomistic and ecological fallacies when interpreting findings.
Conclusion: Our methodological recommendations can advance implementation science in three ways. First, they can improve research quality and replicability by providing benchmarks for the conduct and evaluation of multilevel implementation studies. Second, they can promote a common language and reference point that cuts across the diverse disciplines and service settings involved in implementation research. Third, they position the field to innovate methodologically and theoretically by identifying areas where methods are well-established and areas where further development is needed.
Title: Association of organizational climate for evidence-based practice implementation with clinician fidelity to cognitive behavioral therapy: a lagged analysis
Background: Theory and empirical research suggest organizational climate for evidence-based practice (EBP) implementation may be an important target to improve clinician use of EBPs in healthcare (Weiner et al., 2011; Williams et al., 2020); however, this work has been criticized for overreliance on self-reported implementation outcomes and cross-sectional designs (Meza et al., 2021). Using combined data from two studies spanning 7 years, this study applies multilevel theory and methods to test the hypothesis that higher levels of organizational EBP implementation climate prospectively predict improved clinician adherence to cognitive behavioral therapy (CBT) as rated by expert observers.
Methods: Biennial assessments of EBP implementation climate collected in 10 community mental health agencies (Time 1) were linked to subsequent observer ratings of clinician adherence to CBT in clinical encounters with 108 youth (Time 2). Experts rated CBT adherence using the Therapy Process Observation Coding System-Revised Strategies which generated two primary outcomes (a) maximum CBT adherence per session, and (b) average CBT adherence per session.
Results: On average, Time 2 clinician adherence observations occurred 19.8 months after Time 1 agency climate assessments. Adjusting for agency, clinician, and client covariates, a one standard deviation increase in agency EBP implementation climate at Time 1 predicted a 0.63-point increase in clinicians’ maximum CBT adherence per session at Time 2 (p = 0.000), representing a large effect size (d = 0.93) when comparing agencies in the upper versus lower tertiles of EBP implementation climate. Similar findings were evident for average CBT adherence per session (b = 0.23, p < 0.001, d = 0.72). Length of time between assessments of climate and adherence did not moderate these relationships.
Conclusion: Organizational EBP implementation climate is a promising predictor of clinicians’ observed adherence to CBT. Discussion focuses on implications for multilevel theory and methods in implementation science.
Title: Individual and organizational predictors of community clinicians’ intentions to use parent coaching: A mixed method analysis
Background: Clinician’s intentions to use evidence-based interventions may be influenced by clinician factors (e.g., attitudes, norms, self-efficacy) and organizational factors (e.g., implementation climate, leadership, organizational culture). The relationship between individual and organizational factors in shaping clinicians’ intentions and behavior is not well understood, but has direct implications for developing multi-level implementation strategies. Using parent coaching, a complex psychosocial intervention, as an example, we employed a mixed-methods approach to examine relationships between individual and organizational factors and clinicians intended use of parent coaching.
Methods: We used multilevel path models to test hypothesized relationships among organizational factors, individual factors, and clinician’s self-reported intentions to use five core components of parent coaching in a sample of 256 clinicians from 35 agencies. We also conducted semi-structured qualitative interviews based on the Consolidated Framework for Implementation Research (CFIR) with 36 clinicians to probe for perceived barriers and facilitators to parent coaching.
Results: Adjusting for agency, clinicians’ self-efficacy (b = .32, p = .02), attitudes (b = .22, p = .004), and injunctive norms (b = .31, p = .03) were associated with their intentions to use one of the core parent coaching components. Agencies’ implementation climate was not associated with clinicians’ attitudes, norms or self-efficacy (all p’s > .18). Results were generally similar across parent coaching core components and organizational factors. Qualitative findings expanded on the quantitative results by revealing how clinicians described key constructs across levels, including preferences, beliefs, self-efficacy, and organizational support.
Conclusion: These results have direct implications for developing multi-level implementation strategies specific to context. In settings such as early intervention agencies, where clinicians often work isolated from supervisors and peers, implementation strategies should target individual factors, or strengthen the role of the agency. Results also illustrate how qualitative methods can inform explanations of the relationship between individual- and organizational-level factors.
References
1. Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2014). Assessing the organizational context for EBP implementation: The development and validity testing of the Implementation Climate Scale (ICS). Implementation Science, 9(1). https://doi.org/10.1186/s13012-014-0157-1
2. Fishman, J., Beidas, R., Reisinger, E., & Mandell, D. S. (2018). The Utility of Measuring Intentions to Use Best Practices: A Longitudinal Study Among Teachers Supporting Students With Autism. Journal of School Health, 88(5), 388–395. https://doi.org/10.1111/josh.12618
3. Glisson, C. (2002). The organizational context of children’s mental health services. Clinical Child and Family Psychology Review, 5(4), 233–253.
González-Romá, V., & Hernández, A. (2022). Conducting and Evaluating Multilevel Studies: Recommendations, Resources, and a Checklist. Organizational Research Methods. doi:10.1177/10944281211060712
4. Humphrey, S. E., & LeBreton, J. M. (Eds.). (2019). The handbook of multilevel theory, measurement, and analysis (pp. ix-637). Washington, DC: American Psychological Association.
5. Klein, K. J., & Kozlowski, S. W. (2000). Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions. Jossey-Bass.
6. Kozlowski, S. W. J., & Klein, K. J. (2000). A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 3–90). Jossey-Bass.
7. Meza, R. D., Triplett, N. S., Woodard, G. S., Martin, P., Khairuzzaman, A. N., Jamora, G., & Dorsey, S. (2021). The relationship between first-level leadership and inner-context and implementation outcomes in behavioral health: a scoping review. Implementation Science, 16:69. doi:10.1186/s13012-021-01104-4.
8. Powell, B. J., Fernandez, M. E., Williams, N. J., Aarons, G. A., Beidas, R. S., Lewis, C. C., … & Weiner, B. J. (2019). Enhancing the impact of implementation strategies in healthcare: a research agenda. Frontiers in public health, 7. doi:10.3389/fpubh.2019.00003
Weiner, B. J., Belden, C.M., Bergmire, D. M., & Johnston, M. (2011). The meaning and measurement of implementation climate. Implementation Science, 6:78. doi:10.1186/1748-5908-6-78.
9. Williams, N. J., Wolk, C. B., Becker-Haimes, E. M., & Beidas, R. S. (2020). Testing a theory of strategic implementation leadership, implementation climate, and clinicians’ use of evidence-based practice: a 5-year panel analysis. Implementation Science 15:10. doi:10.1186/s13012-020-0970-7.
10. Whetten, D. A. (1989). What Constitutes a Theoretical Contribution? Academy of Management Review, 14(4), 490–495
New frontiers for implementation science in environmental health equity and climate resilience
Authors
Dr. Gila Neta - National Cancer Institute
Dr. Ashlinn Quinn - Berkeley Air
Dr. Nicole Errett - University of Washington
Dr. Aruni Bhatnagar - University of Louisville
Dr. Gregory Aarons - Department of Psychiatry, University of California, San Diego; Child & Adolescent Services Research Center; University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
Abstract
Our environment, including exposures in the physical, built, and social environments, can negatively affect the health of individuals and communities. These environmental exposures disproportionately impact racial and ethnic minority groups, marginalized and under-resourced communities (Brulle & Pellow, 2006). Given the interconnectedness of the environment with individual- and structural-level social determinants of health, ensuring equitable implementation of interventions, programs, and policies that prevent or mitigate these environmental exposures is urgently needed (Onakomaiya et al., 2019; Boyer et al., 2020). This symposium will highlight real-world examples where implementation science can advance environmental health equity and climate resilience. Speakers will describe a range of research efforts to promote the adoption and integration of environmental health interventions and policies that prevent and mitigate potentially harmful exposures and ultimately advance environmental health equity.
Title: Applications of RE-AIM for evaluating clean cookstove programs in low- and middle-income countries
Abstract: Household air pollution from cooking with biomass fuels (such as wood, dung, and crop residues) is associated with a substantial burden of disease globally and contributes to a range of adverse health conditions: from neonatal respiratory infections to adult cardiovascular disease, cataracts, and cancer. This talk will highlight how the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) implementation science framework has been employed to evaluate programs seeking to scale up the use of cleaner cooking technologies in low- and middle-income settings, where cooking with traditional biomass fuels is common. Using examples from published and ongoing work, we will discuss how applying RE-AIM to retrospective and concurrent evaluations of disparate clean cookstove programs has enabled the identification of key common elements tied to program success and/or lack thereof and has helped focus the research agenda for a clean cookstove researcher and implementer community (Quinn et al., 2019). We will also discuss how the use of RE-AIM in this context has had its challenges, for example: adjusting to competing definitions of key terms, application to programs that were not always explicitly designed around health outcomes and overcoming initial unfamiliarity in a practitioner/researcher community. Using this experience with RE-AIM as an example, we will highlight how the tools and frameworks of implementation science can advance the development, adoption, and scale-up of interventions to protect human health from pollutants in the environment and the risks associated with climate change.
Disclosures of Interest: Ashlinn Quinn reports being formerly employed by NIH Fogarty International Center who supported time spent on the presented work. Ashlinn Quinn also reports being currently employed by Berkeley Air Monitoring Group who supported travel to present the work.
Title: Implementation Science opportunities in climate resilience and emergency preparedness
Abstract: The evidence base necessary to guide the nation’s response to public health emergencies and disasters is underdeveloped. As such, research about short- and long-term health impacts, as well as the implementation and effectiveness of response and recovery strategies, is an essential component of disaster and public health emergency response (Ebi et al., 2021). Implementation science in the context of disasters can identify barriers and facilitators to implementation of specific preparedness, response and recovery strategies; inform incident- and context-specific implementation strategies; and promote learning about the effectiveness of response and recovery strategies at community scale. However, those conducting research during disaster response face unique challenges, including those related to research issue identification and prioritization, funding, institutional review board, logistics, and the development of relationships with local communities and researchers. Studying implementation in the context of disasters introduces additional challenges, including those related to the variety of hazards in which evidence-informed innovations may be implemented, short implementation windows, unique implementation environments, and imperfect and evolving evidence. To build capacity to conduct robust implementation science research in the context of a disaster, we propose the development of research networks that integrate communities, public health disaster practitioners, and researchers; the development of research plans, protocols and processes for advance or rapid institutional review board approval; and training for early career an established professionals to create a competent workforce. We will share lessons learned, tools, and resources developed as part of the 2022 University of Washington and National Institute of Environmental Health Sciences Disaster Research Response (DR2) workshop to build such capacity, including a concept of operations plan to guide collaborative disaster research response between academic public health and public health agencies, and a conceptual framework to guide collaborative public health emergency preparedness and response implementation science research.
Title: Implementing community-wide environmental implementation: The Green Heart Project
Abstract: Studies suggest that living in areas of high residential greenness is associated with better mental and physical health, including lower levels of anxiety and depression, cardiovascular and all-cause mortality, and longer survival after cancer diagnosis (Fyfe-Johnson et al., 2020). Individuals who live in greener areas also are likely to differ systematically in their socioeconomic status and health behavior than those who live in less green areas. The Green Heart Project was designed to examine the longitudinal effects of a greening intervention in a neighborhood on the levels of area pollution and community health. The project involves installation of 8000 to 10,000 mature trees and shrubs in a neighborhood of 2 sq. miles and 25,000 residents in Louisville, KY. The ongoing project offers a unique opportunity to develop and test key elements of implementation science ranging from community interactions to evaluate the alignment of neighborhood priorities with the objectives of the project; assess the local context for appropriate implementation; recruit individual participation in planting large mature trees with backyards and front yards; and identify appropriate dissemination approaches, partners and stakeholders, including members of the community, funders, state and local authorities. Using a hybrid approach of discovery and implementation, project personnel are working with community members to evaluate not only health, but service outcomes as well, and to educate the community about the value of urban greenspaces. This is to ensure that the implementation is acceptable and sustainable and that members of the community participate in taking care of the trees that have been installed in the neighborhood. Overall, the objective is to develop evidence-based policy recommendations for greening neighborhoods worldwide.
References
1. Boyer, C. J., Bowen, K., Murray, V., Hadley, J., Hilly, J. J., Hess, J. J., & Ebi, K. L. (2020). Using Implementation Science For Health Adaptation: Opportunities For Pacific Island Countries. Health affairs (Project Hope), 39(12), 2160–2167.
2. Brulle, R. J., & Pellow, D. N. (2006). Environmental justice: human health and environmental inequalities. Annual review of public health, 27, 103–124. https://doi.org/10.1146/annurev.publhealth.27.021405.102124
3. Ebi, K. L., Vanos, J., Baldwin, J. W., Bell, J. E., Hondula, D. M., Errett, N. A., Hayes, K., Reid, C. E., Saha, S., Spector, J., & Berry, P. (2021). Extreme Weather and Climate Change: Population Health and Health System Implications. Annual review of public health, 42, 293–315. https://doi.org/10.1146/annurev-publhealth-012420-105026
4. Fyfe-Johnson, A. L., Hazlehurst, M. F., Perrins, S. P., Bratman, G. N., Thomas, R., Garrett, K. A., Hafferty, K. R., Cullaz, T. M., Marcuse, E. K., & Tandon, P. S. (2021). Nature and Children's Health: A Systematic Review. Pediatrics, 148(4), e2020049155. https://doi.org/10.1542/peds.2020-049155
5. Onakomaiya, D., Gyamfi, J., Iwelunmor, J., Opeyemi, J., Oluwasanmi, M., Obiezu-Umeh, C., Dalton, M., Nwaozuru, U., Ojo, T., Vieira, D., Ogedegbe, G., & Olopade, C. (2019). Implementation of clean cookstove interventions and its effects on blood pressure in low-income and middle-income countries: systematic review. BMJ open, 9(5), e026517. https://doi.org/10.1136/bmjopen-2018-026517
6. Quinn, A. K., Neta, G., Sturke, R., Olopade, C. O., Pollard, S. L., Sherr, K., & Rosenthal, J. P. (2019). Adapting and Operationalizing the RE-AIM Framework for Implementation Science in Environmental Health: Clean Fuel Cooking Programs in Low Resource Countries. Frontiers in public health, 7, 389. https://doi.org/10.3389/fpubh.2019.00389
Mechanisms of change within external implementation support: A conceptual model and applied case examples
Authors
Ms. Rebecca Roppolo - UNC-Chapel Hill
Dr. Will Aldridge - University of North Carolina at Chapel Hill
Ms. Jacquie Brown - Jacquie Brown Consulting
Dr. Brian Bumbarger - Science, Systems & Communities Consulting, LLC
Dr. Renee Boothroyd - UNC-Chapel Hill
Dr. Stephen Phillippi - Center for Evidence to Practice, Health Sciences Center, Louisiana State University School of Public Health
Dr. Rohit Ramaswamy - Cincinnati Children's Hospital Medical Center
Abstract
What does the work of implementation support providers look like? How do those efforts effect change? The goal of this symposia is to introduce a conceptual model, grounded in well-established theories of behavior and organization change, that labels mechanisms of change within the support process and explains how their use may result in improved practice outcomes in theory and in practice through a series of applied practice case examples. We believe this model presents an innovative approach and opportunity to enhance research and the study of “what works?” for implementation support in complex systems by labeling mechanisms of change and proposing relationships between core practice components and improvements to short- and long-term outcomes for those receiving implementation support. After this introduction, three applied case examples will be presented that demonstrate application of the model across various settings and contexts: (1) describing how implementation support practitioners support implementation knowledge transfer, coaching, and capacity building in a provincial education system; (2) describing how a state-level intermediary works to build the implementation capacity of public systems and treatment providers to serve Medicaid-enrolled children; and (3) describing the process of translating an innovative practice model to address both practice-level and organizational-leadership behavior into county child welfare agency contexts. At the conclusion of the presentations, the discussant will make summary statements and moderate a conversation between the presenters and the symposium attendees. It is expected that considerations for future implementation research, implementation practice, and transferability to other contexts will be discussed.
Title: “Core Practice Components and Outcomes of External Implementation Support: A Conceptual Model to Guide Research and Practice” by Will Aldridge (presenting)
Background: External implementation support (EIS) is a well-recognized feature of implementation science and practice, often under related terms such as technical assistance and facilitation of implementation. Recently, there have been acknowledgements of the need to clarify the mechanisms of change through which EIS influences related outcomes. The goal of this presentation is to propose a model of EIS core practice components to describe how implementation support providers (ISPs) might influence intended practice outcomes over time.
Methods: Existing models of EIS have gaps related to addressing practice outcomes at individual and organizational levels, connecting practice activities to intended outcomes, situating within broader empirical and conceptual literature, or grounding in well-established theories of behavior and organization change. The proposed model addresses these gaps by being empirically informed, conceptually situated within a broader theory of change, and guided by eight practice principles and social cognitive theory, with expanded attention to shaping collective actions and improvements at team, organizational, and systems levels. Results: Our model proposes 10 core practice components as the mechanisms of change of EIS. When used according to underlying theory and principles, they are believed to contribute to favorable practice outcomes at individual, team, organizational, and system levels. The model offers flexibility by recognizing both the need for sequential support processes and the demand to practice in ways that are dynamic and responsive. Conclusion: The proposed model is intended to support prospective studies of EIS by conceptualizing discernable practice components with hypothesized relationships to proximal and distal practice outcomes. In practice, the model can inform training and coaching for ISPs, complimenting and extending competency-based approaches. Over time, the model should be refined based on empirical findings and contributions from ISPs across the field. (Note. Abstract text copied from manuscript submitted for publication to Implementation Research & Practice)
Title: “Engaging, Informing, and Supporting Leadership in Facilitating Effective Implementation” by Jacquie Brown (presenting)
Background: It is critical for leadership in an implementing setting to have an understanding of implementation science and its application. However, leadership should not be viewed as only the senior leaders of the organization, as, in complex systems, leadership is cascaded through the organization reflecting role and expertise.
Methods: The case study being presented describes how implementation support practitioners support implementation knowledge transfer, coaching and capacity building through a school system comprising 72 autonomously governed school boards operating under the policies of the Provincial Ministry of Education. The Ministry of Education has funded an intermediary organization to support the integration of mental health promotion, prevention and early intervention at all schools. Each Board is assigned an implementation support practitioner who is available to them to support their implementation processes. Results: Implementation support structures have been developed at every Board to provide implementation leadership. The leadership structure includes a Superintendent responsible for Mental Health with direct responsibility to the Director of Education, the Board Mental Health Lead with responsibility for facilitating the development and implementation of a 3-year Strategic Mental Health Plan and an annual Action Plan, and the Mental Health Leadership Team which includes the managers of regulated mental health professionals Implementation support practitioners. The implementation support practitioners meet regularly with the leadership at each Board to support the dissemination and implementation of resources, programs, and materials for use throughout the system.
Conclusion: This case study illustrates four of the Core Practice Components described in the practice model: #2 Reinforce leaders and teams’ self-regulation of effective implementation performance; #5 Provide adult learning on implementation science and best practices to leaders and teams; #7. Facilitate leaders and teams’ application of skills, resources, and abilities within their context; and #8. Provide supportive behavioral coaching to leaders and teams.
Title: “An Applied Example of The Core Practice Components of External Implementation Support: Scaling Evidence-Based Behavioral Health Interventions within a State Medicaid System” by Brian Bumbarger (presenting) and Stephen Phillippi.
Background: A state-level intermediary (the “Center”) formed in partnership between a university school of public health and state Office of Behavioral Health (OBH) is promoting statewide scale-up of a diverse menu of evidence-based children’s behavioral health treatment models serving Medicaid-enrolled children.
Methods: This state intermediary center focuses on building implementation capacity of both the public system (i.e. policy context) and local children’s behavioral health treatment providers (i.e. practice context), supporting providers’ capacity for adoption and implementation of twelve treatment models, through training, written treatment model descriptions, guides for referral and appropriate matching, business models and navigating Medicaid reimbursement. The co-creation of this approach focused on studying the applied context, through an implementation framework, to deeply understand the specific practitioner and provider capacities necessary to effectively scale these treatment models. The Center analyzes Medicaid claims data and continually assesses the needs of the field through interviews, focus groups and surveys to provide strategic feedback to OBH, managed care organizations (MCOs) and providers to inform model refinement and facilitate continuous quality improvement; facilitates collective learning and adaptive problem solving through provider and MCO learning communities; and facilitates a statewide Implementation Advisory Committee representing key stakeholder groups.
Results: Core practice components in an ecological model of implementation support have been critical to developing capacities, from the behavioral economics that impact provider and patient access and motivation, to the macro-level funding, policy and regulatory facilitators and constraints necessary to support and sustain the initiative’s ambitious goals, and to adapt implementation supports, policy, and technical assistance to meet new implementation and sustainability challenges (such as the COVID pandemic and subsequent shift to telehealth).
Conclusion: The Center embodies each of the implementation scientist-practitioner collaborative Practice Principles outlined in the EIS model and represents a potentially replicable model for other states.
Title: “Using External Implementation Support (EIS) in California’s Child Welfare Practice Model: Leading with Relationships and Adjusting to Context” by Renee Boothroyd.
Background: As part of a federally funded project, implementation specialists from the author’s institution provided EIS to several county child welfare agencies implementing a practice model. The model was co-designed to address both practice-level and organizational-leadership behaviors. Technical processes of support (e.g., training, brokering resources, and consultation) needed to be translated into local agency contexts to address frustration and distrust issues.
Methods: Implementation specialists employed a series of informal Plan-Do-Study-Act cycles to install, reflect upon, and adjust support approaches and activities with county leadership and implementation teams.
Results: Implementation specialists repeatedly focused on building collaborative relationships with their state and local partners (CPC 1). Initial efforts explored sharing power in implementation practice and using co-creation principles to support community partners’ call for authentic change strategies from agency leadership. Support adjusted over time based on context, including shifting language and ways for teaching implementation science; co-developing action steps with local teams to develop their implementation capacity (CPC 6); and adjusting new learning and organizational capacities through small tests of change and reflective coaching (CPCs 7, 8). Support frequently incorporated adaptive leadership by creating space for collective learning and problem solving with local teams (CPC 9). Over time, support involved coaching strategies to reinforce self-regulation of effective practices (CPC 2) and leaned heavily on the principle of local ownership of progress. These factors enabled FPG implementation specialists to successfully transition from the intensive support role (CPC 10) with sustainable change in place.
Conclusion: Reinforcing working alliances and fostering collaborative learning cultures are critical and ongoing aspects of implementation support. Implementation specialists often enter complex local contexts with long-standing adaptive challenges. Incorporating cycles of learning, even informally, can be critical for understanding and adjusting implementation support approaches that meet local teams where they are and invest in them for change.
References
1. Aldridge, W. A., II, Roppolo, R. H., Brown, J., Bumbarger, B. K., & Boothroyd, R. I. (2022). Core Practice Components and Outcomes of External Implementation Support: A Conceptual Model and Case Examples to Guide Research and Practice [Manuscript submitted for publication].
2. Albers, B., Metz, A., & Burke, K. (2020, May 1). Implementation support practitioners - a proposal for consolidating a diverse evidence base. BMC Health Serv Res, 20(1), 368. https://doi.org/10.1186/s12913-020-05145-1
3. Albers, B., Metz, A., Burke, K., Bührmann, L., Bartley, L., Driessen, P., & Varsi, C. (2021). Implementation Support Skills: Findings From a Systematic Integrative Review. Research on Social Work Practice, 31(2), 147-170. https://doi.org/10.1177/1049731520967419
4. Phillippi, S., Beiter, K., Thomas, C., & Vos, S. (2020). Identifying gaps and using evidence-based practices to serve the behavioral health treatment needs of medicaid-insured children. Children and Youth Services Review, 115, 105089.
5. Phillippi, S., Singh, S., & Bumbarger B., et a (2020). Telehealth Among Behavioral Health Providers in Louisiana During COVID-19: A Story of Adoption and Adaptation. Journal of the Louisiana Public Health Association. 1:15–18.
Examining the implementation roadmap, anticipated and unexpected costs, and novel planning implementation strategies employed to scale up FAIR—a community-based intervention to treat and prevent parental opioid misuse and methamphetamine use
Authors
Dr. Lisa Saldana - Oregon Social Learning Center
Dr. Piper Block - Oregon Social Learning Center
Dr. Gracelyn Cruden - Oregon Social Learning Center
Dr. Jason Chapman - Oregon Social Learning Center
Dr. Ryan Singh - Oregon Social Learning Center
Mr. Mark Campbell - Oregon Social Learning Center
Mrs. Shelley Crawford - Oregon Social Learning Center
Abstract
Implementation requires being flexible and responsive to the real-world. As demonstrated by these three papers, flexibility during implementation does not require extensive adaptations to evidence-based interventions or a sacrifice of clinical quality. Rather, following community and service system partners’ preferences for where to work, how, and with whom dramatically increases implementation reach and impact on health and well-being for those who can benefit most.
The first paper exemplifies how implementation is an endeavor to meet community and service system partners’ needs. Leveraging the NIH HEAL Initiative, Families Actively Improving Relationships (FAIR)—an evidence-based program for families involved with or facing risk of involvement with child welfare—was scaled out to four counties in partnership with county and state child welfare partners. FAIR’s evidence-base was developed within the context of treating parental substance use. A Hybrid 2 trial is testing an adaptation of FAIR—PRE-FAIR—that aims to prevent parental initiation or escalation of opioid misuse and/or methamphetamine use. The trial is being pursued in the United States state with the second highest rate of substance misuse and lowest rate of treatment access. The presentation will highlight not only implementation progress, but how strong implementation strategies, fidelity to implementation, and community partnerships can support successful implementation in clinical settings facing important outer context pressures. The second paper exemplifies how the real costs of implementation include both anticipated and unexpected costs such as preparation meetings and hiring. Paper three describes how a system dynamics simulation model that was developed with insights from graduated FAIR parents and clinicians is being used to support Pre-implementation activities such as clinician training and workflow planning. Discussion will describe how the model is also being used to empirically explore how observed PRE-FAIR dosage patterns are impacting clinical workflow and clinic financial solvency.
Title: Building to Scale: Implementation of a Complex Preventive Intervention
Background: FAIR (Families Actively Improving Relationships) is an evidence-based practice for parents referred to the child welfare system. Treatment addresses substance use (primarily opioids and/or methamphetamine), mental health, parenting, and ancillary needs. FAIR is community-based and relies on collaboration with service and community partners.
As part of the NIH Helping to End Addiction Long-term (HEAL) initiative, FAIR was adapted for upstream prevention to be evaluated for clinical effectiveness in a Hybrid Type 2 trial in four rural counties. Counties were selected in collaboration with state leadership. A clearly defined implementation process was followed at the county, clinic, and provider levels. Challenges and opportunities related to the staggering increase in opioid and methamphetamine use in the participating regions and the COVID-19 pandemic will be highlighted.
Methods: The Stages of Implementation Completion (SIC) provides the FAIR implementation roadmap. Pre-implementation was completed in four counties served by three clinics. Each clinic posed unique challenges: (1) de-adoption of previous services, (2) starting a new clinic, and (3) expanding from an existing FAIR program into a new rural area. Strategies were utilized to maintain a thorough pre-implementation process under each set of conditions. Referrals screened as too severe (active diagnosis) for upstream intervention were referred to regular FAIR treatment.
Results: Strong fidelity to the SIC pre-implementation process facilitated program launch in each community. Unique county circumstances created variability in pre-implementation process behavior and resulting SIC scores for time taken to complete activities (duration = 385; 343; 385; and 525 days). Despite steady referrals and treatment entry, the majority have been screened as ineligible for upstream intervention.
Conclusion: Deliberate pre-implementation completion can help to overcome challenges posed by unanticipated contextual factors. Despite successful implementation outcomes for this Hybrid 2 trial, the significant community need for treatment over prevention limits clinical effectiveness testing.
Disclosures of Interest: Lisa Saldana is the developer of the FAIR program being discussed. FAIR can be purchased for adoption.
Title: Implementation Costs and Resource Allocation of a Complex Preventive Intervention
Background: To successfully implement evidence-based programs (EBPs), both intervention and implementation costs must be accounted. EBPs have fixed, expected intervention costs, but there are unexpected costs at each implementation stage. Institutional decision-makers report that unforeseen implementation costs are a foremost impediment to bringing an EBP to fruition. This presentation will explain the distinction between intervention and implementation costs using an example EBP, Families Actively Improving Relationships (FAIR), and demonstrate effective means of documenting and clarifying likely implementation costs.
Methods: The Stages of Implementation Completion (SIC) provides a roadmap for EBP implementation. The Cost of Implementing New Strategies (COINS) approach maps costs onto implementation efforts within each of the eight SIC stages. Pre-Implementation (SIC Stages 1-3) data from two clinics newly implementing FAIR are presented to exemplify the magnitude and diversity of additional implementation costs.
Results: A major source of often ignored costs are the hours that previously established employees (e.g., clinic director, program supervisor, business manager, and clinicians), expend during Pre-Implementation. Purveyor staff spent 103 and 106 hours preparing and meeting with the two clinics. The two clinics spent 30 and 44 hours meeting with the purveyor, and 49 and 33 hours on other pre-implementation activities respectively, all prior to FAIR-specific hiring and training. These resources, as well as fixed implementation costs, are captured and displayed through the COINS tool so that institutional decision-makers can better plan for the entire implementation process.
Conclusion: To successfully advance EBP implementation, decision-makers need information about both intervention and implementation costs. Tools such as SIC and COINS can assist with properly disentangling and displaying cost-related data. Costs such as employee hours during Pre-Implementation are most often overlooked, but there are additional costs throughout the stages of implementation that must be accounted to know the entire scope of resources necessary to successfully implement EBPs.
Title: Leveraging a System Dynamics Model to Facilitate Clinicians’ Learning During Pre-Implementation of an Intervention to Prevent Parental Opioid and Methamphetamine Use Disorder
Background: An intervention to treat parental opioid use disorder (OUD) and methamphetamine use disorder (MUD)—Families Actively Improving Relationships (FAIR)—was rigorously adapted to prevent parental OUD and MUD. This adapted version—PRE-FAIR—was hypothesized to have a reduced dosage (i.e., intensity and duration) compared to FAIR, resulting in more rapid caseload turnover. Turnover changes could have non-linear, dynamic effects on clinical capacity and sustainability. System dynamics was leveraged to identify how PRE-FAIR implementation will impact the clinical workloads and the financial solvency of implementing clinics.
Methods: Semi-structured interviews with graduated FAIR parents (n = 9), clinicians (n = 9) and administrators (n = 2) were analyzed using content analysis. Emergent themes were translated into causal loop diagrams—qualitative system dynamics models—to guide prevention adaptations and generate hypotheses about the typical PRE-FAIR dosage. Diagrams also informed the structure of a quantitative system dynamics model. The model is designed to support clinicians’ learning about how their PRE-FAIR time commitments impact their workload, burnout likelihood, and clinic solvency. The model was calibrated to current FAIR clinical guidelines.
Results: Model testing is ongoing. Thus, insights might evolve. Current insights highlight the criticality of efficient note-writing. At baseline, the model suggests that even 10% less efficiency in note writing (∼20 versus 18 minutes writing per note) causes a backlog that exceeds what can be cleared without a significant re-prioritization of how clinicians spend their time.
Conclusion: System dynamics can be leveraged for generating implementation hypotheses and supporting Pre-implementation activities such as clinician training. Simulation models provide an approachable, data-driven method for engaging clinicians to anticipate how implementation activities might affect their workflow. Next steps include further calibrating the model with clinic administrative data, and modifying the model as needed based on emerging patterns of PRE-FAIR dosage based on client complexity.
References
1. Cruden, G., Crawford, S., & Saldana, L. (2021). Prevention Adaptation of an Evidence-Based Treatment for Parents Involved With Child Welfare Who Use Substances. Frontiers in Psychology, 12, 689432.
2. Saldana, L., Chamberlain, P., Bradford, W. D., Campbell, M., & Landsverk, J. (2014). The cost of implementing new strategies (COINS): A method for mapping implementation resources using the Stages of Implementation Completion. Children Youth Services Review, 39, 177–182.
3. Saldana, L., Chapman, J. E., Campbell, M., Alley, Z., Schaper, H., & Padgett, C. (2021). Meeting the Needs of Families Involved in the Child Welfare System for Parental Substance Abuse: Outcomes From an Effectiveness Trial of the Families Actively Improving Relationships Program. Frontiers in Psychology, 12, 2592. https://www.frontiersin.org/article/10.3389/fpsyg.2021.689483
Leadership and Organizational Change for Implementation (LOCI): Past, present and future directions to enhance and align inner and outer context leadership and climate
Authors
Dr. Marisa Sklar - Department of Psychiatry, University of California, San Diego; Child and Adolescent Services Research Center; UC San Diego ACTRI Dissemination and Implementation Science Center
Dr. Mark Ehrhart - Department of Psychology, University of Central Florida
Ms. Nallely Vega - Institute for the Study of Behavioral Health and Addiction, Boise State University
Dr. Nate Williams - Institute for the Study of Behavioral Health and Addiction; School of Social Work, Boise State University;
Dr. Ane-Marthe Skar - The Norwegian Centre for Violence and Traumatic Stress Studies
Dr. Karina Egeland - The Norwegian Centre for Violence and Traumatic Stress Studies
Dr. Erika Crable - Department of Psychiatry, University of California, San Diego; Child and Adolescent Services Research Center; UC San Diego ACTRI Dissemination and Implementation Science Center
Dr. Gregory Aarons - Department of Psychiatry, University of California, San Diego; Child & Adolescent Services Research Center; University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
Abstract
Leadership and Organizational Change for Implementation (LOCI) is a multi-faceted implementation strategy which aims to improve the implementation of evidence-based practices (EBPs) in behavioral health service settings (Aarons, 2017). LOCI is guided by six core principles including an emphasis on evidence, alignment of leadership and organizational policies, clear communication in sup ort for implementation, feedback to inform decision-making and implementation planning, sustained change for long-term results, and ongoing adaptation to tailor LOCI for unique contexts. LOCI aims to enhance leadership to develop an organizational climate that supports implementation and sustainment of EBPs (Aarons, 2017). LOCI is grounded within the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework (Aarons, 2011). Initial LOCI research focused on improving climate for EBP implementation in EPIS inner organizational contexts during the Preparation and Implementation phases (Aarons, 2015; Egeland, 2019). Recent LOCI-Systems Level (LOCI-SL) research simultaneously engage multilevel EPIS outer system contexts and inner organizational contexts during Exploration, Preparation and Implementation phases to support system and interorganizational alignment. In this symposium, we provide a rich description of LOCI for improving inner context implementation leadership and climate demonstrated through a case study of a LOCI-facilitated digital measurement-based care implementation effort in the U.S. Two additional presentations feature quantitative and qualitative evidence of LOCI’s effectiveness and perceived utility from a LOCI-facilitated effort to implement post-traumatic stress disorder treatment in child and adult mental specialty care settings in Norway. A final presentation of the new LOCI-SL strategy describes the impetus and approach for advancing LOCI to support system and interorganizational alignment to implement and sustain motivational interviewing and an artificial intelligence quality assurance platform in organizations providing substance use treatment across one U.S. state. This symposium demonstrates the effectiveness and continued evolution of the LOCI strategy for implementation projects conducted across diverse countries, provider organizations, and service systems.
Title: Using the Leadership and Organizational Change for Implementation (LOCI) strategy to implement digital measurement-based care in youth behavioral health services: An intrinsic case study.
Background: Organizational context plays a key role in implementation of evidence-based practices (EBP). The Leadership and Organizational Change for Implementation (LOCI) strategy (Aarons et al., 2017) addresses organizational context by targeting implementation leadership and climate. Implementation leadership refers to the extent to which leaders emphasize and support EBP implementation. Implementation climate refers to the shared perceptions regarding the extent to which the policies, practices, and procedures in an organization are aligned with the goal of EBP implementation. While research has collected data to show that implementation leadership and climate are important constructs, many of these studies lack a rich description of what these constructs look like in practice. This case study fills this gap through rich description of an organization that fully embodied these constructs for successful EBP implementation.
Method: A single organization, intrinsic case study (Crowe et al., 2011) was employed to gain an in-depth and multifaceted understanding of how one organization used LOCI to create a climate for the successful implementation of a digital measurement-based care system for youth. Taking an interpretive epistemological approach that aimed to convey the experience of this organization, the goal of this case study was to provide a narrative that could guide others’ efforts toward implementation (Yin, 2018).
Results: A rich description of this organization’s successful implementation is provided. We highlight a few key points as possible lessons learned from this organization’s experience. These key points include the organization’s authentic, and multilevel, commitment throughout implementation, and the comprehensive integration of the EBP into a wide swath of organizational policies, practices, and procedures.
Conclusion: A primary goal of this case study was to simply describe an example of a successful implementation facilitated by the LOCI strategy. We hope others find this case study valuable for their own research and practice.
Title: The Effect of LOCI on transformational and implementation leadership, and implementation climate.
Background: The Leadership and Organizational Change for Implementation (LOCI) strategy serves as an implementation strategy that aims to build leadership skills and create a positive strategic organizational climate to support effective and sustained implementation of evidence-based practices (EBPs) (Aarons et al., 2017). In this talk, we will present findings from a study evaluating the effect of LOCI on transformational and implementation leadership and on implementation climate in a Norwegian mental health context.
Methods: A stepped-wedge cluster randomized study design was used, enrolling 42 Norwegian child and adult specialized mental health clinics in three cohorts (Egeland et al., 2019). The study was framed within the EPIS framework, focusing mainly on the inner organizational context and the preparation and implementation phases (Aarons et al, 2011). All therapists (n = 790) received training in screening of trauma exposure and posttraumatic stress disorder (PTSD). A subgroup of therapists (n = 248) received training in evidence-based treatment methods for PTSD. Therapists completed surveys at baseline, 4, 8, 12, 16, and 20 months, assessing their perception of their leaders’ (n = 47) transformational and implementation leadership, and the implementation climate. General linear mixed-effects models were used to investigate the hypothesis that the LOCI strategy would lead to improved therapist-provided scores on implementation leadership, transformational leadership, and implementation climate.
Results: The results showed that after LOCI was introduced among first-level leaders, there was a significant increase in therapists’ perception of transformational and implementation leadership, and implementation climate (Skar et al., 2022). This change was maintained over time, whereas the control condition demonstrated a steady decrease in therapist-rated reports.
Conclusion: The LOCI strategy can serve as an effective implementation strategy to strengthen transformational and implementation leadership skills and contribute to a more positive implementation climate for EBP implementation in mental health settings.
Title: Change doesn’t happen by itself: A thematic analysis of first-level leaders’ experiences of the LOCI strategy
Background: The aim of The Leadership and Organizational Change for Implementation (LOCI) strategy is to support leaders in succeeding with evidence-based practice (EBP) implementation by promoting effective transformational and implementation leadership, and implementation climate (Aarons et al., 2017). How leaders experience the LOCI strategy are important for their utilization and effectiveness in supporting EBP implementation. We will present findings from a qualitative study exploring first-level leaders’ experiences of participating in LOCI.
Methods: This study was part of a larger trial implementing evidence-based practices for posttraumatic stress disorder (PTSD) in Norwegian child and adult mental health clinics (Egeland et al., 2019). Eleven first-level leaders participated in semi-structured interviews after completing the LOCI strategy. Data were analyzed through reflexive thematic analysis.
Results: Leadeŕs experiences of participating in LOCI were related to the following four main themes: (1) structuring the EBP implementation, (2) taking responsibility for the EBP implementation, (3) interacting with others about the EBP implementation, and (4) becoming aware of EBP implementation and their own leadership (Borge et al., 2022). Two of the leaders found LOCI to be less useful. In the first case, the practitioners were seemingly so self-sufficient and proficient that the leader concluded the implementation would have succeeded anyway. In the second case, the organizational challenges overshadowed and hampered the EBP implementation from the start. The two narratives illustrate how context matters and how individual characteristics interact with intervention design and delivery to produce differing learning experiences and outcomes.
Conclusion: Most leaders reported great benefits from participating in LOCI. The leaders reported increased awareness of leadership for EBP implementation. The results suggests that the LOCI strategy is effective in engaging and empower first-level leaders responsible for EBP implementation and point to important challenges for future research on implementation strategies.
Title: Landscape Analysis as a Pre-implementation Tool to Support Interorganizational Alignment and Implementation Success in Complex Policy and Service Environments
Background: Research has addressed the importance of organizational factors in implementation of evidence-based practices. However, implementation often takes place in complex multilevel contexts. Few studies examine generating interorganizational alignment across outer system and inner organizational contexts in complex, statewide service systems (Crable, 2022). We describe our process for engaging and aligning systems entities in a NIDA-funded statewide project to implement Motivational Interviewing (MI) and concurrent implementation of an artificial intelligence quality assurance platform (Lyssn.io) in substance use disorder (SUD) treatment agencies.
Methods: We conducted a landscape analysis (Walt, 2008) to map key systems entities, interorganizational relationships, resources, and opportunities to align MI + Lyssn with current policies and systems priorities. Landscape analysis included document review of 32 policy statements and contracts from systems entities, ethnographic observation of 14 state agency public meetings, and 26 member checking interviews to contextualize findings. We used inductive qualitative methods to identify systems partners, relational ties between systems entities, priority initiatives compatible with study goals, and develop four tailored asks for each partner aimed at promoting MI + Lyssn alignment across system and organizational levels.
Results: We describe outer systems contexts characterized by a statute for EBP use and decentralized decision-making such that multiple systems entities work interdependently to advance EBP SUD initiatives with limited authority to enforce change. We identified priority partners for MI + Lyssn implementation given concurrent SUD treatment policy goals and close working relationships. Engagement with participating provider agencies varied and partnerships with managed care organizations was challenging due to competing priorities and payer perspectives.
Conclusion: Landscape analysis is underused and can aid identifying opportunities for multilevel alignment across systems and organizations for EBP implementation and sustainment. Consistent with the Exploration, Preparation, Implementation, Sustainment (EPIS) framework (Aarons 2011), this approach can support exploration and preparation phase activities to understand complex systems and inform implementation and sustainment strategies.
References
1. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. C. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23.
2. Aarons, G.A., Ehrhart, M.G., & Farahnak, L.R., & Hurlburt, M. (2015). Leadership and organizational change for implementation (LOCI): A randomized mixed-method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implementation Science, 10(1), 11.
3. Aarons, G. A., Ehrhart, M. G., Moullin, J. C., Torres, E. M., & Green, A. E. (2017). Testing the Leadership and Organizational Change for Implementation (LOCI) intervention in substance abuse treatment: A cluster randomized trial study protocol. Implementation Science, 12, 29.
4. Borge, R. H., Egeland, K. M., Aarons, G., Ehrhart, M., Sklar, M., & Skar, A.-M. S. (2022). Change doesn’t happen by itself”: A thematic analysis of first-level leaders’ experiences participating in the Leadership and Organizational Change for Implementation (LOCI) strategy intervention. Administration and Policy in Mental Health and Mental Health Services Research. (In press)
5. Crable, E. L., Benintendi, A., Jones, D. K., Walley, A. Y., Hicks, J. M., & Drainoni, M.-L. (2022). Translating Medicaid policy into practice: Policy Implementation Strategies from three US states’ experiences enhancing substance use disorder treatment. Implementation Science, 17(1).
6. Crowe, S., Cresswell, K., Robertson, A., Huby, G., Avery, A., & Sheikh, A. (2011). The case study approach. BMC Medical Research Methodology, 11:100.
7. Egeland, K.M., Solheim Skar, A.M., Endsjø, M., Laukvik, E.H., Bækkelund, H., Babaii, A., Granly, L.B., Husebø, G.K., Ehrhart, M., Sklar, M., Brown, C.H., Aarons, G.A. (2019). Testing the Leadership and Organizational Change for Implementation (LOCI) intervention in Norwegian mental health clinics: A stepped-wedge cluster randomized design study protocol. Implementation Science, 14(1): 28.
8. Skar, A.-M. S., Braathu, N., Peters, N., Bækkelund, H., Endsjø, M., Babaii, A., Borge, R. H., Wentzel-Larsen, T., Ehrhart, M., Sklar, M., Brown, C. H., Aarons, G., & Egeland, K. M. (2022). A stepped-wedge randomized trial investigating the effect of the Leadership and Organizational Change for Implementation (LOCI) intervention on implementation and transformational leadership, and implementation climate. BMC Health Services Research. 22(1):298.
9. Walt, G., Shiffman, J., Schneider, H., Murray, S. F., Brugha, R., & Gilson, L. (2008). ‘doing’ health policy analysis: Methodological and conceptual reflections and challenges. Health Policy and Planning, 23(5), 308–317.
10. Yin, R. (2018). Case Study Research and Applications: Design and Methods (6th ed.). Los Angeles, CA: Sage.
Utilizing the TEAMS implementation-effectiveness trial to assess the joint impact of multilevel implementation strategies and implementation context on autism EBI implementation outcomes in schools and child mental health services
Authors
Dr. Aubyn Stahmer - UC Davis MIND Institute
Dr. Lauren Brookman-Frazee - UC San Diego Dissemination and Implementation Science Center & UC San Diego Department of Psychiatry & Child and Adolescent Services Research Center
Dr. Anna Lau - UCLA Department of Psychology
Dr. Barbara Caplan - UC San Diego Department of Psychiatry & Child and Adolescent Services Research Center
Dr. Scott Roesch - San Diego State University
Dr. Yesenia Mejia - UC San Diego Department
Dr. Taffeta Wood - UC Davis MIND Institute
Dr. Colby Chlebowski - San Diego State University & UC San Diego Department of Psychiatry & Child and Adolescent Services Research Center
Abstract
Education and mental health are two key public service sectors providing care to autistic children. However, while autism evidence-based interventions (EBIs) are available (Wong et al., 2015), they are not routinely delivered in these settings (Brookman-Frazee et al., 2010; Stahmer et al., 2005). This symposium presents findings from the TEAMS Study, two linked randomized Hybrid Type 3 implementation trials testing two implementation strategies on implementation mechanisms and outcomes when paired with training with two ASD EBIs (AIM HI, CPRT). The two implementation strategies were: (1) TLI (Teams Leadership Institute), an adaptation of the Leadership and Organizational Change for Implementation (LOCI) leadership training, and (2) TIPS (TEAMS Individual Provider Strategies), a provider training protocol enhanced with motivational interviewing. These implementation strategies targeted key implementation mechanisms identified in the AIM HI and CPRT effectiveness trials (implementation leadership and climate, provider motivation; Brookman-Frazee & Stahmer, 2018). It is critical to understand the effectiveness of these implementation strategies within the inner/outer implementation contexts (e.g. service system, COVID disruption, community socioeconomic resources, provider factors) to maximize the public health impact of autism EBIs.
This symposium will illustrate key findings related to the joint impact of implementation strategies and context on implementation mechanisms and outcomes (i.e., provider EBI fidelity). First, we will present the primary outcomes of the TEAMS Trial (Paper 1), highlighting the effectiveness of multilevel implementation strategies on implementation mechanisms and outcomes, with consideration of the impact of COVID disruptions. Additional presentations will highlight the relevance of outer context socioeconomic factors (Papers 2 and 3) and individual provider factors (Paper 4), alone and jointly with implementation strategy use, on EBI implementation facilitators and/or outcomes. The panel will end with a discussion of takeaways for enhancing the public health impact of both EBIs and implementation strategies by considering key factors within the implementation context.
Title: Testing the effectiveness of multi-level implementation strategies for two evidence-based autism interventions: Conducting a statewide hybrid type 3 trial in children’s mental health services and schools during the COVID pandemic
Authors: Aubyn Stahmer, Lauren Brookman-Frazee, Anna Lau, Barbara Caplan, and Scott Roesch
Background: Autistic children are a high priority population served in multiple public systems. Evidence-based interventions (EBIs) for autism are available (Wong et al., 2015) but are not routinely delivered in community care (Brookman-Frazee et al., 2010; Stahmer et al., 2005). TEAMS includes two linked randomized Hybrid Type 3 implementation trials to test two implementation strategies when paired with AIM HI or CPRT and examine mechanisms of these strategies (Brookman-Frazee & Stahmer, 2018). The TEAMS Leadership Institute (TLI) targets key mechanisms identified in our community effectiveness trials (i.e., implementation leadership and climate). TEAMS Individualized Provider Strategies (TIPS) embeds motivational interviewing strategies into provider training to increase provider training engagement and EBI implementation. This presentation describes methodological considerations in testing the effectiveness of implementation strategies across service contexts and EBIs and in the context of COVID disruptions, and preliminary findings.
Methods: Thirty four mental health clinics and 31 school districts were randomized to four implementation conditions: (1) Standard EBI Training Only (n = 15 programs), (2) TIPS Enhanced Training (n = 14), (3) TLI with standard training (n = 20), or (4) TIPS + TLI (n = 16). Participants included 387 provider/child dyads across conditions. Provider training was provided in four “cohorts” starting in Fall 2017. The 3rd cohort (2019-2020) was disrupted by COVID emergency measures. The primary implementation outcome was EBI fidelity. Measures of implementation mechanisms include implementation climate and leadership, provider EBP attitudes and motivation for training.
Results: Preliminary analyses indicate providers trained during and following the COVID disruption had lower EBI fidelity compared to providers trained pre-COVID. TLI and TIPS had differential patterns of association with implementation mechanisms. Analytic and methodological considerations will be discussed and main effects of implementation strategies on fidelity indicators will be reported.
Conclusion: The TEAMS trial provides initial support for associations between multilevel implementation strategies and implementation mechanisms and outcomes.
Title: Delivery of evidence-based interventions for autism: Neighborhood context and provider perceptions of key implementation drivers
Authors: Yesenia Mejia, PhD, Barbara Caplan, PhD, Anna Lau, PhD, Aubyn Stahmer, PhD, Lauren Brookman-Frazee, PhD
Background: Neighborhood context and provider and organizational factors are critical to equitable autism evidence-based intervention (EBI) implementation. Provider attitudes toward EBIs and organizational implementation climate have been associated with EBI implementation (Woodard et al., 2021). In addition, neighborhood context (e.g., community resources) surrounding a provider’s agency can affect attitudes toward EBIs (Glisson & Schoenwald, 2005). This study assessed: (1) the association between neighborhood context and provider attitudes toward EBIs and organizational implementation climate, and (2) whether these associations varied by provider type (teachers, therapists).
Methods: This study utilized baseline data from a hybrid type 3 implementation-effectiveness trial of multi-level implementation strategies for two autism EBIs (AIM HI, CPRT). Participants included 401 providers (teachers, therapists) from 54 school districts and mental health programs. Neighborhood context was characterized using a composite score measuring neighborhood social and economic resources (e.g., median household income, single parenthood), drawn from a publicly available geo-coded data (COI, 2022). Providers completed self-report measures about their attitudes towards EBIs (Evidence-Based Practice Attitude Scale; Aarons, 2004) and their perceptions about organizational implementation climate (Implementation Climate Scale; Ehrhart & Aarons, 2014) at baseline. Multilevel models were run to control for the nested nature of the data (Level 1 = provider; Level 2 = program.
Results: Preliminary findings revealed neighborhood context, (i.e., lower-resource neighborhood), was associated with more negative attitudes toward EBI for therapists (B = -0.01, p < .05), but not teachers (B = 0.00, p = .63). Neighborhood context was not associated with provider-rated implementation climate (p > .10), but there was a main effect of provider type (B = -0.25, p < .01), such that therapists indicated more positive ratings of climate than teachers.
Conclusion: Building implementation supports for schools and therapists working in under-resourced neighborhoods may increase the public health impact of autism EBIs.
Title: Implementing classroom pivotal response teaching: Classroom quality and fidelity in the context of school poverty
Authors: Taffeta Wood, PhD; Barbara Caplan, PhD; Aubyn Stahmer, PhD
Background: Autistic children in schools with higher poverty access classrooms of lower quality (Kraemer et al., 2020). School may be their sole intervention (Thomas, Parish & Williams, 2014). Understanding district-level poverty and implementation supports in predicting classroom quality and evidence-based intervention (EBI) fidelity may inform implementation planning to promote equity. The current study examined these associations within the context of an implementation trial of Classroom Pivotal Response Teaching (CPRT; Stahmer et al., 2011).
Methods: The larger implementation-effectiveness trial tested four implementation conditions with CPRT training: (1) EBI training, (2) TEAMS Leadership Institute (TLI) to increase leader EBI support; (3) TEAMS Individual Provider Strategy (TIPS) with motivational strategies, (4) TLI + TIPS. Classroom quality was self-rated by teachers (N = 181) using the Autism Program Environment Rating Scale (APERS). District poverty was measured by percent of students receiving Free and Reduced Lunch (FRL). Raters coded videorecorded teaching sessions for CPRT fidelity. Regressions examined the main effects and interactions of FRL and implementation condition on (1) classroom quality, and (2) CPRT fidelity.
Results: FRL was not significantly associated with APERS (B = .00, p = .99), nor were FRL x implementation condition interactions. There were significant interactions between FRL and implementation condition in predicting CPRT fidelity. There was a negative association between FRL and CPRT fidelity (B = -0.02, p = .017) in the control condition; these associations were not significant in the TLI and TLI + TIPS conditions. In the TIPS condition, FRL and CPRT fidelity were positively associated (B = 0.01, p < .05).
Conclusion: Teachers in districts with higher poverty had greater challenges reaching fidelity and benefited most from TIPS supports. TLI may help mitigate the negative impact of district-level poverty on EBI fidelity.
Title: Predictors of observed EBI strategy use with autistic children in community mental health services: Associations with therapist factors and use of organizational and therapist level implementation Strategies
Authors: Barbara Caplan, PhD; Colby Chlebowski, PhD; Lauren Brookman-Frazee, PhD
Background: Community therapist training in An Individualized Mental Health for Autism (AIM HI) is associated with greater evidenced-based intervention (EBI) strategy use and child/caregiver outcomes compared to usual care (Brookman-Frazee et al., 2022). As therapist factors and leadership support have been linked to EBI implementation (Suhrheinrich et al., 2020), efforts have been made to develop multi-level (provider, leader) implementation strategies. This study aims to examine the link between therapist factors (experience, EBI attitudes, motivation), multilevel implementation strategies, and therapist use of EBI strategies.
Methods: Data are drawn from the TEAMS trial, a hybrid type 3 implementation-effectiveness trial of multi-level implementation strategies for two autism EBIs (AIM HI, CPRT). Participants include 158 therapists from 33 clinic- and school-based mental health programs within the AIM HI TEAMS Study. Organizations were randomly assigned to one of four implementation conditions: (1) standard EBI training, (2) provider training enhanced with motivational interviewing (TIPS) only, (3) leadership training in implementation support (TLI) only, (4) TLI + TIPS. Therapists completed surveys at baseline. Video recordings of therapy sessions were coded for different types of EBI strategies (i.e. AIM HI fidelity) using the AIM HI Coding Observational Coding System [active teaching (child/caregiver); organization/structure; continuity across sessions]. Multilevel models were run to account for the nested nature of the data (Level 1 = Provider/Child; Level 2 = Program).
Results: Relative to standard EBI training, TLI + TIPS was associated with greater active teaching with caregivers (B = 0.71, p = .01). Both TLI only (B = 0.58, p = .02) and TLI + TIPS (B = 0.67, p < .01) were associated with greater session organization/structure. Baseline provider motivation (B = 0.38, p < .05) and TIPS + TLI (B = 0.50, p < .05) were associated with greater session continuity.
Conclusion: Provider- and leader-level implementation strategies may improve the public health impact of autism EBIs by facilitating multiple aspects of EBI fidelity, particularly when used in combination.
References
1. Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The evidence-based practice attitude scale (EBPAS). Mental Health Services Research, 6(2), 61–74. https://doi.org/10.1023/B:MHSR.0000024351.12294.65
2. Brookman-Frazee, L., Chlebowski, C., Villodas, M., Garland, A., McPherson, J., Koenig, Y., & Roesch, S. (2022). The effectiveness of training community mental health therapists in an evidence-based intervention for ASD: Findings from a hybrid effectiveness-implementation trial in outpatient and school-based mental health services. Autism, 13623613211067844. https://doi.org/10.1177/13623613211067844
3. Brookman-Frazee, L., & Stahmer, A. C. (2018). Effectiveness of a multi-level implementation strategy for ASD interventions: Study protocol for two linked cluster randomized trials. Implementation Science, 13(1), 66. https://doi.org/10.1186/s13012-018-0757-2
4. Brookman-Frazee, L., Taylor, R., & Garland, A. F. (2010). Characterizing Community-Based Mental Health Services for Children with Autism Spectrum Disorders and Disruptive Behavior Problems. Journal of Autism and Developmental Disorders, 40(10), 1188–1201. https://doi.org/10.1007/s10803-010-0976-0
5. Diversitydatakids.org. 2022. “Child Opportunity Index 2.0 ZIP Code data”, retrieved from https://data.diversitydatakids.org/dataset/coi20_zipcodes-child-opportunity-index-2-0-zip-code-data?_external = True on Apr 21 2022.
6. Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2014). Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS). Implementation Science, 9(1), 1-11.
7. Glisson, C., & Schoenwald, S. K. (2005). The ARC organizational and community intervention strategy for implementing evidence-based children's mental health treatments. Mental health services research, 7(4), 243-259.
8. Kraemer, B. R., Odom, S. L., Tomaszewski, B., Hall, L. J., Dawalt, L., Hume, K. A., … & Brum, C. (2020). Quality of high school programs for students with autism spectrum disorder. Autism, 24(3), 707-717.
9. Stahmer, A. C., Collings, N. M., & Palinkas, L. A. (2005). Early Intervention Practices for Children With Autism: Descriptions From Community Providers. Focus on Autism and Other Developmental Disabilities, 20(2), 66–79. https://doi.org/10.1177/10883576050200020301
10. Stahmer, A. C., Suhrheinrich, J., Reed, S., Schreibman, L., & Bolduc, C. (2011). Classroom Pivotal Response Teaching for Children with Autism. Guilford Press. 72 Spring Street, New York, NY 10012.
11. Suhrheinrich, J., Rieth, S. R., Dickson, K. S., & Stahmer, A. C. (2020). Exploring Associations Between Inner-Context Factors and Implementation Outcomes. Exceptional Children, 86(2), 155–173. https://doi.org/10.1177/0014402919881354
12. Thomas, K. C., Parish, S. L., & Williams, C. S. (2014). Healthcare expenditures for autism during times of school transition: Some vulnerable families fall behind. Maternal and child health journal, 18(8), 1936-1944.
13. Wong, C., Odom, S. L., Hume, K. A., Cox, A. W., Fettig, A., Kucharczyk, S., Brock, M. E., Plavnick, J. B., Fleury, V. P., & Schultz, T. R. (2015). Evidence-Based Practices for Children, Youth, and Young Adults with Autism Spectrum Disorder: A Comprehensive Review. Journal of Autism and Developmental Disorders, 45(7), 1951–1966. https://doi.org/10.1007/s10803-014-2351-z
14. Woodard, G. S., Triplett, N. S., Frank, H. E., Harrison, J. P., Robinson, S., & Dorsey, S. (2021). The impact of implementation climate on community mental health clinicians’ attitudes toward exposure: An evaluation of the effects of training and consultation. Implementation Research and Practice, 2, 26334895211057883.
Implementation mechanisms: Taking them to the next level
Authors
Ms. Callie Walsh Bailey - Washington University in St. Louis
Dr. Shannon Wiltsey Stirman - Stanford University & National Center for PTSD
Dr. Christian Helfrich - US Dep
Dr. Joanna Moullin - Curtin University
Dr. Per Nilsen - Division of Society and Health, Department of Health, Medicine and Caring Sciences, Linköping University
Dr. Byron Powell - Brown School, Washington University in St. Louis, St. Louis, MO, United States
Dr. Oladunni Oluwoye - Elson S. Floyd College of Medicine at Washington State University
Dr. Elvin Geng - Washington University School of Medicine
Dr. Bo Kim - VA Center for Healthcare Organization and Implementation Research / Harvard Medical School
Dr. Anne Sales - University of Missouri
Dr. Aubyn Stahmer - UC Davis MIND Institute
Dr. Sarah Masyuko - University of Washington
Dr. Maria Fernandez - UTHealth School of Public Health
Ms. Leah Buhrmann - Northumbria Universit
Dr. Anjuli Wagner - University of Washington
Dr. Arianna Means - University of Washington
Dr. Cara Lewis - Kaiser Permanente Washington Health Research Institute
Dr. Lisa Saldana - Oregon Social Learning Center
Dr. Meagan Brown - Kaiser Permanente Washington Health Research Institute
Dr. Timothy Walker - University of Texas Health Science Center at Houston
Dr. Jacob Szeszulski - Texas A&M AgriLife Research
Dr. Donna Shelley - NYU School of Global Public Health
Dr. Gracelyn Cruden - Oregon Social Learning Center
Dr. Erika Crable - University of California, San Diego
Dr. Andrew Quanbeck - University of Wisconsin
Dr. Brian Mittman - Kaiser Permanente Southern California
Dr. Amy Kilbourne - US Department of Veterans Affairs and University of Michigan
Dr. Ingrid Eshun-Wilsonova - Washington University at St. Louis
Dr. Shannon Sweeney - Oregon Health & Science University
Dr. Deborah Cohen - Oregon Health & Science University
Dr. JoAnn Kirchner - VA Behavioral Health QUERI
Dr. Michael Parchman - Kaiser Permanente Washington Health Research Institute
Abstract
Establishing implementation mechanisms can help teams select and match strategies to prioritized determinants and streamline implementation strategies to optimize their impact, affordability, scalability, and efficiency. A 2018 systematic review revealed that no studies have yet to empirically establish mechanisms for implementation strategies. With AHRQ funding (R13HS025632), we convened a Mechanisms Network of Expertise (MNoE) to generate a research agenda for advancing the study of implementation mechanisms. An overview of that process and the resulting research agenda was presented in the morning plenary. This breakout features teams generated by the MNoE who undertook work to begin addressing specific actions highlighted by the research agenda. The first talk offers a cautionary note about the hazards of over-relying on frameworks with the study of mechanisms as a solution. The second talk will feature conceptualizing mechanisms as the crux of the theory of a strategy, which as a process, holds promise for boosting the generalizability of our initiatives. The third talk will discuss the importance of harnessing community partnerships in elucidating mechanisms and what it will take to make it so. The fourth talk will overview implementation mapping as an approach to identifying and describing mechanisms of implementation strategies centering form and function in the design. The fifth talk will introduce systems analytical methods for examining implementation mechanisms. The final talk with center practice facilitation as an example of a complex implementation strategy and how mechanism mapping can improve our understanding of how it works and how to improve its effectiveness. This rich series of talks featuring implementation mechanisms will conclude with a reflection on how implementation mechanisms can improve both science and practice.
Title: The hazards of overreliance on theories, models, and frameworks and how the study of mechanisms can offer a solution
Abstract: Identifying, developing, and testing implementation strategies is a key focus in the field of implementation science. Despite the existence of guidance for specifying and reporting implementation strategies, establishment of a taxonomy of implementation strategies, and hundreds of trials testing strategies, evidence for the effectiveness is limited for many strategies and mixed even amongst well-studied strategies. A prevailing problem with the study of implementation strategies is a lack of attention towards mechanisms, i.e., the processes or events through which implementation strategies operate. Identifying plausible mechanisms and testing these is essential in developing a clear understanding of how, why, and in what contexts implementation strategies work. As mechanisms receive increased attention, we caution implementation scientists against creating and adopting a static, fixed list of mechanisms, as has largely been the case with many models, frameworks, and strategies in implementation science. Premature reliance on models and frameworks without critical reflection on whether they sufficiently capture assumptions about mechanisms may hamper the field’s efforts by precluding innovative research on mechanisms that does not conform to a prevailing taxonomy. We explore the potential hazards of becoming stagnated by over-reliance on models and frameworks too early in the discovery process, propose ways in which the studies of mechanisms can help the field avoid uncritical use of models and frameworks, and suggest alternatives to simply amassing lists and taxonomies. We provide concrete recommendations for approaching mechanisms in implementation research and outline an approach to selecting, describing, and studying mechanisms that involves a meeting of inductive and deductive approaches.
Title: When the Parts are More than the Sum: Mechanisms and Generalization in Implementation Research
Background: Implementation contexts differ organizationally, structurally and culturally. Therefore, implementation strategies — complex, multi-level and coordinated behavior nested within these contexts — are unlikely to have invariant effects. Yet if science necessitates generalization, does implementation science face an intractable problem? We examine how mechanistic analyses of implementation strategies can advance generalizable insights in the absence of context-invariant effects.
Methods: We adopt Craver’s characterization of a mechanism (from philosophy of science), comprising (1) a phenomenon, (2) it’s parts, (3) causings between the parts, and (4) these components’ organization. Aligning to Pearl’s Transportability Framework we using causal diagrams — nodes (parts) and arrows (potential causings) — to represent these four components for a given implementation strategy and selection nodes as context-dependent effects to draw qualitative insights about generalization and explore implications for quantitative analyses.
Results: Mechanistically decomposing an implementation strategy provides a “theory of a strategy,” that when represented as causal diagrams with temporally and conceptually ordered dependent parts, allows specification of how contextual differences influence the mechanism, and therefore the output of the mechanism. This approach makes the interaction of relevant contextual effects on specific steps of the mechanism, which influence anticipated effects across contexts, that go beyond the intuition that context matters. This approach also demonstrates that combining information about the strategy (via its mechanism) and the target settings (via potential contextual effects on the mechanism) can enable theoretically-justified, structurally sound, cross-context generalization. Mechanistic decomposition allows us a set of principles that can be used to develop insights that apply to a wide range of contextual settings (generalize) even when the effects at each of these sites differ.
Conclusion: Theoretically-based, and causally-ordered conceptualizations of implementation strategies elucidate how, where and why contextual differences influence effects. A strategy’s “anatomy,” therefore can facilitate inferring across contexts, shape empirical studies to collect target context data that would enhance cross-context inferences, and advance overall conceptual understanding of a strategy. Mechanistic analyses also suggest that generalizing is not a strategy’s feature, but rather accessed through combining knowledge about how a strategy works with information about contextual levers on that chain of events in diverse target settings.
Title: Harnessing Community Partnerships and Multidisciplinary Perspectives to Elucidate Implementation Mechanism
Abstract: Implementation science research increasingly focuses on the integration of evidence-based practices into community programs. In this translation process, community partnerships play an important role in ensuring that outcomes are valid and meaningful for community stakeholders. As the science of implementation develops, increasing attention is being paid to the identification of mechanisms underlying effective implementation strategies to understand how a specific implementation strategy works in a particular setting or evidence-based intervention. However, thus far, community stakeholders have been only minimally involved in the identification of mechanisms related to implementation strategies. This could be due to several reasons, such as the lack of available guidance to involve community stakeholders in mechanistic research, a lack of awareness of the benefits of doing so, and that the language used to describe mechanisms and determinants has become increasingly idiosyncratic, making communication with community partners from different discipline challenging.
While rigorous experimental studies will help improve our identification and understanding of implementation mechanism, the process will be complex, slow and may or may not be relevant to the service systems who hope to affect. Therefore, we propose the use of community-partnered and use of co-creation or exchange models in the elucidation of implementation mechanisms to enhance relevance to the community. This presentation will highlight the importance of partnering with community stakeholders to elucidate implementation mechanisms through bi-directional collaboration. We will provide examples of methods from other fields with established strategies for engaging stakeholders and discuss the importance of a multidisciplinary perspective for understanding these complex concepts. Based on this and learnings from the field of implementation science over the last years, we will provide recommendations for working with community stakeholders to identify implementation mechanisms.
Title: Identifying and describing mechanisms of implementation strategies: An Implementation Mapping approach
Background: Implementation Mapping is a systematic process for developing, selecting, and tailoring implementation strategies that is based on Intervention Mapping, a framework for designing multi-level interventions in health promotion. During the Implementation Mapping process, planners create and use Implementation Mapping-specific logic models to help understand and illustrate the mechanisms through which implementation strategies influence implementation and effectiveness outcomes. These logic models explicitly depict the relations between implementation tasks, determinants and contextual factors influencing these outcomes. Personal determinants of implementation behaviors (e.g., self-efficacy for implementation), as well as contextual factors across levels of the implementation environment (e.g., organizational culture) are considered during the process of developing the logic models. Planners – whether developing or selecting implementation strategies - also use these logic models (by reading them from right to left) to make decisions about what specific implementation strategies would influence identified-determinants and the pathways through which this influence is expected to occur.
This paper describes how the use of Implementation Mapping, and in particular, how Implementation Mapping logic models can provide insight about mechanisms of implementation strategy both prospectively (during the development of implementation strategies) and retrospectively (for describing mechanisms of previously developed strategies).
Methods: The paper will demonstrate how both the form and function of implementation strategies are described using Implementation Mapping. Examples will illustrate the process that leads planners in clearly delineating theoretical methods of change used to influence implementation determinants and tasks (function), and how those methods are operationalized by the implementation strategy (form). The paper will also include example tables listing specific change methods that can be selected to influence specified determinants of implementation.
Results: Through the use of logic models, the Implementation Mapping process clearly articulates the mechanisms by which implementation strategies operate. This information is critical to inform planning (and evaluation efforts), and to help advance the field of implementation science.
Title: A structured approach to applying systems analytical methods for examining implementation mechanisms
Background: Systems analytical methods are multi-disciplinary approaches that use qualitative or quantitative modeling to examine complex systems. Applied to studying implementation mechanisms, the methods can help: (i) Identify and manage conditions that may or may not activate mechanisms (expected mechanisms targeted by an implementation strategy and unexpected mechanisms that the methods help detect) and (ii) adapt the strategy to address emergent contextual influences that were not foreseen when the strategy was initially selected for use.
Methods: Building on Wagner, Crocker, and colleagues’ systems engineering approach to studying health systems, we specified four steps to apply systems analytical methods for studying implementation mechanisms. As a case example, we used Lewis and colleagues’ depression screening implementation. We additionally conducted a narrative review of the literature to identify examples of systems analytical methods applied to elucidating and testing mechanisms, to help outline the steps’ strengths, limitations, and implications for future implementation mechanisms research.
Results: The four steps are: 1) Identify potential mechanisms activated by planned implementation strategies, 2) assess the expected impact of the strategies and implementation contexts on the mechanisms, 3) refine the strategies to enable better mechanism activation, and 4) examine the refined strategies’ impact on mechanism activation. Although implementation research in general is increasingly embracing systems analytical methods, reviewed examples of mechanisms research use the methods predominantly for Steps 1-2 and less for Steps 3-4.
Conclusion: Our four-step approach to applying systems analytical methods can encourage more mechanisms research efforts to consider these methods, and in turn fuel both (i) rigorous comparisons of these methods to alternative mechanisms research approaches and (ii) an active discourse across the field to better characterize when these methods are more or less appropriate to use for advancing mechanisms-related knowledge.
References
1. Bareinboim, E. & Pearl, J. (2016). Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences, 113(27), 7345-7352.
2. Geng, E.H., Baumann, A.A. & Powell, B.J. (2022). Mechanism mapping to advance research on implementation strategies. PLoS medicine, 19(2), e1003918.
3. Westreich, D., Edwards, J.K., Lesko, C.,R., Cole, S.R. & Stuart, E., A. (2019). Target validity and the hierarchy of study designs. American journal of epidemiology, 188(2), 438-443.
ORAL PRESENTATIONS
Assessing real-world access to evidence-based trauma-informed mental health services for youth: A mystery shopper study
Authors
Ms. Danielle Adams - University of Chicago
Ms. Nancy Jacquelyn Pérez-Flores - Brown School of Social Work, Washington University in St. Louis
Ms. Fatima Mabrouk - Silver School of Social Work, New York University
Ms. Carolyn Minor - American Blues Theater, Chicago Illinois
Background: Less than half of adolescents with a mental health (MH) disorder receive any MH care. Medicaid insures 38% of U.S. youth, making safety-net agencies key MH providers for Medicaid-enrolled adolescents. Yet the ability to access evidence-based trauma treatment at these agencies remains difficult. This study examines how access to trauma-informed MH services delivered in safety-net agencies varies by insurance type, race of the caller, and organizational type.
Methods: Using a mystery shopper methodology, three pseudo mothers (White, Latina, and Black voice actresses) called eligible community mental health centers (CMHCs) and federally qualified health centers (FQHCs) (N = 229) in Cook County, Illinois, requesting to schedule a MH appointment for their adolescent child who recently witnessed a traumatic event. The women called each agency twice in the spring and summer of 2021 with alternating Medicaid and private insurance coverage. A generalized linear model was used to assess the impact of contributing factors on scheduling an appointment.
Results: Only 17% (n = 78) of pseudo mothers could schedule an appointment. Primary reasons for denial of an appointment given by schedulers at agencies involved capacity or waitlist (n = 87) and an administrative requirement to switch their primary care provider into the organization's network (n = 71). Thirty-eight percent (n = 115) of agencies reported offering trauma-informed treatment, but only 39% (n = 45) of those could name a specific trauma-treatment offered by the agency. Only 11 agencies reported offering a trauma-treatment that was evidence-based. Insurance and organizational type did not predict MH access. Black and Latina pseudo mothers were 18% more likely to be denied an appointment than the White caller (p = .019).
Conclusion: Results have implications for the dissemination of evidence-based trauma-treatments and the availability and accessibility of MH services in safety-net agencies. Organizational-level strategies to reduce administrative requirements, enforce anti-discrimination policies, and increase the availability of evidence-based trauma treatments will be developed.
References
1. Cummings, J. R., Case, B. G., Ji, X., & Marcus, S. C. (2016). Availability of youth services in US mental health treatment facilities. Administration and Policy in Mental Health and Mental Health Services Research, 43(5), 717-727.
2. Cummings, J. R., Wen, H., & Druss, B. G. (2013). Improving access to mental health services for youth in the United States. JAMA, 309(6), 553-554.
2. Kugelmass, H. (2016). “Sorry, I’m Not Accepting New Patients” an audit study of access to mental health care. Journal of Health and Social Behavior, 57(2), 168-183.
4. Olin, S. C. S., O’Connor, B. C., Storfer-Isser, A., Clark, L. J., Perkins, M., Scholle, S. H., … & Horwitz, S. M. (2016). Access to care for youth in a state mental health system: A simulated patient approach. Journal of the American Academy of Child & Adolescent Psychiatry, 55(5), 392-399.
Disclosures of Interest: None declared.
Lyssn ClientBot: An AI-based MI training tool
Authors
Dr. David Atkins - Lyssn
Dr. Michael Tanana - Lyssn
Dr. Brian Pace - Lyssn
Ms. Angela Klipsch - Lyssn
Ms. Roisín Slevin - Lyssn
Dr. David Rosengren - Prevention Research Institute
Dr. Rita Dykstra - Prevention Research Institute
Dr. Cynthia Weaver - Annie E. Casey Foundation
Background: Workshops alone are insufficient for changing clinical practice.1 Case consultation or supervision with feedback improve skill acquisition, but are resource-intensive and rely on access to a content expert. Chatbots may offer a more feasible and scalable alternative to support skill development. We present Phase 1 findings from a fast-track SBIR project to develop and evaluate the effectiveness of ClientBot, which simulates a client experiencing mental health or substance use issues and allows clinicians to engage in Motivational Interviewing (MI)-focused conversations while receiving immediate, automated MI fidelity feedback to support skill development.2,3
Methods: Implementation of ClientBot is being planned or is underway at two partner organizations: Covenant House (CH), a youth homelessness crisis care shelter in New York City, and the Prevention Research Institute’s Prime for Life (PFL) program, an evidence-based prevention program specifically designed for people making high-risk alcohol and drug choices. User-centered design interviews and focus groups at both sites provided feedback to inform the ongoing development of ClientBot, which were complemented with quantitative data on usability from PFL trainees (System Usability Scale, SUS).
Results: Ten PFL trainees found ClientBot straightforward to use (MSUS = 86, range 0-100), and perceived MI skills feedback as accurate (M = 8.2, range 0-10). In focus groups, CH staff (n = 13) found ClientBot easy to navigate and expressed interest in future practice with ClientBot. ClientBot’s machine learning MI fidelity score achieved 96% agreement with human evaluated MI fidelity. We are currently in Phase II implementation at both sites, and will share quantitative data on usability, acceptability, appropriateness, and feasibility from the Phase II PFL pilot field trial.
Conclusion: Leveraging technology to improve clinical skills may offer a more feasible route to broadening access to evidence-based psychotherapies at scale. The development of such technologies through a community-partnered, user-centered design process may improve adoption speed in clinical practice.
References
1. Schwalbe, C. S., Oh, H. Y., & Zweben, A. (2014). Sustaining motivational interviewing: A meta-analysis of training studies. Addiction, 109, 1287–1294. http://dx.doi.org/10.1111/add.12558
2. Tanana, M. J., Soma, C. S., Srikumar, V., Atkins, D. C., & Imel, Z. E. (2019). Development and Evaluation of ClientBot: Patient-Like Conversational Agent to Train Basic Counseling Skills. Journal of medical Internet research, 21(7), e12529. https://doi.org/10.2196/12529
3. Imel, Z. E., Pace, B. T., Soma, C. S., Tanana, M., Hirsch, T., Gibson, J., Georgiou, P., Narayanan, S., & Atkins, D. C. (2019). Design feasibility of an automated, machine-learning based feedback system for motivational interviewing. Psychotherapy (Chicago, Ill.), 56(2), 318–328. https://doi.org/10.1037/pst0000221
Disclosures of Interest: David Atkins is a co-founder and CEO with an equity stake in Lyssn.io, Inc.
The relational and transactional strategy continuum measure
Authors
Dr. Leah Bartley - University of North Carolina, Chapel Hill & Kaye Implementation & Evaluation
Dr. William Oscar Fleming - UNC-Chapel Hill
Prof. Allison Metz - University of North Carolina at Chapel Hill
Background: As the field of implementation science seeks to identify mechanisms of change related to specific strategies, focusing more on relational aspects of the implementation process might offer insight and further account for unexplained variance in implementation outcomes. One opportunity is to further define the types of strategies and interactions that occur within implementation using Relational Theory.
Methods: The purpose of the overall study was to conduct a secondary review of ERIC implementation strategies1 and the degree to which they are relational or transactional in nature. First though, a categorical coding scheme was developed using Relational Theory to categorize strategies from highly transactional alliances to highly relational. The Relational and Transactional Strategy Continuum Measure was based on the Relational Continuum2 and Relational Theory3. The continuum was used to code the 73 ERIC implementation strategies. A two staged deductive and inductive coding process was conducted between three independent reviewers using distinct excel databases.
Results: Results from the coding suggested more implementation strategies featured relational alliance (highly relational, n = 18, semi-relational, n = 19) compared to transactional (highly transactional n = 9, semi-transactional, n = 10) and 18 strategies coded as operational alliances, which were strategies that involved a working exchange between parties with minor technical and relational characteristics.
Conclusion: This study is the first attempt to offer a critical perspective on implementation strategies using Relational Theory. It highlights the relational nature of the ERIC strategies compilation, as half of the strategies (n = 37) were coded as highly-relational or semi-relational. The Relational and Transactional Strategy Continuum could be used in practice to consider elements of strategy implementation and specify not only the actors, action targets, outcomes as well as how alliances are formed, interactions among actors, considerations of power dynamics and accountability and potential outcomes of relationships. Implementation researchers can begin to study the differences in effects when strategies are implemented in transactional or relational methods.
References
1. Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., … & Kirchner, J. E. (2015). A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10(1), 1-14.
2. Fletcher, J. K. (1998). Relational practice: A feminist reconstruction of work. Journal of management inquiry, 7(2), 163-186.
3. Fawcett, S. E., & Waller, M. A. (2015). Designing the supply chain for success at the bottom of the pyramid. Journal of Business Logistics, 36(3), 233-239.
Disclosures of Interest: Leah Bartley reports working for Kaye Implementation and Evaluation, a for-profit evaluation firm. This firm is not connected to the data gathered or analysis process of the measure.
Mechanism mapping to explain the success of a CHW-delivered intervention to improve screening for gestational diabetes in an Indian slum community.
Authors
Ms. Kathryn Broderick - Weill Cornell
Mr. Matthew Ponticiello - Weill Cornell Medicine
Ms. Vaishali Kulkarni - Deep Griha Society
Ms. Andrea Chalem - Weill Cornell Medicine
Dr. Puja Chebrolu - Weill Cornell Medicine
Ms. Ashelsha Onawale - Deep Griha Society
Dr. Jyoti Mathad - Weill Cornell Medicine
Dr. Radhika Sundararajan - Weill Cornell Medicine
Background: Gestational diabetes (GDM) is a common disease of pregnancy and results in increased risk of perinatal complications, type 2 diabetes, and cardiovascular disease1–3. The World Health Organization recommends the oral glucose tolerance test (OGTT) as gold standard screening for GDM. In India, 13% of pregnant women have GDM4, but our preliminary data found that only 2% of women delivering at a government hospital in a slum community had completed an OGTT. To bridge this evidence-practice gap, we hypothesized that an implementation strategy utilizing community health workers (CHW)5,6 to deliver home-based screening could increase OGTT uptake.
Methods: Between October 2021 and April 2022 we screened 247 pregnant women in an urban slum of Pune, India, and 224 women (90.7%) accepted the CHW-delivered OGTT. We conducted 30 interviews with purposively-sampled participants and all five CHWs to examine factors contributing to intervention implementation. Interview transcripts were analyzed using content analysis,7 and we apply a mechanism mapping approach8 to better understand “why” the CHW implementation strategy resulted in high uptake of this evidence-based screening tool.
Results: We found CHWs act through three distinct pathways: 1) affective, 2) cognitive, and 3) logistic. Within the context of low social distance, the CHW can create affective, social bonds with pregnant women, which allows the CHW to influence health behavior as a trusted advisor. The CHW cognitive influence comes through translation of evidence-based information about GDM for a population with low health literacy and education. Logistically, CHWs reduces structural barriers to GDM screening by visiting the patient in their home. The confluence of these pathways facilitates engagement, absorption of health information and, ultimately, acceptance of OGTT.
Conclusion: Our data suggest that CHW are uniquely positioned to effectively increase GDM screening uptake. Mechanism mapping, as in this example, can inform intervention generalizability by identifying constituent context and pathways contributing to success.
References
1. Vounzoulaki, E., Khunti, K., Abner, S. C., Tan, B. K., Davies, M. J., & Gillies, C. L. (2020). Progression to type 2 diabetes in women with a known history of gestational diabetes: Systematic review and meta-analysis. BMJ, m1361. https://doi.org/10.1136/bmj.m1361
2. Plows, J., Stanley, J., Baker, P., Reynolds, C., & Vickers, M. (2018). The Pathophysiology of Gestational Diabetes Mellitus. International Journal of Molecular Sciences, 19(11), 3342. https://doi.org/10.3390/ijms19113342
3. Kramer, C. K., Campbell, S., & Retnakaran, R. (2019). Gestational diabetes and the risk of cardiovascular disease in women: A systematic review and meta-analysis. Diabetologia, 62(6), 905–914. https://doi.org/10.1007/s00125-019-4840-2
4. Zhu, Y., & Zhang, C. (2016). Prevalence of Gestational Diabetes and Risk of Progression to Type 2 Diabetes: A Global Perspective. Current Diabetes Reports, 16(1), 7. https://doi.org/10.1007/s11892-015-0699-x
5. Jeet, G., Thakur, J. S., Prinja, S., & Singh, M. (2017). Community health workers for non-communicable diseases prevention and control in developing countries: Evidence and implications. PLOS ONE, 12(7), e0180640. https://doi.org/10.1371/journal.pone.0180640
6. Namazzi, G., Okuga, M., Tetui, M., Muhumuza Kananura, R., Kakaire, A., Namutamba, S., Mutebi, A., Namusoke Kiwanuka, S., Ekirapa-Kiracho, E., & Waiswa, P. (2017). Working with community health workers to improve maternal and newborn health outcomes: Implementation and scale-up lessons from eastern Uganda. Global Health Action, 10(sup4), 1345495. https://doi.org/10.1080/16549716.2017.1345495
7. Hsieh, H.-F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288.
8. Geng, E. H., Baumann, A. A., & Powell, B. J. (2022). Mechanism mapping to advance research on implementation strategies. PLOS Medicine, 19(2), e1003918. https://doi.org/10.1371/journal.pmed.1003918
Disclosures of Interest: None declared.
Context, adaptation, and adoption of a clinical decision support for cardiovascular risk reduction among women Veterans in a multisite trial: a longitudinal, mixed-method analysis
Authors
Dr. Julian Brunner - Center for the Study of Healthcare Innovation, Implementation, & Policy, VA Greater Los Angeles Healthcare System
Dr. Bevanne Bean-Mayberry - Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System; David Geffen School of Medicine, UCLA
Dr. Melissa Farmer - Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System
Dr. Catherine Chanfreau-Coffinier - VA Informatics and Computing Infrastructure (VINCI), VA Salt Lake City Healthcare System, Salt Lake City, UT
Dr. Claire Than - Center for the Study of Healthcare Innovation, Implementation and Policy, VA Greater Los Angeles Healthcare System
Dr. Tannaz Moin - Center for the Study of Healthcare Innovation, Implementation and Policy, VA Greater Los Angeles Healthcare System; David Geffen School of Medicine, UCLA
Dr. Alison Hamilton - Center for the Study of Healthcare Innovation, Implementation and Policy, VA Greater Los Angeles Healthcare System; David Geffen School of Medicine, UCLA
Dr. Erin Finley - Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System; UT Health San Antonio
Background: Evaluations of clinical decision support implementation often struggle to measure and explain heterogeneity in uptake over time and across settings, and to account for the impact of context and adaptation on implementation success (Harry et al., 2019). In 2017-2020, the EMPOWER QUERI implemented a computerized template aimed at identifying and documenting women Veterans’ cardiovascular (CV) risk (Bean-Mayberry et al., 2022) across five Veterans Healthcare Administration sites, using an enhanced Replicating Effective Programs (REP) implementation strategy (Hamilton et al., 2017).
Methods: We conducted 39 periodic reflections (Finley et al., 2018) with implementation team members before, during, and after implementation. Reflections were coded for theory-based and emergent contextual factors, implementation strategies, and adaptations to the CV template; member checking with the implementation team was used to validate findings and pinpoint key events. We used longitudinal joint displays (Fetters et al., 2013) of qualitative and quantitative findings to explore (1) how contextual factors emerged across sites, (2) how the template and implementation strategies were adapted in response to contextual factors, and (3) how contextual factors and adaptations coincided with template uptake across sites and over time.
Results: We identified site structure, staffing changes, relational authority of champions, and external leadership as important contextual factors. These factors gave rise to adaptations such as splitting the template into multiple parts, pairing the template with a computerized reminder, conducting academic detailing, creating cheat sheets, and using small-scale pilot-testing. All five sites exhibited variations in utilization over the months of implementation, though later sites exhibited higher template utilization immediately post-launch, possibly reflecting a “preloading” of adaptations from previous sites.
Conclusion: Our findings underscore the importance of adaptive approaches to implementation, with intentional shifts in intervention and strategy to meet the needs of individual sites, as well as the value of integrating mixed-method data sources in conducting longitudinal evaluation of implementation efforts.
References
1. Bean-Mayberry, B., Moreau, J., Hamilton, A. B., Yosef, J., Joseph, N. T., Batuman, F., Wight, S. C., & Farmer, M. M. (2022). Cardiovascular Risk Screening among Women Veterans: Identifying Provider and Patient Barriers and Facilitators to Develop a Clinical Toolkit. Women’s Health Issues, S1049386721001894. https://doi.org/10.1016/j.whi.2021.12.003
2. Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving Integration in Mixed Methods Designs-Principles and Practices. Health Services Research, 48(6pt2), 2134–2156. https://doi.org/10.1111/1475-6773.12117
3. Finley, E. P., Huynh, A. K., Farmer, M. M., Bean-Mayberry, B., Moin, T., Oishi, S. M., Moreau, J. L., Dyer, K. E., Lanham, H. J., Leykum, L., & Hamilton, A. B. (2018). Periodic reflections: A method of guided discussions for documenting implementation phenomena. BMC Medical Research Methodology, 18(1), 153. https://doi.org/10.1186/s12874-018-0610-y
4. Hamilton, A. B., Farmer, M. M., Moin, T., Finley, E. P., Lang, A. J., Oishi, S. M., Huynh, A. K., Zuchowski, J., Haskell, S. G., & Bean-Mayberry, B. (2017). Enhancing Mental and Physical Health of Women through Engagement and Retention (EMPOWER): A protocol for a program of research. Implementation Science, 12(1). https://doi.org/10.1186/s13012-017-0658-9
5. Harry, M. L., Truitt, A. R., Saman, D. M., Henzler-Buckingham, H. A., Allen, C. I., Walton, K. M., Ekstrom, H. L., O’Connor, P. J., Sperl-Hillen, J. M., Bianco, J. A., & Elliott, T. E. (2019). Barriers and facilitators to implementing cancer prevention clinical decision support in primary care: A qualitative study. BMC Health Services Research, 19(1), 534. https://doi.org/10.1186/s12913-019-4326-4
Disclosures of Interest: None declared.
Improving the implementation capacities that sustain evidence-based programs over time: A close look at changing social capital in Evidence2Success communities
Authors
Dr. Sarah Chilenski - Edna Bennett Pierce Prevention Research Center, Pennsylvania State University
Dr. Jochebed Gayles - Edna Bennett Pierce Prevention Research Center, Pennsylvania State University
Dr. Aaron Luneke - Evidence-based Prevention and Implementation Suport, Penn State University
Dr. Daphne Lew - Division of Biostatistics and Center for Population Health Informatics, Washington University School of Medicine
Dr. Francisco Villaruell – Michigan State University
Ms. Mary Lisa Penilla – Edna Bennett Pierce Prevention Research Center, Pennsylvania State University
Mr. Charles Henderson - Kearns Community Coalition
Dr. Lisa Gary - Keecha Harris And Associates
Ms. Mildred Johnson - Annie E. Casey Foundation
Ms. Sylvia Lin - Pennsylvania State University
Background: Theoretically, community prevention systems are social capital interventions. Social capital describes the resources that become available through the interconnectedness of people and groups. This study explores how the interconnections and resources among individuals and organizations change during the implementation of the Evidence2Success framework. Can the Evidence2Success framework, adopted by and implemented in low income communities of color, change the connections between individuals and organizations to improve the social capital that sustains EBIs?
Methods: Semi-structured interviews with Community Board members and Key Leaders in five Evidence2Success communities were conducted at three time points: (1) as the Evidence2Success framework was adopted; (2) after the Community Board selected risk/protective factor and outcome priorities, and EBIs; and (3) three to six months after implementing selected EBIs. Evaluators tracked how people and organizations work together to select, plan for, implement, and evaluate youth and family programs at all time points. Measures were matched with four common conceptualizations of social capital: bonding, bridging, linking, and collective capability. The linear effect of time was tested in multi-level models controlling for individual-level race/ethnicity and education; given the community-level n of 5, community was a fixed effect in the model.
Results: Significant changes over time were reported by Community Board members for measures of bonding, bridging, linking, and collective capability. Effect sizes were small to moderate across all significant models. In contrast, reports from Key Leaders did not change significantly over time.
Conclusion: Reports of the interconnections among people and organizations, and the social capital needed to sustain equitable implementation of EBIs changed significantly over time throughout the implementation of the Evidence2Success framework. These changes did not extend to reports from Key Leaders. Discussion will review the findings and implications for community-systems prevention efforts aimed at supporting the equitable implementation of EBIs in low income communities of color.
References
1. Chilenski, S., Ang, P., Greenberg, M., Feinberg, M., & Spoth, R. (2014). The Impact of a Prevention Delivery System on Perceived Social Capital: the PROSPER Project. Prevention Science, 15(2), 125-137. doi: 10.1007/s11121-012-0347-5
2. Javdani, S., & Allen, N. E. (2011). Proximal Outcomes Matter: A Multilevel Examination of the Processes by Which Coordinating Councils Produce Change. American Journal of Community Psychology, 47(1-2), 12-27. doi: 10.1007/s10464-010-9375-0
3. Petersen, D. M. (2002). The potential of social capital measures in the evaluation of comprehensive community-based health initiatives. American Journal of Evaluation, 23(1), 55-64. doi: 10.1016/S1098-2140(01)00167-9
Disclosures of Interest: None declared.
Examining the influence of the organizational implementation
context in schools
Authors
Dr. Catherine Corbin - University of Washington
Dr. Yanchen Zhang - University of Minnesota
Dr. Clayton Cook - Character Strong Organization
Prof. Aaron Lyon - University of Washington
Dr. Mark Ehrhart - Department of Psychology, University of Central Florida
Background: School-based social, emotional, and behavioral (SEB) programming can prevent mental health issues for millions of students. For this promise to materialize, these programs must be implemented with quality. Characteristics of organizational implementation context (OIC) are crucial for high quality implementation to occur. Theoretically, implementation leadership (IL) operates through implementation climate (IC) to influence implementation citizenship behaviors (ICB). Evidence supports this outside the education sector, but little is known about whether this would replicate within schools. Further, linking this model to relevant outcomes including teachers’ attitudes toward implementation can highlight how the aspects of the inner-setting might be leveraged to incite change among characteristics of individual implementers known to influence use of EBPs.
Methods: Data were collected from 350 teachers across 39 elementary schools, all of which were implementing a universal prevention SEB intervention. Teachers reported on their school’s IL, IC, and ICB, and their attitudes toward implementation at the beginning and end of the school year. Concurrent and longitudinal mediation models were run in Mplus v8 using bootstrapped standard errors. Difference scores were created for each OIC construct, though models will be re-examined using latent change scores prior to the conference.
Results: Results supported IL operating through IC to influence ICBS, both concurrently and over time (concurrent indirect effect = .50, p < .001; longitudinal indirect effect = .25, p < .001). Additionally, this mediation model emerged as significantly and positively associated with change in teachers’ attitudes toward implementation over time. Specifically,IL positively influenced IC, which positively influenced ICB, ultimately positively influencing teachers’ attitudes toward implementation (total indirect effect = 0.11, p < .001).
Conclusion: The OIC is crucial in laying the foundation for successful implementation. These results align with findings outside the education sector and extends that work by linking aspects of the OIC to teachers’ attitudes toward implementation.
References
1. Aarons, G. A., Erhart, M. G., Farahnak, L. R., & Sklar, M. (2014). Aligning leadership across systems and organizations to develop a strategic climate for evidence-based practice implementation. Annual Review of Public Health, 35, 255-274. https://doi.org/10.1146/annurev-publhealth-032013-182447
2. Duong, M. T., Bruns, E. J., Lee, K., Cox, S., Coifman, J., Mayworm, A., & Lyon, A. R. (2021). Rates of mental health service utilization by children and adolescents in schools and other common service settings: A systematic review and meta-analysis. Administration and Policy in Mental Health and Mental Health Services Research, 48, 1-20. https://doi.org/10.1007/s10488-020-01080-9
3. Ehrhart, M. G., Aarons, G. A., Farahnak, L. R. (2014). Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS). Implementation Science, 9, 157. https://doi.org/10.1186/s13012-014-0157-1
4. Lyon, A. R., Cook, C. R., Brown, E. C., Locke, J., Davis, C., Ehrhart, M., & Aarons, G. A. (2018). Assessing organizational implementation context in the education section: Confirmatory factor analysis of measures of implementation leadership, climate, and citizenship. Implementation Science, 13. https://doi.org/10.1186/s13012-017-0705-6
5. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38, 65-76. https://doi.org/10.1007/s10488-010-0319-7
Disclosures of Interest: None declared.
The effect of caregiver key opinion leaders on increasing caregiver demand for evidence-based practices to treat youth anxiety
Authors
Mrs. Margaret Crane – Temple University
Ms. Gillian Dysart – Temple University
Ms. Sydney Keller – Temple University
Ms. Olivia Brauer – Temple University
Ms. Sirina Tiwari – Temple University
Mr. John Lestino – Magowan Elementary School
Dr. Thomas Olino – Temple University
Dr. Sara Becker – Northwestern University Feinberg School of Medicine
Dr. Jonathan Purtle – Department of Public Health Policy and Management, New York University School of Global Public Health
Dr. Marc Atkins – University of Illinois at Chicago
Dr. Philip Kendall – Temple University
Background: Despite implementation efforts, most anxious youth do not receive evidence-based treatments (EBT). Dissemination initiatives can increase consumer knowledge of EBTs. Although key opinion leaders (KOLs) have been used in public health campaigns, this strategy has not been examined for the dissemination of mental health treatments. This study uses the theory of planned behavior to test the dissemination strategy of involving a KOL to increase caregiver demand for EBT for youth anxiety.
Methods: Participants (N = 264; 92% Female; 69% White) were caregivers who registered for a virtual presentation on youth anxiety and how to seek treatment. Caregivers were cluster-randomized by school (N = 24) to the KOL condition (presented by a researcher and a local caregiver KOL) or the researcher-only condition (presented by two researchers). Presentations occurred from May 2021-2022. Measures were completed pre- and post-presentation, as well as three months later. Results were analyzed quantitatively using multilevel models, logistic regressions, and t-tests.
Results: 106 participants attended the presentations. Relative to the researcher co-presenter, participants rated the KOL as significantly more relatable, familiar, similar, and understanding of their community ts(93) ≥ 2.00, ps < .05, but significantly less credible t(93) = 3.26, p = .002. Across conditions, there was a significant increase in participants’ knowledge of, subjective norms related to, attitudes about, intention to seek EBT for youth anxiety, bs ≥ 0.58, ps < .05, but not stigma, b = 0.59, p = .12. There was no significant difference between conditions in change on these measures, bs ≤ 0.55, ps > .05, or on rates of seeking youth anxiety EBT three months later, b = 0.11, p = .48.
Conclusion: The involvement of a caregiver KOL did not increase caregiver demand for EBT for youth anxiety. However, the outreach presentation was an effective dissemination strategy. Although involving KOLs may not be a necessary strategy for all consumer audiences, it may be beneficial for individuals with higher levels of stigma or scientific skepticism than participants in this study.
References
1. Becker, S. J. (2015). Direct-to-consumer marketing: A complementary approach to traditional dissemination and implementation efforts for mental health and substance abuse interventions. Clinical Psychology: Science and Practice, 22(1), 85–100. https://doi.org/10.1111/cpsp.12086
2. Crane, M. E., Atkins, M. S., Becker, S. J., Purtle, J., Olino, T. M., and Kendall, P. C. (2021). The effect of caregiver key opinion leaders on increasing caregiver demand for evidence-based practices to treat youth anxiety: protocol for a randomized control trial. Implementation Science Communications, 2(1), 107. https://doi.org/10.1186/s43058-021-00213-x
3. Flodgren, G., O’Brien, M. A., Parmelli, E., and Grimshaw, J. M. (2019). Local opinion leaders: Effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews, 2019(6), CD000125. https://doi.org/10.1002/14651858.cd000125.pub5
4. Purtle, J., Marzalik, J. S., Halfond, R. W., Bufka, L. F., Teachman, B. A., and Aarons, G. A. (2020). Toward the data-driven dissemination of findings from psychological science. American Psychologist, 75(8), 1052–1066. https://doi.org/10.1037/amp0000721
Disclosures of Interest: None declared.
To push or to pull: Strengthening outer setting factors to facilitate implementation and sustainability of evidence-based practices in children’s behavioral health settings
Authors
Ms. Amy Doyle - Baker Center for Children and Families/Harvard Medical School
Ms. Angelina Ruiz - Baker Center for Children and Families/Harvard Medical School
Ms. Michaela Harris - Baker Center for Children and Families/Harvard Medical School
Dr. Rachel Kim - Baker Center for Children and Families/Harvard Medical School
Dr. Daniel Cheron - Baker Center for Children and Families/Harvard Medical School
Dr. Robert Franks - Baker Center for Children and Families/Harvard Medical School
Background: Successful implementation and sustainability of evidence-based practices (EBPs) depend on various factors that may be influenced by outer setting constructs as defined by the Consolidated Framework for Implementation Research (CFIR). Little is known about the relationship between outer setting factors and implementation outcomes. State or county-based infrastructure that fosters inter-agency collaboration, provider motivation and shared-learning, and external policies that provide sufficient resources and incentives may all contribute to successful implementation and sustainability at the organizational level.
Methods: This presentation will operationalize key outer setting constructs associated with implementation of a modular EBP in community mental health centers (CMHCs) through three Enhanced Learning Collaboratives (ELCs) in state or county systems of care. Strategies to engage CMHCs and systems’ leaders, such as pre-implementation activities to raise awareness, influence infrastructure, increase stakeholder engagement, and improve inter-organizational collaboration, will be shared. Implementation process and outcome measures collected during and 12-months following the completion of each initiative will be discussed.
Results: Preliminary findings suggest that a “pull” from providers and CMHC leadership contributes to improved clinical integrity, higher organizational capacity, and greater sustainability compared to a “push” from policymakers. Clinicians in the ELC that exhibited greater “pull” demonstrated statistically significant higher use of at least one MATCH module per session (M = 95.2%) compared to clinicians in the “push” ELCs (M = 93.2%); (t(19) = .993, p = .039). The percentage of “pull” clinicians who achieved certification in MATCH was substantially higher (M = 77.3%) than “push” clinicians (M = 68.7%); (t(19) = .850, p = .057).
Conclusion: Outer setting factors including peer pressure, cosmopolitanism, and external policies have potential to contribute to implementation success. Strategies can be employed by researchers, administrators, and implementation practitioners to strengthen outer setting factors, especially during the pre-planning phase, to increase the “pull” from providers and CMHC leadership and sustainability of the intervention.
References
1. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. doi:10.1186/1748-5908-4-50.
2. King DK, Shoup JA, Raebel MA, et al. Planning for Implementation Success Using RE-AIM and CFIR Frameworks: A Qualitative Study. Frontiers in Public Health. 2020;8. Available from: https://www.frontiersin.org/article/10.3389/fpubh.2020.00059.
3. McHugh S, Dorsey CN, Mettert K, Purtle J, Bruns E, Lewis CC. Measures of outer setting constructs for implementation research: A systematic review and analysis of psychometric quality. Implementation Research and Practice. 2020;1:263348952094002. doi:10.1177/2633489520940022.
4. Constructs – The Consolidated Framework for Implementation Research. Available from: https://cfirguide.org/constructs/.
5. Lukas CV, Holmes SK, Cohen AB, Restuccia J, Cramer IE, Shwartz M, et al. Transformational change in health care systems: an organizational model. Health Care Manage Rev. 2007 Dec;32(4):309–20.
6. Lang JM, Franks RP, Epstein C, Stover C, Oliver JA. Statewide dissemination of an evidence-based practice using Breakthrough Series Collaboratives. Child Youth Serv Rev. 2015;55:201–9.
7. Chorpita BF, Weisz JR. Modular Approach to Therapy for Children with Anxiety, Depression, Trauma, or Conduct Problems (MATCH-ADTC). Satellite Beach, FL: PracticeWise, LLC; 2009.
8. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011 Jan;38(1):4–23.
9. Scaccia JP, Cook BS, Lamont A, Wandersman A, Castellow J, Katz J, et al. A practical implementation science heuristic for organizational readiness: R = MC2. J Community Psychol. 2015 Apr;43(4):484–501.
Disclosures of Interest: None declared.
A mixed methods stakeholder-focused cost analysis to deploy implementation strategies for school-based universal prevention
Authors
Dr. Andria Eisman - Wayne State University
Ms. Christine Koffkey - Wayne State University
Dr. Lawrence Palinkas - University of Southern California
Ms. Christina Harvey - Oakland County Schools, Michigan
Ms. Judy Fridline - Genesee Intermediate School District, Michigan
Dr. Amy Kilbourne - University of Michigan
Dr. David Hutton - University of Michigan
Background: Obtaining information on implementation strategy costs from multiple perspectives is essential to data-driven decision-making about resource allocation for EBI implementation. This mixed methods study aims to identify the costs and priorities of deploying Enhanced Replicating Effective Programs (REP) to implement the Michigan Model for Health,TM a universal school-based prevention EBI, from key stakeholder perspectives including the implementer, leadership, and organizational perspectives.
Methods: We used a modified cost-calculator microcosting approach to map activity costs across implementation phases for Enhanced REP. We used an experimental embedded mixed methods approach, incorporating qualitative data collection concurrently with quantitative data, to explore cost considerations, expand data collection and explain the costs and priorities across stakeholder groups.
Results: We estimate the startup costs at $1497/school and pre-implementation costs at $5618/school. During the implementation phase, the mean facilitation cost is $1244/school. Qualitative data, however, indicate that Enhanced REP activities represented largely opportunity costs for teachers. We found a misalignment between professional development (PD) time allocation and teacher priorities; Implementation activities, including training, lesson prep, and facilitation with the health coordinator, are in addition to regular teaching responsibilities. This was partly due to leadership priorities (e.g., sticking to district PD schedule) and organizational priorities (e.g., budget).
Conclusion: Our results indicate significant opportunity costs of time spent by teachers outside of school hours when engaging in implementation strategy activities. Previous research findings indicate that, from the societal perspective, universal prevention is a good return on investment. Notable misalignment in cost burden and priorities, however, exists across stakeholder groups. Additional multi-level strategies are needed to improve the alignment of costs and benefits to enhance implementation success and sustainment. As perspectives included in economic evaluation reflect value judgments, we focus on those perspectives informed by the analysis. We also discuss opportunities to create greater alignment across perspectives.
References
1. Eisman, A. B., Quanbeck, A., Bounthavong, M., Panattoni, L., & Glasgow, R. E. (2021). Implementation science issues in understanding, collecting, and using cost estimates: a multi-stakeholder perspective. Implementation science, 16(1), 75.
2. Chamberlain, P., Snowden, L. R., Padgett, C., Saldana, L., Roles, J., Holmes, L., et al. (2011). A strategy for assessing costs of implementing new practices in the child welfare system: Adapting the English cost calculator in the United States. Administration and Policy in Mental Health and Mental Health Services Research, 38, 24–31.
3. Drummond, M., Sculpher, M., Claxton, K., Stoddart, G., Torrance, G. (2015) Methods for the economic evaluation of health care programmes (4th ed.). Oxford: Oxford University Press.
Disclosures of Interest: None declared.
Assessing the Massachusetts Academic Public Health Corps response to COVID-19: A mixed methods evaluation guided by the Consolidated Framework for Implementation Research (CFIR)
Authors
Dr. Patricia Elliott - Boston University School of Public Health
Ms. Sarah Fielman - Boston University School of Public Health
Ms. Hiba Abousleiman - Boston University School of Public Health
Ms. Alyson Codner - Boston University School of Public Health
Ms. Zoë Wangstrom - Boston University School of Public Health
Ms. Allyson Cogan - Boston University School of Public Health
Dr. Jacey Greece - Boston University School of Public Health
Background: The Massachusetts Academic Health Department Consortium (AHD) established the Academic Public Health Volunteer Corps (APHVC) to support Local Health Departments (LHDs) by placing APHVC volunteers to help meet rapidly emerging needs during COVID-19, and to engage students in professional development opportunities. A mixed-methods evaluation aligned with the Consolidated Framework for Implementation Research (CFIR) was conducted to generate lessons learned from LHDs’ perspectives, and to inform recommendations for replication and sustainability.
Methods: An online survey was emailed to all 351 Massachusetts LHDs and was completed by LDHs who used (n = 27) and did not use (n = 45) the APHVC. The survey assessed LHD characteristics and communities served, impact of COVID-19 on local efforts and resources, successes and challenges of utilizing APHVC, and future use of APHVC. Results informed probes for qualitative data collection via focus groups and interviews (n = 11). Participants provided contextual information including facilitators and barriers that influenced adoption, implementation, and maintenance of APHVC. Themes identified in the survey aligned with themes from the qualitative interviews.
Results: The mixed-methods evaluations leveraged CFIR to develop data collection tools, code, organize and analyze data, and produce recommendations. Survey findings were stratified by LHDs who used and did not use the APHVC; interview findings were categorized into the five CFIR constructs using a deductive approach, allowing the two coders to consider multi-level factors. Differences were resolved by consensus.
Conclusion: APHVC filled resource gaps, built capacity, and provided high quality deliverables. However, LHDs experienced issues with reliability and communication of volunteers, and lack of time to train volunteers. CFIR aided in evaluating APHVC in the context in which it was delivered, producing actionable recommendations to inform best practices, dissemination, and future iterations of the program. Results will enhance program effectiveness and sustainability, community health, and health equity across Massachusetts, and may inform academic practice-based programs across the United States.
References
1. Erwin, P. C., & Brownson, R. C. (2019). The Academic Health Department: Aging Gracefully? Journal of Public Health Management and Practice, 25(1), 32–33. https://doi.org/10.1097/PHH.0000000000000924
2. Keck, C. W. (2019). Academic Health Department Partnerships: Bridging the Gap Between Town and Gown. American Journal of Public Health, 109(5), 665–666. https://doi.org/10.2105/AJPH.2019.305039
3. Keck, W. C. (2000). Lessons Learned from an Academic Health Department: Journal of Public Health Management and Practice, 6(1), 47–52. https://doi.org/10.1097/00124784-200006010-00008
4. Keith, R. E., Crosson, J. C., O’Malley, A. S., Cromp, D., & Taylor, E. F. (2017). Using the Consolidated Framework for Implementation Research (CFIR) to produce actionable findings: A rapid-cycle evaluation approach to improving implementation. Implementation Science, 12(1), 15. https://doi.org/10.1186/s13012-017-0550-7
5. Massachusetts Department of Public Health. (2019). Blueprint for Public Health Excellence Recommendations for Improved Effectiveness and Efficiency of Local Public Health Protections (pp. 1–111). https://www.mass.gov/files/documents/2019/07/15/blueprint-public-health-excellence-2019.pdf
Disclosures of Interest: None declared.
Co-design and implementation of common elements-based academic support in child welfare services: Results and lessons from a pragmatic hybrid trial in Norway
Authors
Dr. Thomas Engell - Regional Centre for Child and Adolescent Mental Health
Background: Child Welfare Services (CWS) need effective practices for academic support that can reach wide (1). Implementation in complex and capacity strained settings pose demands on the implementability of practices and pragmatism in implementation (2). The KOBA study (3) co-designed, implemented, and evaluated Enhanced Academic Support (EAS), a flexible intervention for elementary school children in CWS. There is growing interest in using common elements-methodologies for hypothesis generation, intervention (re)design, and pragmatic implementation. We combined novel common elements-methodology with co-design to develop and test implementable evidence-informed practices tailored to individuals and contexts. This session presents our results, successes, and failures.
Methods: The study used Integrated Knowledge Translation (4), moving through exploration, synthesis, collaborative development, implementation and evaluation, and informing policy and practice. For synthesis, we distilled common practice, process, and implementation elements in systematically reviewed academic interventions for children at risk (5). We identified the most common elements and combinations in effective interventions accounted for in ineffective interventions, and used these elements as “building blocks” for co-design with practitioners and clients. We used a pragmatic hybrid RCT with mixed-methods to evaluate, and collected data on implementation climate, readiness, implementability of EAS, implementation quality, and intervention effectiveness and value. Analyses included psychometrics, multiple regressions, mixed-effects models, content analyses, and convergence and expansion analyses.
Results: Implementation capacity was low, and practitioners’ job satisfaction predicted implementation climate (β=1.12, p < .001, [6]). EAS was implementable in CWS due to specific design characteristics such as cross-domain flexibility, but the implementation strategies were inadequate for sufficient implementation quality (7). Receiving more academic support was associated with favorable perceptions about intervention value, but limitations in implementation quality and statistical power restrict conclusions about effects.
Conclusion: EAS is implementable in CWS, but adaptations and improved implementation strategies are needed to fulfill its potential. The study provides implications about design features, contextual sensitivity, and strategic tailoring to support ethical, effective, and sustainable implementation in complex practice settings.
References
1. Kirkøen, B., Engell, T., Follestad, I. B., Holen, S., & Hagen, K. A. (2021). Early academic struggles among children with home-based support from child welfare services. Children and Youth Services Review, 131, 106268.
Lyon, A. R., & Koerner, K. (2016). User-centered design for psychosocial intervention development and implementation. Clinical Psychology: Science and Practice, 23(2), 180-200.
2. Engell, T., Follestad, I. B., Andersen, A., & Hagen, K. A. (2018). Knowledge translation in child welfare—improving educational outcomes for children at risk: study protocol for a hybrid randomized controlled pragmatic trial. Trials, 19(1), 714.
Jull, J., Giles, A., & Graham, I. D. (2017). Community-based participatory research and integrated knowledge translation: advancing the co-creation of knowledge. Implementation science, 12(1), 1-9.
3. Engell, T., Kirkøen, B., Hammerstrøm, K. T., Kornør, H., Ludvigsen, K. H., & Hagen, K. A. (2020). Common elements of practice, process and implementation in out-of-school-time academic interventions for at-risk children: A systematic review. Prevention science, 21(4), 545-556.
4. Engell, T., Kirkøen, B., Aarons, G. A., & Hagen, K. A. (2020). Individual level predictors of implementation climate in child welfare services. Children and Youth Services Review, 119, 105509.
5. Engell, T., Løvstad, A. M., Kirkøen, B., Ogden, T., & Hagen, K. A. (2021). Exploring how intervention characteristics affect implementability: A mixed methods case study of common elements-based academic support in child welfare services. Children and Youth Services Review, 129, 106180.
Disclosures of Interest: None declared.
Implementation of a cardiovascular toolkit in primary care increased women veterans’ participation in behavioral change programs
Authors
Dr. Melissa Farmer - Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System, Los Angeles, CA
Dr. Alison Hamilton - Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System, Los Angeles, CA; David Geffen School of Medicine at UCLA, Los Angeles, CA
Dr. Erin Finley - University of Texas Health Science Center at San Antonio; VA Greater Los Angeles
Dr. Martin Lee - Center for the Study of Healthcare Innovation, Implementation and Policy, VA Greater Los Angeles Healthcare System
Dr. Alexis Huynh - Center for the Study of Healthcare Innovation, Implementation and Policy, VA Greater Los Angeles Healthcare System
Dr. Catherine Chanfreau-Coffinier - VA Informatics and Computing Infrastructure (VINCI), VA Salt Lake City Healthcare System, Salt Lake City, UT
Dr. Claire Than - Center for the Study of Healthcare Innovation, Implementation and Policy, VA Greater Los Angeles Healthcare System
Dr. Julian Brunner - Center for the Study of Healthcare Innovation, Implementation, & Policy, VA Greater Los Angeles Healthcare System
Dr. C. Amanda Schweizer - Center for the Study of Healthcare Innovation, Implementation and Policy, VA Greater Los Angeles Healthcare System
Dr. Tannaz Moin - Center for the Study of Healthcare Innovation, Implementation and Policy, VA Greater Los Angeles Healthcare System; David Geffen School of Medicine, UCLA
Dr. Bevanne Bean-Mayberry - Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System; David Geffen School of Medicine, UCLA
Background: Cardiovascular (CV) disease is the leading cause of death for women in the US, and women have more difficulty managing CV risk factors and have higher obesity and inactivity rates compared to men (McSweeney et al., 2016). We implemented and evaluated a CV toolkit (Bean-Mayberry et al., 2022) targeting women Veterans to increase identification of CV risk, enhance patient/provider communication about risk, and increase women’s engagement in relevant behavior change programs.
Methods: The CV toolkit included three components: patient CV self-screener, provider CV risk template in the electronic medical record, and a facilitated goal-setting group tailored for women. We evaluated the effect of toolkit implementation on participation in behavior change programs that target reducing CV risk including: 1) VA’s MOVE! weight loss program, and 2) health promotion and disease prevention programs and complementary integrative health services (HPDP/CIH). Using a non-randomized stepped wedge design (Hamilton et al., 2017), we utilized a three-level non-linear fixed effect model and estimated models stratified by age (65 + and <65).
Results: Five VA health clinics implemented the CV toolkit between 6/2017-3/2020. Among women who visited primary care at least once during this time (n = 6009), the mean age was 45 years, and 48% had one or more CV risk factors. For women age 65+ (n = 540), active toolkit implementation resulted in increased odds of participating in MOVE! (OR = 1.09; 95%CI:1.030,1.152) compared to when the toolkit was not active either within or between sites. Women <65 (n = 5469) had increased odds (OR = 1.01; 95%CI:1.002-1.020) of using HPDP/CIH services during active toolkit implementation.
Conclusion: The CV toolkit intervention was effective in increasing women Veterans’ participation in behavior change programs. Given that participation in programs varied by age group, variety in VA programs may be key to supporting CV targeted behavior change for women across the age spectrum.
References
1. Bean-Mayberry, B., Moreau, J., Hamilton, A. B., Yosef, J., Joseph, N. T., Batuman, F., Wight, S. C., & Farmer, M. M. (2022). Cardiovascular Risk Screening among Women Veterans: Identifying Provider and Patient Barriers and Facilitators to Develop a Clinical Toolkit. Women’s Health Issues, S1049386721001894. https://doi.org/10.1016/j.whi.2021.12.003
2. Hamilton, A. B., Farmer, M. M., Moin, T., Finley, E. P., Lang, A. J., Oishi, S. M., Huynh, A. K., Zuchowski, J., Haskell, S. G., & Bean-Mayberry, B. (2017). Enhancing Mental and Physical Health of Women through Engagement and Retention (EMPOWER): A protocol for a program of research. Implementation Science, 12(1). https://doi.org/10.1186/s13012-017-0658-9
3. McSweeney, J.C., Rosenfeld, A.G., Abel, W.M., Braun, L.T., Burke, L.E., Daugherty, S.L., Fletcher, G.F., Gulati, M., Mehta, L.S., Pettey, C., Reckelhoff, J.F. and on behalf of the American Heart Association Council on Cardiovascular and Stroke Nursing, Council on Clinical Cardiology, Council on Epidemiology and Prevention, Council on Hypertension, Council on Lifestyle and Cardiometabolic Health, and Council on Quality of Care and Outcomes Research. (2016). Preventing and Experiencing Ischemic Heart Disease as a Woman: State of the Science. Circulation, 133(13), 1302–1331. https://doi.org/10.1161/CIR.0000000000000381.
Disclosures of Interest: None declared.
Honoring Tribal sovereignty through implementation science: Lessons learned from a Tribal-academic partnership
Authors
Dr. Tommi Gaines - University of California, San Diego
Dr. Jessica Montoya - University of California, San Diego
Dr. Richard Armenta - California State University - San Marcos
Background: American Indians and Alaska Natives (AI/AN) endure a disproportionate burden of HIV-related disparities, yet they are often overlooked in HIV prevention initiatives. Consistent with recommendations to advance health equity within implementation science (IS),1 our academic team partnered with Southern California Tribes to explore the social determinants of health in context of HIV. A prominent theme is historical and ongoing oppression that has functioned to disparage and undermine AI/AN identity, which in turn has led to contemporary health behaviors associated with HIV risk and substance misuse.2,3 In response to these health disparities, Tribes have adopted community-oriented approaches to healing.
Methods: The Tribal-Academic partnership – inclusive of three AI faculty from Southern California Universities and Tribal leadership (Vice Chairman, Secretary, and Resource Management Director) from a Southern California Tribe – obtained federal grant funding to co-develop a multifaceted implementation strategy for integrating HIV self-testing into an existing Tribal Wellness Program.
Results: The grant development process occurred over a 5-month period and capitalized on the Tribe’s priority to strengthen their capacity to prevent substance misuse. During monthly calls, the Tribal-Academic team co-conceptualized a community-engaged mixed-method IS study to examine the feasibility, acceptability, and appropriateness of implementing HIV self-testing within an existing Tribal Wellness Program. Input from Tribal leadership functioned to clarify whether HIV prevention is an appropriate health priority for the Tribe, to identify important stakeholders, and to establish roles for partners throughout the implementation process.
Conclusion: The role of the Tribe in equity-related considerations4, including setting the agenda, defining and prioritizing health problems, and developing measurement protocols to evaluate IS outcomes was essential to ensuring the Tribe exercised their sovereign right to protect the health and wellbeing of their community members. Through a shared and dynamic decision-making process, the Tribal-Academic partnership used an equity lens throughout the early phases of IS activities.
References
1. Brownson, R.C., Kumanyika, S.K., Kreuter, M.W., Haire-Joshu, D. (2021). Implementation science should give higher priority to health equity. Implementation Science, 16(1): 28. https://doi.org/10.1186/s13012-021-01097-0
2. Armenta, R., Kellog, D., Montoya, J.L., Romero, R., Armao, S., Calac, D., Gaines, T.L. (2021). “There is a lot of practice in not thinking about that": Structural, interpersonal, and individual-level barriers to HIV/STI prevention among reservation based American Indians. International Journal of Environmental Research and Public Health, 18(7):3566. https://doi.org/10.3390/ijerph18073566
3. Walters, K.L., Beltran, R., Evans-Campbell, T., Simoni, J.M. (2011). Keeping Our Hearts from Touching the Ground: HIV/AIDS in American Indian and Alaska Native Women. Womens Health Issues, 21(6): S261-S5. https://doi.org/10.1016/j.whi.2011.08.005
4. Kerkhoff, A.D., Farrand, E., Marquez, C., Cattamanchi, A., Handley, M.A.. Addressing health disparities through implementation science—a need to integrate an equity lens from the outset. Implementation Science,17: 13. https://doi.org/10.1186/s13012-022-01189-5
Disclosures of Interest: None declared.
The development of an IEP team implementation strategy: Enhancing the implementation of EBPs for students with ASD
Authors
Mr. Blaine Garman-McClaine - Indiana University
Mr. Kane Carlock - Indiana University
Background: Schools are the primary setting where youth with autism spectrum disorder (ASD) receive behavioral services (Stichter et al., 2016). Federal legislation requires school personnel to support students with ASD using evidence-based behavioral practices (EBBPs). However, EBBPs are implemented at a low rate in schools (Lushin et al., 2020). An Individualized Education Program (IEP), a student’s educational roadmap (Ruble et al., 2013), brings together a diverse set of school-based professionals to support students with ASD. Thus, a student’s IEP team is a pre-existing structure with potential to increase successful adoption and implementation of EBBPs. Recently, numerous implementation strategies have been identified for the school context (Cook et al., 2019). Given the high priority of successful implementation in schools, implementation strategies must be appropriate and feasible for the context, and therefore designed and tailored in collaboration with school-based professionals. The objective of this study was to develop an IEP team implementation strategy designed to enhance the adoption and implementation of EBBPs for students with ASD across school personnel and settings.
Methods: To develop the implementation strategy, the current project was conducted in four stages adapted from Lewis and colleagues (2018). First, 125 IEP team members (e.g., special education teachers, school psychologists) from public elementary schools indicated which determinants of implementation exist in their respective schools, based on the Consolidated Framework for Implementation Research (Damschroder et al., 2009). The authors then selected discrete implementation strategies from the SISTER project (Cook et al., 2019) based on determinants identified during the first stage. Next, ten discrete implementation strategies were presented to the IEP team members who participated in stage one to assess the feasibility and importance of each discrete strategy. Over 40% of participants responded to the second survey. Descriptive analyses were performed for both surveys. Lastly, based on the feedback from IEP team members, a multifaceted, multiphase implementation strategy was developed using the Exploration, Preparation, Implementation, Sustainment Framework (Aarons et al., 2011) and defined using Proctor and colleagues (2013) reporting and specifying guidelines.
Results: Four barriers emerged as most salient to IEP team members: time to collaborate (59.2%), professional development opportunities (48.8%), planning time (48.4%), and resistance from caregivers (32.0%). Similarly, four leading facilitators of implementation were identified from the data: administrative support (71.6%), retaining specialists (71.2%), observing successful implementation (68.8%), and protected planning time (65.6%).
Of the ten discrete implementation strategies that IEP team members rated, monitoring the progress of an implementation effort was regarded as the most feasible (M = 3.8/5) and most important (M = 4.6/5) discrete strategy. IEP team members rated improving implementers’ buy-in (M = 2.9/5) as the least feasible strategy and providing local technical assistance (M = 3.9/5) as the least important strategy. Taking IEP team members’ feedback into account, the final IEP team implementation strategy consists of seven discrete strategies from four SISTER domains (Cook et al., 2019).
Conclusion: There is a critical need for implementation science tools and frameworks to support school-based professionals’ implementation of EBBPs. The implementation strategy presented above is the first to leverage the IEP team structure to enhance the adoption and implementation of EBBPs. Future research should evaluate the overall effectiveness of the implementation strategy and determine which key mechanisms of the multifaceted strategy must remain or be adapted to ensure effectiveness in schools. Furthermore, adaptations to the strategy can be addressed using FRAME-IS (Miller et al., 2021) for use with other evidence-based practices or programs (e.g., school mental health, literacy).
References
1. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4-23. https://doi.org/10.1007/s10488-010-0327-7
2. Cook, C. R., Lyon, A. R., Locke, J., Waltz, T., & Powell, B. J. (2019). Adapting a compilation of implementation strategies to advance school-based implementation research and practice. Prevention Science, 20(6), 914-935. https://doi.org/10.1007/s11121-019-01017-1
3. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(1), 1-15. https://doi.org/10.1186/1748-5908-4-50
4. Lewis, C. C., Scott, K., & Marriott, B. R. (2018). A methodology for generating a tailored implementation blueprint: an exemplar from a youth residential setting. Implementation Science, 13(1), 1-13. https://doi.org/10.1186/s13012-018-0761-6
5. Lushin, V., Mandell, D., Beidas, R., Marcus, S., Nuske, H., Kaploun, V., Seidman, M., Gaston, D., & Locke, J. (2020). Trajectories of evidence based treatment for school children with sutism: What’s the right level for the implementation?. Journal of Autism and Developmental Disorders, 50(3), 881-892. https://doi.org/10.1007/s10803-019-04304-6
6. Miller, C. J., Barnett, M. L., Baumann, A. A., Gutner, C. A., & Wiltsey-Stirman, S. (2021). The FRAME-IS: a framework for documenting modifications to implementation strategies in healthcare. Implementation Science, 16(1), 1-12. https://doi.org/10.1186/s13012-021-01105-3
7. Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013). Implementation strategies: recommendations for specifying and reporting. Implementation Science, 8(1), 1-11. https://doi.org/10.1186/1748-5908-8-139
8. Ruble, L., & McGrew, J. H. (2013). Teacher and child predictors of achieving IEP goals of children with autism. Journal of Autism and Developmental Disorders, 43(12), 2748-2763. https://doi.org/10.1007/s10803-013-1884-x
9. Smith, J. D., Li, D. H., & Rafferty, M. R. (2020). The Implementation Research Logic Model: a method for planning, executing, reporting, and synthesizing implementation projects. Implementation Science, 15(1), 1-12. https://doi.org/10.1186/s13012-020-01041-8
10. Stichter, J. P., Riley-Tillman, T. C., & Jimerson, S. R. (2016). Assessing, understanding, and supporting students with autism at school: Contemporary science, practice, and policy. School Psychology Quarterly, 31(4), 443-449. https://doi.org/10.1037/spq0000184
Disclosures of Interest: None declared.
Using clinician self-report to evaluate fidelity to cognitive processing therapy and cognitive behavior therapy
Authors
Dr. Nicole Gumport - Stanford University
Ms. Samantha Hernandez - National Center for PTSD
Dr. Alayna Park - University of Oregon
Mr. Jiyoung Song - University of California, Berkeley
Dr. Amber Calloway - University of Pennsylvania
Dr. Kimberlye Dean - Massachusetts General Hospital
Dr. Dawne Vogt - VA Boston Healthcare System and Boston University
Dr. Soo Youn - Massachusetts General Hospital
Ms. Clara Johnson - University of Washington
Dr. Rob DeRubeis - University of Pennsylvania
Dr. Luana Marques - Massachusetts General Hospital & Harvard Medical School
Dr. Torrey Creed - University of Pennsylvania
Dr. Shannon Wiltsey Stirman - Stanford University & National Center for PTSD
Background: Assessing fidelity is an important part of assessing key implementation outcomes. Although observer ratings are the gold-standard for assessing fidelity, conducting these ratings is burdensome and often infeasible. Developing scalable measures of fidelity is a priority for implementation efforts. This study aimed to validate a more efficient measure of fidelity—clinician-rated checklists completed as part of routine treatment—to cognitive processing therapy (CPT) and to cognitive behavior therapy (CBT), which are frontline treatments for PTSD and depression/anxiety, respectively. It was hypothesized that clinician rated-fidelity of CPT and CBT would be associated with (1) observer-rated fidelity and (2) treatment outcome.
Methods: Clinicians (N = 164) who treated patients (N = 403) with CPT for PTSD or CBT for depression or anxiety in routine care settings (e.g., VA, medical center, community clinics) completed a clinician-rated checklist of fidelity to either CPT (N = 687 checklists) or CBT (N = 312 checklists) at the end of each session. Checklists were scored for the percentage of required treatment elements that were applied (CPT: mean = 64%, SD = 29%; CBT: mean = 82%, SD = 17%). Trained observers rated sessions for fidelity (N = 552 [CPT]; 268 [CBT]). Treatment outcome was measured using the PTSD Checklist for DSM-5 (PCL-5), Beck Anxiety Inventory (BAI), and Patient Health Questionnaire-9 (PHQ-9). Multilevel modeling was used.
Results: Clinician-rated fidelity was significantly associated with observer-rated fidelity for both CPT (Beta = 0.26, p < .000) and CBT (Beta = 0.49, p < .000). Analyses revealed that clinician-rated fidelity predicted change on the PCL-5 following CPT (Beta = -0.01, p = 0.04). Analyses revealed that clinician-rated fidelity predicted change on the PHQ-9 (Beta = -0.02, p = 0.04) following CBT but on the BAI.
Conclusion: These results offer preliminary evidence that clinician-rated checklists can efficiently, accurately assess fidelity in routine care settings. Although not associated with all outcome measures, clinician ratings were highly concordant with observer ratings, suggesting they are an option for regularly monitoring fidelity. Implications for different measurement options and treatment outcome will be discussed.
References
1. Stirman, S. W., Marques, L., Creed, T. A., Gutner, C. A., Derubeis, R., Barnett, P. G., Kuhn, E., Suvak, M., Owen, J., Vogt, D., Jo, B., & Schoenwald, S. (2018). Leveraging routine clinical materials and mobile technology to assess CBT fidelity : the Innovative Methods to Assess Psychotherapy Practices (imAPP) study. Implementation Science, 13, 1–11. https://doi.org/10.1186/s13012-018-0756-3
2. Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2012). Toward the effective and efficient measurement of implementation fideltiy. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 32–43. https://doi.org/10.1007/s10488-010-0321-0.
3. Schoenwald, S. K. (2011). It’s a Bird, It’s a Plane, It’s … Fidelity Measurement in the Real World. Clinical Psychology: Science and Practice, 18(2), 142–147. https://doi.org/10.1111/j.1468-2850.2011.01245.x
Disclosures of Interest: None declared.
Understanding the multilevel determinants of clinicians’ imaging decision-making: Setting the stage for de-implementation of low-value imaging
Authors
Mrs. Soohyun Hwang - University of North Carolina- Chapel Hill
Dr. Sarah A Birken - Wake Forest School of Medicine
Dr. Matthew Nielsen - University of North Carolina- Chapel Hill
Dr. Jennifer Elston-Lafata - University of North Carolina- Chapel Hill
Dr. Stephanie Wheeler - University of North Carolina- Chapel Hill
Dr. Lisa Spees - University of North Carolina-
Chapel Hill
Background: De-implementation requires targeting multilevel determinants of low-value care. The objective of this study was to identify multilevel determinants of imaging for prostate cancer (PCa) and asymptomatic microhematuria (AMH), two common urologic conditions that have contributed substantially to the approximately $30 billion spent annually on unnecessary imaging in the US.
Methods: We used a convergent mixed-methods approach involving survey and interview data. Using a survey, we asked 33 clinicians (55% response rate) to indicate their imaging approach to 8 clinical vignettes designed to elicit responses that would demonstrate guideline-concordant/discordant imaging practices for describing patients with PCa or AMH. A subset of survey respondents (N = 7) participated in semi-structured interviews guided by the Theoretical Domains Framework (TDF) and Consolidated Framework for Implementation Research (CFIR), whose combination has offered a comprehensive understanding of multilevel implementation determinants. We analyzed the interviews using a directed content analysis approach. For each clinical condition, we identified relevant constructs/domains based on the frequencies with which the construct/domain was mentioned across interviews. We then identified subthemes within each construct/domain and examined subtheme similarities and differences across the two clinical conditions.
Results: The majority of clinicians chose guideline-concordant imaging behaviors for PCa; guideline-concordant imaging intentions were more varied for AMH. Imaging decisions for both PCa and AMH were often driven by national guidelines from major professional societies. However, when clinicians felt that guidelines were inadequate, they reported that their decision-making was influenced by their knowledge of recent scientific evidence, past clinical experiences, and the anticipated benefits of imaging (or not imaging) to both the patient and the clinician. In particular, clinicians referred to patients’ anxiety and uncertainty which were, at times, resolved through more intensive diagnostic imaging. Patients’ clinical factors also informed clinicians’ imaging decisions. For AMH patients, clinicians additionally expressed concerns regarding legal liability risk.
Conclusion: Our study identified comprehensive multilevel determinants of imaging to inform the development of de-implementation interventions to reduce low-value imaging, using CFIR + TDF, which we found useful for identifying determinants of de-implementation. De-implementation interventions should be tailored to address the contextual determinants that are specific to each clinical condition.
References
1. Augustsson, H., Ingvarsson, S., Nilsen, P. et al. (2021). Determinants for the use and de-implementation of low-value care in health care: a scoping review. Implement Sci Commun 2, 13. https://doi.org/10.1186/s43058-021-00110-3
2. Li, S., & Brantley, E. (2015). Malpractice Liability Risk and Use of Diagnostic Imaging Services: A Systematic Review of the Literature. Journal of the American College of Radiology : JACR, 12(12 Pt B), 1403–1412. https://doi-org.libproxy.lib.unc.edu/10.1016/j.jacr.2015.09.015
2. Smith, M., Saunders, R., Stuckhardt, L., McGinnis, J. M., Committee on the Learning Health Care System in America, & Institute of Medicine (Eds.). (2013). Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. National Academies Press (US).
Disclosures of Interest: None declared.
Evaluating VHA’s response to intimate partner violence among women primary care patients
Authors
Dr. Katherine Iverson - National Center for PTSD, VA Boston Healthcare System & Department of Psychiatry, Boston University School of Medicine
Ms. Kelly Stolzmann - VA Center for Healthcare Organization and Implementation Research, VA Boston Healthcare System
Ms. Omonyele Adjognon - Center for Healthcare Organization and Implementation Research, VA Boston Healthcare System & Boston University
Ms. Julianne Brady - VA Center for Healthcare Organization and Implementation Research, VA Boston Healthcare System
Dr. Melissa Dichter - Center for Health Equity Research and Promotion, Corporal Crescenz VA Medical Center & Department of Social Work, Temple University
Dr. Robert Lew - VA Boston Healthcare System and Department of Public Health, Boston University
Dr. Megan Gerber - Albany Medical College & Albany VA Medical Center
Dr. Samina Iqbal - Palo Alto VA Healthcare System
Dr. Galina Portnoy - VA Pain Research, Informatics, Multi-morbidities, and Education Center, VA Connecticut & Yale School of Medicine
Dr. Christopher Miller - VA Boston Healthcare System and Harvard Medical School
Background: Intimate partner violence (IPV) is a public health problem that disproportionately impacts women veterans. The Veterans Health Administration (VHA) recommends evidence-based IPV screening in primary care to identify women who may benefit from support services1. To enhance the integration of IPV screening programs at late-adopting primary care clinics, VHA initiated implementation facilitation (IF) consisting of interactive, flexible, tailored support that can include action planning, goal identification, and engaging stakeholders. We are evaluating IF’s impact on implementation (i.e., reach of IPV screening) and clinical effectiveness (i.e., disclosure rates) outcomes for this operations-led initiative.
Methods: A cluster randomized, stepped wedge, Hybrid Type II design compared IF to implementation as usual within VHA2. IF was led by VHA Office of Women's Health leadership and occurred in two waves across nine sites. Using RE-AIM3 as an analytic framework, we examined medical records to identify changes in IPV screening rates (reach) and IPV disclosure rates (effectiveness) from pre- to post-IF for the wave 1 sites (n = 5, treating n = 5,938 women). Chi-square tests were conducted to examine change in outcomes over time.
Results: Across wave 1 sites, there were significant increases in the proportion of women receiving IPV screening pre-IF to post-IF (32% to 47%, p < .0001), with a non-significant trend in disclosure rates increasing across sites from pre-IF (8.3%) to post-IF (10.6%; p = .10). Notably, the disclosure rate of IPV during the IF period is nearly double the national IPV disclosure rate for women VHA patients (6.3%), suggesting that IF may increase the effectiveness of IPV screening relative to implementation as usual.
Conclusion: IF increases reach of IPV screening programs in VHA primary care clinics and may improve the clinical effectiveness of IPV screening programs by increasing disclosure of IPV. Findings may benefit other healthcare systems aiming to implement effective IPV screening programs, thereby enhancing the healthcare response to IPV.
References
1. United States Department of Veterans Affairs, Veterans Health Administration (VHA). (2018). VHA Directive 1198. Intimate Partner Violence Assistance Program.
2. Iverson K. M., Dichter, M. E., Stolzmann K. M., et al., Miller, C. J. (2020). Assessing the Veterans Health Administration's response to intimate partner violence among women: Protocol for a randomized hybrid type 2 implementation-effectiveness trial. Implement Science, 15, 29, 1-10.
3. Glasgow, R. E., Vogt, T. M., Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89, 1322–1327.
Disclosures of Interest: None declared.
R3 supervisor implementation strategy increases evidence-based supervisor-caseworker interactions
Authors
Dr. Jamie Jaramillo - Oregon Social Learning Center
Dr. Michael Stoolmiller - Oregon Research Institute
Dr. Zoe Alley - Oregon Social Learning Center
Dr. Holle Schaper - Oregon Social Learning Center
Dr. Patricia Chamberlain - Oregon Social Learning Center
Dr. Lisa Saldana - Oregon Social Learning Center
Background: Family engagement in child welfare systems (CWSs) is critical for positive system outcomes (Toros et al., 2018), yet CWSs often operate with a deficit-model that reduces such engagement (Harris, 2011; Tuck, 2013). Strength-based caseworker-family interactions, which promote engagement (Arbeiter & Toros, 2017; Bijleveld, et al., 2015), could be increased by leveraging an understudied CWS dyad: supervisor-caseworker. The R3 Supervisor Strategy is an implementation approach targeting the interactions between supervisors and caseworkers to facilitate positive evidence-based interactions between caseworkers and families. The present study explores R3 supervisor-caseworker interactions over time (measured via microsocial coding) and is the first step toward understanding such dynamics as a potential mechanism for increasing the implementation of evidence-based family engagement strategies and subsequent positive system outcomes.
Methods: Caseworker-supervisor microsocial interactions were coded for positive R3 behaviors over 463 group supervision sessions across 102 supervisors and 372 caseworkers (4.5 sessions per supervisor, range = 2-6). Fifteen-minute session excerpts were coded in 45 consecutive 20-second intervals. Multi-level Cox models with random effects were used to evaluate both the extent to which 1) supervisor and caseworker R3 behaviors increased from baseline to the final post- R3 -training assessment and 2) caseworker’s R3 behaviors when unprompted by the supervisor also increased.
Results: By the final post-training assessment, supervisors (hazard rate = 2.03, p < .001) and caseworkers (hazard rate = 1.80, p = .009) were significantly more likely to use an R3 positive interaction strategy. Furthermore, caseworkers showed a significant increase in their use of R3 strategies without prompting from the supervisor (hazard rate = 1.80, p < .001), with no such increase for supervisor-prompted R3 strategies (hazard rate = 0.824, p > .05).
Conclusion: The R3 supervisor implementation strategy successfully increased positive evidence-based engagement behaviors of both supervisors and caseworkers, with caseworkers independently demonstrating these skills over time. Future analyses will examine the impact of this on caseworker-family interactions and subsequent system level outcomes.
References
1. Arbeiter, E., & Toros, K. (2017). Participatory discourse: Engagement in the context of child protection assessment practices from the perspectives of child protection workers, parents and children. Children and Youth Services Review, 74,17–27. https:// doi.org/10.1016/j.childyouth.2017.01.020.
2. Bijleveld, G. G., Dedding, C. W. M., & Bunders-Aelen, J. F. G. (2015). Children’s and young people’s participation within child welfare and child protection services: A state-of-the-art review. Child & Family Social Work, 20(2), 129–138. https://doi.org/10.1111/cfs.12082
2. Harris, N. (2011). Does responsive regulation offer an alternative? Questioning the role of formalistic assessment in child protection investigations. British Journal of Social Work, 41, 1383–1403. http://dx.doi.org/10.1093/bjsw/bcr112.
3. Toros, K., DiNitto, D. M., & Tiko, A. (2018). Family engagement in the child welfare system: A scoping review. Children and Youth Services Review, 88, 598–607. https://doi.org/10.1016/j.childyouth.2018.03.011
4. Tuck, V. (2013). Resistant parents and child protection: Knowledge base, pointers for practice and implications for policy. Child Abuse Review, 22,5–19. http://dx.doi.org/ 10.1002/car.1207.
Disclosures of Interest: None declared.
Uncovering determinants of perceived feasibility of an evidence-based therapy through coincidence analysis
Authors
Ms. Clara Johnson - University of Washington
Dr. Chris Gray - Duke University
Ms. Chang(Lucy) Liu - University of Washington
Mr. Augustine Wasonga - Ace Africa
Dr. Kate Whetten - Duke University
Dr. Shannon Dorsey - University of Washington
Background: A mental health provider’s perception of how well an intervention can be carried out in their context (i.e., feasibility) is an important indicator of successful implementation of an intervention. Despite the importance of feasibility and recent calls to better understand determinants of implementation success, no studies to our knowledge have identified determinants of feasibility. The current study aims to uncover which implementation determinants are associated with perceived feasibility of an evidence-based therapy through a case-based causal approach. Construct selection was based on the integration of implementation and social change theories, stakeholder perspectives, and review of the implementation literature.
Methods: Data come from an implementation-effectiveness study (BASIC) which aims to identify implementation practices and policies that lead to successful implementation of culturally-adapted Trauma-focused Cognitive Behavioral Therapy (TF-CBT) in western Kenya. We utilized coincidence analysis (CNA), a case-based mathematical approach based on Boolean algebra, given its ability to identify complex causality and equifinality of determinants and outcomes. Using self-reports from 240 lay-counselors that delivered TF-CBT to children who experienced a parental death, CNA identified multiple “pathways”, or combinations of determinants, that led to the same outcome (i.e., high/low levels of feasibility).
Results: CNA revealed one pathway of determinants that led to high levels of feasibility: high counselor TF-CBT self-efficacy. This pathway explained 76% of high feasibility with 80% reliability. Results highlighted three potential pathways of determinants that led to low levels of feasibility, all of which included work-related stressors (i.e., burnout, fatigue, poor satisfaction). These pathways explained 77% of low feasibility with 76% reliability.
Conclusion: The results indicate that self-efficacy and work-related stressors may be important determinants of mental health intervention feasibility, particularly in contexts similar to that of this study. The presenter will further explore implications for the study sites and present possible strategies to improve feasibility-related determinants.
References
1. Dorsey, S., Gray, C. L., Wasonga, A. I., Amanya, C., Weiner, B. J., Belden, C. M., … & Whetten, K. (2020). Advancing successful implementation of task-shifted mental health care in low-resource settings (BASIC): Protocol for a stepped wedge cluster randomized trial. BMC psychiatry, 20(1), 1-14. https://doi.org/10.1186/s12888-019-2364-4
2. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., … & Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65-76. https://doi.org/10.1007/s10488-010-0319-7
3. Williams, N. J., & Beidas, R. S. (2019). Annual research review: The state of implementation science in child psychology and psychiatry: A review and suggestions to advance the field. Journal of Child Psychology and Psychiatry, 60(4), 430-450. https://doi.org/10.1111/jcpp.12960
Disclosures of Interest: None declared.
Expanding implementation strategy taxonomies to reflect published practice: Mapping between ERIC, EPOC, and a living review of global HIV implementation research
Authors
Dr. Christopher Kemp - Johns Hopkins University
Ms. Sita Lujintanon - Johns Hopkins University
Ms. Noelle Le Tourneau - Washington University School of Medicine
Dr. Laura Beres - Johns Hopkins University
Dr. Sheree Schwartz - Johns Hopkins University
Dr. Byron Powell - Brown School, Washington University in St. Louis, St. Louis, MO, United States
Dr. Stefan Baral - John Hopkins University
Mr. Ryan Thompson - Johns Hopkins University
Mr. Branson Fox - Washington University School of Medicine
Dr. Elvin Geng - Washington University School of Medicine
Dr. Ingrid Eshun-Wilson - Washington University in St. Louis
Background: Implementation strategy taxonomies have been developed to promote the clarity and comparability of implementation research. Our objective was to use a dataset of implementation strategies from the Living Database of HIV Implementation Science in Low-and-Middle Income Countries (LIVE) to identify opportunities to refine strategy taxonomies to better reflect the breadth of published research.
Methods: We conducted a systematic review and included studies of any design, published from 2004-2022, in low- or middle-income countries (LMICs), describing HIV-related intervention implementation and reporting at least one HIV cascade outcome. Implementation strategies were abstracted through inductive coding of actors, actions, and action targets (Proctor et al., 2013). We mapped actions to the Expert Recommendations for Implementing Change (ERIC) and Effective Practice and Organisation of Care (EPOC) strategy taxonomies (Powell et al., 2015; EPOC, 2015).
Results: We screened 42,595 abstracts; 418 studies met inclusion criteria. We abstracted 3,253 implementation strategies (median 6 per study, range 1-44) that used 79 different actions. Strategies to engage patients (e.g., educating on HIV treatment or prevention) were the most common (45.5%). Approximately half (41, 51.9%) of the 79 actions identified in LIVE mapped to both the ERIC and EPOC taxonomies. An additional 10 (12.7%) mapped only to ERIC and 17 (21.5%) mapped only to EPOC. 11 (13.9%) were not represented in ERIC or EPOC. These un-mapped actions included providing psychosocial support to patients, performing contact tracing, and offering patients a menu of services. 33 (45.2%) of the ERIC strategies were not represented in EPOC and 57 (54.3%) of the EPOC strategies were not represented in ERIC.
Conclusion: We identified opportunities to expand established implementation strategy taxonomies, including by incorporating patient-level strategies used in published studies, and by using complementary taxonomies to fill gaps in strategy types.
References
EPOC. (2015). EPOC Taxonomy. https://epoc.cochrane.org/epoc-taxonomy
Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015). A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10(1), 21.
Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013). Implementation strategies: Recommendations for specifying and reporting. Implementation Science, 8(1), 139.
Disclosures of Interest: None declared.
Decision-makers’ experiences with rapid evidence summaries to support real-time evidence informed decision-making in crises: A mixed methods study
Authors
Dr. Ahmad Firas Khalid - Ottawa Hospital Research Institute
Dr. Jeremy Grimshaw - Ottawa Hospital Research Institute
Dr. Salim Sohani - Canadian Red Cross
Dr. Faiza Rab - Canadian Red Cross
Ms. Nandana Parakh - McMaster University
Dr. Rana Charide - McMaster University
Background: A strong health system depends on increasing the use of evidence-based policies.1 There is a clear need for research evidence to drive policymaking and emergency responses so that lives are not lost, and resources are not wasted. To improve the use of research evidence in policy and practice, it is important to provide corresponding products tailored to the target audience, such as rapid evidence summaries. 2, 3 This study aims to gain real-world insights from decision-makers about how they use evidence summaries to inform real-time decision-making in crisis-settings, and to use our findings to improve the format of evidence summaries.
Methods: This study used an explanatory sequential mixed method study design. First, we used a survey to identify the views and experiences of those who were directly involved in crisis response, and who may or may not have used evidence summaries. Second, we used the insights generated from the survey to help inform qualitative interviews with decision-makers in crisis zones to derive an in-depth understanding of how they use evidence summaries and their desired format for evidence summaries to gain real-time support in using evidence to inform decision-making.
Results: A diverse set of 26 decision-makers working in health and humanitarian emergencies participated in this study. Decision-makers preferred the following components in evidence summaries: title of evidence summary, target audience, presentation of key findings in an actionable checklist or infographic format, implementation considerations, assessment of the quality of evidence presented, citation and hyperlink to the full review, funding sources, language of full review, and other sources on information on the topic. Decision-makers identified specific suggestions about how to improve online platforms hosting evidence summaries, many of which can also be applied to other evidence websites.
Conclusion: Our evidence summary template can streamline the process of responding to decision-makers knowledge needs during a crisis.
References
1. Blanchet, K., Sistenich, V., Ramesh, A., Frison, S., Warren, E., Smith, J., & Roberts, B. (2013). An evidence review of research on health interventions in humanitarian crises. London: London School of Hygiene & Tropical Medicine.
2. Giguere, A., Labrecque, M., Haynes, R. B., Grad, R., Pluye, P., Légaré, F., … & Carmichael, P. H. (2014). Evidence summaries (decision boxes) to prepare clinicians for shared decision-making with patients: a mixed methods implementation study. Implementation Science, 9(1), 1-13.
3. Petkovic, J., Welch, V., Jacob, M. H., Yoganathan, M., Ayala, A. P., Cunningham, H., & Tugwell, P. (2016). The effectiveness of evidence summaries on health policymakers and health system managers use of evidence from systematic reviews: a systematic review. Implementation Science, 11(1), 1-14.
Disclosures of Interest: None declared.
StrategEase: An interactive tool to support the selection of implementation strategies using theory
Authors
Dr. Sobia Khan - The Center for Implementation
Dr. Julia Moore - The Center for Implementation
Ms. Valentina Gestaldo - The Center for Implementation
Ms. Vanessa Trenton - The Center for Implementation
Dr. Lauren Tessier - The Center for Implementation
Dr. Maoliosa Donald - The Center for Implementation
Background: Researchers have identified two important components of fostering individual level change using implementation strategies: 1) identifying determinants of change and linking these theoretically to intervention functions [the theoretical domains framework (TDF) and capability-opportunity-motivation-behaviour (COM-B) theory, mapped to intervention functions in the behaviour change wheel]; 2) taxonomies of implementation strategies that support change [e.g., expert recommendations for implementing change (ERIC)]. However, there are multiple gaps in selecting strategies in practice. First, the two components have not been linked to one another, making it difficult to select both theoretically-sound and pragmatically-specific strategies. Second, there is little guidance for researchers new to implementation science and implementation practitioners to undergo this process. To bridge these gaps, we developed StrategEase, a freely available interactive online tool that aids in identifying implementation strategies by linking theory to concrete strategies.
Methods: The StrategEase tool builds on the behaviour change wheel algorithm developed by Michie et al (2014): the domains of TDF, which is grounded in the COM-B theory, are linked to intervention functions. Next, we developed a list of implementation strategies drawn from multiple sources: ERIC, Effective Practice and Organisation of Care (EPOC), behavior change techniques, intervention mapping, and a crowdsourcing activity. We mapped specific strategies to the intervention functions. A prototype tool was developed in Genially and usability tested during two stages of the design phase.
Results: The StrategEase tool is interactive; it allows users to take categorized barriers and facilitators, and select from lists of specific implementation strategies suggested to them that were compiled using theory.
Conclusion: This freely available interactive online StrategEase tool will enable thousands of professionals around the world to select implementation strategies to directly address the underlying barriers and facilitators to change.
References
1. Atkins, L., Francis, J., Islam, R., O’Connor, D., Patey, A., Ivers, N., Foy, R., Duncan, E. M., Colquhoun, H., Grimshaw, J. M., Lawton, R., Michie, S. (2017). A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems. Implementation Science, 12, 77. https://doi.org/10.1186/s13012-017-0605-9
2. Cane, J., O’connor, D., & Michie, S. (2012). Validation of the theoretical domains framework for use in behavior change and implementation research. Implementation Science, 7(37),1-17.
2. Powell, B.J., Waltz, T.J., Chinman, M.J., Damschroder, L.J., Smith, J.L., Matthieu, M.M.,
Proctor, E.K., & Kirchner, J.E. (2015). A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10(21), 1-14.
Disclosures of Interest: None declared.
Embodying equity in implementation: synthesis of guiding questions and actions across multiple implementation pathways
Authors
Dr. Sobia Khan - The Center for Implementation
Dr. Julia Moore - The Center for Implementation
Dr. Lauren Tessier - The Center for Implementation
Ms. Gail D'Souza - Pennsylvania State University
Background: Implementation may not be bridging, and could be perpetuating, inequities. Recognizing this, the fields of implementation science and practice have prioritized equity. Despite this focused attention, it is difficult for implementation researchers and practitioners to embed equity actions in implementation. The purpose of this work was to understand key considerations and recommendations to enhance the practice of equity across implementation processes (designing for implementation; implementation, spread, and scale; implementation teams).
Methods: First, we conducted a scoping review of equity literature published in the field of implementation science and in related fields: prevention science, health services research, and public health. Literature was identified through an online search and consultation with experts. Two analysts abstracted key equity considerations and mapped these to implementation process models. A third analyst resolved differences and facilitated discussion to arrive at consensus. Lack of concrete equity actions was a key gap identified in the literature. This resulted in the conduct of a second phase of this work: a crowdsourcing activity. Using the results of phase one, we sought input from implementation researchers and practitioners to determine what types of equity strategies have been used in their work to address specific equity considerations.
Results: This work resulted in the development of the “equity iceberg” that describes the multiple levels of equity considerations in implementation: actions, relationships, systems and structures, and mental models. Guiding questions to reflect on equity were developed for each of these levels. We also synthesized equity actions that would provide implementers with guidance on how to enhance equity based on how they responded to the guiding questions.
Conclusion: Equity is paramount and yet is difficult to practice. The equity iceberg, guiding questions, and actions can help implementers deeply reflect and act on equity at all steps and stages of implementation.
References
1. Baumann, A.A., Cabassa, L.J. Reframing implementation science to address inequities in healthcare delivery. BMC Health Serv Res 20, 190 (2020). https://doi.org/10.1186/s12913-020-4975-3
2. Yousefi Nooraie, R., Kwan, B., Cohn, E., AuYoung, M., Clarke Roberts, M., Adsul, P., & Shelton, R. (2020). Advancing health equity through CTSA programs: Opportunities for interaction between health equity, dissemination and implementation, and translational science. Journal of Clinical and Translational Science, 4(3), 168-175. doi:10.1017/cts.2020.10
3. Woodward, E.N., Matthieu, M.M., Uchendu, U.S. et al. The health equity implementation framework: proposal and preliminary study of hepatitis C virus treatment. Implementation Sci 14, 26 (2019). https://doi.org/10.1186/s13012-019-0861-y
Disclosures of Interest: None declared.
Caregiver home practice of intervention skills and their impact on targeted parenting skills
Authors
Dr. Joanna Kim - Arizona State University
Prof. Nancy Gonzales - Arizona State University
Background: Home practice (HP) of intervention skills is an active ingredient in evidence-based interventions and often the theorized mechanism of change (Kazantzis et al., 2010). In caregiver-mediated interventions, caregiver HP is necessary for interventions to reach the intended target—the child. However, caregiver HP rates are relatively unknown (Chacko et al., 2016). Few studies have examined caregiver HP rates’ associations with intervention targets (one exception, Berkel et al., 2018). We examined 1) how often caregivers complete assigned HP and 2) HP’s relation to intervention outcomes. Given historical differences in intervention engagement and implementation outcomes for diverse families, we also examined how HP rates and associations varied by program language delivery (English vs. Spanish).
Methods: Of the 414 primary caregivers (42.8% Spanish delivery) who participated in the family-based preventative intervention Bridges, 233 caregivers (56.3%) reported how often they completed assigned HP. Caregivers self-reported on their use of parenting skills taught within the intervention (solicitation of child activities, consistent discipline, effective communication, and family cohesion) at pre- and post-intervention. We conducted a multigroup structural equation model (SEM) to identify HP’s associations with intervention targets, controlling for pre-intervention parenting and family processes, caregiver intervention attendance, number of caregivers participating in the program (i.e., program dosage), and child sex. The multigroup framework was used to identify differential associations by program language.
Results: Caregivers in the Spanish-language implementation of the program were significantly more likely to complete HP than those in English-language groups. The SEM revealed that caregiver HP was significantly associated with post-intervention gains in consistent discipline for both groups (B = .07, p < .05), but gains in solicitation (BEnglish = .03, p > .05; BSpanish = .17, p < .01) and parent-child communication (BEnglish = -.03, p > .05; BSpanish = .13, p < .001) were only evident for the Spanish-language participants.
Conclusion: Findings suggest that HP effectively promotes targeted intervention skills, particularly among Latinx caregivers when services are available in their preferred language.
References
1. Berkel, C., Sandler, I. N., Wolchik, S. A., Brown, C. H., Gallo, C. G., Chiapa, A., Mauricio, A. M., & Jones, S. (2018). “Home Practice Is the Program”: Parents’ Practice of Program Skills as Predictors of Outcomes in the New Beginnings Program Effectiveness Trial. Prevention Science, 19(5), 663–673. https://doi.org/10.1007/s11121-016-0738-0
2. Chacko, A., Jensen, S. A., Lowry, L. S., Cornwell, M., Chimklis, A., Chan, E., Lee, D., & Pulgarin, B. (2016). Engagement in Behavioral Parent Training: Review of the Literature and Implications for Practice. Clinical Child and Family Psychology Review, 19(3), 204–215. https://doi.org/10.1007/s10567-016-0205-2
3. Kazantzis, N., Whittington, C., & Dattilio, F. (2010). Meta-Analysis of Homework Effects in Cognitive and Behavioral Therapy: A Replication and Extension. Clinical Psychology: Science and Practice, 17(2), 144–156. https://doi.org/10.1111/j.1468-2850.2010.01204.x
Disclosures of Interest: None declared.
Applying the Matrixed Multiple Case Study method to identify factors related to sustainability of the evidence-based Collaborative Chronic Care Model in outpatient mental health clinics
Authors
Dr. Bo Kim - VA Center for Healthcare Organization and Implementation Research / Harvard Medical School
Dr. Jennifer Sullivan - Providence VA Medical Center and Brown University School of Public Health
Ms. Madisen Brown - VA Boston Healthcare System
Dr. Samantha Connolly - VA Boston Healthcare System and Harvard Medical School
Dr. Elizabeth Spitzer - VA Boston Healthcare System and Harvard Medical School
Dr. Christopher Miller - VA Boston Healthcare System and Harvard Medical School
Background: Sustaining evidence-based practices (EBPs) in healthcare is crucial to ensuring ongoing care quality and addressing health disparities. Approaches to identifying the most pertinent factors affecting sustainability are sorely needed. One such approach is Matrixed Multiple Case Study (MMCS; Kim et al., 2020), which we applied to identify factors related to sustainability of the evidence-based Collaborative Chronic Care Model (CCM; Miller et al., 2013) at nine mental health clinics.
Methods: We conducted directed content analysis of thirty provider interviews, using CCM elements and Integrated Promoting Action on Research Implementation in Health Services (i-PARIHS; Harvey & Kitson, 2016) domains as codes. Based on CCM code summaries, we designated each site as high/medium/low sustainability. We used i-PARIHS code summaries to identify relevant factors for each site, the extent of their presence, and whether they had enabling/neutral/hindering/unclear influences. We organized these data into a matrix and assessed cross-site trends affecting sustainability.
Results: CCM sustainability status was split among the sites, with three sites each being high, medium, and low. Twenty-six factors were identified, of which three exhibited strong trends by CCM sustainability status. (i) “Staff/Leadership turnover” was present with a hindering influence on sustainability at most high, medium, and low sites. (ii) “Collaborativeness/Teamwork” was present and enabling at most high and medium sites, while only somewhat present with a neutral influence at most low sites. (iii) “Having a consistent/strong internal facilitator” was present and enabling at high sites, while variably present with a hindering, neutral, or unclear influence at low sites.
Conclusion: MMCS was useful in identifying that CCM sustainability may be impacted by knowledge retention during staff/leadership transitions, provider collaboration, and availability of skilled facilitators. A trial is underway to prospectively examine whether enhancing the abovementioned factors affects sustainability. Using MMCS on other EBPs may identify sustainability-related factors applicable across multiple contexts.
References
1. Harvey, G., & Kitson, A. (2016). PARIHS revisited: from heuristic to integrated framework for the successful implementation of knowledge into practice. Implementation science : IS, 11, 33. https://doi.org/10.1186/s13012-016-0398-2
2. Kim, B., Sullivan, J. L., Ritchie, M. J., Connolly, S. L., Drummond, K. L., Miller, C. J., Greenan, M. A., & Bauer, M. S. (2020). Comparing variations in implementation processes and influences across multiple sites: What works, for whom, and how? Psychiatry research, 283, 112520. https://doi.org/10.1016/j.psychres.2019.112520
3. Miller, C. J., Grogan-Kaylor, A., Perron, B. E., Kilbourne, A. M., Woltmann, E., & Bauer, M. S. (2013). Collaborative chronic care models for mental health conditions: cumulative meta-analysis and metaregression to guide future research and implementation. Medical care, 51(10), 922–930. https://doi.org/10.1097/MLR.0b013e3182a3e4c4
Disclosures of Interest: None declared.
Factors influencing the implementation of guideline-recommended practices for post-concussive sleep disturbance and headache in the Veterans Health Administration
Authors
Dr. Adam Kinney - Rocky Mountain MIRECC
Dr. Nazanin Bahraini - Rocky Mountain MIRECC
Dr. Jeri Forster - Rocky Mountain MIRECC
Dr. Lisa Brenner - Rocky Mountain MIRECC
Background: The VA/DoD 2016 Clinical Practice Guideline (CPG) for mild traumatic brain injury (mTBI) includes evidence-based treatment recommendations for post-concussive sleep disturbance and headache. Determinants of implementing these recommendations in the Veteran Health Administration (VHA) is unknown. The purpose of this study was to understand barriers and facilitators to the implementation of recommended practices for post-concussive sleep disturbance and headache.
Methods: Convergent parallel mixed methods design. 20 VHA stakeholders (14 clinicians; 4 researchers; 2 policymakers) across 10 national VHA sites rated the quality of recommendations for sleep disturbance and headache using the AGREE-REX instrument. A descriptive analysis of item scores was performed to rate recommendations with respect to: 1) clinical credibility (e.g., evidence quality); 2) alignment with stakeholder values; and, 3) implementability. We conducted semi-structured interviews with stakeholders and used Descriptive and Interpretive Analysis to identify themes. After analyzing the AGREE-REX and interview data, we synthesized the results into coherent conclusions by comparing and contrasting respective findings.
Results: Stakeholders highlighted that the mTBI CPG includes features which reflect clinical priorities (e.g., alignment with Veteran preferences), making it an appropriate standard of care and facilitating its implementation. However, stakeholders also identified that the design of the mTBI CPG, along with provider- (e.g., knowledge) and facility-level factors (e.g., resources), may obstruct implementation. In addition to these potential barriers, stakeholders noted a lack of comprehensive systematic efforts designed to promote the uptake of these recommendations. Findings also revealed stakeholder recommendations for addressing potential barriers (e.g., decision support).
Conclusion: With the recent release of the 2021 mTBI CPG, decisionmakers are encouraged to incorporate information gathered from previous implementation efforts to promote adherence to updated recommendations. Study findings, including recommended changes suggested by stakeholders, offer information that can be leveraged to design such efforts and promote care quality and associated outcomes for Veterans with mTBI.
References
1. The Management of Concussion-mild Traumatic Brain Injury Working Group. (2016). VA/DoD clinical practice guideline for management of concussion-mild traumatic brain injury (mTBI). The Office of Quality and Performance, VA & Quality Management Directorate, US Army Medical Command.
2. The Management of Concussion-mild Traumatic Brain Injury Working Group. (2021). Management and Rehabilitation of Post-Acute Mild Traumatic Brain Injury (mTBI). The Office of Quality and Performance, VA & Quality Management Directorate, US Army Medical Command. https://www.healthquality.va.gov/guidelines/rehab/mtbi/
3. AGREE-REX Research Team. (2019). The Appraisal of Guidelines Research & Evaluation—Recommendation EXcellence (AGREE-REX) [Electronic Version]. https://www.agreetrust.org/wp-content/uploads/2021/07/AGREE-REX-Tool-PDF-version.pdf
Disclosures of Interest: None declared.
Key steps in implementation facilitation: Development and recommendations
Authors
Dr. JoAnn Kirchner - VA Behavioral Health QUERI
Dr. Mona Ritchie - VA Behavioral Health QUERI
Dr. Katherine Dollar - VA Center for Integrated Healthcare
Mr. Jeffrey Smith - VA Behavioral Health QUERI
Dr. Eva Woodward - VA Center for Mental Health and Outcomes Research
Background: Implementation Facilitation (IF), a frequently applied, multifaceted, evidence-based implementation strategy, incorporates multiple complementary activities and strategies depending upon the context within which implementation occurs, as well as characteristics of the recipients of the implementation and the innovation itself (Harvey & Kitson, 2015) . IF is complex, difficult to describe and the core components can be challenging to operationally define. In collaboration with IF trainees, practitioners and researchers, we conducted a multi-phase process to develop an overview of the primary IF activities (steps) critical to the process of facilitation.
Methods: First, the lead author drafted the steps and incorporated feedback from other authors. Second, we piloted the steps with participants in an IF training to confirm their usefulness for understanding IF (Kirchner et al., 2022). Third, we recruited an expert panel of 9 IF practitioners, researchers, and regional and national clinical leaders from our IF Learning Collaborative (IFLC), conducted 2 rounds of review, and incorporated the panel’s feedback. Next, we reviewed and made further revisions to the steps, presented them to the full IFLC, and incorporated their additional suggestions. Lastly, the expert panel reviewed and approved the final version of the IF Key Steps document.
Results: The resulting “Key Steps in Implementation Facilitation” are framed within two principles: ensuring the ongoing presence of the facilitator across all phases of implementation and providing a “safe” facilitator environment. The document includes 8 steps with some minimal guidance for their application and links for each step to additional resources provided within a comprehensive, publicly available IF Training Manual (Ritchie et al., 2020).
Conclusion: Incorporating input from a broad set of facilitation interest groups, we identified key steps in the IF process. This document can be used for training facilitators as well as explaining this very complex implementation strategy to those who are novice to the concept of IF, ultimately better defining and generalizing the core features of facilitation.
References
1. Harvey, G., & Kitson, A. (2015). Translating evidence into healthcare policy and practice: Single versus multi-faceted implementation strategies—Is there a simple answer to a complex question? International Journal of Health Policy and Management, 4(3), 123–126. PubMed. https://doi.org/10.15171/ijhpm.2015.54
2. Kirchner J.E., Dollar K.M., Smith J.L., Pitcock J.A., Fletcher T.L., Curtis N.D., Morris K.K., & Toper D.R. (2022) Development and preliminary evaluation of an implementation facilitation training program. Implementation Research and Practice. (in press)
3. Ritchie, M. J., Dollar, K. M., Miller, C. J., Smith, J. L., Oliver, K. A., Kim, B., Connolly, S. L., & Kirchner, J. E. (2020). Using Implementation Facilitation to Improve Care in the Veterans Health Administration (Version 3). Veterans Health Administration, Quality Enhancement Research Initiative (QUERI) for Team-Based Behavioral Health, 2020. Available at: Https://www.queri.research.va.gov/tools/implementation/Facilitation-Manual.pdf.
Disclosures of Interest: JoAnn E. Kirchner is a paid consultant for ViiV Healthcare.
Identifying usability issues for redesign of an evidence-based behavioral intervention for use in schools
Authors
Dr. Jill Locke - University of Washington
Dr. Karen Bearss - University of Washington
Background: Educators endorse disruptive behavior as a considerable concern for autistic students, which is compounded by the lack of adequate resources for behavioral intervention planning in the classroom. The RUBI program is an evidence-based, low-intensity manualized intervention, initially developed for parents of autistic children ages 3-14 and co-occurring disruptive behavior. Utilizing the Discover, Design/Build, Test (DDBT) framework (Lyon et al., 2019), which combines human-centered design and implementation science, usability issues of the original RUBI intervention were identified for redesign to ensure intervention-setting fit when used in the education sector.
Methods: RUBI intervention content was collaboratively and iteratively redesigned with elementary school stakeholders (41 school staff from 28 schools) to ensure the usability, feasibility, acceptability, and appropriateness of the redesigned intervention, RUBI in Educational Settings (RUBIES). As part of the Discover Phase, we conducted contextual evaluations (N = 8) and cognitive walkthroughs (N = 15) with educators to identify usability issues of the original RUBI intervention. As part of the Design/Build Phase, we engaged in collaborative redesign sessions (N = 6) and user testing (N = 12) to generate potential solutions to identified usability issues. In all iterative phases, we used the Intervention Usability Scale (Lyon et al., 2021), Feasibility of Intervention Measure, Acceptability of Intervention Measure, and Intervention Appropriateness Measure (Weiner et al., 2017).
Results: Conventional content analysis was used to code qualitative data and identify usability issues. Two usability issues were identified: time demands for creating visuals and inclusion of culturally responsive text and pictorial examples. Recommendations were provided for modifications to RUBI sessions to address the needs of the school context and end-users to develop RUBIES. RUBIES was deemed usable (78.4), feasible (4.3), acceptable (4.6), and appropriate (4.5) when considered in schools.
Conclusion: Identifying and addressing usability issues may promote greater utility of RUBIES in school contexts, which is currently being tested.
References
1. Lyon, A. R., Munson, S. A., Renn, B. N., Atkins, D. A., Pullmann, M. D., Friedman, E., &
Areán, P. A. (2019). Use of human-centered design to improve implementation of evidence-based psychotherapies in low-resource communities: Protocol for studies applying a framework to assess usability. Journal of Medical Internet Research Protocols, 8(10):e14990. https://dx.doi.org/10.2196/14990
2. Lyon, A.R., Pullman, M.D., Jacobson, J., Osterhage, K., Al Achkar, M., Renn, B.N…Arean, P.A. (2021). Assessing the usability of complex psychosocial interventions: The Intervention Usability Scale. Implementation Research and Practice, 2, 1-9. https://doi.org/10.1177/2633489520987828
2. Weiner, B. J., Lewis, C.C., Stanick, C., Powell, B.J., Dorsey, C.N., Clary, A.S., … & Halko, H. (2017). Psychometric assessment of three newly developed implementation outcome measures. Implementation Science, 12, 108. https://doi.org/10.1186/s13012-017-0635-3
Disclosures of Interest: Karen Bearss reports receiving publication royalties from Oxford University Press, does consulting with Attend Behavior and was an invited speaker for ABAC, Inc.
Implementation mapping to design the integration of adolescent depression services within primary care clinics of Mozambique
Authors
Dr. Kate Lovero - Columbia University
Dr. Palmira dos Santos - Ministry of Health
Background: Globally, psychiatric disorders are the largest contributor to burden of disease in adolescents1. However, most adolescents in low- and middle-income countries (LMIC), do not have access to treatment, and contextually appropriate strategies for delivering evidence-based care are needed to expand services to these areas2,3. Here, we employed Implementation Mapping4 to develop a multilevel strategy for integrating adolescent depression services within primary care clinics of Maputo, Mozambique.
Methods: Implementation planners comprised Mozambican Ministry of Health staff and US-based researchers. We conducted qualitative investigation of implementation determinants with adolescents, caregivers, policymakers, mental health specialists, and primary care providers, guided by the Consolidated Framework for Implementation Research5. In a series of workshops with policymakers, providers, and NGO staff implementing adolescent primary healthcare, we selected, prioritized, and specified implementation strategies based on the Expert Recommendations for Implementing Change6 and Implementation Logic Models7 developed with stakeholders.
Results: We identified determinants to implementation across all levels of the CFIR, 25% of which were unique to LMIC adolescent-specific services. We developed 42 potential strategies to target these determinants, of which 33 were prioritized as important and feasible and subsequently included in the final implementation plan. Though implementation frameworks employed provided important structure for data collection, adaptations were required for application in this context.
Conclusion: To our knowledge, this is one of the first studies to systematically develop a strategy for implementation of adolescent mental health services and, specifically, to apply Implementation Mapping in LMIC. We identified unique implementation determinants and strategies important for adolescent mental health care integration that have not previously been noted for implementation of adult mental health care in LMIC. Findings from this study will inform implementation of integrated adolescent mental health services in Mozambique and may serve as a model for efforts in other LMIC and low-resource settings.
References
1. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science, 4(1), 50.
2. Fernandez, M. E., ten Hoor, G. A., van Lieshout, S., Rodriguez, S. A., Beidas, R. S., Parcel, G., Ruiter, R. A. C., Markham, C. M., & Kok, G. (2019, 2019-June-18). Implementation Mapping: Using Intervention Mapping to Develop Implementation Strategies [Methods]. Frontiers in Public Health, 7(158). https://doi.org/10.3389/fpubh.2019.00158
3. Gore, F. M., Bloem, P. J., Patton, G. C., Ferguson, J., Joseph, V., Coffey, C., Sawyer, S. M., & Mathers, C. D. (2011, Jun 18). Global burden of disease in young people aged 10-24 years: a systematic analysis. Lancet, 377(9783), 2093-2102. https://doi.org/10.1016/S0140-6736(11)60512-6
4. Kieling, C., Baker-Henningham, H., Belfer, M., Conti, G., Ertem, I., Omigbodun, O., Rohde, L. A., Srinath, S., Ulkuer, N., & Rahman, A. (2011, Oct 22). Child and adolescent mental health worldwide: evidence for action. Lancet, 378(9801), 1515-1525. https://doi.org/10.1016/s0140-6736(11)60827-1
5. Patel, V., Kieling, C., Maulik, P. K., & Divan, G. (2013, May). Improving access to care for children with mental disorders: a global perspective. Arch Dis Child, 98(5), 323-327. https://doi.org/10.1136/archdischild-2012-302079
6. Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015). A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10(1), 21.
7. Smith, J. D., Li, D. H., & Rafferty, M. R. (2020, 2020/09/25). The Implementation Research Logic Model: a method for planning, executing, reporting, and synthesizing implementation projects. Implementation Science, 15(1), 84. https://doi.org/10.1186/s13012-020-01041-8
Disclosures of Interest: None declared.
Stay interviews: An innovative tool to inform development, implementation, and evaluation of strategies to address primary care employee retention
Authors
Dr. Tanya Olmos-Ochoa - Department of Veterans Affairs
Dr. Matthew McCoy - Department of Veterans Affairs, HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System
Dr. Shay Cannedy - 1Department of Veterans Affairs, HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System, Los Angeles, CA;; David Geffen School of Medicine at UCLA, Los Angeles, CA
Ms. Kristina Oishi - 1Department of Veterans Affairs, HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System, Los Angeles, CA;; David Geffen School of Medicine at UCLA, Los Angeles, CA
Dr. Alison Hamilton - Department of Veterans Affairs, HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System, Los Angeles, CA; David Geffen School of Medicine at UCLA, Los Angeles, CA
Background: Primary care employee turnover is associated with increased demands on remaining staff, burnout, and poor patient and organizational outcomes. Strategies to address turnover rely predominantly on the perspectives of departing employees, with less known about the employees who remain. Stay interviews—a management tool to assess the organizational and individual-level factors influencing employee retention—are underutilized in implementation science and may have important applications for intervention development, implementation, and evaluation of employee retention strategies. We piloted the use of stay interviews in the Veterans Health Administration (VA) to explore factors contributing to retention of primary care women’s health employees.
Methods: Drawing on the Job Demands-Resources Model, we conducted 24 stay interviews with women’s health employees at sites experiencing employee turnover. Interview guide domains included: 1) training and role responsibilities, 2) work resources and intention to remain in role, 3) work demands and available supports, 4) ability to alter tasks and the work environment; and 5) recommendations for employee retention. We used rapid qualitative analysis methods to generate initial recommendations for development of a women’s health employee retention agenda.
Results: Stay interviews elicited rich data, including employees’ perceptions of the organizational and individual level factors that influenced retention (e.g., support from colleagues, flexible work schedules). Employees found the interviews acceptable and in several cases cathartic—“I am just very thankful to be able to share what I’ve been going through.” Stay interviews were feasible to conduct and elicited key contextual information about employees’ challenges, coping mechanisms, and motivation to remain in their role.
Conclusion: Stay interviews are acceptable to employees and feasible for research teams to conduct. Interview data can inform development, implementation, and evaluation of interventions to support employee retention. Use of stay interviews in implementation science should promote transparency about how employee data will be used.
References
1. Bakker, A.B., Demerouti, E., & Sanz-Vergel, A.I. (2014). Burnout and Work Engagement: The JD-R Approach. Annual Review of Organizational Psychology and Organizational Behavior, 1(1), 389-422. https://doi.org/10.1146/annurev-orgpsych-031413-091235
2. Finnegan RP. (2015). How to conduct stay interviews. SAM Advanced Management Journal, 80(2):49.
3. Waldman, J.D., Kelly, F., Arora, S., & Smith, H.L. (2014). The shocking cost of turnover in health care. Health Care Management Review, 29(1), 2-7. https://doi.org/10.1097/00004010-200401000-00002
Disclosures of Interest: None declared.
Adaptations to coordinated specialty care during the COVID-19 pandemic: Using the Framework for Reporting Adaptations and Modifications for Evidence-Based Interventions (FRAME)
Authors
Dr. Sapana Patel - Columbia University Vagelos College of Physicians and Surgeons
Dr. Elaina Monague - The New York State Psychiatric Institute
Ms. Anna Giannicci - The New York State Psychiatric Institute
Dr. Ana Stefancic - New York State Psychiatric Institute, Columbia University Medical Center
Ms. Shannon Pagdon - The New York State Psychiatric Institute
Dr. Sapna Mendon-Plasek - Division of Behavioral Health Services and Policy Research, Columbia University – New York State Psychiatric Institute, New York, NY
Dr. Leopoldo J. Cabassa - Washington University in St. Louis
Dr. Iruma Bello - New York State Psychiatric Institute, Columbia University Medical Center
Ms. Reanne Rahim - The New York State Psychiatric Institute
Dr. Lisa Dixon - Columbia University Department of Psychiatry and The New York State Psychiatric Institute
Background: Throughout the COVID-19 pandemic, OnTrackNY teams continued to provide coordinated specialty care (CSC) in a radically shifted environment. This presentation will describe adaptations to OnTrack NY’s multidisciplinary, team-based intensive care model implemented during the pandemic.
Methods: OnTrackNY providers were recruited to participate in in-depth, qualitative interviews conducted using phone and video platforms. The project team co-developed the qualitative interview guides with OnTrack Central trainers for each of the six team member roles [e.g., Outreach and Recruitment Coordinator, Peer Specialist, Supported Employment and Education Specialist (SEES), Registered Nurse (RN), Primary Clinician, and Psychiatric Care Provider (PCP)]. A coding team used the FRAME to identify the top three role-based adaptations to the OnTrackNY model from transcripts of the qualitative interviews.
Results: Twenty-three providers (n = 3-4 providers per role) discussed challenges and adaptations of providing CSC services during the pandemic. Use of telehealth was a major adaptation applied across all roles. Adaptations to outreach included narrowing community outreach to inpatient and emergency settings, increasing communication with referral sources, increasing contact with newly referred participants and families. Peer specialist adaptations include conducting physically-distanced groups, discussing current events and expanding online resources for engagement. SEES adaptations included monitoring evolving employment opportunities, conducting mock job interviews remotely, and supporting online learning. Adaptations to PCP/RN roles included sending equipment home for monitoring heath, changing methods and frequency of administering medication, and providing education regarding COVID-19 and vaccination. Adaptations to the Primary Clinician role included increasing informal “check-ins,” using screen sharing to complete assessments and safety plans and addressing increased stress due to the pandemic.
Conclusion: Adaptations to CSC were common with providers most frequently making changes to format and setting of care delivery and content modifications (tailoring content, loosening structure and increasing frequency). Future work will examine if implications of adaptations on OnTrackNY fidelity indicators and care outcomes.
References
1. Patel SR, Bello I, Cabassa LJ, Nossel IR, Wall MM, Montague E, Rahim R, Mathai CM, Dixon LB. Adapting coordinated specialty care in the post-COVID-19 era: study protocol for an integrative mixed-methods study. Implement Sci Commun. 2021 Jul 5;2(1):72. doi: 10.1186/s43058-021-00178-x. PMID: 34225817; PMCID: PMC8256216.
2. Wiltsey-Stirman, S., Baumann, A. A., & Miller, C. J. (2019). The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implementation science : IS, 14(1), 58. https://doi.org/10.1186/s13012-019-0898-y
3. Bello, I., Lee, R., Malinovsky, I., Watkins, L., Nossel, I., Smith, T., Ngo, H., Birnbaum, M., Marino, L., Sederer, L. I., Radigan, M., Gu, G., Essock, S., & Dixon, L. B. (2017). OnTrackNY: The Development of a Coordinated Specialty Care Program for Individuals Experiencing Early Psychosis. Psychiatric services (Washington, D.C.), 68(4), 318–320. https://doi.org/10.1176/appi.ps.201600512
Disclosures of Interest: None declared.
Five principles to guide co-created implementation collaboratives with three applied case study examples
Authors
Dr. Monica Perez-Jolles - University of Colorado School of Medicine
Dr. Cathleen Willging - Center Director, Senior Research Scientist Pacific Institute for Research and Evaluation
Dr. Nicole Stadnick - University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
Dr. Erika Crable - University of California - San Diego
Dr. Rebecca Lengnick-Hall - Brown School, Washington University, St. Louis, MO, United States
Dr. Jemma Hawkins - Cardiff University
Dr. Gregory Aarons - Department of Psychiatry, University of California, San Diego; Child & Adolescent Services Research Center; University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
Background: Implementation science (IS) must meaningfully engage diverse and sometimes disenfranchised stakeholders to improve buy-in, fidelity, outcome relevance, equity and evidence-based practice sustainment (Brownson, 2021, Stanton, 2022). Including diverse voices requires cultural humility and skills for welcoming and integrating multi-stakeholder perspectives and values. Despite calls for greater community engagement, IS lacks guidance on structuring collaborative approaches that promote co-created synergistic implementation processes. We address this gap by identifying co-creation principles that researchers and community partners can use to guide implementation collaborations.
Methods: We conducted a narrative review of organizational, community-engaged, and patient-centered research to identify guiding principles of co-creation processes applicable to IS. Seven researchers, experienced in IS, co-creation, and community engagement, discussed the narrative review and reflected upon experiences facilitating implementation collaborations. We employed consensus decision-making to develop core principles promoting co-creation IS research. The Exploration, Preparation, Implementation, Sustainment framework (Aarons, 2011) and three case studies are used to illustrate each principle.
Results: We describe five guiding principles for co-created implementation collaboratives: 1) Equity in relationship building among stakeholders with end-user knowledge to ensure that perspectives are equally valued and all members have shared responsibility and access to decision-making power throughout the duration of the collaboration, 2) Reflexivity to create spaces for self-reflection on and address power imbalances, unintended consequences, and use of dominant research frameworks, 3) Reciprocity and Mutuality to support knowledge sharing, skill-building among end-users, mutually beneficial relationships, and co-creating project outputs, 4) Transformative and Personalized collaborative processes that facilitate skill-building among community-based stakeholders, 5) Relationships Facilitated through inclusive implementation networks, iterative processes to identify absent stakeholders, and active communication.
Conclusion: Operationalization of co-created implementation collaborations is critical to sparking synergy, and meaningfully addressing differential power, privilege, and access to resources. Implementation researchers should employ and test these principles for their impact on developing meaningful, co-creation collaborations.
References
1. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. C. (2010). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23. https://doi.org/10.1007/s10488-010-0327-7
2. Brownson, R. C., Kumanyika, S. K., Kreuter, M. W., & Haire-Joshu, D. (2021). Implementation science should give higher priority to health equity. Implementation Science, 16(1). https://doi.org/10.1186/s13012-021-01097-0
3. Stanton, M. C., Ali, S. B., & the SUSTAIN Center Team. (2022). A typology of power in implementation: Building on the exploration, preparation, implementation, sustainment (epis) framework to advance mental health and HIV health equity. Implementation Research and Practice, 3, 263348952110642. https://doi.org/10.1177/26334895211064250
Disclosures of Interest: None declared.
Eleven reasons for adaptation of Swedish parenting programs
Authors
Mr. Kristoffer Pettersson - Mälardalen university
Dr. Pernilla Liedgren - Mälardalen university
Dr. Fabrizia Giannotta - Mälardalen university
Prof. Aaron Lyon - University of Washington
Prof. Henna Hasson - Karolinska Institutet
Prof. Ulrica von Thiele Schwarz - Mälardalen university
Background: While questions about adaptation and fidelity are commonly discussed in implementation science (von Thiele Schwarz et al., 2019; Wiltsey Stirman et al., 2019), little research has examined the explicit decision-making processes involved when modifying evidence-based interventions (EBIs). This presentation will include findings from the first of a series of studies(von Thiele Schwarz et al., 2021) that explore the “fidelity-adaptation dilemma” through the lens of normative as well as descriptive decision theories(Gigerenzer & Gaissmaier, 2011; Schoemaker, 1982). The study aim was to explore antecedents to adaptation decisions (i.e., reasons for adaptations) that can arise when leading voluntary parenting support groups in a community setting. Reasons for adaptation among parenting programs have previously not been studied directly.
Methods: Eight service providers from both rural and urban areas of Sweden participated in the study. Seventeen group leaders of five different parenting programs were interviewed individually using a semi-structured interview format. Questions focused on their experience of the “fidelity-adaptation dilemma” and the circumstances that give rise to adaptation decisions. Data were analyzed using thematic analysis.
Results: Eleven reasons for adaptations were identified and organized into four separate areas: characteristics of group leaders(supplementary skills and knowledge, preferred ways of working), characteristics of families (problem complexity, diverse or limited educational experience, non-parenting needs for support, colliding value systems), group incidents(criticism and challenges, excessive questions or discussions), and didactic challenges (lack of focus or engagement, limitations of the material, language differences).
Conclusion: The study shows common factors that trigger adaptation and fidelity decisions in the provision of parenting programs in community settings, even in well-implemented programs. Some of the identified reasons might be difficult to predict, making it challenging for implementation practitioners or researchers to successfully plan for them during early stages of implementation. Knowledge about the range of reasons that occur in this context can be used to inform decision-making during implementation planning, and potentially increase the chances of ultimate sustainment.
References
1. Gigerenzer, G., & Gaissmaier, W. (2011). Heuristic decision making. Annual Review of Psychology, 62, 451-482. https://doi.org/10.1146/annurev-psych-120709-145346
Schoemaker, P. J. H. (1982). The Expected Utility Model: Its Variants, Purposes, Evidence and Limitations. Journal of Economic Literature, 20(2), 529-563.
2. von Thiele Schwarz, U., Aarons, G. A., & Hasson, H. (2019). The Value Equation: Three complementary propositions for reconciling fidelity and adaptation in evidence-based practice implementation. BMC Health Services Research, 19(1), 1-10. https://doi.org/10.1186/s12913-019-4668-y
3. von Thiele Schwarz, U., Lyon, A. R., Pettersson, K., Giannotta, F., Liedgren, P., & Hasson, H. (2021). Understanding the value of adhering to or adapting evidence-based interventions: a study protocol of a discrete choice experiment. Implementation Science Communications, 2(1). https://doi.org/10.1186/s43058-021-00187-w
4. Wiltsey Stirman, S., Baumann, A. A., & Miller, C. J. (2019). The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implementation Science, 14(58), 1-10.
Disclosures of Interest: None declared.
Factors influencing the implementation of mental health recovery into services: Findings from a systematic mixed studies review
Authors
Dr. Myra Piat - McGill University, Department of Psychiatry
Dr. Megan Wainwright - Durham University, UK
Ms. Eleni Sofouli - Douglas Hospital Research Centre
Dr. Brigitte Vachon - University of Montreal
Background: Mental health recovery is increasingly the focus of mental health policy, guidelines, and action plans worldwide1. However, no known systematic review, to date, has been published on how recovery has been implemented into services from an implementation science perspective.
Methods: We conducted a systematic mixed studies review following a convergent qualitative2 synthesis design to address the question: How has mental health recovery been implemented into services for adults, and what factors influence the implementation of recovery-oriented services? We applied the best-fit framework synthesis method using the Consolidated Framework for Implementation Research (CFIR)3. Librarians ran searches in seven databases including Ovid- MEDLINE, Cochrane Library, and Scopus. Two reviewers independently screened studies for inclusion or exclusion using DistillerSR. Qualitative, quantitative, and mixed methods peer-reviewed studies published since 1998 were included if they reported a new effort to transform adult mental health services towards a recovery orientation, and reported findings related to implementation experience, process, or factors. The Mixed Methods Appraisal Tool4 was used to critically appraise all included studies. Data was extracted in NVivo12 to the 38 CFIR constructs. The synthesis included a within-case and a cross-case thematic analysis of data coded to each CFIR construct. Cases were types of recovery-oriented innovations.
Results: Seventy studies met our inclusion criteria5. These were grouped into seven types of recovery-oriented innovations. Common CFIR implementation factors across innovations are: Intervention Characteristics (flexibility, relationship building, lived experience); Inner Setting (traditional biomedical vs. recovery-oriented approach, the importance of organizational and policy commitment to recovery-transformation, staff turnover, lack of resources to support personal recovery goals, information gaps about new roles and procedures, interpersonal relationships), Characteristics of Individuals (variability in knowledge about recovery, characteristics of recovery-oriented service providers); Process (the importance of planning, early and continuous engagement with stakeholders). Very little data from included studies was extracted to the outer setting domain, and therefore, we present only some initial observations and note that further research on outer setting implementation factors is needed.
Conclusion: The CFIR required some adaptation for use as an implementation framework in this review. The common implementation factors reported are an important starting point for stakeholders to consider when implementing recovery-oriented services.
References
1. Slade, M., Leamy, M., Bacon, F., Janosik, M., Le Boutillier, C., Williams, J., & Bird, V. (2012). International differences in understanding recovery: systematic review. Epidemiology and psychiatric sciences, 21(4), 353-364
2. Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health. 2014;35(1):29–45. https://doi.org/10.1146/annurev-publhealth-032013-182440
3. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science, 4, 50. https://doi.org/10.1186/1748-5908-4-50
4. Hong QN, Fàbregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, et al. The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Educ Inf. 2018;34(4):285–91.
5. Piat, M., Wainwright, M., Sofouli, E. et al. Factors influencing the implementation of mental health recovery into services: a systematic mixed studies review. Syst Rev 10, 134 (2021). https://doi.org/10.1186/s13643-021-01646-0
Disclosures of Interest: None declared.
Expect the unexpected: Researcher awareness of the ripple effects of EBT implementation
Authors
Dr. Michael Pullmann - University of Washington
Dr. Catherine Corbin - University of Washington
Mr. Ian Muse - University of Washington
Background: Implementation efforts are primarily focused on the critical outcomes of adoption and delivery of interventions with fidelity. However, any change in complex systems can lead to a cascading array of unplanned, non-linear, or unexpected ripple effects. Ripple effects as outcomes that are caused by EBT implementation efforts and are indirect, unplanned, unanticipated, and/or more salient to stakeholders other than researchers. Ripple effects may have profound impact on the success of implementation. For instance, training on evidence-based practices could lead to increased job satisfaction and therapist retention, or greater burnout and turnover. Researchers often lack awareness and measurement of ripple effects.
Methods: We surveyed 82 participants across multiple roles; including children’s therapists, youth who received mental health treatment, caregivers, policy makers, and researchers. During the first survey, participants rated which of 18 common children’s mental health EBT implementation strategies were relevant for their role, and then brainstormed nominations for hundreds of possible ripple effects. Qualitative analyses coded these into individual ripple effects and ripple effect categories. Later surveys collected ratings of the likelihood of ripple effect for each implementation strategy.
Results: We qualitatively coded 66 unique ripple effects, grouped into 12 categories, positive and negative. A few examples of categories and specific codes include general job-related ripple effects (e.g., satisfaction, burnout), general knowledge, skills, and attitudes towards EBTs (e.g. skills, intervention fatigue), and equity and stigma (e.g. prejudice, equity, stigma around mental health). Researchers were often less likely than participants to endorse the likelihood of certain ripple effects such as caregiver strain, re-traumatization of clients, equity, and stigma.
Conclusion: More research on ripple effects will better expand our knowledge of the cascading impact of implementation strategies and lead to more balanced implementation efforts.
References
1. Lee, D., Kim, Y., & Devine, B. (2022). Spillover effects of mental health disorders on family members’ health-related quality of life: Evidence from a US sample. Medical Decision Making, 41(1), 80–93. https://doi.org/10.1177/0272989X211027146
2. Lewis, C. C., Klasnja, P., Powell, B. J., Lyon, A. R., Tuzzio, L., Jones, S., Walsh-Bailey, C., & Weiner, B. (2018). From classification to causality: Advancing understanding of mechanisms of change in implementation science. Frontiers in Public Health, 6, 136. https://doi.org/10.3389/fpubh.2018.00136
3. Lipsitz, L. A. (2012). Understanding health care as a complex system: The foundation for unintended consequences. JAMA, 308(3), 243–244. PubMed. https://doi.org/10.1001/jama.2012.7551
4. Mannion, R., & Braithwaite, J. (2012). Unintended consequences of performance measurement in healthcare: 20 salutary lessons from the English National Health Service: consequences of performance measurement. Internal Medicine Journal, 42(5), 569–574. https://doi.org/10.1111/j.1445-5994.2012.02766.x
5. Park, A. L., Tsai, K. H., Guan, K., & Chorpita, B. F. (2018). Unintended consequences of evidence-based treatment policy reform: Is implementation the goal or the strategy for higher quality care? Administration and Policy in Mental Health and Mental Health Services Research, 45(4), 649–660. https://doi.org/10.1007/s10488-018-0853-2
6. Powell, B. J., Beidas, R. S., Lewis, C. C., Aarons, G. A., McMillen, J. C., Proctor, E. K., & Mandell, D. S. (2017). Methods to improve the selection and tailoring of implementation strategies. The Journal of Behavioral Health Services & Research, 44(2), 177–194. https://doi.org/10.1007/s11414-015-9475-6
7. Rogers, E. M. (2003). Diffusion of innovations. Free Press.
8. Rogers, E. M. (2004). A prospective and retrospective look at the diffusion model. Journal of Health Communication, 9(sup1), 13–19. https://doi.org/10.1080/10810730490271449
Disclosures of Interest: None declared.
Implementation & evaluation of Lock2Live, a decision aid to support patients at risk of suicide limit access to firearms
Authors
Dr. Julie Richards - Kaiser Permanente Washington Health Research Institute
Dr. Elena Kuo - Kaiser Permanente Washington Health Research Institute
Dr. Ursula Whiteside - NowMattersNow.org
Ms. Lisa Shulman - Kaiser Permanente Washington Health Research Institute
Ms. Angela Garza Mcwethy - Kaiser Permanente Washington
Ms. Rebecca Parrish - Kaiser Permanente Washington
Dr. Gregory Simon - Kaiser Permanente Washington Health Research Institute
Dr. Jennifer Boggs - Kaiser Permanent Colorado Institute for Health Research
Dr. Ali Rowhani-Rabhar - University of Washington School of Public Health
Dr. Christine Stewart - Kaiser Permanente Washington Health Research Institute
Dr. Marian Betz - University of Colorado School of Medicine
Background: Lock To Live (L2L) is a novel web-based decision aid designed to help people at risk of suicide limit access to firearms and other lethal means. The project goal was to use mixed-methods and RE-AIM framework concepts to support implementation and evaluate reach and adoption of L2L in a large regional healthcare system.
Methods: Implementation of L2L augmented existing clinical practices supporting suicide prevention in primary care (PC) and mental health specialty (MH). Training materials (e.g., huddle card, slides, video) were developed to reinforce clinicians using a smart phrase in the electronic health record (EHR) to add the L2L URL and QR code to the lethal means section of a safety planning template. Qualitative interviews were conducted with both purposefully sampled patients, including firearm owners, and clinicians responsible for safety planning, to inform implementation refinement. Descriptive statistical analyses of adoption and reach of L2L included adult patients with at least one PC or MH encounter 1/1/2020-12/31/2021.
Results: Semi-structured interviews with patients (N = 36) elicited 5 themes for facilitating L2L implementation: have open conversations, validate experiences and feelings, share what to expect, walk through L2L, and make L2L accessible and memorable. Clinician interviews (N = 30) indicated most remembered and liked L2L, but few reported using it regularly. Among adults who reported prior month suicidal intent and/or planning, 13% of 2,134 seen in PC and 26% of 2524 seen in MH received L2L during the 2-year evaluation period. Comparing the first six months to the last six months, PC rates increased from 2% to 29%, and MH increased from <1% to 48%.
Conclusion: Qualitative findings informed refinement of L2L implementation, including improvements to clinician-facing training resources and EHR-based clinical decision support tools. These improvements increased adoption of L2L and reach among adults at risk of suicide. Findings will support firearm suicide prevention practices in healthcare nationwide.
References
1. Betz, M. E., Knoepke, C. E., Siry, B., Clement, A., Azrael, D., Ernestus, S., & Matlock, D. D. (2018). ‘Lock to Live': development of a firearm storage decision aid to enhance lethal means counselling and prevent suicide. Inj Prev. https://doi.org/10.1136/injuryprev-2018-042944
2. Creswell, J. W., & Clark, V. L. P. (2017). Designing and conducting mixed methods research. Sage publications.
3. Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health, 89(9), 1322-1327. https://www.ncbi.nlm.nih.gov/pubmed/10474547
Disclosures of Interest: None declared.
Characterising processes and outcomes of tailoring implementation strategies in healthcare: A protocol for a scoping review
Authors
Dr. Fiona Riordan - School of Public Health, University College Cork
Dr. Bianca Albers - University of Zurich
Prof. Lauren Clack - University of Zurich
Ms. Nickola Pallin - School of Public Health, University College Cork, Cork
Dr. Geoffrey Curran - University of Arkansas for Medical Sciences
Dr. Cara Lewis - Kaiser Permanente Washington Health Research Institute
Dr. Byron Powell - Brown School, Washington University in St. Louis
Dr. Justin Presseau - Ottawa Hospital Research Institute, Ottawa, Canada
Prof. Luke Wolfenden - School of Medicine and Public Health, College of Health, Medicine, and Wellbeing, the University of Newcastle, Australia
Dr. Sheena McHugh - School of Public Health, University College Cork
Background: Tailoring strategies to target the salient barriers to and enablers of implementation is considered a critical step in supporting successful delivery of evidence based interventions in healthcare. Theory, evidence, and stakeholder engagement are considered key ingredients in the process however, these ingredients can be combined in different ways(Powell et al., 2017; Wensing & Grol, 2019). There is no consensus on the definition of tailoring or single method for tailoring strategies to optimize impact, ensure transparency, and facilitate replication. The purpose of this scoping review is to describe how tailoring has been undertaken within healthcare, to answer questions about how it has been conceptualised, described, and conducted in practice, and to identify research gaps.
Methods: The review is being conducted in accordance with best practice guidelines(Danielle Levac, 2010) and the Preferred Reporting Items for Systematic Reviews and Meta-analysis extension for scoping reviews (PRISMA-ScR) will be used to guide the reporting. Searches have been conducted of MEDLINE, Embase, Web of Science, Scopus, from 2005 to present. Reference lists of included articles will be searched. Grey literature will be searched on Google Scholar. Screening and data extraction is being conducted by four members of the research team. Any discrepancies will be resolved through discussion. Initial analysis will be quantitative involving a descriptive numerical summary of the characteristics of the studies and the tailoring process. Qualitative content analysis aligned to the research questions will also be conducted, and data managed using NVivo where applicable. This scoping review is pre-registered with the Open Science Framework.
Results: In total, 5936 articles have been identified through database searches. Pilot screening of title and abstracts is underway.
Conclusion: The findings will serve as a resource for implementation researchers and practitioners to guide future research in this field and facilitate systematic, transparent, and replicable development of tailored implementation strategies.
References
1. Danielle Levac, H. C. (2010). Scoping studies: advancing the methodology. Implementation Science, 1–18. https://doi.org/10.1017/cbo9780511814563.003
2. Powell, B. J., Beidas, R. S., Lewis, C. C., Aarons, G. A., McMillen, J. C., Proctor, E. K., & Mandell, D. S. (2017). Methods to Improve the Selection and Tailoring of Implementation Strategies. Journal of Behavioral Health Services and Research, 44(2), 177–194. https://doi.org/10.1007/s11414-015-9475-6
3. Wensing, M., & Grol, R. (2019). Knowledge translation in health: How implementation science could contribute more. BMC Medicine, 17(1), 1–6. https://doi.org/10.1186/s12916-019-1322-9
Disclosures of Interest: None declared.
Implementation strategies used and reported in brief suicide prevention intervention studies
Authors
Dr. Brittany Rudd - University of Illinois at Chicago
Dr. Molly Davis - Children's Hospital of Philadelphia
Dr. Stephanie Doupnik - Children's Hospital of Philadelphia
Ms. Catalina Ordorica - University of Illinois at Chicago
Dr. Steven Marcus - University of Pennsylvania
Dr. Rinad Beidas - University of Pennsylvania
Background: A systematic review and meta-analysis of 14 brief suicide prevention interventions (BSPs) delivered in healthcare settings demonstrated their efficacy and effectiveness at reducing suicide attempts and increasing treatment initiation (Doupnik et al., 2020). Implementation strategies (Proctor et al., 2013) were used in each study to embed the intervention in the healthcare setting for the evaluation. However, due to journals’ space constraints or implementation science terminology being unfamiliar to those outside the field, published manuscripts of efficacy/effectiveness studies may not fully document implementation strategies used. While implementation strategies used to support an intervention during efficacy/effectiveness studies may differ from those needed in real-world routine care, documenting the former provides an opportunity to inform the latter.
Methods: Two authors independently reviewed each publication from the systematic review to abstract reported strategies using the Pragmatic Implementation Strategy Reporting Tool (Rudd et al., 2020) and came to consensus. We emailed the Reporting Tool to corresponding authors asking them to endorse strategies used in their study (90% response rate). We compared the number of implementation strategies reported in publications to those endorsed by authors, in total and by each implementation strategy category using two-sided, paired-samples Wilcoxon tests. Analyses were conducted in R version 4.1.1.
Results: On average, authors endorsed using 26 implementation strategies when testing the efficacy/effectiveness of BSPs in healthcare settings – a large and statistically significant difference from the average of 4 described in publications (mean difference = -21.83, p = .017). Similarly large differences were observed for each implementation strategy category. Training and educating stakeholders were the most frequently reported implementation strategies in publications even though developing stakeholder relationships were most used.
Conclusion: For the advancement of the field, we recommend that journals publishing intervention research consider requiring authors to report implementation strategies used. We urge collaboration between implementation researchers and clinical researchers to ensure interventions are designed and tested with an eye toward accelerating research-to-practice implementation.
References
1. Doupnik SK, Rudd B, Schmutte T, et al. Association of suicide prevention interventions with subsequent suicide attempts, linkage to follow-up care, and depression symptoms for acute care settings: A systematic review and meta-analysis. JAMA Psychiatry. 2020;77(10):1021-1030. doi:10.1001/jamapsychiatry.2020.1586
2. Proctor E, Powell BJ, McMillen JC. Implementation strategies: Recommendations for specifying and reporting. Implementation Sci. 2013;8(1):139. doi:10.1186/1748-5908-8-139
3. Rudd BN, Davis M, Beidas RS. Integrating implementation science in clinical research to maximize public health impact: A call for the reporting and alignment of implementation strategy use with implementation outcomes in clinical research. Implement Sci. 2020;15(1). doi:10.1186/s13012-020-01060-5
Disclosures of Interest: None declared.
Examining adaptations to a community-based peer-recovery program for substance use disorder amid COVID-19
Authors
Dr. Mechelle Sanders - University of Rochester
Mrs. Teraisa Mullaney - University of Rochester
Dr. Holly Ann Russell - University of Rochester
Ms. Kara Izzo - Rocovery Fitness
Mr. Brian Smith - University of Rochester
Background: During the COVID-19 pandemic, most peer recovery programs had to end face-to-face services and move to an online and or phone-based format. It is not clear what impact if any this may have had on individual client referrals, contact and services rendered. The goal of this study was to explore the impact of the COVID-19 pandemic on a community-based peer-recovery program.
Methods: We employed a before-after cross-sectional study design, dividing the individuals referred to the program client-sample into 2 groups; those referred before the switch to exclusively tele-remote services and those referred after. We then characterized modifications and adaptations to the program using the Framework for Reporting Adaptions and Modifications, and examined the characteristics of the referrals and contact engagements between individual clients and peers before and after the switch to exclusively tele-remote services.
Results: Overall there were a total of n = 5 recovery-peers with n = 134 individuals clients referred to the program. Thirteen adaptions were made to the program within 60-days of the switch in order to continue service delivery. There were no statistically significant changes in the number of days between referral and contact or the mean time the peers spent with individuals clients. There was a statistically significant association between referral setting and time period, p < 0.001.
Conclusion: A community-based peer-recovery program was agile and responded to the rapidly changing demands of the COVID-19 pandemic. There were no substantial changes in days between contact communication with individuals clients. The lessons learned can serve as a model for programs that may need to do the same in the future.
References
1. Rogers AH, Shepherd JM, Garey L, Zvolensky MJ. Psychological factors associated with substance use initiation during the COVID-19 pandemic. Psychiatry research. 2020;293:113407.
2. Volkow ND. Collision of the COVID-19 and addiction epidemics. In: American College of Physicians; 2020.
3. DeJong CA, Verhagen JGD, Pols R, Verbrugge CA, Baldacchino A. Psychological impact of the acute COVID-19 period on patients with substance use disorders: we are all in this together. Basic and clinical neuroscience. 2020;11(2):207.
4. Abuse S. Mental Health Services Administration. Key substance use and mental health indicators in the United States: Results from the 2016 National Survey on Drug Use and Health (HHS Publication No. SMA 17-5044, NSDUH Series H-52). Rockville, MD: Center for Behavioral Health Statistics and Quality. Substance Abuse and Mental Health Services Administration. 2017.
5. Czeisler MÉ, Lane RI, Petrosky E, et al. Mental health, substance use, and suicidal ideation during the COVID-19 pandemic—United States, June 24–30, 2020. Morbidity and Mortality Weekly Report. 2020;69(32):1049.
6. Eddie D, Hoffman L, Vilsaint C, et al. Lived experience in new models of care for substance use disorder: A systematic review of peer recovery support services and recovery coaching. Frontiers in psychology. 2019;10:1052.
7. Boisvert RA, Martin LM, Grosek M, Clarie AJ. Effectiveness of a peer-support community in addiction recovery: participation as intervention. Occupational therapy international. 2008;15(4):205-220.
8. Lin G, Zhang T, Zhang Y, Wang Q. Statewide Stay-at-Home Directives on the Spread of COVID-19 in Metropolitan and Nonmetropolitan Counties in the United States. The Journal of Rural Health. 2020.
9. Baillargeon J, Polychronopoulou E, Kuo Y-F, Raji MA. The Impact of Substance Use Disorder on COVID-19 Outcomes. Psychiatric Services. 2020:appi. ps. 202000534.
10. Wang QQ, Kaelber DC, Xu R, Volkow ND. COVID-19 risk and outcomes in patients with substance use disorders: analyses from electronic health records in the United States. Molecular psychiatry. 2021;26(1):30-39.
11. Adams WE, Rogers ES, Edwards JP, Lord EM, McKnight L, Barbone M. Impact of COVID-19 on Peer Support Specialists in the United States: Findings From a Cross-Sectional Online Survey. Psychiatric Services. 2021:appi. ps. 202000915.
12. Schmidt LA, Rieckmann T, Abraham A, et al. Advancing recovery: Implementing evidence-based treatment for substance use disorders at the systems level. Journal of Studies on Alcohol and Drugs. 2012;73(3):413-422.
13. Satinsky EN, Doran K, Felton JW, Kleinman M, Dean D, Magidson JF. Adapting a peer recovery coach-delivered behavioral activation intervention for problematic substance use in a medically underserved community in Baltimore City. PloS one. 2020;15(1):e0228084.
14. Bergman BG, Kelly JF14. Bergman BG, Kelly JF. Online digital recovery support services: An overview of the science and their potential to help individuals with substance use disorder during COVID-19 and beyond. Journal of substance abuse treatment. 2020:108152.
15. Bergman BG, Kelly JF15. Bergman BG, Kelly JF, Fava M, Evins AE. Online recovery support meetings can help mitigate the public health consequences of COVID-19 for individuals with substance use disorder. In: Elsevier; 2020.
16. Inc. RF. https://www.rocoveryfitness.org/about.html. Published n.d. Accessed 5/3/2021, 2021.
17. Stirman SW, Baumann AA, Miller CJ. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implementation Science. 2019;14(1):1-10.
18. Prochaska JO, Redding CA, Evers KE. The transtheoretical model and stages of change. Health behavior: Theory, research, and practice. 2015;97.
19. Sutton S. Back to the drawing board? A review of applications of the transtheoretical model to substance use. Addiction. 2001;96(1):175-186.
20. Uscher-Pines L, Sousa J, Raja P, Mehrotra A, Barnett M, Huskamp HA. Treatment of opioid use disorder during COVID-19: Experiences of clinicians transitioning to telemedicine. Journal of substance abuse treatment. 2020;118:108124.
21. Sisak MR, Bleiberg J, Dazio S. Police Shift Priorities as Novel Coronavirus Crisis Grows. PBS News Hour. 2020.
22. Clay JM, Parker MO. Alcohol use and misuse during the COVID-19 pandemic: a potential public health crisis? The Lancet Public Health. 2020;5(5):e259.
23. McKnight-Eily LR, Okoro CA, Strine TW, et al. Racial and ethnic disparities in the prevalence of stress and worry, mental health conditions, and increased substance use among adults during the COVID-19 pandemic—United States, April and May 2020. Morbidity and Mortality Weekly Report. 2021;70(5):162.
24. Guerrero EG, Marsh JC, Khachikian T, Amaro H, Vega WA. Disparities in Latino substance use, service use, and treatment: Implications for culturally and evidence-based interventions under health care reform. Drug and alcohol dependence. 2013;133(3):805-813.
25. Mennis J, Stahler GJ. Racial and ethnic disparities in outpatient substance use disorder treatment episode completion for different substances. Journal of substance abuse treatment. 2016;63:25-33.
26. Marotta PL, Tolou-Shams M, Cunningham-Williams RM, Washington Sr DM, Voisin D. Racial and ethnic disparities, referral source and attrition from outpatient substance use disorder treatment among adolescents in the United States. Youth & Society. 2020:0044118X20960635.
Disclosures of Interest: None declared.
Supporting cancer screening and prevention at hospital systems through the development of communities of practices teams and a multipronged capacity building approach
Authors
Mrs. Sarah Shafir - The American Cancer Society
Ms. Jessie Sanders - The American Cancer Society
Mrs. Dani Schenk - The American Cancer Society
Dr. Donoria Evans - The American Cancer Society
Background: Cancer is the second most common cause of death in the United States, exceeded only by heart disease (Centers for Disease Control and Prevention, 2022). The Hospital System Capacity Building (HSCB) Initiative aims to decrease cancer morbidity by increasing cancer screening and prevention efforts across the United States. The initiative engages representatives from hospital systems, local and/or state health departments, the author’s institution, and selected community organizations to form Communities of Practice (COP) teams (Wenger, McDermott, & Snyder, 2002). The COP approach utilizes multi-sector partnerships and the implementation of Evidence based Interventions (EBIs) to incorporate cancer screening and prevention into each hospital systems’ mission, priority setting, quality standards and investment practices. The initiative supports coordination and collaboration across partners, improves access to care, and leads to comprehensive approaches to cancer screening and prevention. Since the project began in 2019 the author’s institution has recruited and provided capacity building assistance (CBA) to 21 COP teams across the country located in each Health and Human Service Region.
Methods: COP work is grounded by collaborative action plans (CAP) updated annually by COP teams and supported and reviewed by national office staff and subject matter experts (SMEs). CAPs include the project aim, EBIs and activities, and how partners will collaborate to implement work. The work of each COP team is supported by a multipronged CBA approach provided through our national office, regional staff that sit on each COP team, and SMEs. CBA is conducted through quarterly calls with SMEs, monthly calls with national staff, resources and materials to support EBI implementation, webinars and other didactic learning, and peer to peer learning including project ECHO. Project evaluation includes tracking and monitoring EBI implementation, measuring system-level cancer screening and vaccination rates, and other key partnership related outcomes (Mattessich, Murray-Close, & Monsey, B, 2001; National Collaborating Centre for Methods and Tools, 2008; Partnership Assessment Tool for Health, n.d.). Screening and evaluation data are collected from each COP team bi-annually.
Results: Seventeen sites implemented 148 activities in support of 61 EBIs in 2021. Sixty-seven percent of the 15 sites for whom we have data saw increases in their cancer screening or HPV vaccination rates in 2021. This is especially significant given the ongoing impact of COVID-19 on cancer prevention and screening rates (Bakouny et al., 2021; Mast, Deckert, & Muñoz del Río, 2022).
Conclusion: 2021 screening rates and evaluation data demonstrate the success of the COP approach in supporting hospital systems in incorporating cancer prevention and screening interventions. The partnerships developed and EBIs implemented during the year supported most systems in increasing rates despite COVID-19. Results indicate that the COP approach and CBA model used could be 1) adopted by other hospital systems across the country to support cancer screening and prevention efforts and 2) applied across different systems and topic areas to support the development of multi-sector partnerships and implement EBIs.
References
1. Bakouny, Z., Paciotti, M., Schmidt, A., Lipsitz, S., Choueiri, T., & Trinh, Q. (2021). Cancer screening tests and cancer diagnoses during the COVID-19 pandemic. JAMA Oncol. 7(3), 458–460. doi:10.1001/jamaoncol.2020.7600
2. Centers for Disease Control and Prevention. (2022, February 28). An update on cancer deaths in the United States. An Update on Cancer Deaths in the United States | CDC
3. Mast, C., Deckert, J., & Muñoz del Río, A. (2022, January 18). Troubling cancer screening rates still seen nearly two years into the pandemic. Epic Research. Troubling Cancer Screening Rates Still Seen Nearly Two Years Into the Pandemic (epicresearch.org)
4. Mattessich, P., Murray-Close, M., & Monsey, B. (2001). Wilder collaboration factors inventory. St. Paul, MN: Wilder Research. https://www.wilder.org/Wilder-Research/Research-Services/Pages/Wilder-Collaboration-Factors-Inventory.aspx
5. National Collaborating Centre for Methods and Tools (2008). Partnership evaluation: The partnership self-assessment tool. Hamilton, ON: McMaster University. https://www.nccmt.ca/resources/search/1
6. Partnership Assessment Tool for Health (n.d). Partnership for healthy outcomes. Partnership-Assessment-Tool-for-Health_-FINAL.pdf (chcs.org)
7. Wenger, E., McDermott, R., & Snyder, W. (2002, March 25). Principles for cultivating communities of practices. Harvard Business School. https://hbswk.hbs.edu/archive/cultivating-communities-of-practice-a-guide-to-managing-knowledge-seven-principles-for-cultivating-communities-of-practice
Disclosures of Interest: None declared.
Moderators of implementation strategy effectiveness to support CBT delivery at Michigan high schools: Results from the Adaptive School-based Implementation of CBT (ASIC) Clustered-SMART
Authors
Dr. Shawna Smith - University of Michigan
Dr. Daniel Almirall - University of Michigan
Dr. Seo Youn Choi - University of Michigan
Dr. Elizabeth Koschmann - TRAILS at Tides Center
Ms. Amy Rusch - University of Michigan
Dr. Emily Bilek - University of Michigan
Ms. Annalise Lane - University of Michigan
Dr. James Abelson - University of Michigan
Dr. Daniel Eisenberg - UCLA Fielding School of Public Health
Dr. Joseph Himle - University of Michigan
Dr. Kate Fitzgerald - Columbia University Department of Psychiatry
Ms. Celeste Liebrecht - University of Michigan
Dr. Amy Kilbourne - University of Michigan
Background: Youth metal health services are increasingly provided in schools. However, implementation of mental health evidence-based practices (EBPs) in schools remains deficient,1 in part due to heterogeneous barriers to school-based delivery of mental health EBPs.2 While implementation strategies hold promise for addressing a wide range of barriers, it is important to understand which implementation strategies work best to address which barriers in school settings.3
Methods: The Adaptive School-based Implementation of CBT (ASIC) study recruited 94 Michigan high schools to examine the effects of different implementation strategies (including some adaptive strategies) on school professional (SP) delivery of cognitive behavioral therapy (CBT). Initially, all schools were randomized to receive skills-based Coaching or not. After 8 weeks, schools deemed “slower-responders” were re-randomized to either receive Facilitation or continue with the previous strategy. The primary outcome was the number of SP-delivered CBT sessions over the course of the study. Longitudinal multilevel models examined whether the effects of Coaching or (among slower-responders) Facilitation on CBT delivery were moderated by school-level variables, including SPs’ formal training in CBT, pre-randomization CBT delivery, school EBP support, and/or barriers to CBT delivery.
Results: Overall, SPs who received Coaching reported less CBT delivery (b = -.40 [90% CI = -.62,-.17]); among slower-responders, those who received Facilitation reported more CBT delivery (b = .54 [CI = .23,.85]). Coaching improved delivery more in schools where fewer SPs had formal CBT training (b = -.92 [CI = -1.57,-.28]), and where SPs did not deliver CBT pre-randomization (b = -.94 [CI = -1.37,-.50]). Among slower-responders, Facilitation improved delivery more in schools where SPs reported on average 2 + barriers (b = 1.10 [CI = -.89,3.10]), where baseline EBP support was higher (b = .11 [CI = -.25,.46]), and where school administrator support was lower (b = -.36 [CI = -.69,-.03]).
Conclusion: Moderator analyses can inform how to target and tailor implementation strategy provision. Here, Coaching worked best when SP experience with CBT was minimal; Facilitation worked best when clear organizational barriers were apparent.
References
1. Owens, J. S., Lyon, A. R., Brandt, N. E., Warner, C. M., Nadeem, E., Spiel, C., & Wagner, M. (2014). Implementation Science in School Mental Health: Key Constructs in a Developing Research Agenda. School Mental Health, 6(2), 99–111. https://doi.org/10.1007/s12310-013-9115-3
2. Whitaker, K., Fortier, A., Bruns, E. J., Nicodimos, S., Ludwig, K., Lyon, A. R., Pullmann, M. D., Short, K., & McCauley, E. (2018). How Do School Mental Health Services Vary Across Contexts? Lessons Learned from Two Efforts to Implement a Research-Based Strategy. School Mental Health, 10(2), 134–146. https://doi.org/10.1007/s12310-017-9243-2
3. Kilbourne, A. M., Smith, S. N., Choi, S. Y., Koschmann, E., Liebrecht, C., Rusch, A., Abelson, J. L., Eisenberg, D., Himle, J. A., Fitzgerald, K., & Almirall, D. (2018). Adaptive School-based Implementation of CBT (ASIC): Clustered-SMART for building an optimized adaptive implementation intervention to improve uptake of mental health interventions in schools. Implementation Science, 13(1), 119. https://doi.org/10.1186/s13012-018-0808-8
Disclosures of Interest: None declared.
Walk the Talk online toolkit: A knowledge mobilization tool that builds organizational capacity in implementation science and mental health recovery
Authors
Ms. Eleni Sofouli - Douglas Hospital Research Centre
Dr. Myra Piat - McGill University, Department of Psychiatry
Dr. Megan Wainwright - Durham University, UK
Background: Transforming mental health services to recovery-orientation is a policy priority for numerous countries worldwide1. To accelerate the implementation of mental health recovery guidelines in Canada, our team developed an innovative implementation strategy: 3-phase facilitated planning process grounded in implementation science2. The Consolidated Framework for Implementation Research (CFIR)3, the CFIR-ERIC Matching tool4 and the Implementation Science Research Development Tool (ImpRes)5 were integrated into the process. After successfully rolling it out into seven mental health organisations, materials were converted into an online toolkit- walkthetalktoolkit.ca
Methods: The research team worked with a social enterprise specialising in website and digital content development to create the online toolkit/website. The website is designed to accompany and support implementation facilitators through a 3-step process: 1) Establishing an implementation team, 2) Conducting the 12-meeting planning process and 3) Ongoing implementation coaching. Each phase has clear objectives and deliverables. The website includes resources in lay language where all of the facilitator materials: handouts for teams, videos synthesizing guideline content and implementation science research can be viewed and downloaded. By the end of the process, an organization is expected to have implemented a recovery-oriented innovation based on the needs of its clientele, and its local context.
Results: A bilingual (EN-FR) website was launched in February 2022. Two large community-based organisations that offer a wide-range of services to people with mental health challenges are in the process of implementing the online toolkit. Each organization assigned an internal facilitator (staff member) who is receiving coaching from toolkit creators. So far, 450 users across the globe have accessed the website.
Conclusion: Walk the Talk is a knowledge mobilization tool for building capacity in implementation science and mental health recovery in non-academic implementation facilitators at the organisational level.
References
1. Slade, M., Leamy, M., Bacon, F., Janosik, M., Le Boutillier, C., Williams, J., & Bird, V. (2012). International differences in understanding recovery: systematic review. Epidemiology and psychiatric sciences, 21(4), 353-364.
2. Piat, M., Sofouli, E., Wainwright, M., Albert, H., Rivest, M. P., Casey, R., … & Kasdorf, S. (2022). Translating mental health recovery guidelines into recovery-oriented innovations: A strategy combining implementation teams and a facilitated planning process. Evaluation and Program Planning, 91, 102054.
3. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science, 4, 50. https://doi.org/10.1186/1748-5908-4-50
4. Waltz, T. J., Powell, B. J., Fernandez, M. E., Abadie, B., & Damschroder, L. J. (2019). Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions. Implementation Science, 14(1), 42.
5. Hull, L., Goulding, L., Khadjesari, Z., Davis, R., Healey, A., Bakolis, I., & Sevdalis, N. (2019). Designing high-quality implementation research: development, application, feasibility and preliminary evaluation of the implementation science research development (ImpRes) tool and guide. Implementation Science, 14(1), 80.
Disclosures of Interest: None declared.
Best practices for equity-centered exploration using listen-in sessions
Authors
Dr. Megan Stanton - Eastern Connecticut State University
Dr. Samira Ali - University of Houston
Ms. Katie McCormick - University of Texas at Austin
Mrs. Neena Smith-Bankhead - Emory University School of Public Health
Ms. Althea Hart - The Joyce 432 Organization
Background: Stanton et al. (2022) raise critical questions regarding meaningful community engagement throughout implementation, as articulated in the EPIS Model (Aarons, 2011). They argue that power is generated in the exploration phase through establishing the narrative of the social problem (discursive power), evaluating whose knowledge counts as evidence (epistemic power) and assessing equity in implementation-related needs and investments (material power). This presentation will discuss best practices emerging from the exploration phase of a large scale, multi-stakeholder initiative designed to build systems-level HIV service capacity in the U.S. South.
Methods: A multi-disciplinary team conducted equity-centered listen-in sessions (LIS) to define initiative scope, refine intervention methods and gather information on implementation feasibility, accessibility, and appropriateness, as well as to understand inner, outer, bridging and innovation factors impacting potential implementation sites. Best practices in equity-centered LIS methodology were identified through several rounds of reflexive discussion with the LIS team.
Results: Four best practices for equity-centered, exploratory LIS were identified: 1) people living with HIV were involved in all aspects of the LIS process, including methodological decision making; 2) the team drew on diverse sources of data- accessing a range of epistemic positionalities- including epidemiological mapping data, multi-sectoral participant observation, focus groups and interviews; 3) the team developed transparent decision making methods for LIS site selection to center marginalized voices and prioritize communities disproportionately impacted by the HIV epidemic; and 4) the team developed community accountability structures to act on LIS data, track its integration into initiative implementation and collect ongoing data.
Conclusion: The pursuit of health equity through implementation is a critical goal (Baumann & Cabassa, 2020) which, without operationalization, may remain unmet. Methods for equity-centered exploration presented here apply to a range of contexts and provide a starting point for implementation stakeholders.
References
1. Aarons, G. A., Hurlburt, M., Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38, 4–23. https://doi.org/10.1007/s10488-010-0327-7
2. Baumann, A. A., Cabassa, L. J. (2020). Reframing implementation science to address health inequities in healthcare delivery. BMC Health Services Research, 20. https://doi.org/10.1186/s12913-020-4975
3. Stanton, M. C., Ali, S. B., & SUSTAIN, the. (2022). A typology of power in implementation: Building on the exploration, preparation, implementation, sustainment (EPIS) framework to advance mental health and HIV health equity. Implementation Research and Practice. https://doi.org/10.1177/26334895211064250
Disclosures of Interest: None declared.
Fidelity, Attitudes, and Influence Typologies (FAIT): A tool for implementation facilitation
Authors
Dr. Taren Swindle - University of Arkansas for Medical Sciences
Dr. Julie Rutledge - Education and Research in Children’s Health Center, College of Applied and Natural Sciences, Louisiana Tech University
Ms. Janna Martin - University of Arkansas for Medical Sciences
Dr. Geoffrey Curran - University of Arkansas for Medical Sciences
Background: Implementation facilitation has been described as a “black box” with over 72 distinct roles and 22 complex skills.1,2 Tools to inform implementation facilitation can offer tangible approaches for demystifying the process. The current study sought to: (1) describe a novel classification approach for types of implementer (i.e., end user) behavior and (2) illustrate application of the approach to inform implementation facilitation.
Methods: A small-scale, cluster randomized Hybrid Type III implementation trial was conducted in 38 early care and education classrooms to implement a nutrition promotion intervention. External facilitators (N = 3) and a data collection supervisor used a modified Rapid Assessment Procedure Informed Clinical Ethnography3 to complete immersive observations and thematic content analyses of interviews to identify characteristics of teachers’ behavior at varying levels of implementation fidelity. Expanding on the Diffusions of Innovation,4 three key factors – attitudes toward the innovation, fidelity/adaptations, and influence – were identified that the research team used to develop the novel classification approach, Fidelity, Attitudes, and Influence Typologies (FAIT).
Results: FAIT includes 8 typologies delineated by combinations of characteristics: (1) achieving fidelity targets or not (i.e., Adopting vs. Non-Adopting), (2) valence of attitudes toward the innovation (i.e., Supporter vs. Resister), and (3) influence in the context (i.e., Active vs. Passive). For example, an implementer with poor fidelity but a positive attitude toward the innovation that she shares with her peers would be a Non-Adopting Active Supporter. Unique facilitation responses are conceptualized for each typology. Teachers without fidelity and a negative attitude (i.e., non-adopting resisters) are a prime target for facilitation with the goal of supporting a shift to a more positive attitude before attending to fidelity.
Conclusion: FAIT provides a guide to target the efforts of implementation facilitation. Future work could examine the hypothesis that application of FAIT leads to improved tailoring and outcomes of implementation facilitation.
References
1. Sullivan, G., Blevins, D., & Kauth, M. R. (2008). Translating clinical training into practice in complex mental health systems: Toward opening the ‘Black Box’ of implementation. Implementation Science, 3(1), 1-7.
2. Ritchie, M. J., Parker, L. E., & Kirchner, J. E. (2020). From novice to expert: a qualitative study of implementation facilitation skills. Implementation science communications, 1(1), 1-12.
3. Palinkas, L. A., & Zatzick, D. (2019). Rapid assessment procedure informed clinical ethnography (RAPICE) in pragmatic clinical trials of mental health services implementation: methods and applied case study. Administration and Policy in Mental Health and Mental Health Services Research, 46(2), 255-270.
4. Dearing, J. W., & Cox, J. G. (2018). Diffusion of innovations theory, principles, and practice. Health affairs, 37(2), 183-190.
Disclosures of Interest: Dr. Taren Swindle and UAMS have a financial interest in the technology (WISE) discussed in this presentation/publication. These financial interests have been reviewed and approved in accordance with the UAMS conflict of interest policies.
Mixed methods, iterative, stakeholder-engaged adaptation of collaborative decision skills training for veterans with psychosis: A progress report
Authors
Dr. Emily Treichler - VA San Diego MIRECC/University of California, San Diego
Mr. Elijah Sosa - VA San Diego MIRECC/University of California, San Diego
Ms. Kasey Yu - UC San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center, University of California San Diego, La Jolla, CA
Ms. Jillian Abasta - VA San Diego MIRECC/University of California, San Diego
Ms. Sydney Seaton - VA San Diego MIRECC/University of California, San Diego
Ms. Elissa Gomez - VA San Diego MIRECC/University of California, San Diego
Ms. Joanna Jain - VA San Diego MIRECC/University of California, San Diego
Dr. Borsika Rabin - University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center &
Background: Adapting interventions for local contexts and populations is critical. Continuous adaptation throughout the early implementation period can increase effectiveness and likelihood of maintenance. Collaborative Decision Skills Training (CDST) aims to empower people with mental illness during treatment decision-making. CDST is undergoing adaptation for Veterans with psychosis participating in VA services. We present a mixed methods, iterative, and stakeholder-engaged adaptation, and provide a progress report on our current adaptation of CDST following two iterations.
Methods: An Adaptation Resource Team (ART) including five Veterans with psychosis, two VA clinicians, and four researchers was formed. The ART participated in an iterative process including qualitative interviews, group meetings, and surveys to develop and integrate adaptations. CDST was then implemented among nine Veterans with psychosis. Veterans and clinicians who participated in the implementation provided mixed methods feedback. The ART added new members, including one VA clinician and four Veterans with psychosis who participated in the CDST implementation. A similar iterative adaptation process was completed. The expanded framework for reporting adaptations and modifications to evidence-based treatments (FRAME) was used to specify adaptations.
Results: Across both rounds of adaptations, response to CDST was broadly positive. The first round led to 164 unique adaptations, most of which were small additions or expansion of existing components, and intended to improve CDST’s effectiveness. Implementation of CDST was feasible and acceptable. The second round of adaptations led to a further 35 unique adaptations. These adaptations were again mostly small additions or expansions intended to increase CDST’s effectiveness or implementation feasibility.
Conclusion: Mixed methods, iterative, and stakeholder engaged adaptation facilitated CDST’s feasibility and acceptability. Pre- and post-implementation adaptations allowed for increased optimization. Adaptations will continue for the duration of the associated clinical trial with the aim to fully optimize CDST and identify potential delivery strategies for similar VA settings.
References
1. Chambers, D. A., & Norton, W. E. (2016). The Adaptome: Advancing the Science of Intervention Adaptation. American Journal of Preventive Medicine, 51(4, Supplement 2), S124–S131. https://doi.org/10.1016/j.amepre.2016.05.011
2. Rabin, B. A., McCreight, M., Battaglia, C., Ayele, R., Burke, R. E., Hess, P. L., Frank, J. W., & Glasgow, R. E. (2018). Systematic, Multimethod Assessment of Adaptations Across Four Diverse Health Systems Interventions. Frontiers in Public Health, 6. https://doi.org/10.3389/fpubh.2018.00102
3. Treichler, E. B. H., Rabin, B. A., Spaulding, W. D., Thomas, M. L., Salyers, M. P., Granholm, E. L., Cohen, A. N., & Light, G. A. (2021). Skills-based intervention to enhance collaborative decision-making: Systematic adaptation and open trial protocol for veterans with psychosis. Pilot and Feasibility Studies, 7(1), 89. https://doi.org/10.1186/s40814-021-00820-4
4. Wiltsey Stirman, S., Baumann, A. A., & Miller, C. J. (2019). The FRAME: An expanded framework for reporting adaptations and modifications to evidence-based interventions. Implementation Science, 14(1), 58. https://doi.org/10.1186/s13012-019-0898-y
Disclosures of Interest: None declared.
Fidelity to the ACT SMART toolkit: An assessment of implementation strategy fidelity
Authors
Ms. Jessica Tschida - Michigan State University, Department of Psychology
Dr. Amy Drahota - Michigan State University, Department of Psychology; Child and Adolescent Services Research Center (CASRC)
Background: Fidelity, comprised of adherence, dose, participant responsiveness, quality, and differentiation, has rarely been assessed for implementation strategies despite the opportunity to advance our knowledge of strategy use and adaptations (Akiba et al., 2021; Berry et al., 2021). We provide one of the first models of assessing implementation strategy fidelity in an evaluation of fidelity to the Autism Community Toolkit: Systems to Measure and Adopt Research-Based Treatments (ACT SMART). ACT SMART is a blended implementation strategy aiming to increase utilization of evidence-based practices (EBPs) for autism spectrum disorder (ASD) in community settings.
Methods: Participants included implementation teams (N = 6) composed of agency leaders and staff at ASD community agencies participating in the ACT SMART pilot trial.
Using an instrumental case study approach and scoring rubric for implementation strategy fidelity (Slaughter et al., 2015), we assessed implementation team adherence, dose, and participant responsiveness for ACT SMART overall and for each phase. To examine differences in fidelity across phases, we conducted repeated measures ANOVAs with Bonferroni post-hoc tests.
Results: Adherence to the ACT SMART toolkit averaged 87% (SD = 6.0), average dose was moderate, and average participant responsiveness was very engaged. There was a significant main effect of phase for dose (F(2,8) = 11.38, MSE = .190, p = .005, η2=.74, 95% CI [.16, .84]), such that dose was significantly lower in the planning for implementation phase compared to the implementation phase (d = 3.98, 95% CI [1.05, 6.88]). Assessing implementation strategy fidelity can identify overall potential for strategies to be used with high fidelity and possible adaptations to specific components with lower fidelity.
Conclusion: We found overall high fidelity to ACT SMART along with potential need to adapt the planning for implementation phase. This assessment has important implications for increasing EBP use in ASD community agencies as well as informing evaluations of implementation strategy fidelity generally.
References
1. Akiba, C. F., Powell, B. J., Pence, B. W., Nguyen, M. X., Golin, C., & Go, V. (2021). The case for prioritizing implementation strategy fidelity measurement: benefits and challenges. Translational Behavioral Medicine, ibab138. https://doi.org/10.1093/tbm/ibab138
2. Berry, C. A., Nguyen, A. M., Cuthel, A. M., Cleland, C. M., Siman, N., Pham-Singer, H., & Shelley, D. R. (2021). Measuring implementation strategy fidelity in HealthyHearts NYC: A complex intervention using practice facilitation in primary care. American Journal of Medical Quality, 36(4), 270–276. https://doi.org/10.1177/1062860620959450
3. Slaughter, S. E., Hill, J. N., & Snelgrove-Clarke, E. (2015). What is the extent and quality of documentation and reporting of fidelity to implementation strategies: A scoping review. Implementation Science, 10(1), 129. https://doi.org/10.1186/s13012-015-0320-3
Disclosures of Interest: None declared.
Consensus-based process evaluation reporting guidelines (CONPHES) for public health intervention studies conducted within and/or parallel to an effectiveness trial
Authors
Dr. Femke Van Nassau - Department of Public and Occupational Health, Amsterdam UMC Location University of Amsterdam
Mr. Bart Cillekens - Department of Public and Occupational Health, Amsterdam UMC Location University of Amsterdam
Dr. Judith Jelsma - Department of Public and Occupational Health, Amsterdam UMC Location University of Amsterdam
Mr. Christiaan Vis - VU University
Dr. Wieneke Mokkink - Department of Epidemiology and Data Science, Amsterdam Public Health research institute, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
Prof. Shaun Treweek - Health Services Research Unit, University of Aberdeen, Aberdeen, United Kingdom,
Dr. Hidde Van der Ploeg - Department of Public and Occupational Health, Amsterdam UMC Location University of Amsterdam
Background: Many researchers conduct a process evaluation (Moore et al., 2015) within and/or parallel to an effectiveness trial of a public health intervention. Yet, there is currently no standardized, scientifically recognized guideline for reporting such process evaluations, impeding interpretation of the study results as well as comparison between studies.
Methods: Therefore, we conducted an e-Delphi study (Keeney et al., 2006), in which international experts participated. Based on a scoping review, a first draft of the reporting guidelines existed of 32 items, which was proposed to panellist in round 1. Participants were asked to rate the proposed items (on a 5-point Likert scale), and provide comments and suggestions. In the next two rounds adjustments to the items and descriptions were proposed to the e-Delphi panel until a consensus of 67% was reached (Mokkink et al., 2010).
Results: Of the 137 invited experts, 72 (53%) participated in round 1. Of those 64 (89% of 72) completed round 2; 55 (76% of 72) completed round 3. This resulted in 19 items included in the CONPHES guidelines. Items include the description of the intervention, as well as the implementation strategies applied, and the role of the delivery and support team. The guideline also includes describing what framework was used for evaluation and how evaluation outcomes were assessed. Lastly, providing a detailed description of analyses applied (both quantitative and qualitative) as well as quality assurance are included in the guidelines. The guidelines are accompanied with an Explanation and Elaboration (E&E) document, in which also practical examples are provided.
Conclusion: The consensus-based e-Delphi study has resulted in a well-balanced and practical applicable CONPHES reporting guideline for researchers and practitioners involved in conducting and reporting process evaluations. Implementation of these guidelines will results in higher quality reporting of process evaluations of interventions aimed at promoting public health.
References
1. Keeney, S., Hasson, F., & McKenna, H. (2006). Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs, 53(2), 205-212. https://doi.org/10.1111/j.1365-2648.2006.03716.x
2. Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M., & de Vet, H. C. (2010). The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol, 63(7), 737-745. https://doi.org/10.1016/j.jclinepi.2010.02.006
3. Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., Moore, L., O'Cathain, A., Tinati, T., Wight, D., & Baird, J. (2015). Process evaluation of complex interventions: Medical Research Council guidance. Bmj, 350, h1258. https://doi.org/10.1136/bmj.h1258
Disclosures of Interest: None declared.
Practice and research-driven Dutch implementation research agenda
Authors
Dr. Femke Van Nassau - Department of Public and Occupational Health, Amsterdam UMC Location University of Amsterdam
Mrs. Anouk Driessen - Department of Public and Occupational Health, Amsterdam UMC Location University of Amsterdam
Dr. Leti van Bodegom - Leiden University Medical Center
Dr. Bethany Hipple Walters - Trimbos Institute
Dr. Erwin Ista - Erasmus MC
Dr. Wouter Keijser - Universiteit Twente / DIRIMI Instituut
Dr. Rianne Van der Kleij - Van der Sluis - Leiden University Medical Center
Mr. Christiaan Vis - VU University
Dr. Gera Welker - UMC Groningen
Prof. Michel Wensing - Heidelberg Universiteit, Duitsland; Radboudumc, Netherlands
Background: In recent years, implementation research has gained more attention in the Netherlands. Yet, scarce funding of implementation research has resulted in mainly context specific knowledge and lacks generalizability to other contexts. Therefore, the Netherlands Implementation Collective (NIC) (Nederlands Implementatie Collectief, 2022) was asked by the national funding agency ZonMw to develop a Dutch implementation research agenda.
Methods: To do so, we 1) held interviews with Dutch implementation researchers; 2) conducted a 2-round e-Delphi study (Keeney et al., 2006); and 3) reached out to implementation professionals to share their implementation barriers in an online survey. In the e-Delphi study, panelists were asked to provide research questions in round 1, which were then merged into 31 proposed research topics. Delphi panelist scored these topics (on a 5-point Likert scale). Consensus was reached if 67% agreed with inclusion of the topic (Mokkink et al., 2010). These topics guided the thematic analyses of the input of the survey among practice professionals.
Results: Of the 47 invited researchers, 26 (55%) participated in round 1 (222 research questions). Twenty participants (77% of 26) completed round 2, in which consensus was reached on 14 topics. The survey among 74 practice professionals provided 230 barriers. Topics were categorised into 7 themes linked to implementation, sustainability, scale-up and de-implementation, such as knowledge on how to link determinants to strategies, tailoring of strategies, using innovative research designs. But also need for capacity for implementation and implementation research was expressed, and the need for practical tools to apply evidence-based implementation in practice.
Conclusion: By combining both input from implementation researchers (how does it work) as well as professionals (how to apply), the research agenda addresses topics relevant for both fields. The research agenda is widely disseminated among funding organisations, policy makers, researchers and practice to guide future implementation research.
References
1. Keeney, S., Hasson, F., & McKenna, H. (2006). Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs, 53(2), 205-212. https://doi.org/10.1111/j.1365-2648.2006.03716.x
2. Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M., & de Vet, H. C. (2010). The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol, 63(7), 737-745. https://doi.org/10.1016/j.jclinepi.2010.02.006
3. Nederlands Implementatie Collectief. (2022). https://nederlandsimplementatiecollectief.nl/
Disclosures of Interest: None declared.
Co-developing an implementation theory-informed intervention to support blood plasma donation among newly eligible men who have sex with men in Canada: A community and health system stakeholder participatory approach
Authors
Dr. Elisabeth Vesnaver - Clinical Epidemiology, Ottawa Hospital Research Institute, Ottawa, Canada
Ms. Amelia Palumbo - Clinical Epidemiology, Ottawa Hospital Research Institute, Ottawa, Canada
Ms. Castillo Gisell - Clinical Epidemiology, Ottawa Hospital Research Institute, Ottawa, Canada
Ms. Emily Gibson - Clinical Epidemiology, Ottawa Hospital Research Institute, Ottawa, Canada
Mx. Terrie Butler-Foster - Canadian Blood Services
Mr. Don Lapierre - Canadian Blood Services
Mr. Andrew Rosser - London Community Advisory Group
Mr. Richard MacDonagh - London Community Advisory Group
Mr. Glenndl Miguel - Calgary Community Advisory Group
Mr. Marco Reid - Calgary Community Advisory Group
Dr. Paul MacPherson - The Ottawa Hospital
Mr. Nolan Hill - Calgary Community Advisory Group
Mr. Kyle Rubini - London Community Advisory Group
Mr. Taylor Randall - London Community Advisory Group
Mr. William Osbourne-Sorrell - London Community Advisory Group
Dr. Sheila O'Brien - Canadian Blood Services
Dr. William Bridel - Calgary Community Advisory Group
Dr. Joanne Otis - Université du Québec à Montréal
Mr. Mark Greaves - Calgary Community Advisory Group
Mr. Tail Al-Bakri - London Community Advisory Group
Mr. Maximilian Labrecque - Calgary Community Advisory Group
Dr. Marc Germain - Hema-Quebec
Mr. Shane Orvis - Calgary Community Advisory Group
Mr. Andrew Clapperton - Calgary Community Advisory Group
Dr. Dana Devine - Canadian Blood Services
Dr. Justin Presseau - Clinical Epidemiology, Ottawa Hospital Research Institute, Ottawa, Canada
Background: Canada does not collect enough blood plasma to meet the needs of Canadians. Yet, gay, bisexual, and all men who have sex with men (gbMSM) only recently became eligible to donate plasma through a pilot program in two cities. While policy changes are necessary for increased donation opportunity in this population, they may not be sufficient to support successful implementation. We aimed to understand which barriers and enablers would impact on plasma donation and to work with stakeholders to co-develop strategies to encourage donation that addresses their needs and concerns.
Methods: The study was a collaboration with Canadian Blood Services (national blood operator) and community advisors identifying as gbMSM. Guided by French’s model for designing theory-informed implementation interventions1, we used the Theoretical Domains Framework2 (TDF) and qualitative interviews to identify donation barriers/enablers in men identifying as gbMSM (N = 27) and staff (N = 28). TDF-linked barriers/enablers were mapped behaviour change techniques3 (BCT). We collaborated with community advisors to co-develop suitable strategies that operationalize BCTs delivered using feasible and acceptable channels.
Results: BCTs were operationalized using an independent website and video, and a set of strategies for delivery by Canadian Blood Services. For example, for delivery by video, 7 TDF domains were mapped to 11 BCTs (eg Barrier: concern of not being welcome in centre [domain: Beliefs about consequences] addressed by showing positive authentic clinic staff interaction with newly eligible male donor [BCT: information about social consequences]).
Conclusion: We co-developed multimodal interventions to support plasma donation by newly eligible men identifying as gbMSM and new criteria implementation by staff. Focus groups to determine broader acceptability of interventions are in process. Our participatory approach, rooted in theory and lived experience, can continue to support these critical stakeholders as policies evolve.
References
1. French, S. D., Green, S. E., O’Connor, D. A., McKenzie, J. E., Francis, J. J., Michie, S., … & Grimshaw, J. M. (2012). Developing theory-informed behaviour change interventions to implement evidence into practice: a systematic approach using the Theoretical Domains Framework. Implementation Science, 7(1), 1-8.
2. Atkins, L., Francis, J., Islam, R., O’Connor, D., Patey, A., Ivers, N., … & Michie, S. (2017). A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems. Implementation science, 12(1), 1-18.
3. Michie, S., Richardson, M., Johnston, M., Abraham, C., Francis, J., Hardeman, W., … & Wood, C. E. (2013). The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Annals of behavioral medicine, 46(1), 81-95.
Disclosures of Interest: Terrie Butler-Foster, Don Lapierre, Mindy Goldman, Sheila O'Brien, and Dana Devine work for Canadian Blood Services who administered the grant. Marc Germain works for Héma-Québec, another Canadian blood operator. The authors report no other conflicts of interest.
Using the dynamic adaptation process to implement evidence-informed practices to achieve mental health equity for gender and sexual minority high school students
Authors
Dr. Cathleen Willging - Pacific Institute for Research and Evaluation
Mr. Adrien Lawyer - Transgender Resource Center of New Mexico
Dr. Daniel Shattuck - Pacific Institute for Research and Evaluation
Dr. Bonnie Richard - Pacific Institute for Research and Evaluation
Dr. Mary Ramos - University of New Mexico Health Sciences Center
Background: Reducing adolescent suicide in the United States is a public health priority, and gender and sexual minority (GSM) youth are at elevated risk (National Academies of Sciences, Engineering, and Medicine, 2022). The Centers for Disease Control and Prevention tracks use of six evidence-informed school-based practices (EIPs) to enhance mental health equity for GSM students (Shattuck et al., 2022). Guided by the Exploration, Preparation, Implementation, and Sustainment framework, our community-engaged study assesses the extent to which high schools implemented EIPs using the Dynamic Adaptation Process (DAP), a participatory and multifaceted implementation strategy (Aarons et al., 2012).
Methods: Our mixed-method cluster randomized controlled trial involved 42 New Mexico high schools. Data sources for this analysis include annual surveys and individual and small group qualitative interviews with school professionals, most of whom took part in Implementation Resource Teams charged with instantiating the EIPs. Other sources included structured assessments of EIP implementation and periodic reflections with implementation coaches and technical assistance experts. Qualitative data were analyzed using deductive and inductive coding techniques; quantitative data were analyzed using linear regressions.
Results: The DAP facilitated collaboration among school professionals and community intermediary organizations to shift knowledge and attitudes and execute contextually-responsive implementation strategies. The DAP fostered relationship-building and leadership, encouraging school leaders to legitimate EIP-implementation efforts and champion health equity for GSM students, an often invisible and neglected population. The DAP led to statistically significant change in the adoption of safe spaces, prohibitions on bullying and harassment based on GSM identity, inclusive health education materials, staff professional development, and facilitation of students’ access to GSM-affirming behavioral, sexual, and reproductive healthcare.
Conclusion: Participatory implementation science models like the DAP can help address mental health equity for marginalized populations by enabling the uptake of practices likely to contribute to wellbeing. This mixed-methods study provides a rich example for future implementation science research tackling health disparities for GSM people in schools and other complex systems.
References
1. Aarons, G. A., Green, A. E., Palinkas, L. A., Self-Brown, S., Whitaker, D. J., Lutzker, J. R., … & Chaffin, M. J. (2012). Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implementation Science, 7(1), 1-9.
2. Shattuck, D. G., Rasberry, C. N., Willging, C. E., & Ramos, M. M. (2022). Positive Trends in School-Based Practices to Support LGBTQ Youth in the United States Between 2010 and 2018. Journal of Adolescent Health, 70(5), 810-816.
3. National Academies of Sciences, Engineering, and Medicine. (2022). Strategies and Interventions to Reduce Suicide: Proceedings of a Workshop. Washington, DC: The National Academies Press.
Disclosures of Interest: None declared.
Consultation as an implementation strategy to increase fidelity of measurement-based care
Authors
Ms. Grace Woodard - University of Miami
Ms. Elizabeth Casline - University of Miami
Dr. Golda Ginsburg - University of Connecticut School of Medicine
Dr. Jill Ehrenreich-May - University of Miami
Dr. Amanda Jensen-Doss - University of Miami
Background: Measurement-based care (MBC), defined as regularly administering outcome measures to clients to inform clinical decision making, has mixed evidence of effectiveness in youth (Bergman et al., 2018). MBC has the potential to improve mental health treatment broadly due to its applicability across diagnoses and theoretical orientations, but only about 20% of clinicians regularly use MBC (Lewis et al., 2019). Training and consultation are widely used implementation strategies to increase the use of evidence-based practices (EBP). While training alone is not enough to change clinical practice, the optimal content or dose of post-training consultation is unknown. It is critical to justify the high cost of consultation by understanding its effects on the use of an EBP. Little research has studied consultation focused on MBC. The current study will assess the association between time spent in consultation (“dosage”) and MBC fidelity.
Methods: Participants included community mental health clinicians (N = 30) in the MBC condition of a randomized controlled trial and the youth clients (N = 56) treated for at least one session. Dosage was extracted from consultation call notes. MBC fidelity was measured using the implementation index (Bickman et al., 2016), which combines the rates of administering and viewing questionnaires, using objective data from the online MBC system. Multilevel modeling was used to account for nesting within clinicians (ICC = 0.26).
Results: For every 30-minutes discussing a case during the consultation period, fidelity increased by 6.6% (b = 0.22, SE = 0.06, p < .01). At the average consultation dosage, a case’s fidelity was 72% (b = 72.07, SE = 4.65, p < .001).
Conclusion: These results help justify the cost of consultation by showing that each minute of discussion increases fidelity. Future studies should elucidate the mechanisms by which consultation is effective for increasing MBC fidelity, like engagement of clinicians, consultant techniques, or content discussed.
References
1. Bergman, H., Kornor, H., Nikolakopoulou, A., Hanssen-Bauer, K., Soares-Weiser, K., & Tollefsen, T. K. (2018). Client feedback in psychological therapy for childen and adolescents with mental health problems. Cochrane Database of Systematic Reviews, 8, CD011729.
2. Bickman, Leonard, Douglas, S. R., De Andrade, A. R. V., Tomlinson, M., Gleacher, A., Olin, S., & Hoagwood, K. (2016). Implementing a measurement feedback system: A tale of two sites. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 410–425. https://doi.org/10.1007/s10488-015-0647-8
3. Lewis, C. C., Boyd, M., Puspitasari, A., Navarro, E., Howard, J., Kassab, H., Hoffman, M., Scott, K., Lyon, A., Douglas, S., Simon, G., & Kroenke, K. (2019). Implementing measurement-based care in behavioral health: A review. JAMA Psychiatry, 76(3), 324–335. https://doi.org/10.1001/jamapsychiatry.2018.3329
Disclosures of Interest: None declared.
Moderators of EBP implementation in school systems in California
Authors
Dr. Yue Yu - UC Davis MIND Institute
Dr. Melina Melgarejo - San Diego state un
Mrs. Patricia Schetter - UC Davis MIND
Ms. Soo Park - UC Davis MIND Institute
Dr. Gregory Young - UC Davis MIND Institute
Dr. Jessica Suhrheinrich - San Diego State University
Dr. Aubyn Stahmer - UC Davis MIND Institute
Background: Ensuring effective use of Evidence-based Practice (EBP) for autism in schools is imperative. Research indicates implementation climate and EBP resources (Lyon et al., 2013) relate to successful implementation. Little is known about contextual factors at system levels that impact EBP implementation for school-based providers. Understanding these factors is crucial for selecting and tailoring implementation strategies to support EBP scale up. Using an implementation science framework (Aarons et al., 2011), we explored provider and system level moderating factors on provider EBP implementation and student outcomes.
Methods: Data were collected from California school personnel. School-based providers (N = 1083), representing 333 districts completed surveys about implementation climate (ICS; Ehrhart et al., 2014), provider experience with autism, EBP resources, and EBP implementation outcomes (fidelity, competence, knowledge). Student outcomes were obtained from the California Department of Education. Multilevel modeling was conducted to analyze the relationship between EBP Implementation and student outcomes and the moderation effects of provider and school district level factors.
Results: Higher ICS (χ2= 50.63, df = 1, p < 0.001) predicted better EBP implementation outcomes, and provider ASD experience moderated this relationship (χ2= 10.91, df = 1, p < 0.001). ICS was more impactful on EBP implementation when provider ASD experience was low. EBP resources predicted student outcomes, moderated by District poverty level (χ2= 8.3, df = 1, p < 0.005). Greater EBP resources predicted a higher percentage of students who met Math standards only when district poverty was high. Greater EBP implementation was related to lower school district suspension rates (χ2= 8.64, df = 1, p = 0.003).
Conclusion: Data suggested moderating effects on EBP implementation from both provider and system level factors. Implementation climate seems especially important to EBP implementation when provider autism-related experience is low. Preliminary data suggested an impact of implementation factors and implementation outcomes on student outcomes.
References
1. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and policy in mental health and mental health services research, 38(1), 4-23.
2. Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2014). Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS). Implementation Science, 9(1), 1-11.
3. Lyon, A. R., & Bruns, E. J. (2019). From evidence to impact: joining our best school mental health practices with Our Best Implementation Strategies. School Mental Health, 11(1), 106-114. doi:10.1007/s12310-018-09306-w
Disclosures of Interest: None declared.
POSTER PRESENTATIONS
Clinical supervision: A potential implementation strategy to improve EBT implementation climate
Authors
Mr. Rashed AlRasheed - University of Washington
Mr. Noah Triplett - University of Washington
Ms. Clara Johnson - University of Washington
Dr. Michael Pullmann - University of Washington
Dr. Shannon Dorsey - University of Washington
Background: Implementing evidence-based treatments (EBTs) in community mental health settings can be challenging. High EBT implementation climate—organizational members’ perception that EBT use is expected, supported and rewarded—has theoretically been linked to successful EBT implementation. Little research has investigated factors that facilitate a high EBT implementation climate. Clinical supervisors are in a unique position to shape clinicians’ organizational perceptions and thus may sustainably bolster an organization’s EBT implementation climate via supervision. This study assessed whether EBT-focused supervision practices and supervisor- and clinician-level factors affect EBT implementation climate.
Methods: Data came from a randomized controlled trial examining the effect of supervision practices on implementation and clinical outcomes in community mental health. Participating clinicians who delivered Trauma-focused Cognitive Behavioral Therapy were randomized to one of two EBT-focused supervision conditions: Symptom & Fidelity Monitoring (SFM) or SFM + Behavioral Rehearsal (BR). Supervisors (N = 37) and their clinicians (N = 120), who were predominantly White, female, and master’s level, completed self-report measures at baseline and at one-year follow-up. The impact of EBT-focused supervision practices and supervisor- and clinician-level factors on implementation climate was examined via a two-level multilevel model (MLM) with random effects at the supervisor level to account for clustering. Exploratory analyses examined whether EBT-focused supervision practices affected EBT implementation climate under specific conditions.
Results: Clinicians’ EBT implementation climate scores did not significantly change from baseline to follow-up. However, higher supervisor self-efficacy was associated with a higher EBT implementation climate (β = 0.31, p = 0.02). Exploratory analyses demonstrated that in the SFM + BR condition (vs. SFM alone), the effect of self-efficacy on implementation climate was stronger.
Conclusion: This study highlights that supervision can be an implementation strategy that impacts implementation climate under specific conditions. In this study, supervisors needed to feel confident and capable in supervising, particularly when supervision includes an “active” element (e.g., BR).
References
1. Weiner, B. J., Belden, C. M., Bergmire, D. M., & Johnston, M. (2011). The meaning and measurement of implementation climate. Implementation Science : IS, 6, 78. https://doi.org/10.1186/1748-5908-6-78
2. Birken, S., Clary, A., Tabriz, A. A., Turner, K., Meza, R., Zizzi, A., Larson, M., Walker, J., & Charns, M. (2018). Middle managers’ role in implementing evidence-based practices in healthcare: A systematic review. Implementation Science: IS, 13(1), 149. https://doi.org/10.1186/s13012-018-0843-5
3. Dorsey, S., Pullmann, M. D., Deblinger, E., Berliner, L., Kerns, S. E., Thompson, K., Unützer, J., Weisz, J. R., & Garland, A. F. (2013). Improving practice in community-based settings: A randomized trial of supervision - study protocol. Implementation Science: IS, 8, 89. https://doi.org/10.1186/1748-5908-8-89
4. Williams, N. J., Ehrhart, M. G., Aarons, G. A., Marcus, S. C., & Beidas, R. S. (2018). Linking molar organizational climate and strategic implementation climate to clinicians’ use of evidence-based psychotherapy techniques: Cross-sectional and lagged analyses from a 2-year observational study. Implementation Science, 13(1), 85. https://doi.org/10.1186/s13012-018-0781-2
5. Williams, N. J., Wolk, C. B., Becker-Haimes, E. M., & Beidas, R. S. (2020). Testing a theory of strategic implementation leadership, implementation climate, and clinicians’ use of evidence-based practice: A 5-year panel analysis. Implementation Science : IS, 15, 10. https://doi.org/10.1186/s13012-020-0970-7
6. Southam-Gerow, M. A., Rodríguez, A., Chorpita, B. F., & Daleiden, E. L. (2012). Dissemination and implementation of evidence based treatments for youth: Challenges and recommendations. Professional Psychology: Research and Practice, 43(5), 527–534. https://doi.org/10.1037/a0029101
7. Aarons, G. A., Ehrhart, M. G., Farahnak, L. R., & Hurlburt, M. S. (2015). Leadership and organizational change for implementation (LOCI): A randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implementation Science, 10(1), 11. https://doi.org/10.1186/s13012-014-0192-y
8. Aarons, G. A., Ehrhart, M. G., Farahnak, L. R., & Sklar, M. (2014). Aligning Leadership Across Systems and Organizations to Develop a Strategic Climate for Evidence-Based Practice Implementation. Annual Review of Public Health, 35(1), 255–274. https://doi.org/10.1146/annurev-publhealth-032013-182447
Disclosures of Interest: None declared
A qualitative study to examine feasibility of incorporating a critical consciousness lens into a school-based prevention intervention for newcomer Latinx immigrant youth
Authors
Ms. Alejandra Arce - University of California, San Francisco
Dr. Kelly Edyburn - University of California, San Francisco
Ms. Tatiana Baquero-Devis - University of California, San Francisco
Dr. William Martinez - University of California, San Francisco
Background: Scholars have previously called for more interventions that directly address impacts of systemic inequities (Shelton et al., 2021). Critical consciousness (CC) is the developmental process by which marginalized groups become aware of and challenge systemic inequities via action. A growing literature documents the benefits of critical consciousness-based interventions for promoting positive health outcomes in youth of color (McWhirter et al., 2021; Rapa et al., 2020). However, few interventions have been developed for newcomer Latinx immigrant youths (NLIY) with varying exposure to and awareness of systemic inequities in the host country. The current study examines ways in which Fuerte, a school-based prevention intervention for NLIY, could be augmented by incorporating CC into its curriculum. Fuerte is implemented in partnership with the local School District.
Method: As part of our program evaluation efforts, we conducted seven focus groups with current participants (N = 30) to help determine the feasibility of creating a module focused on consciousness raising around immigration and systemic inequities. Participants were NLIY in high school who had lived in the U.S. for less than five years. Focus groups were conducted in Spanish using a semi structured interview guide. Participants were asked to report on the obstacles that they have faced as immigrants in the U.S. Thematic analysis will be used to code data.
Results: Preliminary analyses show that NLIY identify both individual and systemic obstacles in their migration/acculturation journey. NLIY also compare current obstacles and social conditions in the U.S. to those in their native countries and feel confident in their ability to overcome them.
Conclusion: NLIY are impacted by multiple systems of oppression. Interventions aimed at promoting their well-being should consider applying a CC lens to explicitly address health impacts of systemic inequities among these youth to inform implementation efforts with this population.
References
1. McWhirter, E. H., Cendejas, C., Fleming, M., Martínez, S., Mather, N., Garcia, Y., Romero, L., Ortega, R. I., & Rojas-Araúz, B. O. (2021). College and career ready and critically conscious: Asset-building with Latinx immigrant youth. Journal of Career Assessment, 29(3), 525–542. https://doi.org/10.1177/1069072720987986
2. Rapa, L. J., Diemer, M. A., & Roseth, C. J. (2020). Can a values-affirmation intervention bolster academic achievement and raise critical consciousness? Results from a small-scale field experiment. Social Psychology of Education: An International Journal. https://doi.org/10.1007/s11218-020-09546-2
3. Shelton, R. C., Adsul, P., Oh, A., Moise, N., & Griffith, D. M. (2021). Application of an antiracism lens in the field of implementation science (IS): Recommendations for reframing implementation research with a focus on justice and racial equity. Implementation Research and Practice. https://doi.org/10.1177/26334895211049482
Disclosures of Interest: None declared
R3 supervisor strategy in substance use disorder treatment settings: a qualitative formative evaluation of provider perceptions in Arkansas
Authors
Dr. Jure Baloh - University of Arkansas for Medical Sciences
Dr. Geoffrey Curran - University of Arkansas for Medical Sciences
Dr. Lisa Saldana - Oregon Social Learning Center
Background: The R3 supervisor strategy was developed to support implementation of evidence-based practices (EBPs) in child welfare systems (CWS) by reinforcing effort, roles and relationships, and taking small steps (Saldana et al., 2016). The R3 strategy improved CWS organizational social contexts (e.g. implementation climate) and systems’ performance (e.g. case closure), but has not yet been tested in other settings. To assess its fit with substance use disorder (SUD) settings, we qualitatively examined SUD provider perceptions of the R3 strategy to inform future modifications and testing.
Methods: We conducted a qualitative formative evaluation (Stetler et al., 2006) with 25 program leaders and counselors at 8 community SUD programs in Arkansas to learn about R3 strategy perceptions, barriers, and facilitators. Semi-structured interviews were based on the i-PARIHS framework (Harvey & Kitson, 2016), lasted ∼1 hour (phone/Zoom), and were recorded, transcribed, and thematically analyzed.
Results: The R3 supervision strategy was generally found acceptable and feasible for use in SUD settings. It fits current supervisory practices (e.g. case summaries, group supervision), and participants welcomed it provided an evidence-based structure and support to both to counselors and supervisors. Participants thought that most counselors would find R3 supervision beneficial, particularly those less experienced or learning new skills/practices. Key barriers included time availability (for supervisors), and recording of supervision sessions, which may induce anxiety and “acting”. To help secure buy-in, participants stressed the need to emphasize the goal of R3 supervision is to support counselors and improve client outcomes. Participants identified several potential benefits, both for counselors (e.g. self-efficacy, reduced burnout) and clients (e.g. treatment engagement, retention).
Conclusion: The R3 supervisor strategy is a promising intervention to improve supervision, organizational social contexts, counselor well-being and performance, and client outcomes in SUD settings. Future studies need to further evaluate its fit and effectiveness in these settings.
References
1. Harvey, G., & Kitson, A. (2016). PARIHS revisited: from heuristic to integrated framework for the successful implementation of knowledge into practice. Implementation science: IS, 11, 33. https://doi.org/10.1186/s13012-016-0398-2
2. Saldana, L., Chamberlain, P., & Chapman, J. (2016). A supervisor-targeted implementation approach to promote system change: The R3 Model. Administration and policy in mental health, 43(6), 879–892. https://doi.org/10.1007/s10488-016-0730-9
3. Stetler, C. B., Legro, M. W., Wallace, C. M., Bowman, C., Guihan, M., Hagedorn, H., Kimmel, B., Sharp, N. D., & Smith, J. L. (2006). The role of formative evaluation in implementation research and the QUERI experience. Journal of general internal medicine, 21 Suppl 2(Suppl 2), S1–S8. https://doi.org/10.1111/j.1525-1497.2006.00355.x
Disclosures of Interest: None declared
How social capital, trust, and relationships relate to organizational culture and self-efficacy
Authors
Dr. Leah Bartley - University of North Carolina, Chapel Hill & Kaye Implementation & Evaluation
Dr. Sarah Kaye - Kaye Implementation and Evaluation
Ms. Stephanie Clone - Kaye Implementation and Evaluation
Background: There have been numerous studies on the benefits of social capital, particularly in public health. In workplace implementation efforts, social capital suggests staff who have the opportunity to work in groups with strong cohesion can result in mutual trust and reciprocity. What is the relationship between social capital and staff reported self-efficacy and organizational culture?
Methods: This implementation study is part of a statewide type one hybrid CRT to assess effectiveness of a kinship and adoption navigation program to support children and families. The CFIR1 was used to identify relevant study constructs including organization culture, social capital and implementation outcomes of reach and fidelity. Navigators and agency staff were surveyed using the Social Capital Scale2, Learning Climate and Culture Stress subscales3 and self-efficacy. Results were analyzed using descriptive and bivariate statistics by different roles.
Results: Preliminary results were positive. There were no statistically significant differences between groups regarding organizational learning culture; navigators had an average rating of 4.5 for learning culture, compared to 4.4 of agency staff (N = 56, 1 = strongly disagree, 5 = strongly agree). There was a statistically significant difference regarding stress between agency staff and navigators [t(40) = -3.37, p = .002]. Generally, agency staff felt neutrally about stress, while navigators indicated less stress. Regarding the social capital of relationships and trust, responses were positive; staff “agreed” (4.5) positively regarding relationships and trust. Final analyses will explore the inter-relationships between the constructs.
Conclusion: Social capital, which includes workplace relationships and “whether people feel that they are respected, valued, and treated as equals at work, rather than feeling that it is all a matter of seniority in their hierarchy2” (p. 3) is an important aspect to consider in the implementation process. This study provides an opportunity to consider the relationship between social capital, organization culture and practitioner self-efficacy.
References
1. Damschroder, L., Hall, C., Gillon, L., Reardon, C., Kelley, C., Sparks, J., & Lowery, J. (2015). The Consolidated Framework for Implementation Research (CFIR): progress to date, tools and resources, and plans for the future. Implementation Science : IS, 10(Suppl 1), A12. https://doi.org/10.1186/1748-5908-10-S1-A12
Kouvonen, A., Kivimäki M, Vahtera, J., Oksanen, T., Elovainio, M., Cox, T., Virtanen, M., Pentti, J., Cox, S. J., & Wilkinson, R. G. (2006). Psychometric evaluation of a short measure of social capital at work. BioMed Central Public Health, 6, 251–251
1. Fernandez, M. E., Walker, T. J., Weiner, B. J., Calo, W. A., Liang, S., Risendal, B., Friedman, D. B., Tu, S. P., Williams, R. S., Jacobs, S., Herrmann, A. K., & Kegler, M. C. (2018). Developing measures to assess constructs from the inner setting domain of the consolidated framework for implementation research. Implementation Science, 13(1). https://doi.org/10.1186/s13012-018-0736-7
Disclosures of Interest: None declared
Accelerating research use in courts: Development of a conceptual behavioral research use tool
Authors
Ms. Ella Baumgarten - Department of Psychiatry and Behavioral Sciences, University of Washington School of Medicine
Ms. Kristin Vick - Department of Psychiatry and Behavioral Sciences, University of Washington School of Medicine
Dr. Sarah Walker - Department of Psychiatry and Behavioral Sciences, University of Washington School of Medicine
Background: In recent years, there have been demands for substantial reform of the US youth criminal-legal system. Behavioral research evidence has the potential to guide juvenile justice reform and there is a push for individuals to use research-based information when making decisions that impact youth and families. While importance of research use is acknowledged, ways that individuals within the juvenile justice system obtain and use research-based information is generally unknown. Currently, there is no validated measure of how behavioral research-based information is used within this system. This study aims to fill that gap and develop a survey tool that assesses conceptual research use (CRU) among juvenile court leaders.
Methods: This study employed theoretical review, Delphi methods for expert review, and cognitive interviewing in developing the CRU measure. 10 CRU-measure cognitive interviews were completed with juvenile probation leaders. Responses were transcribed, organized by item, and thematically coded using a content coding approach for measure feedback, question comprehension, points of confusion, and research use responses.
Results: Three themes emerged from cognitive interviews: Definition and sources of research, patterns of research evidence use, and CRU item wording. Regarding research definitions and sources, many participants did not think of research as study findings but as information shared through colleagues or trusted sources. For research use patterns, respondents often discussed research guiding current initiatives in their courts rather than theoretical information and indicated that research use occurs over long periods of time. Regarding item wording, respondents felt that some questions were confusing and seemed redundant.
Conclusion: Cognitive interviews identified survey areas that needed rewording to fit patterns of research use among juvenile court leaders. References to studies were revised to reflect sources of research information in addition to edits to clarify wording. A revision was found to be more understandable by participating juvenile court leaders.
References
1. Cusworth Walker, S., Vick, K., Gubner, N. R., Herting, J. R., & Palinkas, L. A. (2021). Accelerating the conceptual use of behavioral health research in juvenile court decision-making: study protocol. Implementation Science Communications, 2(1), 14-14.
2. Hsieh, M.-L., Woo, Y., Hafoka, M., Van Wormer, J., Stohr, M. K., & Hemmens, C. (2016). Assessing the current state of juvenile probation practice: A statutory analysis. Journal of Offender Rehabilitation, 55(5), 329-354.
3. National Research Council. (2013). Reforming Juvenile Justice: A Developmental Approach (R. J. B. Committee on Assessing Juvenile Justice Reform, Robert L. Johnson, Betty M. Chemers, and Julie A. Schuck Ed.). Committee on Law and Justice, Division of Behavioral and Social Sciences and Education. The National Academic Press.
Disclosure of Interest: None declared
RE-AIMing for health equity: Using RE-AIM to evaluate equitable implementation of the Family Check-Up 4 Health program in primary care settings
Authors
Dr. Cady Berkel - Arizona State University
Ms. Nalani Thomas - Arizona State University
Ms. Elisabeth Williams - Arizona State University
Dr. James Merle - University of Utah
Ms. Lizeth Alonso - Arizona State University
Prof. Anne Mauricio - University of Oregon
Dr. Jenna Rudo-Stern - Phoenix Children's Hospital
Dr. JD Smith - University of Utah
Background: The field of prevention has begun to look to implementation science (IS) to address the pervasive and persistent health disparities confronted by Black, Indigenous, People of Color (BIPOC) communities (Brownson et al, 2021). To date, IS frameworks have largely ignored their potential role in promoting health equity. IS frameworks should address two goals: 1) promoting implementation only for programs that at a minimum do not perpetuate disparities and 2) promoting equitable implementation of programs for BIPOC communities within routine service settings. RE-AIM is a highly cited IS framework that has adopted an explicit focus on equity (Glasgow et al, 2013; Shelton et al, 2020). This study makes use of RE-AIM in evaluating the equitable implementation and effectiveness of the Family Check-Up 4 Health (FCU4Health) delivered in BIPOC-serving primary care organizations.
Methods: FCU4Health is an individually-tailored preventive intervention, adapted from the Family Check-Up for primary care settings. Data came from a type 2 effectiveness-implementation hybrid trial conducted with children with elevated BMI in primary care. We report on the implementation dimensions of RE-AIM in chronological order (adoption, reach, implementation, maintenance), followed by effectiveness, focusing on issues related to equity for each.
Results: Adoption was evaluated at the clinic and the pediatrician level. The three clinics approached served similar proportions of BIPOC families (∼85%). All three adopted the program. At the pediatrician level (N = 67), we examined variability in the proportion of BIPOC families referred. Rates surpassed clinic demographics. Reach was evaluated as a process from initial referral to program initiation. BIPOC families were significantly more likely to be referred, but equally likely to enroll in the study, participate in data collection, and initiate services. Implementation was assessed via multiple indicators. There was no difference in observed fidelity or active in-session participation. BIPOC families participated in fewer sessions, however, the total length of time in sessions did not differ. Maintenance was evaluated based on the number of BIPOC-serving clinics who implemented after the trial. None continued to implement, however all remain on our advisory board and provide insight as we develop strategies to enhance the potential for scale-up to BIPOC serving organizations. Effectiveness was evaluated by examining program effects on parenting and child behavioral health within BIPOC families only. Results supported the mediational program effects.
Conclusion: Results demonstrate how RE-AIM can be used to assess the ability of preventive interventions to promote equity as well as the equitable implementation of these programs in routine care settings.
References:
1. Brownson, R. C., Kumanyika, S. K., Kreuter, M. W., & Haire-Joshu, D. (2021). Implementation science should give higher priority to health equity. Implementation Science, 16(1), 28-28. https://doi.org/10.1186/s13012-021-01097-0
2. Glasgow, R., Askew, S., Purcell, P., Levine, E., Warner, E., Stange, K., Colditz, G., & Bennett, G. (2013). Use of RE-AIM to address health inequities: Application in a low-income community health center-based weight loss and hypertension self-management program. Translational Behavioral Medicine, 3(2), 200-210. https://doi.org/10.1007/s13142-013-0201-8
3. Shelton, R. C., Chambers, D. A., & Glasgow, R. E. (2020). An extension of RE-AIM to enhance sustainability: Addressing dynamic context and promoting health equity over time. Frontiers in Public Health, 8(134). https://doi.org/10.3389/fpubh.2020.00134
Disclosures of Interest: Anne Marie Mauricio has ownership interest in Northwest Prevention Science, Inc (NPS) and serves as a consultant for fees at NPS. NPS is licensed to disseminate the Family Check-Up for profit. Family Check-Up4Health, related to the research described in this paper, is a derivative of the Family-Check-Up.
Validation of three instruments used in evidence-based practice implementation
Authors
Ms. Nora Braathu - The Norwegian Centre for Violence and Traumatic Stress Studies
Ms. Randi Hovden Borge - The Norwegian Centre for Violence and Traumatic Stress Studies
Ms. Nadina Peters - The Norwegian Centre for Violence and Traumatic Stress Studies
Ms. Mathilde Endsjø - The Norwegian Centre for Violence and Traumatic Stress Studies
Dr. Ane-Marthe Skar - The Norwegian Centre for Violence and Traumatic Stress Studies
Mr. Erlend Høen Laukvik - The Norwegian Centre for Violence and Traumatic Stress Studies, Oslo, Norway
Dr. Karina Egeland - The Norwegian Centre for Violence and Traumatic Stress Studies
Background: The implementation of evidence-based practices (EBPs) is of crucial importance in health care institutions. Successful implementation can be influenced by factors like strategic (i.e. implementation) leadership, positive implementation climate and organizational citizenship behavior. The current paper combines three psychometric studies on assessment tools used to measure these three constructs in Norwegian mental health care services.
Methods: Strategic leadership was measured by the Implementation Leadership Scale (ILS; N = 795); climate was measured by the Implementation Climate Scale (ICS; N = 383); and extra-role behaviors performed by the employees in support of the EBP was assessed by the Implementation Citizenship Behavior Scale (ICBS; N = 152). All instruments were subjected to a confirmatory factor analysis (CFA).
Results: Results revealed that all scales showed acceptable psychometric properties, validating the use of these instruments in Norway. The ILS demonstrated an excellent model fit (χ2 (48) = 112.575, p < 0.001; CFI = 0.999, TLI = 0.999; RMSEA = 0.043; SRMR = 0.010). The ICS had an acceptable model fit (χ2 (119) = 342.897, p < 0.001; CFI = 0.989; TLI = 0.986, RMSEA = 0.070, SRMR = 0.061), and the ICBS revealed an excellent model fit (χ2 (8) = 8.15, p = .42; CFI = 1.00; TLI = 1.00; RMSEA = .011, SRMR = 0.015)
Conclusion: Research has established that implementation leadership, implementation climate, as well as employee behavior towards the EBP as important factors for successful implementation. There is a need for efficient and valid instruments that could be used to evaluate implementation efforts. The results suggest that the ICS, ILS and the ICBS are valid and reliable tools to measure these concepts.
References
1. Aarons, G. A., Ehrhart, M. G., & Farahnak, L. R. (2014c). The Implementation Leadership Scale (ILS): Development of a brief measure of unit level implementation leadership. Implementation Science: IS, 9(1), 45. https://doi.org/10.1186/1748-5908-9-45
2. Weiner, B. J., Belden, C. M., Bergmire, D. M., & Johnston, M. (2011). The meaning and measurement of implementation climate. Implementation Science, 6(1), 1-12
3. Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2015). Going above and beyond for implementation: the development and validity testing of the Implementation Citizenship Behavior Scale (ICBS). Implementation Science, 10(1), 1-9.
Disclosures of Interest: None declared
Using the matrixed multiple case study methodology to understand site differences in a hybrid Type 1 trial of a peer-led healthy lifestyle intervention for people with serious mental illness
Dr. Leopoldo J. Cabassa - Brown School of Social Work, Washington University in St. Louis, Missouri
Dr. Ana Stefancic - Columbia University Department of Psychiatry, New York
Ms. Daniela Tuda - Washington University in St. Louis
Dr. Lauren Bochicchio - School of Nursing, Columbia University, New York
Dr. Byron Powell - Brown School, Washington University in St. Louis, St. Louis
Mr. Mark Hawes - Brown School of Social Work, Washington University in St. Louis
Background: Site differences in implementation trial outcomes are common but often not examined. In a Hybrid-Type-1 trial of a peer-led healthy lifestyle intervention (PGLB) for people with SMI in three supportive housing agencies, outcomes differed by study sites. Sites 1 and 2 reported null findings, and at site 3 PGLB significantly outperformed usual care (UC) on clinically significant weight loss and cardiovascular disease risk reductions. The matrixed-multiple-case-study methodology was used to examine how implementation factors contributed to site differences.
Methods: This mixed-method approach is used to systematically examine site differences in implementation studies. Three implementation factors were examined: 1) PGLB fidelity ratings using audio recording of sessions; 2) participants’ acceptability of PGLB and UC using the client satisfaction questionnaire; and 3) changes in the integration of healthcare services at the study sites captured from two sets of qualitative interviews with 12 agency leaders (4 per study site). The first interviews were done before the trial started. The second interviews were done at the end of the trial. ANOVAs were used to examine site differences in fidelity ratings and client satisfaction. Content analysis was used to analyze leadership interviews.
Results: A trend approaching significance indicated that site 3 reported the highest fidelity ratings. High levels of satisfaction with PGLB were reported at all sites. There were differences in participants’ satisfaction with UC, with site 3 reporting the lowest levels of satisfaction. Agency leaders reported an increase in their prioritization of clients’ health throughout the trial. Sites differed in how these priorities were put into action; with site 1 having the most substantial integration of healthcare services.
Conclusion: Differences in fidelity, satisfaction with UC, and integration of healthcare services contributed to site differences. The matrixed-multiple-case-study is a useful methodology to identify implementation factors contributing to the heterogeneity of implementation trial results.
References
1. Cabassa. L. J., Stefancic, A., Lewis-Fernández, R., Luchsinger, J., Weinstein, L., Guo, S., Palinkas, L., Bochicchio, L., Wang, X., O’Hara, K., Blady, M., Simiriglia, C., & McCrudy, M. (2021). Main outcomes of the peer-led healthy lifestyle intervention for people with serious mental illness in supportive housing. Psychiatric Services, 72 (5), 555-562.
2. Cabassa, L. J., Stefancic, A., Bochicchio, L., Tuda, D., Weatherly, C. & Lengnick-Hall (2021). Organization leaders’ decisions to sustain a peer-led healthy lifestyle intervention for people with serious mental illness in supportive housing. Translational Behavioral Medicine, 11 (5), 1151-1159.
3. Kim, B., Sullivan J.L., Ritchie, M. J., Connolly, S. L., Drummond, K. L., Miller, C. J., Greenan, M. A., & Bauer, M. S. (2020). Comparing variations in implementation processes and influence across multiple site: What works, for whom, and how? Psychiatric Research 283, 3-9.
Disclosure of Interest: None declared
Understanding barriers to and facilitators of mobile health implementation in a multi-county digital mental health project
Authors
Mr. Eduardo Ceballos-Corro - University of California-Irvine
Mr. Jorge Castro - University of California-Irvine
Ms. Kera Mallard-Swanson - Stanford University, California
Ms. Kristina Palomares - University of California-Irvine
Dr. Nicole Stadnick - University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center, San Diego, California
Dr. Stephen M. Schueller - University of California-Irvine
Dr. Elizabeth V. Eikey - University of California San Diego
Dr. Margaret Schneider - University of California-Irvine
Dr. Kai Zheng - University of California-Irvine
Dr. Dana B. Mukamel - University of California-Irvine
Dr. Dara H. Sorkin - University of California-Irvine
Background: Mobile Health (mHealth) technologies are increasingly common due to their accessibility, affordability, and efficacy. Since 2017, a multi-county collaborative project in California, Help@Hand,1 has been exploring use of mHealth technologies. This study sought to understand barriers to and facilitators of implementing mHealth technologies within these county service settings.
Methods: Between June 2018 and March 2021, 12 site visits were completed across six counties participating in the Help@Hand project. Visits included pre- and post-implementation interviews with 43 clinic leaders and service providers interviewed at multiple points to evaluate their mHealth implementation. We developed an interview codebook based on the Exploration, Preparation, Implementation, Sustainment (EPIS) framework2 to identify barriers and facilitators within the domains. A thematic analysis of 72 transcripts with coding and analysis was completed using NVIVO.
Results: Results indicate that barriers to and facilitators of mHealth implementation coalesced around EPIS outer context, inner context, and innovation factors. Identified barriers included user interfaces being overwhelming and time-consuming to set up (innovation-context fit), concerns surrounding clients’ data privacy (outer context), and competing time demands among clinic leaders and service providers (inner organizational context environment). Primary facilitators included clients’ openness to and flexibility around using mHealth technologies (client characteristics), supportive leadership around the implementation of mHealth technologies (inner context implementation climate), and ease of using mHealth technologies as part of a client’s treatment (innovation factors).
Conclusion: Consistent with conceptualizations of mHealth as technology-enabled services,3 rather than products, these barriers and facilitators touch both technology and service components of these tools. Application of the EPIS model suggested that implementation of mHealth technologies in mental health services will be facilitated by cultivating support from leadership, openness among clients, and ease of use of the mHealth technology. In contrast, implementation is hindered by complexity of the technology, threats to client privacy, and insufficient personnel time allocated to implementers.
References
1. Help@Hand.org (2018). Retrieved April 22, 2022, from https://helpathandca.org/
2. Moullin, J. C., Dickson, K. S., Stadnick, N. A., Rabin, B., & Aarons, G. A. (2019). Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implementation Science : IS, 14(1), 1. https://doi.org/10.1186/s13012-018-0842-6
3. Mohr, D. C., Weingardt, K. R., Reddy, M., & Schueller, S. M. (2017). Three Problems With Current Digital Mental Health Research . . . and Three Things We Can Do About Them. Psychiatric services (Washington, D.C.), 68(5), 427–429. https://doi.org/10.1176/appi.ps.201600541
Disclosures of Interest: None declared
What about “It”?: Adapting implementation strategies to support psychiatric residential implementation teams in defining usable innovations
Authors
Dr. Shannon Chaplo - University of North Carolina at Chapel Hill
Dr. Robin Jenkins - University of North Carolina at Chapel Hill
Dr. Renee Boothroyd - UNC-Chapel Hill
Ms. Jessica Reed - Cornerstone
Background: The Building Bridges Initiative (BBI) and the Six Core Strategies are two complementary, trauma-informed organizational culture change frameworks that have demonstrated efficacy in preventing residential staff’s use of restraint and seclusion while improving youth and family voice into treatment. Following state-wide training by BBI/Six Core trainers, our team provided implementation support to 7 youth psychiatric residential treatment (PRTF) facilities to develop readiness and implement these evidence-based frameworks. An ongoing challenge in our support model was the principle-based nature of these frameworks that necessitated working with each PRTF implementation team to define these innovations.
Methods: To address this challenge, we used a common theory of change and implementation support practice model that allowed us to build readiness with each site to co-create how the frameworks could be operationalized at various organizational levels. This included developing a context-specific capacity assessment and tailoring a mental model of each framework. We also integrated other support roles to build readiness and plan for sustainability by involving other key players including the state child behavioral health team, framework developers, and managed care organization representatives.
Results: These strategies lead to each PRTF’s ability to create practice profiles of BBI/Six Core principles that fit their agency context. After this critical step, teams were better able to build capacity and make progress in other areas to sustain other practices in the frameworks including building workforce development and data system capacities.
Conclusion: Our work demonstrated the importance in adapting our approach to co-creation and implementation strategies to address barriers inherent to implementing interventions that, though evidence-supported, allow for great flexibility, and therefore, the need for extensive exploration processes to define the intervention within each PRTF to make it a true usable innovation.
References
1. Glisson, C., & Williams, N. J. (2015). Assessing and changing organizational social contexts for effective mental health services. Annual Review of Public Health, 36, 507-523.
2. Greenhalgh, T., Jackson, C., Shaw, S., & Janamian, T. (2016). Achieving research impact through co-creation in community-based health services: literature review and case study. The Milbank Quarterly, 94(2), 392-429.
3. Weiner, B. J. (2009). A theory of organizational readiness for change. Implementation science, 4(1), 1-9.
Disclosures of Interest: None declared
A thematic analysis of stakeholder reported structural inequities affecting people with HIV
Authors
Ms. DeAujZhane Coley - Penn Center for Mental Health, Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, Penn Medicine Nudge Unit, University of Pennsylvania Health System
Ms. Chynna Mills - Penn Center for Mental Health, Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania
Dr. Kathleen Brady - AIDS Activities Coordinating Office, Philadelphia Department of Public Health, Philadelphia, PA, Penn Center for AIDS Research, University of Pennsylvania Children’s Hospital of Philadelphia, The Wistar Institute
Dr. Robert Gross - Perelman School of Medicine, University of Pennsylvania
Dr. Florence Momplaisir - Perelman School of Medicine, University of Pennsylvania
Dr. Rinad Beidas - University of Pennsylvania
Dr. Amanda Sanchez - Perelman School of Medicine at the University of Pennsylvania
Background: Adherence and retention in care is imperative to achieve favorable health outcomes for people with HIV (PWH) (1). Challenges with retention and adherence are pronounced for marginalized communities facing structural inequities including stressors related to poverty, experiences of racism, and unequal distributions of power (2). In preparation for a hybrid type 2 effectiveness implementation trial, we aimed to understand barriers and facilitators to community-health-worker delivery of an evidence-based behavioral intervention, Managed Problem Solving (MAPS), in community settings (3). In this planning study, we prioritized understanding structural inequities affecting both patient needs and the MAPS intervention to increase the likelihood of successful implementation and reduction of inequities.
Methods: Semi-structured, qualitative interviews developed based on the Consolidated Framework for Implementation Research (CFIR) were conducted to understand stakeholder perceptions (N = 31). Stakeholders represented prescribing clinics, non-prescribing clinical team members, clinic administrators, and policymakers. Analyses were guided by an integrated approach combining identification of a priori attributes of interest (i.e., CFIR categories), and modified grounded theory. We focus on the results of a thematic analysis of two codes: structural inequities (i.e., policies, and practices leading to inequities in healthcare for individuals from marginalized identities) and dignity and trust (i.e., actions to address structural inequities).
Results: All interviews have been coded and are currently undergoing thematic analysis. Preliminary structural inequity sub-themes include access to care, stigma, and mistrust in healthcare systems. The dignity and trust code illustrates stakeholder input on ameliorating inequities by demonstrating respect for patients, building trusting relationships, enabling patient autonomy, hiring representative staff, and meeting patients where they are.
Conclusion: These results can inform implementation strategies to improve the effectiveness of EBPs in communities facing structural inequities and highlight the ways in which EBPs, such as MAPS, may employ dignity and trust to mitigate disparities in adherence and retention in HIV care.
References
1. Schaecher K. L. (2013). The importance of treatment adherence in HIV. The American journal of managed care, 19(12 Suppl), s231–s237.
2. Katelin Hoskins, PhD, Amanda L. Sanchez, PhD, Carlin Hoffacker, BA, Florence Momplaisir, MD, Robert Gross, MD, et. Al. (2021) Implementation Mapping to Increase the Use of Evidence-Based Interventions in the HIV Treatment Continuum of Care
3. Gross R, Bellamy SL, Chapman J, Han X, O’Duor J, Palmer SC, et al. Managed Problem Solving for Antiretroviral Therapy Adherence: A Randomized Trial. JAMA Internal Medicine. (2013) 173: 300-306. doi:10.1001/jamainternmed.2013.2152
Disclosures of Interest: None declared
Usability evaluation of a leadership-focused implementation strategy to support universal prevention programs in schools
Authors
Mx. Vaughan Collins - University of Washington
Dr. Catherine Corbin - University of Washington
Dr. Clayton Cook - Character Strong Organization
Dr. Mark Ehrhart - University of Central Florida
Dr. Jill Locke - University of Washington
Prof. Aaron Lyon - University of Washington
Background: While implementation strategies tend to be complex, socially-mediated interventions, this complexity can interfere with their widespread application to improve evidence-based practice use in schools. Usability evaluation of implementation strategies is critical to ensure that they align with user needs, expectations, and contextual constraints.3 In service of the iterative development of Helping Educational Leaders Mobilize Evidence (HELM), an adaptation of the evidence-based organizational intervention Leadership and Organizational Change for Implementation (LOCI),1 2 this presentation will focus on usability testing of the implementation strategy.
Methods: The Cognitive Walkthrough for Implementation Strategies (CWIS), a mixed-methods approach to evaluate implementation strategy usability,3 was utilized to evaluate a HELM strategy prototype. Preconditions were determined along with a hierarchical task analysis and task prioritization process before converting tasks to scenarios. Facilitators conducted three group testing sessions, guiding elementary school principals through the intervention in a leader (n = 10), 4 scenarios and 13 subtasks, or coach role (n = 5), 3 scenarios and 8 subtasks. Users reported their anticipated success of completing each subtask and provided qualitative rationales. Following sessions, users completed a quantitative assessment of strategy usability (e.g., Intervention Usability Scale [IUS]).4
Results: To identify, classify, and prioritize usability issues, users’ subtask ratings (scale 1-4) were calculated. Leaders tended to rate their anticipated success of knowing what to do (mean = 3.70) and knowing they did it successfully (mean = 3.70) higher than doing it (mean = 3.66) whereas coaches tended to rate knowing what to do (mean = 3.75) and doing it (mean = 3.83) higher than knowing they did it successfully (mean = 3.59). Leaders’ IUS ratings (scale 0-100) ranged from 52.5 to 100 with a mean of 77.8 (median = 82.5; SD = 15.5) while coaches’ ranged from 80 to 100 with a mean of 87.5 (median = 85, SD = 7.9), demonstrating the consistent positive opinions of HELM’s overall usability. Redesign solutions will be finalized by the presentation.
Conclusion: The CWIS methodology is expected to yield a more usable HELM strategy and information that may be more broadly applicable to the development of complex implementation strategies in real-world settings like schools.
References
1. Aarons, G., Ehrhart, M. G., Farahnak, L. R., & Hurlburt, M. S. (2015). Leadership and Organizational Change for Implementation (LOCI): A randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implementation Science, 10(1), 11. https://doi.org/10.1186/s13012-014-0192-y
2. Aarons, G. A., Ehrhart, M. G., Moullin, J. C., Torres, E. M., & Green, A. E. (2017). Testing the Leadership and Organizational Change for Implementation (LOCI) intervention in substance abuse treatment: A cluster randomized trial study protocol. Implementation Science, 12, 29. https://doi.org/10.1186/s13012-017-0562-3
3. Lyon, A. R., Coifman, J., Cook, H., McRee, E., Liu, F. F., Ludwig, K., Dorsey, S., Koerner, K., Munson, S. A., & McCauley, E. (2021). The Cognitive Walkthrough for Implementation Strategies (CWIS): A pragmatic method for assessing implementation strategy usability. Implementation Science Communications, 2(1), 78. https://doi.org/10.1186/s43058-021-00183-0
4. Lyon, A. R., Pullman, M. D., Jacobson, J., Osterhage, K., Al Achkar, M., Renn, B. N., Munson, S. A., & Areán, P. A. (2021). Assessing the usability of complex psychosocial interventions, The Intervention Usability Scale. Implementation Research and Practice. 2(1). https://doi.org/10.1177/2633489520987828
Disclosure of Interest: Vaughan Collins reports being employed by Dr. Aaron Lyon (Associate Editor, Implementation Research and Practice) on two research projects, including the one that this poster is based on. Vaughan Collins also reports being employed on a research training program of which Dr. Lisa Sanetti (Editorial Board, Implementation Research and Practice) is a paid Core Faculty member who dictates the work. Vaughan Collins also reports being employed on a research project in which Dr. Mark Ehrhardt (Editorial Board, Implementation Research and Practice) consults, including the one that this poster is based on. Vaughan Collins also reports being employed on a research training program of which Dr. Lawrence Palinkas (Associate Editor, Implementation Research and Practice), Cara Lewis (Co-founding Editor, Implementation Research and Practice), Enola Proctor (Editorial Board, Implementation Research and Practice), Ross Brownson (Editorial Board, Implementation Research and Practice), and Byron Powell (Editorial Board, Implementation Research and Practice) served as consultants.
Preliminary effectiveness of a simulated patient implementation strategy to improve the integration of screening for mental health within HIV settings in Kenya
Authors
Ms. Tessa Concepcion - University of Washington School of Public Health
Mr. Peter Mogere - Partners in Health and Research Development
Dr. Kenneth Ngure - Partners in Health and Research Development
Dr. Njoroge Mwathi - Partners in Health and Research Development
Mr. Roy Njiru - Partners in Health and Research Development
Mr. Boaz Kipkorir - Partners in Health and Research Development
Dr. Pamela Kohler - University of Washington
Dr. Bradley H Wagenaar - University of Washington School of Public Health
Dr. Shannon Dorsey - University of Washington
Ms. Catherine Kiptinness - Partners in Health and Research Development
Ms. Emmah Owidi - Partners in Health and Research Development
Mr. Gakuo Maina - Partners in Health and Research Development
Dr. Jennifer Velloza - University of California San Francisco
Background: Kenyan adolescent girls and young women (AGYW) experience a dual burden of HIV and common mental disorders (CMD). HIV clinics are a key entry point for AGYW; however, rates of screening and referral for CMDs are low. We adapted and piloted an effective provider training implementation strategy, simulated patient encounters (SPEs), in a Kenyan HIV clinic.
Methods: This pilot study was conducted in a public HIV clinic in Thika, Kenya from January to November 2021. The SPE implementation strategy included case script development from prior qualitative work, patient actor training, and four standardized mock clinical encounters followed by provider competency assessment. We conducted an interrupted time series analysis using abstracted HIV and CMD data from AGYW ages 16-25 years visiting the clinic 7-months before and 3-months after SPE training. We used generalized linear models to assess changes in CMD screening rates after training.
Results: We conducted the three-day SPE training with ten HIV providers in August 2021. Provider competency ratings improved across four mock encounters (mean score from 8.1 to 13.7 between first and fourth encounters). We abstracted all medical records (n = 1154) including from 888 (76%) AGYW seeking HIV treatment, 243 (21%) seeking prevention services, and 34 (3%) seeking other services. CMD screening rates increased immediately following training from 8.25% to 20.51% (relative risk [RR] = 2.32, 95% confidence interval [CI] = 1.19-4.52, p = 0.01). The trend of CMD screening rates over time was 11% higher in the 7-months pre-SPE than the 3-months post-SPE training period (95% CI = 1.07-1.22, p < 0.01). Finally, 0.9% of all pre-SPE screens resulted in referral versus 4.7% of post-SPE screens (p = 0.07).
Conclusion: The SPE model is a promising implementation strategy for improving HIV provider competencies and CMD service delivery for adolescents in HIV clinics. Future research is needed to explore long-term effects on adolescent clinical outcomes in larger trials.
References
1. Chibanda D. (2017). Depression and HIV: integrated care towards 90-90-90. International health, 9(2), 77–79. https://doi.org/10.1093/inthealth/ihw058
2. Larsen, A., Abuna, F., Owiti, G., Kemunto, V., Sila, J., Wilson, K. S., Owens, T., Pintye, J., Richardson, B. A., Kinuthia, J., John-Stewart, G., & Kohler, P. (2022). Improving Quality of PrEP Counseling for Adolescent Girls and Young Women in Kenya With Standardized Patient Actors: A Dose-Response Analysis. Journal of acquired immune deficiency syndromes (1999), 89(1), 34–39. https://doi.org/10.1097/QAI.0000000000002814
3. Mall, S., Sorsdahl, K., Struthers, H., & Joska, J. A. (2013). Mental health in primary human immunodeficiency virus care in South Africa: a study of provider knowledge, attitudes, and practice. The Journal of nervous and mental disease, 201(3), 196–201. https://doi.org/10.1097/NMD.0b013e3182845c24
4. Stockton, M. A., Udedi, M., Kulisewa, K., Hosseinipour, M. C., Gaynes, B. N., Mphonda, S. M., Maselko, J., Pettifor, A. E., Verhey, R., Chibanda, D., Lapidos-Salaiz, I., & Pence, B. W. (2020). The impact of an integrated depression and HIV treatment program on mental health and HIV care outcomes among people newly initiating antiretroviral therapy in Malawi. PloS one, 15(5), e0231872. https://doi.org/10.1371/journal.pone.0231872
5. Velloza, J., Celum, C., Haberer, J. E., Ngure, K., Irungu, E., Mugo, N., Baeten, J. M., Heffron, R., & Partners Demonstration Project Team (2017). Depression and ART Initiation Among HIV Serodiscordant Couples in Kenya and Uganda. AIDS and behavior, 21(8), 2509–2518. https://doi.org/10.1007/s10461-017-1829-z
6. Velloza, J., Heffron, R., Amico, K. R., Rowhani-Rahbar, A., Hughes, J. P., Li, M., Dye, B. J., Celum, C., Bekker, L. G., Grant, R. M., & HPTN 067/ADAPT Study Team (2020). The Effect of Depression on Adherence to HIV Pre-exposure Prophylaxis Among High-Risk South African Women in HPTN 067/ADAPT. AIDS and behavior, 24(7), 2178–2187. https://doi.org/10.1007/s10461-020-02783-8
7. Weaver, M., & Erby, L. (2012). Standardized patients: a promising tool for health education and health promotion. Health promotion practice, 13(2), 169–174. https://doi.org/10.1177/1524839911432006
8. Xu, T., de Almeida Neto, A. C., & Moles, R. J. (2012). A systematic review of simulated-patient methods used in community pharmacy to assess the provision of non-prescription medicines. The International journal of pharmacy practice, 20(5), 307–319. https://doi.org/10.1111/j.2042-7174.2012.00201.x
Disclosures of Interest: None declared
Crowd sourcing a remote research team to support junior implementation science researchers’ skills development and networking
Authors
Dr. Erika Crable - Department of Psychiatry, University of California, San Diego; Child & Adolescent Services Research Center; University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
Dr. Daisy Le - Department of Policy, Populations and Systems, School of Nursing; Department of Prevention and Community Health, Milken Institute School of Public Health, The George Washington University
Dr. Ashley Helle - Department of Psychological Sciences, University of Missouri
Dr. Patrick Kierkegaard - CRUK Convergence Science Centre, Institute for Cancer Research & Imperial College London; NIHR London In Vitro Diagnostics Co-operative, Department of Surgery and Cancer, Imperial College London
Mr. Samuel Packard - Columbia University Mailman School of Public Health
Ms. Samreen Fathima - Northern Light Health, Eastern Maine Medical Center, Clinical Research
Dr. Allyson Varley - Health Services Research & Development, Birmingham VA Health System
Dr. LauraEllen Ashcraft - Center for Health Equity Research and Promotion, Corporal Crescenz VA Medical Center
Background: Team science (TS), the collaborative work of scientists conducting interdependent tasks to actualize a shared goal, is increasingly common in implementation science (IS) and critical to the advancement of early career researchers (ECRs) (Dickson, 2021; Cree-Green, 2020). Clear expectations, communication, and goal setting are key strategies to facilitate TS (Aarons, 2020). TS promotes innovation, productivity, networking, and interdisciplinary mentoring. Limited networking and research opportunities for ECRs during the COVID-19 pandemic (e.g., cancelled/ineffective conferences, hiring freezes) highlighted the need for clear processes and practical applications of TS strategies to support ECRs outside of formal institutional supports. We present a case study describing processes and outcomes of crowd sourcing a research team of ECRs on Twitter, with the goal of facilitating IS and methodological skills development, enhancing manuscript productivity, and broadening networks.
Methods: In 2020, an ECR tweeted a call for collaborators to conduct qualitative analyses for 2 behavioral health IS projects. A multi-institutional and cross-disciplinary team of 8 ECRs, with varying levels of expertise in behavioral health, IS, and qualitative methods, formed, including 6 women and 2 men working across 8 research institutes.
Results: During the first video call, ECRs discussed their research expertise, motivation and capacity for in-kind collaboration. We co-developed a formal collaborator agreement detailing key strategies for TS: division of labor expectations, communication processes, and shared goals related to project deliverables and skill growth. Researchers with IS and qualitative expertise facilitated multiple skills-building sessions to support analysis for 2 IS manuscripts. TS challenges include tailoring interdependent research tasks to individual skills, identifying software to facilitate cross-institution analysis, and communicating across time zones.
Conclusions: Pandemic-initiated telework culture amplified opportunities for multi-institutional TS, enabling ECRs to gain skills, publish, and networks. We provide a strategic approach and formal agreement template for ECRs to develop productive, virtual IS collaborations.
References
1. Aarons, G. A., Reeder, K., Miller, C. J., & Stadnick, N. A. (2019). Identifying strategies to promote team science in dissemination and Implementation Research. Journal of Clinical and Translational Science, 4(3), 180–187. https://doi.org/10.1017/cts.2019.413
2. Cree-Green, M., Carreau, A.-M., Davis, S. M., Frohnert, B. I., Kaar, J. L., Ma, N. S., Nokoff, N. J., Reusch, J. E., Simon, S. L., & Nadeau, K. J. (2020). Peer mentoring for professional and personal growth in academic medicine. Journal of Investigative Medicine, 68(6), 1128–1134. https://doi.org/10.1136/jim-2020-001391
3. Dickson, K. S., Glass, J. E., Barnett, M. L., Graham, A. K., Powell, B. J., & Stadnick, N. A. (2021). Value of peer mentoring for early career professional, research, and personal development: A case study of implementation scientists. Journal of Clinical and Translational Science, 5(1). https://doi.org/10.1017/cts.2021.776
Disclosures of Interest: Ashley C. Helle’s work was supported by NIH K08AA028543 (PI: Helle).
A cost evaluation protocol for a quasi-experimental study on multidisciplinary pain clinics for veterans with chronic pain
Authors
Dr. Sarah Daniels - VA Center for Innovation to Implementation (Ci2i), VA Palo Alto Healthcare System
Ms. Shayna Cave - VA Center for Innovation to Implementation (Ci2i), VA Palo Alto Healthcare System
Dr. Todd Wagner - VA Health Economics Research Center, Stanford University
Ms. Taryn Perez - Center for Innovation to Implementation (Ci2i), VA Palo Alto Healthcare System
Dr. William Becker - Pain Research, Informatics, Multimorbidities and Education (PRIME) Center, VA Connecticut Healthcare System, Yale School of Medicine
Dr. Amanda Midboe - VA Center for Innovation to Implementation, Stanford University School of Medicine
Background: Opioid-related morbidity and mortality affect over 2 million Americans, and opioid use disorder and fatal overdoses were estimated to cost $1.02 trillion in 2017 alone1. Veterans are at increased risk and in need of guideline-concordant interventions that address risky opioid regimens in the context of treatment for chronic pain as well as opioid use disorder. We will implement an evidenced-based, multi-disciplinary, integrated pain clinics to address these needs among Veterans2. This protocol outlines a comprehensive cost evaluation of a multi-disciplinary pain clinic from the perspective of the VA healthcare system.
Methods: A budgetary impact analysis will estimate the short-term economic effects of the clinics, with implementation, intervention, and downstream costs3. Implementation costs will be assessed with manually logged, hourly contribution data describing implementation facilitation activities. Intervention and downstream costs will be calculated using workload data from the Managerial Cost Accounting System (MCA), a VA activity-based cost accounting system.
Results: Implementation costs will be calculated by multiplying an hourly wage by the number of logged hours of implementation activities. For intervention costs, a difference-in-differences model will be used to compare treated patients to a control group of clinically similar patients undergoing the standard of care at neighboring sites. Control VA sites will be selected based on location, presence of a traditional pain clinic, and parallel pretrend assessment of opioid tapering. Downstream costs will be computed using a similar difference-in-differences approach.
Conclusion: Incorporating both implementation and intervention costs into this evaluation provides the rare opportunity to assess the entirety of budget impacts for this study, and yields a reproducible method for costing future VHA implementation trials. Underlying cost estimates for both starting-up and employing complex interventions can motivate decisionmakers to maintain and expand effective programming.
References
1. Florence, C., Luo, F., & Rice, K. (2020). The economic burden of opioid use disorder and fatal opioid overdose in the United States, 2017. Drug and Alcohol Dependence, 108350. https://doi.org/10.1016/j.drugalcdep.2020.108350
2. Becker, W. C., Edmond, S. N., Cervone, D. J., Manhapra, A., Sellinger, J. J., Moore, B. A., & Edens, E. L. (2017). Evaluation of an Integrated, Multidisciplinary Program to Address Unsafe Use of Opioids Prescribed for Pain. Pain Medicine, 19(7), 1419–1424. https://doi.org/10.1093/pm/pnx041
3. Gold, H. T., McDermott, C., Hoomans, T., & Wagner, T. H. (2022). Cost data in Implementation Science: categories and approaches to costing. Implementation Science: IS, 17, 11. https://doi.org/10.1186/s13012-021-01172-6
Disclosures of Interest: None declared
Implementation of nudge types to encourage use of complementary and integrative health therapies in the Veterans Health Administration
Authors
Dr. Claudia Der-Martirosian - US Department of Veterans Affairs
Ms. Marlena Shin - US Department of Veterans Affairs
Dr. Hannah Gelman - US Department of Veterans Affairs
Ms. Michelle Upham - US Department of Veterans Affairs
Ms. Briana Lott - US Department of Veterans Affairs
Dr. Steven Zeliadt - US Department of Veterans Affairs
Dr. Stephanie Taylor - US Department of Veterans Affairs
Dr. A. Rani Elwy - US Department of Veterans Affairs
Background: The Comprehensive Addiction and Recovery Act mandates complementary and integrative health (CIH) therapy provision in the Veterans Health Administration (VA). In 2018, 18 VA medical centers focused on implementing practitioner-delivered (acupuncture, chiropractic care, therapeutic massage) and self-care (yoga, Tai Chi/Qigong, meditation/mindfulness) evidence-based CIH therapies. However, little is known about the best practices for encouraging patients to use these therapies. Since much of our work is about implementation practitioners in CIH, our objective was to identify strategies implemented by the 18 sites to encourage (i.e., nudge) Veterans to use CIH therapies. We identified different nudge types to self-care, practitioner-delivered, or both CIH therapies (i.e., dual care).
Methods: Since 2018, CIH program/clinical directors at each site participated in site visits/calls with the study team, where we learned about implementation approaches used to nudge patients to CIH therapies. We first categorized these strategies into specific nudge types, and then, we used a consensus-approach to map these strategies to the 73 Expert Recommendations for Implementing Change (ERIC) strategies.
Results: We identified eight specific nudge types: Gateway, Incentive, On Pathway, Off Pathway, Referral, Site Structure, Advertising/Marketing/Outreach, Availability of Resources. More frequently used nudge types included Referral and Advertising/Marketing/Outreach. Specific examples of frequently used nudges will be presented for each type of CIH therapy (practitioner-delivered, self-care, or dual care). We also identified specific ERIC strategies that map to the eight nudge types. For example, several ERIC strategies, such as develop and distribute educational materials, conduct educational meetings, outreach visits, and use of mass media, mapped to the Advertising/Marketing/Outreach nudge type (e.g., use of flyers, website, digital waiting room information screens, etc. to encourage Veterans to use CIH therapies).
Conclusion: The study findings provide valuable information about implementation practices on how to nudge patients to engage in self-care, as well as dual care, CIH therapies.
References
1. Elwy, A. R., Johnston, J., M., Bormann, J.E., et al. (2014). A systematic scoping review of complementary and alternative medicine mind and body practices to improve the health of veterans and military personnel. Medical Care, 52, S70-82. doi: 10.1097/MLR.0000000000000228
2. Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., Kirchner, J. (2015) E. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10:21. doi: 10.1186/s13012-015-0209-1
2. Zeliadt, S. B., Coggeshall, S., Gelman, H., Shin, M. H., Elwy, A. R., Bokhour, B. G., Taylor, S.L. (2020). Assessing the relative effectiveness of combining self-care with practitioner-delivered complementary and integrative health therapies to improve pain in a pragmatic trial. Pain Medicine. 21(S2), S100–S109. doi: 10.1093/pm/pnaa349
Disclosures of Interest: None declared
Using ADAPT-ITT and FRAME-IS to guide adaptation of an evidence-based family home visiting intervention in Sierra Leone
Authors
Dr. Alethea Desrosiers - Brown University
Ms. Carolyn Schafer - North Western University
Ms. Julia D'Ambrosio - Boston University
Ms. Musu Jambai - Caritas-Freetown
Dr. Rebecca Esliker - University of Makeni
Dr. Theresa S. Betancourt - Boston College
Background: Evidence-based interventions (EBIs) to reduce family violence and improve early childhood development are sparse in resource-constrained settings like Sierra Leone. Given the costs and time necessary to develop, implement, and evaluate EBIs, culturally adapting an existing EBI may accelerate implementation for a new cultural context. This study sought to systematically adapt and track adaptations of an evidence-based family home visiting intervention—the Family Strengthening Intervention for Early Childhood Development (FSI-ECD)—for vulnerable families in Sierra Leone. We applied a modified ADAPT-ITT framework to guide FSI-ECD adaptation and the FRAME-IS to track adaptations. ADAPT-ITT draws on community-based participatory research approaches to engage key stakeholders throughout the adaptation process. FRAME-IS categorizes adaptations into different types (e.g., content, context, training) and levels of modifications (e.g., patient, group).
Methods: We convened two rounds of focus groups with ECD experts from Sierra Leone’s Ministry of Education and UNICEF (N = 8; 6 females/2 males) and two Community Advisory Board meetings (N = 10; 9 females/1 male) between February and May 2021 to obtain feedback on FSI-ECD module content. Adaptations were independently coded by two research assistants using code definitions in the FRAME-IS. A final code was assigned to each modification after discussion.
Results: Qualitative data analysis supported 27 content modifications, which were coded as ‘tweaking’ under the FRAME-IS framework. Other key modifications included changing images and vignette names to represent the cultural context and adding healthcare system and hygiene information consistent with Government of Sierra Leone suggested practices. We also made three context modifications, including translation of the manual. We will continue tracking additional adaptations during FSI-ECD implementation.
Conclusion: Findings suggest that ADAPT-ITT and FRAME-IS are feasible and easily applied in an LMIC. Systematically tracking adaptations may help accelerate the adaptation process for EBIs across cultures and contexts and elucidate the relationship between adaptations and implementation outcomes.
References
1. Betancourt, T.S., Jensen, S.G., Barnhart, D.A., Brennan, R.T., Murray, S.M., Yousafzai, A.K., et al. (2020). Promoting parent-child relationships and preventing violence via home-visiting: a pre-post cluster randomised trial among Rwandan families linked to social protection programmes. BMC Public Health, 20(1), 621.
2. Munro-Kramer, M.L., Rominski, S.D., Seidu, A.A, Darteh, E.K.M., Huhman, A., & Stephenson, R. (2020). Adapting a Sexual Violence Primary Prevention Program to Ghana Utilizing the ADAPT-ITT Framework.Violence Against Women, 26(1), 66-88.
3. Miller, C., Barnett, M.L., Baumann, A.A., Gutner, C.A., & Whitsey-Stirman, S. (2021). The FRAME-IS: A framework for documenting modifications to implementation strategies in healthcare. Implementation Science, 16, 36.
Disclosures of Interest: None declared
Evaluating the impact of implementation factors on mental health task-sharing: A causal analysis
Authors
Dr. Saloni Dev - Tufts University
Dr. John Griffith - Bouve College of Health Sciences, Northeastern University
Dr. Collette Ncube - Institute for Health Equity and Social Justice Research, Northeastern University
Dr. Vikram Patel - Harvard University TH Chan School of Public Health / Harvard Medical School
Dr. Alisa Lincoln - Institute for Health Equity and Social Justice Research, Northeastern University
Background: Mental health task-sharing is an efficacious strategy for bridging the global mental health treatment gap. However, more research is needed to enhance its implementation in routine care settings. This study aims to evaluate the causal association between implementation factors and participant outcomes within a mental health task-sharing intervention to strengthen the implementation-to-outcome link within global mental health.
Methods: This is a secondary analysis of cohort study data from the Program for Improving Mental Health Care (PRIME) implemented in Sehore district, India whereby trained non-specialist health workers delivered manualized treatment for depression and alcohol use disorder (AUD). We used propensity scores and inverse probability of treatment weights to study the causal impact of (1) treatment attendance at participant level (n = 430), and (2) therapy quality at the provider level (n = 6) on participants’ symptom severity at 3- (midline) and 12-month (endline) follow up.
Results: Within the depression cohort, higher treatment session attendance led to lower symptom severity at midline but not at endline. Furthermore, we found a null impact of therapy quality on symptom severity at both time-points. In the AUD cohort, we found a null impact of treatment session attendance on symptom severity. Lastly, an imbalance of confounders rendered an analysis of the causal impact of therapy quality on AUD symptom severity irrelevant.
Conclusion: Our findings have implications for enhancing treatment session attendance among those with depression within task-sharing based mental health interventions. Further research is needed to understand the impact of therapy quality on participant outcomes for depression and AUD.
References
1. World Health Organization (2009). Mental health systems in selected low- and middle-income countries: a WHO-AIMS cross-national analysis. Retrieved from: https://apps.who.int/iris/bitstream/handle/10665/44151/9789241547741_eng.pdf?sequence = 1&isAllowed = y
2. Van Ginneken, N., Tharyan, P., Lewin, S., Rao, G. N., Meera, S. M., Pian, J., … & Patel, V. (2013). Non-specialist health worker interventions for the care of mental, neurological and substance-abuse disorders in low-and middle-income countries. Cochrane database of systematic reviews, (11).
3. Shidhaye, R., Shrivastava, S., Murhar, V., Samudre, S., Ahuja, S., Ramaswamy, R., & Patel, V. (2016). Development and piloting of a plan for integrating mental health in primary care in Sehore district, Madhya Pradesh, India. The British journal of psychiatry, 208(s56), s13-s20.
Disclosures of Interest: None declared
Implementing evidence-based leading practices to recognize and respond to Postpartum Hemorrhage (PPH) cases across the Military Health System (MHS)
Authors
Dr. Susanna Didrickson - Defense Health Agency, Office of Clinical Support, Directorate of Medical Affairs
Dr. Faye Curran - Defense Health Agency, Research & Engineering Directorate
Ms. Daniella Kanyer - Defense Health Agency, Research & Engineering Directorate
Background: The Military Health System (MHS) previously lacked practices for implementing clinical recommendations across its large network of medical facilities. However, this system provides opportunities for shaping an effective implementation framework. In 2020, the Research & Engineering Implementation Science Branch (ISB) at the Defense Health Agency (DHA), in partnership with Women’s Health subject matter experts, leveraged implementation science principles to facilitate the standardization of a multi-component postpartum hemorrhage (PPH) patient safety bundle at 12 Military Treatment Facilities (MTFs). PPH is the leading cause of preventable maternal death worldwide (World Health Organization, 2017). Recently, MHS rates were at 7.4 deaths per 100,000 live births (TRICARE, 2021). While preventing PPH entirely is impossible, prompt assessment, recognition, and communication of maternal hemorrhage risk, in accordance with Alliance for Innovation on Maternal Health (AIM) guidelines, may help reduce maternal morbidity and mortality and improve patient outcomes.
Methods: From December 2020 to October 2021, multiple implementation strategies were used to execute the bundle across MTFs including: identifying and coaching champions at MTFs, hosting peer-to-peer work groups, creating informative newsletters, developing standardized resource guides and trainings, and creating a centralized web platform to share information. Additionally, a self-report instrument was designed to regularly solicit MTFs’ implementation status and support MTFs’ compliance with the PPH bundle.
Results: In December of 2020, a baseline implementation assessment measured an average compliance of 63% across all MTFs. By the end of the campaign in October 2021, all MTFs reached implementation scores above the campaign’s pre-determined target– 80%, with an overall MHS-wide average of 97%.
Conclusions: MTF self-report data and lessons from the implementation project helped inform a current large-scale implementation across an additional 31 MTFs worldwide. A next step is to understand the mediating influence of this implementation on relevant clinical outcomes and gain a better understanding of real-world impact.
References
1. Agency for Healthcare Research and Quality (AHRQ). (n.d.). About TeamSTEPPS. Retrieved March 2, 2022, from https://www.ahrq.gov/teamstepps/about-teamstepps/index.html
2. Military Health System. (n.d.). Postpartum Hemorrhage. Retrieved March 2, 2022, from https://www.health.mil/Military-Health-Topics/Total-Force-Fitness/Preventive-Health/Womens-Health/Postpartum-Hemorrhage
3. TRICARE Newsroom. (2021, November 24). How Health Care for New Mothers is Improving Across the MHS. Retrieved March 21, 2022, from https://newsroom.tricare.mil/Local-News/Article/2854375/how-health-care-for-new-mothers-is-improving-across-the-mhs
4. World Health Organization. (2017, June 23). WHO Recommendations on Prevention and Treatment of Postpartum Hemorrhage and the WOMAN Trial. Retrieved March 2, 2022, from https://www.who.int/reproductivehealth/topics/maternal_perinatal/pph-woman-trial/en/
Disclosures of Interest: None declared
Assessing community-level parent support networks: Leveraging community partners to co-create and implement a social network analysis to support program implementation and scale-up
Authors
Ms. Sandra Diehl - The Impact Center, Frank Porter Graham Child Development Institute, University of North Carolina at Chapel Hill
Dr. Devon Minch - The Impact Center, Frank Porter Graham Child Development Institute, University of North Carolina at Chapel Hill
Ms. Margaret Sullivan - Department of School Psychology, University of Minnesota
Ms. Lynne Carter - Pitt County Health Department, Greenville, NC
Ms. Jennifer Hardee - Pitt County Health Department, Greenville, NC
Background: Social Network Analysis (SNA) contributes detailed data and network maps to improve program implementation and scale-up (Valente, 2015). Network maps illustrate partnerships and behavior across key people, community organizations and systems (Glegg, 2019). Collaborating with community members to co-create survey administration and response processes ensures successful approaches that are most relevant to local contexts (CCIS, 2021). A community-academic partnership co-created and piloted the process, methods and real world application of an SNA to assess parenting supports to enhance Triple P implementation in one NC region.
Methods: Academic partners collaboratively developed the content, methods and real-world application of a survey to assess parenting support with a local health department (LHD) leading Triple P implementation efforts in the region. To ensure contextual responsiveness to family needs, academic partners (1) worked with the LHD to identify childcare center families as the population of interest, (2) collaboratively developed an assessment plan and timeline, and (3) co-created the survey and participant recruitment strategies. Partners will co-develop a visual map of network connections and network metrics using R studio, descriptive data using SPSS, and recommendations by July 31st 2022.
Results: The SNA will be used in connection with provider locations and family service data to improve family access to and participation in Triple P services. Key people, places and platforms in the community will be identified and leveraged for marketing, media, collaboration and program recruitment efforts.
Conclusion: This presentation will describe the process and application for co-creating SNA as part of implementation support for improved program reach, collaboration and increased awareness, access and participation in Triple P services among families in the community. Co-creating the design and implementation of SNA ensures responsive approaches and builds trust with communities. This process can be applied to other communities and programs in the future.
References
1. Valente, T.W., Palinkas L.A., Czaja S., Chu K.H., Brown C.H. (2015). Social Network Analysis for Program Implementation. PLoS ONE 10(6), 1-18. e0131712. doi:10.1371/journal.pone.0131712
2. Glegg, S.M.N., Jenkins, E., Kothari, A. (2019). How the study of networks informs knowledge translation and implementation: a scoping review. Implementation Science 14(34), 1-26. https://doi.org/10.1186/s13012-019-0879-1
3. Consortium for Cancer Implementation Science (2021, July). Resources for stakeholder and community engagement. Community Participation Capacity Building Task Group of CCSI. Retrieved from: https://cancercontrol.cancer.gov/sites/default/files/2021-08/CCIS_Engagement-Bibliography_080931_508.pdf
Disclosures of Interest: None declared
Elevating community partner voices in the redesign of an online learning website for improved access to virtual implementation support for Triple P scale-up
Authors
Ms. Sandra Diehl - The Impact Center, Frank Porter Graham Child Development Institute, University of North Carolina at Chapel Hill
Ms. Marais Pletsch - The Impact Center, Frank Porter Graham Child Development Institute, University of North Carolina at Chapel Hill
Ms. Julie Chin - The Impact Center, Frank Porter Graham Child Development Institute, University of North Carolina at Chapel Hill
Ms. Capri McDonald - The Impact Center at FPG Child Development Institute, University of North Carolina at Chapel Hill
Background: Our university-based workgroup provides intermediary implementation support that applies best practices while honoring local context and nurturing community-engaged partnerships. Support delivery for Triple P scale-up shifted from primarily in-person support to remote during the pandemic. Redeveloping our existing online learning website ‘The Sim Lab’ became a top priority in order to (1) build system partners’ knowledge and skills around effective implementation, (2) make available self-management tools, and (3) promote multi-sector support networks within statewide systems. Partnering with the learning community to ensure The Sim Lab was user-informed from the beginning of the redesign process was a vital step to ensure its usability, sustainability, and effectiveness.
Methods: We convened and facilitated 3 semi-structured 1-hour focus groups (n = 14) with varied user audiences to co-create the redesign of The Sim Lab. User audiences represented multiple system levels including funders, intermediary organizations, regional Triple P implementation teams, intervention developers, and state leadership. The focus groups gathered feedback over 2 development phases and produced a wish-list outlining desired changes related to accessibility, navigation, functionality, and relevance. Team members summarized group feedback into actionable redesign goals.
Results: The focus group summaries produced themes related to met and unmet needs. Key themes include (1) Accessibility: participants desired more visuals and less words and overall, less academic jargon; (2) Navigation: participants voiced a need for more descriptive labels in navigation and found the order of the current website navigation too complicated; (3) Functionality: participants had difficulty easily seeking the information needed and gave valuable suggestions on website layout; and (4) Relevance: participants helped us to realize connecting implementation tools and target audiences and embedding multi-sector efforts into the site was fundamental.
Conclusion: Using a co-creation approach ensures greater application of an equitable online resource, enabling us to increase the awareness, reach, and accessibility of implementation support.
References
1. Facilitating Power. (2019). Spectrum of Family & Community Engagement for Educational Equity. https://www.oregon.gov/ode/students-and-family/equity/AfricanAmericanBlackStudentEducation/Documents/Spectrum-ofFamily-Community-Engagement-For-Educational-Equity.pdf
2. Washington State Department of Health. (n.d.). Community Engagement Guide. 30. https://www.doh.wa.gov/Portals/1/Documents/1000/CommEngageGuide.pdf
CDC. (2015). Chapter 1: What Is Community Engagement? | Principles of Community Engagement | ATSDR. https://www.atsdr.cdc.gov/communityengagement/pce_what.html
Disclosures of Interest: None declared
Supporting Providers and Reaching Kids (SPARK): The importance of community partnerships to promote engagement and utilization of a tablet-based provider tool
Authors
Dr. Hannah Espeleta - Medical University of Sou
Dr. Tatiana Davidson - Medical University of South Carolina
Ms. Hannah Sebald - Medical University of South Carolina
Ms. Nicole Litvitskiy - Medical University of South Carolina
Ms. Gabriela Becerra - Medical University of South Carolina
Dr. Rochelle Hanson - Medical University of South Carolina
Ms. Tonya Hazelton - Medical University of South Carolina
Dr. Kenneth Ruggiero - Medical University of South Carolina
Dr. Leigh Ridings - Medical University of South Carolina
Background: Child mental health care varies widely in community settings. Technology-based tools can improve the effectiveness of evidence-based practice delivery, targeting provider training, fidelity monitoring, and youth engagement. Partnered research can attenuate gaps between research and practice and improve quality of care; yet barriers may affect establishing and maintaining strong community partnerships. We developed a technology-delivered support tool, SPARK, comprised of interactive components to improve TF-CBT delivery. SPARK is being tested in a multi-site Hybrid I trial to establish strategies for engaging partners while evaluating intervention uptake, efficacy, and integration into practice. This presentation will describe two unique perspectives of the study: 1) processes and challenges in establishing community partnerships and 2) provider experience and SPARK utilization in clinical practice.
Methods: Research activities are discussed with community clinics and agency leaders identify site champions and providers. Enrolled providers (N = 95) are randomized to TF-CBT delivery with or without SPARK. Providers (N = 46) integrate the tool based on clinical preference. Families (N = 232) complete assessments at treatment initiation and four follow-up timepoints. Providers complete interviews about their experiences with the research design and toolkit.
Results: Providers and champions are excited to partner with the study. Providers in community-based settings have more concerns about implementation than providers in specialty settings. Strategies to maintain and promote partnerships will be discussed. The study has maintained high retention of families (65% at 12 months) and providers (62%). Providers demonstrated variable SPARK usage across treatment components. Thematic interviews indicate that SPARK enhanced TF-CBT delivery, highlighting usefulness of tools that explain session goals, facilitate fidelity, and promote session preparation.
Conclusion: Strong champions, shared goals, and ongoing communication contributed to successful partnerships and engagement. Provider engagement with SPARK is encouraging. If SPARK efficacy is strong, it supports the potential of technology to promote treatment fidelity and effectiveness for children and families.
References
1. Anton, M. T., Ridings, L. E., Hanson, R., Davidson, T., Saunders, B., Price, M., … & Ruggiero, K. J. (2020). Hybrid type 1 randomized controlled trial of a tablet-based application to improve quality of care in child mental health treatment. Contemporary clinical trials, 94, 106010.
2. Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., Hinder, S., … & Shaw, S. (2017). Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. Journal of medical Internet research, 19(11), e8775.
3. Lattie, E. G., Nicholas, J., Knapp, A. A., Skerl, J. J., Kaiser, S. M., & Mohr, D. C. (2020). Opportunities for and tensions surrounding the use of technology-enabled mental health services in community mental health care. Administration and Policy in Mental Health and Mental Health Services Research, 47(1), 138-149.
Disclosures of Interest: None declared
Assessing pre-implementation factors for acute care settings: A qualitative examination of trauma center readiness to implement stepped care mental health programming
Authors
Dr. Hannah Espeleta - Medical University of South Carolina
Ms. Kristen Higgins - Medical University of South Carolina
Ms. Olivia Bravoco - Medical University of South Carolina
Dr. Leigh Ridings - Medical University of South Carolina
Dr. Kenneth Ruggiero - Medical University of South Carolina
Dr. Tatiana Davidson - Medical University of South Carolina
Background: National trauma center guidelines require mental health screening and referral after traumatic injury; however, few centers have implemented protocols to address this. The Trauma Resilience and Recovery Program (TRRP) is an evidence-informed, scalable, technology-enhanced stepped model of care – one of the few in the US – that provides early intervention and services after injury. TRRP results in a ten-fold increase in patient access to mental health follow-up. Consistent with the conference theme, “to accelerate evidence into practice” this presentation will explore the determinants of TRRP adoption during pre-implementation at four US trauma centers.
Methods: A semi-structured field note template, informed by CFIR, was used during on-site and virtual trainings to assess organizational readiness and attitudes towards TRRP implementation. Documented information included access to resources, patient population, reactions to the model, and relationships among team members. Codes were used to organize data based on the CFIR Domains: 1) Intervention characteristics; 2) Inner setting; and 3) Outer setting.
Results: Similar themes emerged across centers. Implementation facilitators included having licensed providers to deliver TRRP, availability of trainees, involvement in screening or mental health services, and team cohesion. Barriers to implementation included limited provider time, staffing, large number of admitted patients, limited mental health training, and competing demands. Sustainability facilitators were leadership enthusiasm and using a “scale up” model, selecting a narrow patient population to implement TRRP and expanding patient eligibility over time. The main barrier to sustainability was a lack of dedicated resources (e.g., staffing).
Conclusions: Stepped-care models offer potential to improve access of mental health care across a range of populations. Yet, little is known about how to effectively integrate these models into hospital-based settings. This study explores implementation determinants with the goal of identifying strategies to maximize adoption and sustained use of stepped-care protocols and improve access to mental health services.
References
1. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation science, 4(1), 1-15.
2. Geerligs, L., Rankin, N. M., Shepherd, H. L., & Butow, P. (2018). Hospital-based interventions: a systematic review of staff-reported barriers and facilitators to implementation processes. Implementation Science, 13(1), 1-17.
3. Ruggiero, K. J., Davidson, T. M., Anton, M. T., Bunnell, B., Winkelmann, J., Ridings, L. E., … & Fakhry, S. M. (2020). Patient engagement in a technology-enhanced, stepped-care intervention to address the mental health needs of trauma center patients. Journal of the American College of Surgeons, 231(2), 223-230.
Disclosures of Interest: None declared
Implementing an innovation tournament to gather care teams’ ideas for behavioral science-based workflow improvements
Authors
Ms. Madeline Fagen - University of Pennsylvania
Mr. Daniel Blumenthal - University of Pennsylvania
Ms. Christina Johnson - University of Pennsylvania
Mrs. Adina Lieberman - University of Pennsylvania
Dr. M. Kit Delgado - University of Pennsylvania
Dr. Srinath Adusumalli - University of Pennsylvania
Dr. Rinad Beidas - University of Pennsylvania
Background: Health system leaders typically drive decision making around healthcare delivery. Nevertheless, care teams involved in daily workflows also have critical insights into process improvement, so integrating both groups’ perspectives is vital.1,2 Innovation tournaments are a promising, but underused approach for engaging a range of healthcare stakeholders in idea generation and implementation.1,3 Penn Medicine’s Nudge Unit launched a system-wide innovation tournament to gather stakeholders’ ideas for how to utilize behavioral and implementation science to promote equitable, evidence-based care. Given the untapped benefits of innovation tournaments for health systems, this study aims to evaluate the acceptability of our process.
Methods: Our innovation tournament consisted of four phases. In phase 1, we invited stakeholders from across the health system to submit ideas through a REDCap-based template. In phase 2, groups of experts (e.g., Nudge Unit, clinical decision support) assessed applications using a custom rubric. In phase 3, select applicants will present proposals to health system executives to determine winners for implementation. In phase 4, we will evaluate the innovation tournament process to refine it for next year and inform the design of other innovation tournaments in healthcare organizations. We will survey selected and non-selected applicants and expert evaluators using a multi-methods approach: quantitative ratings will address acceptability and appropriateness, while qualitative responses will allow open-ended feedback. Additional learnings will be gathered by reviewing process collateral (e.g., meeting minutes) and thematically coding submissions.
Results: Our tournament had substantial engagement, with 74 proposals submitted. Proposal selection, process evaluation, and analysis are ongoing and will be completed by August 2022.
Conclusion: Our findings can help behavioral and implementation scientists gain a more practical understanding of how to systematically include healthcare stakeholders in idea generation through innovation tournaments. Additionally, findings will provide an evidence base to inform participatory design strategies at other healthcare organizations.
References
1. Terwiesch, C., Mehta, S.J., & Volpp, K.G. (2013). Innovating in health delivery: The Penn medicine innovation tournament. Healthcare, 1(1-2), 37-41. doi: 10.1016/j.hjdsi.2013.05.003
2. Asch, D.A., Terwiesch, C., Mahoney, K.B., & Rosin, R. (2014). Insourcing health care innovation. The New England Journal of Medicine, 370(19), 1775-1777. DOI: 10.1056/NEJMp1401135
3. Stewart, R. E., Williams, N., Byeon, Y. V., Buttenheim, A., Sridharan, S., Zentgraf, K., … & Beidas, R. S. (2019). The clinician crowdsourcing challenge: Using participatory design to seed implementation strategies. Implementation Science, 14(1), 1-8. doi.org/10.1186/s13012-019-0914-2
Disclosure of Interest: Srinath Adusumalli reports being employed by CVS Health.
Rinad Beidas reports being principal at Implementation Science & Practice, LLC. Rinad Beidas receives royalties from Oxford University Press. Rinad Beidas receives consulting fees from United Behavioral Health and OptumLabs. Rinad Beidas serves on the advisory boards for Optum Behavioral Health, AIM Youth Mental Health Foundation, and the Klingenstein Third Generation Foundation.
Ethnography lost in translation: An anthropological implementation science
Authors
Dr. Elissa Faro - Carver College of Medicine, University of Iowa
Dr. Aaron Seaman - Carver College of Medicine, University of Iowa
Dr. Ellen Rubinstein - Sociology and Anthropology, North Dakota State University
Dr. Peter Taber - University of Utah School of Medicine
Dr. Gemmae Fix - Boston University School of Medicine
Dr. Heather Reisinger - Carver College of Medicine, University of Iowa
Background: Ethnography is increasingly used in implementation science (IS): a recent scoping review identified 73 articles that explicitly used “ethnography,” concluding that it holds great potential for IS (Gertner et al., 2021). Yet questions exist about its boundaries as a methodological construct (Ramanadhan et al., 2021), especially when not explicitly named, because those boundaries often don’t align clearly with the parameters of granting institutions and publication outlets (Eakin & Mykhalovskiy, 2003). As practicing, applied anthropologists we (co-authors) adopt an ethnographic perspective to answer questions about how IS publications are defining ethnography and to describe what types of IS projects are employing ethnography as part of their work.
Methods: We worked with a health services librarian to systematically search relevant databases for studies that include the components of “ethnographic” methods (i.e., observation, triangulation) used in “implementation science” projects, regardless of researcher, methodological rigor, or article type. We included English-language, peer-reviewed publications of IS conducted in healthcare or community settings using elements of ethnography (labeled as such or not).
Results: Our initial search yielded 2,449 abstracts. We reviewed a subset to develop exclusion criteria; then each abstract was reviewed by two team members with discrepancies adjudicated by a third. Another round of the 423 abstracts focused on more granular inclusion criteria yielded 289 articles for full review. Each full article was reviewed and abstracted. While reconciliation is ongoing, as we iteratively and reflexively interrogated our process and results at each step, it has emerged that the anthropology of ethnography gets lost in its translation to IS.
Conclusion: Ethnography gets relegated as a method in IS rather than an overarching epistemology that provides theoretical foundations to guide design decisions for multi-method projects – a fundamental element of anthropological ethnography. To reach its full potential in IS, ethnography should be more than method.
References
1. Eakin, J. M., & Mykhalovskiy, E. (2003). Reframing the evaluation of qualitative health research: reflections on a review of appraisal guidelines in the health sciences. J Eval Clin Pract, 9. https://doi.org/10.1046/j.1365-2753.2003.00392.x
2. Gertner, A. K., Franklin, J., Roth, I., Cruden, G. H., Haley, A. D., Finley, E. P., Hamilton, A. B., Palinkas, L. A., & Powell, B. J. (2021). A scoping review of the use of ethnographic approaches in implementation research and recommendations for reporting. Implementation Research and Practice, 2. https://doi.org/10.1177/2633489521992743
3. Ramanadhan, S., Revette, A. C., Lee, R. M., & Aveling, E. L. (2021, Jun 29). Pragmatic approaches to analyzing qualitative data for implementation science: an introduction. Implement Sci Commun, 2(1), 70. https://doi.org/10.1186/s43058-021-00174-1
Disclosures of Interest: None declared
Enhancing multilevel stakeholder engagement in implementation research: perspectives of VA implementation scientists
Authors
Dr. Karissa Fenwick - Department of Veterans Affairs, HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System, Los Angeles, CA
Dr. Susan Frayne - Department of Veterans Affairs, HSR&D Center for Innovation to Implementation, VA Palo Alto Healthcare System, Menlo Park, CA; Department of Medicine, Stanford University, Stanford, CA
Dr. Elizabeth Yano - Department of Veterans Affairs, HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System, Los Angeles, CA; David Geffen School of Medicine at UCLA
Ms. Diane Carney - Department of Veterans Affairs, HSR&D Center for Innovation to Implementation, VA Palo Alto Healthcare System, Menlo Park, CA
Dr. Alison Hamilton - Department of Veterans Affairs, HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System, Los Angeles, CA; David Geffen School of Medicine at UCLA, Los Angeles, CA
Background: Stakeholder engagement improves the quality, clinical relevance, and contextual fit of healthcare interventions (Woolf et al., 2016). However, there is limited work characterizing how researchers perceive stakeholder engagement or how research institutions can support it (Concannon et al., 2014). The objective of this project is to describe how experienced researchers engaged multilevel stakeholders in implementation and intervention studies, and the benefits, challenges, and needs they encountered.
Methods: We identified Veterans Health Administration (VA) researchers with histories of stakeholder-engaged intervention/implementation research through a combination of literature review and snowball sampling. We conducted semi-structured telephone interviews with 28 participants in 2016. Two research team members analyzed interview data using the constant comparative method (Miles et al., 2014).
Results: Participants stated that engaging patients, providers, and administrators is essential for successful implementation and described how multilevel stakeholder input substantively improved their research. Common engagement approaches varied in intensity and included interviews and focus groups, advisory committees, and stakeholder membership on research teams. Participants expressed a desire—even a moral imperative—to increase their engagement efforts. Despite the benefits of stakeholder engagement, some participants stated that it is time-consuming, complex, and incompatible with professional incentives (e.g., grants, manuscripts), deterring researchers from pursuing it. Additionally, stakeholder engagement may be a poor methodological fit for researchers who prioritize authority and control over the research process. Participants suggested that VA can support stakeholder engagement research by reconceptualizing funding mechanisms, providing resources, and fostering an organizational culture that aligns with engagement principles.
Conclusions: Results illustrate that multilevel stakeholder engagement is both critical for implementation research and difficult to achieve. Findings highlight a need for greater alignment between stakeholder engagement activities, professional incentives, and organizational infrastructure, and can inform efforts to strengthen stakeholder engagement research in VA and other healthcare settings.
References
1. Concannon, T. W., Fuster, M., Saunders, T., Patel, K., Wong, J. B., Leslie, L. K., & Lau, J. (2014). A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. Journal of General Internal Medicine, 29(12), 1692-1701.
2. Miles, M. B., Huberman, A. M., &, Saldaña, J. Qualitative data analysis: a methods sourcebook (3rd ed.) Thousand Oaks, CA: Sage Publications, Inc.
3. Woolf, S. H., Zimmerman, E., Haley, A., & Krist, A. H. (2016). Authentic engagement of patients and communities can transform research, practice, and policy. Health Affairs, 35(4), 590-594.
Disclosures of Interest: None declared
Methodologies to support bi-directional collaborations and promote engagement in implementation science
Authors
Dr. Danielle Fettes - Department of Psychiatry, University of California, San Diego; Child and Adolescent Services Research Center; UC San Diego ACTRI Dissemination and Implementation Science Center
Ms. Amanda Farr - Department of Psychiatry, University of California, San Diego
Ms. Emily Velandia - Department of Psychiatry, University of California, San Diego
Dr. David Sommerfeld - Department of Psychiatry, University of California, San Diego; Child
Background: Equitable implementation incorporates community experience to improve outcomes (Loper et al., 2021). Successful community implementation encounters setbacks when practices do not adequately fit the needs and resources of the community it seeks to support. Community engaged research centers those with lived experiences – local experts – as true partners in research (Wallerstein et al., 2020), though effectively engaging community members in the research process can be difficult.
Methods: We employed two distinct collaborative research methods to promote engagement in the implementation process: the nominal group technique (NGT) process and Photovoice. NGT is a method of small-group discussion for priority-setting in implementation (Rankin et al., 2016) which centers participants’ voice – the researchers listen, observe, and facilitate the discussion. Answers to a focus question are systematically captured and ranked to create action items that can be implemented and measured. NGT sessions were conducted with former foster youth. Photovoice is designed to empower participants (Budig et al., 2018), communicating their lived expertise through photographs, analyzed using the SHOWeD method. Photovoice was conducted with transitional age youth experiencing homelessness to illustrate how service participation impacted their lives.
Results: The NGT process resulted in lived-experience experts creating a proposal which influenced Child Welfare System policy on funding allocation for education/career resources for foster youth. With Photovoice, formerly homeless youth creatively documented and analyzed their experiences, serving as catalysts for change by outlining key program elements and benefits.
Conclusion: In innovative approaches such as NGT and Photovoice, participants drive the research process, defining their needs and self-evaluating services without biased frameworks that can come from traditional research methods. These methods also promote engagement – community participants actively collaborate to drive policy development, implementation, and evaluation as co-creators and decisionmakers. Utilization of bi-directional methods advances implementation science, prioritizing the knowledge and expertise of the communities in the research process.
References
1. Budig, K., Diez, J., Conde, P., Sastre, M., Hernán, M. & Franco, M. (2018). Photovoice and empowerment: evaluating the transformative potential of a participatory action research project. BMC Public Health 18, 432. https://doi.org/10.1186/s12889-018-5335-7
2. Loper, A., Woo, B., & Metz, A. (2021). Equity Is Fundamental to Implementation Science. Stanford Social Innovation Review, 19(3), A3–A5. https://doi.org/10.48558/QNGV-KG05
3. Rankin, N. M., McGregor, D., Butow, P. N., White, K., Phillips, J. L., Young, J. M., Pearson, S. A., York, S., & Shaw, T. (2016). Adapting the nominal group technique for priority setting of evidence-practice gaps in implementation science. BMC medical research methodology, 16(1), 110. https://doi.org/10.1186/s12874-016-0210-7
4. Wallerstein, N., Oetzel, J. G., Sanchez-Youngman, S., Boursaw, B., Dickson, E., Kastelic, S., Koegel, P., Lucero, J. E., Magarati, M., Ortiz, K., Parker, M., Peña, J., Richmond, A., & Duran, B. (2020). Engage for Equity: A Long-Term Study of Community-Based Participatory Research and Community-Engaged Research Practices and Outcomes. Health education & behavior : the official publication of the Society for Public Health Education, 47(3), 380–390. https://doi.org/10.1177/1090198119897075
Disclosures of Interest: None declared
Community-academic partnership for translational use of research evidence in policy, program, and practice: A mixed-methods analysis
Authors
Dr. Danielle Fettes - Department of Psychiatry, University of California, San Diego; Child & Adolescent Services Research Center
Ms. Margo Fudge - County of San Diego Child Welfare Services
Dr. Cathleen Willging - Center Director, Senior Research Scientist Pacific Institute for Research and Evaluation
Ms. Emily Velandia - Department of Psychiatry, University of California, San Diego
Ms. Tiffany Lagare - Department of Psychiatry, University of California, San Diego
Ms. Amanda Farr - Department of Psychiatry, University of California, San Diego
Dr. Gregory Aarons - Department of Psychiatry, University of California, San Diego; Child & Adolescent Services Research Center; University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
Background: The Community-Academic Partnership for Translational Use of Research Evidence (CAPTURE) is a collaboration for improving use of research evidence (URE) in child welfare services (CWS). Incorporating CWS policy and decision-makers in co-creation with academic investigators is essential to build trust and establish pathways for URE (Metz and Barley, 2015). CAPTURE is grounded in the Exploration, Preparation, Implementation, Sustainment (EPIS) implementation framework to consider URE through implementation phases and in identifying outer and inner context and bridging factor determinants and mechanisms (Moullin et al., 2019).
Methods: A stepped-wedge, mixed-methods design (Aarons, et al. 2012) guides CAPTURE evaluation, focusing on utility and impact of change mechanisms (e.g., cultural exchange, evidence-building capacity) affecting URE in CWS programs, policy, and practice. Three annual waves of data collection include: (a) system-wide web surveys, ascertaining participants’ perceptions, intentions, and climate towards URE; and (b) semi-structured interviews with CAPTURE partners (i.e., system, academic, and community stakeholders).
Results: Quantitative results indicate upper-level leadership’s greater importance on URE compared to first-level leaders and providers. Systemwide expectations of URE increased over time, while URE intentions decreased. The local socio-political context, changed structure of the CWS landscape, and the COVID-19 pandemic impacted URE activities. Qualitative results highlight importance of cultivating consistent and productive communication, negotiating responsibilities, and tackling tensions between research processes and unanticipated system changes.
Conclusion: Community-engaged partnerships have potential to impact URE in program, policy, and practice. Structures within CWS are not inherently designed to facilitate processes to URE, and there are considerable challenges to meet funders’ expectations while creating large scale system/culture (Boaz et al., 2018). CAPTURE has served as a bridging factor for outer and inner contexts of URE system-change in CWS. Our evidence highlights the need to nurture relationships, establish trust, and truly engage with system partners for bidirectional co-creation for URE.
References
1. Aarons, G. A., Fettes, D. L., Sommerfeld, D. H., & Palinkas, L. A. (2012). Mixed methods for implementation research: application to evidence-based practice implementation and staff turnover in community-based organizations providing child welfare services. Child Maltreatment, 17(1), 67-79.
2. Boaz, A., Hanney, S., Borst, R., O’Shea, A., & Kok, M. (2018). How to engage stakeholders in research: design principles to support improvement. Health research policy and systems, 16:60.
3. Metz A., & Bartley L. (2015). Co-Creating the Conditions to Sustain the Use of Research Evidence in Public Child Welfare. Child Welfare, 94(2), 115-139.
4. Moullin, J. C., Dickson, K. S., Stadnick, N. A., Rabin, B., & Aarons, G. A. (2019). Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implementation Science, 14(1).
Disclosures of Interest: None declared
Intrinsic and extrinsic context of implementation strategies
Authors
Dr. Lindsey Filiatreau - Washington University School of Medicine
Dr. Aaloke Mody – Washington University School of Medicine
Dr. Elvin Geng - Washington University School of Medicine
In dissemination and implementation research “context” is considered critical when considering the effects of an implementation strategy. Additional specification of the ways in which context interacts with implementation strategies can help advance conceptualization of the mechanisms of a strategy and how it may work under different conditions.
We use directed acyclic graphs (DAG) to decompose the effects of an implementation strategy (e.g., a health care worker performance-based incentive or “P4P”) on behavioral outcomes (e.g., offer of PrEP) into different mechanistic pathways and explore ways in which context influences these effects. Specifically, we express effects through different organizational and psychological mechanisms and consider links in the causal chain potentially effected by organizational or regional context. We use structural differences to distinguish potential effects of context on an implementation strategy and describe hierarchical relationships between unique contextual variables.
Strategies have direct and indirect effects on behavioral outcomes. Context can influence the strategy itself, the mediator of a strategy, or the behavioral outcome. In the case of P4P, we hypothesize that organizational morale likely affects the mediated pathway from P4P to offering PrEP. Conversely, supply chain issues (i.e., medication stock-outs) directly influence offering PrEP. We categorize elements of context that influence the strategy itself or mediators on the strategy/outcome pathway as “intrinsic” and those that act on the effects of the strategy as “extrinsic”.
Decomposing context’s relationship to implementation strategies into intrinsic and extrinsic effects helps to extend and sharpen our understanding of the effects of implementation strategies in context. Strategies highly dependent on intrinsic context must consider adaptations in order to achieve desired effects in target settings with differing intrinsic contexts. Conversely, in strategies highly dependent on extrinsic context, scale-out to settings with differing extrinsic contexts may inappropriate altogether. Strategies dependent on both should consider the differences in each between the trial and target setting and the relative effects of extrinsic and intrinsic contextual factors.
References
1. Lewis, C. C., Klasnja, P., Powell, B. J., Lyon, A. R., Tuzzio, L., Jones, S., Walsh-Bailey, C., & Weiner, B. (2018). From Classification to Causality: Advancing Understanding of Mechanisms of Change in Implementation Science. Frontiers in Public Health, 6, 136. https://doi.org/10.3389/FPUBH.2018.00136/BIBTEX
2. Lipsky, A. M., & Greenland, S. (2022). Causal Directed Acyclic Graphs. JAMA, 327(11), 1083–1084. https://doi.org/10.1001/JAMA.2022.1816
3. Nilsen, P., & Bernhardsson, S. (2019). Context matters in implementation science: A scoping review of determinant frameworks that describe contextual determinants for implementation outcomes. BMC Health Services Research, 19(1), 1–21. https://doi.org/10.1186/s12913-019-4015-3
Disclosures of Interest: None declared
A theory-based Implementation Evaluation Template (IET) for pragmatic planning and evaluation: Building science into the service contract
Authors
Dr. Erin Finley - University of Texas Health Science Center at San Antonio; VA Greater Los Angeles
Dr. Holly Lanham - University of Texas Health Science Center, San Antonio
Dr. Tara Karns-Wright - University of Texas Health Science Center, San Antonio
Ms. Karla Ramirez - University of Texas Health Science Center, San Antonio
Dr. Jennifer Sharpe Potter - University of Texas Health Science Center, San Antonio
Background: Population health initiatives are often delivered in the context of short-term contracts, for which rapid turnaround deadlines and demanding programmatic requirements can make it challenging to integrate the theory and methods of implementation science. We conducted a synthesis of key implementation principles and frameworks to develop a streamlined template for implementation evaluation across Be Well Texas, a program of state-contracted initiatives increasing access to evidence-based practices (EBPs) for substance use disorders (SUD).
Method: Building on recommended best practices in implementation science, we developed a templated process for structuring project evaluation plans in five steps: (1) defining core elements, including the quality gap and selected EBP; (2) identifying factors likely to affect implementation across the five domains (e.g., outer context) of the Consolidated Framework for Implementation Research (Damschroder et al., 2009); (3) describing and specifying implementation strategies to be used (Proctor et al., 2013); (4) summarizing intended outcomes within the extended RE-AIM (Shelton et al., 2020) and Translational Science Benefits Models (Luke et al., 2018); and (5) developing a comprehensive evaluation summary describing key methods, data sources, timing, and analysis (Huynh et al., 2018). The resulting Implementation Evaluation Template (IET) was adopted as a core planning tool and iteratively refined over a six-month series of project planning efforts.
Results: We present three case studies in rapid implementation project planning, execution, and evaluation, related to telementoring for medication-assisted treatment, recovery support services, and an innovative virtual SUD clinic. We found the IET to be pragmatic, acceptable, feasible, and adaptable as needed for supporting evaluation of individual implementation initiatives, with the added benefit of allowing Be Well Texas to demonstrate cumulative, state-level impact over time.
Conclusion: We believe the Implementation Evaluation Template provides a valuable tool for supporting pragmatic, implementation science-driven evaluation in service-oriented contexts.
References
1. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, 50.
2. Huynh, A. K., Hamilton, A. B., Farmer, M. M., Bean-Mayberry, B., Stirman, S. W., Moin, T., & Finley, E. P. (2018). A Pragmatic Approach to Guide Implementation Evaluation Research: Strategy Mapping for Complex Interventions. Frontiers in Public Health, 6. https://doi.org/10.3389/fpubh.2018.00134
3. Luke, D. A., Sarli, C. C., Suiter, A. M., Carothers, B. J., Combs, T. B., Allen, J. L., Beers, C. E., & Evanoff, B. A. (2018). The Translational Science Benefits Model: A New Framework for Assessing the Health and Societal Benefits of Clinical and Translational Sciences. Clinical and Translational Science, 11(1), 77–84. https://doi.org/10.1111/cts.12495
4. Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013). Implementation strategies: Recommendations for specifying and reporting. Implementation Science, 8(1). https://doi.org/10.1186/1748-5908-8-139
5. Shelton, R. C., Chambers, D. A., & Glasgow, R. E. (2020). An Extension of RE-AIM to Enhance Sustainability: Addressing Dynamic Context and Promoting Health Equity Over Time. Frontiers in Public Health, 8, 134. https://doi.org/10.3389/fpubh.2020.00134
Disclosures of Interest: None declared
Where are the service providers? Mapping the workforce to inform statewide family first prevention services implementation
Authors
Ms. Natalie Finn - Virginia Commonwealth University
Dr. Rafaella Sale - Virginia Commonwealth University
Ms. Gabriela Aisenberg - Virginia Commonwealth University
Ms. Navneet Kaur - Virginia Commonwealth University
Dr. Michael Southam-Gerow - Virginia Commonwealth University
Background: University-affiliated researchers and state government leaders established a partnership to expand the availability of evidence-based programs (EBPs) for children and families in Virginia. Statewide EBP training initiatives are funded through Title IV-E via the Family First Prevention Services Act. As part of a larger needs assessment and gaps analysis project, researchers assessed workforce capacity within mental and behavioral health services, as well as availability and accessibility of extant EBPs across the state-supervised, locally administered social services system. Geographical data visualization can serve as an integral tool in communicating mental health services data to policy decision-makers (Chung et al., 2002; Poot et al., 2018; Saxena et al., 2006). For this project, in partnership with state leaders, results and accompanying maps were used to inform targeted agency recruitment and selection for EBP training initiatives.
Methods: To compile a comprehensive database of provider agencies in the state, the research team first updated and converged provider databases available from state department directories. Next, the team collected additional directories available through local community service boards and community referral lists. Internet searches were conducted to identify additional private provider agencies by county/city. The database of agencies was then coded for (a) agency location/region, (b) service type provided (e.g., outpatient, intensive in-home, substance use treatment, crisis stabilization, skill building/mentoring, care coordination), and (c) availability of prevention focused EBPs.
Results: A series of geographical maps were created for data visualization and context analysis of agency location, service type, and EBP availability across the state. Areas with an existing provider workforce but low EBP availability were identified (e.g., potential targets for training).
Conclusion: Initial mapping of statewide agency workforce capacity informed targeted “requests for applications” and decision making for allocation of EBP training funding. This work aims to fill geographical gaps and maximize coverage of EBPs in the state.
References
1. Chung, Y., Bagheri, N., Salinas-Perez, J. A., Smurthwaite, K., Walsh, E., Furst, M., … & Salvador-Carulla, L. (2020). Role of visual analytics in supporting mental healthcare systems research and policy: A systematic scoping review. International Journal of Information Management, 50, 17-27.
2. Poot, C. C., Van Der Kleij, R. M., Brakema, E. A., Vermond, D., Williams, S., Cragg, L., … & Chavannes, N. H. (2018). From research to evidence-informed decision making: a systematic approach. Journal of Public Health, 40, 3-12.
3. Saxena, S., Sharan, P., Garrido, M., & Saraceno, B. (2006). World Health Organization's mental health atlas 2005: implications for policy development. World Psychiatry, 5, 179-184.
Disclosures of Interest: None declared
Expanding the reach of evidence-based mental health interventions to private practice: Qualitative assessment using a policy ecology framework
Authors
Dr. Hannah Frank - The Warren Alpert Medical School of Brown University
Ms. Lauren Milgram - The Warren Alpert Medical School of Brown University
Dr. Jennifer Freeman - The Warren Alpert Medical School of Brown University
Dr. Kristen Benito - The Warren Alpert Medical School of Brown University
Background: Evidence-based interventions (EBIs) for mental health disorders are underutilized in routine clinical practice. Exposure therapy for anxiety disorders is one particularly difficult-to-implement EBI that has robust empirical support1. Previous research has examined EBI implementation determinants in publicly-funded mental health settings2, but few studies have examined EBI implementation determinants in private practice. Private practice clinicians likely face unique barriers to implementation, including setting-specific contextual barriers to EBI use. The Policy Ecology framework3 considers broad systemic determinants, including organizational, regulatory, social, and political contexts, which are likely relevant but have not been examined in private practice settings.
Methods: Qualitative interviews were conducted with private practice clinicians (N = 20) to assess EBI implementation determinants using the Policy Ecology framework. Clinicians were asked about implementing mental health EBIs broadly and exposure therapy specifically. Mixed methods analyses compared qualitative responses from clinicians working in solo versus group private practice and clinicians who reported high versus low organizational support for exposure therapy.
Results: Responses include determinants related to organizational support (e.g., colleagues using EBIs), payer restrictions (e.g., lack of reimbursement for longer sessions), fiscal incentives (e.g., payment for attending training), and consumer demand for EBIs. There were notable differences in barriers faced by clinicians who work in group private practices compared to those working in solo practices. Solo private practice clinicians described ways in which their practice setting limits their degree of colleague support (e.g., for consultation or exposure therapy planning), while also allowing for flexibility (e.g., in their schedules and practice location) that may not be available to clinicians in group practice.
Conclusion: Using the Policy Ecology framework provides a broad understanding of contextual factors that impact private practice clinicians’ use of EBIs, including exposure therapy. Findings point to potential implementation strategies that may address barriers that are unique to clinicians working in private practice.
References
1. Becker-Haimes, E. M., Okamura, K. H., Wolk, C. B., Rubin, R., Evans, A. C., & Beidas, R. S. (2017). Predictors of clinician use of exposure therapy in community mental health settings. Journal of Anxiety Disorders, 49, 88–94. https://doi.org/10.1016/j.janxdis.2017.04.002
2. Powell, B. J., Patel, S. V., Haley, A. D., Haines, E. R., Knocke, K. E., Chandler, S., Katz, C. C., Seifert, H. P., Ake, G., 3rd, Amaya-Jackson, L., & Aarons, G. A. (2020). Determinants of implementing evidence-based trauma-focused interventions for children and youth: A systematic review. Administration and Policy in Mental Health, 47(5), 705–719. https://doi.org/10.1007/s10488-019-01003-3
3. Raghavan, R., Bright, C. L., & Shadoin, A. L. (2008). Toward a policy ecology of implementation of evidence-based practices in public mental health settings. Implementation science: IS, 3, 26. https://doi.org/10.1186/1748-5908-3-26
Disclosures of Interest: None declared
The use of a technical assistance strategy to support the integration of data-driven elements in coordinated specialty care
Authors
Ms. Elizabeth R. Fraser - Elson S. Floyd College of Medicine, Washington State University
Mr. Gordon Kordas - Elson S. Floyd College of Medicine, Washington State University
Mrs. Bryony Stokes - Elson S. Floyd College of Medicine, Washington State University
Mr. Sheldon Stokes - Elson S. Floyd College of Medicine, Washington State University
Dr. Michael McDonell - Washington State University
Dr. Oladunni Oluwoye - Elson S. Floyd College of Medicine at Washington State University
Background: Coordinated specialty care (CSC) is an early intervention model that utilizes a shared decision-making approach to deliver evidence-informed treatment to individuals experiencing early psychosis [1]. In more recent years, CSC has moved toward the harmonization and delivery of measures across programs to improve quality of care and assist state and federal policymakers as evident by efforts of the National Institute of Mental Health-funded Early Psychosis Intervention Network and their Core Assessment Battery [2]. The first step to providing data-informed care is the integration and delivery of evidence-based measures [3]. However, it remains unclear what strategies can be utilized to support the implementation of assessment batteries.
Methods: A multifaceted technical assistance (TA) strategy was used to support the integration of a measurement battery, which included client self-report and clinician-rated measures, in nine CSC programs. The multifaceted TA strategy included an initial two-day in-person training, booster sessions, and continuous monthly TA virtual meetings. Clinicians completed satisfaction surveys after the in-person training (4-point Likert scale) and TA staff administratively pulled and calculated measurement completion rates. Separate generalized linear mixed effects models were used to examine measure completion (client self-report and clinician-rated measures completed) across 12 months.
Results: Clinicians were highly satisfied (M = 3.67; SD = 0.49) with the in-person training. Approximately 69.5% of scheduled monthly TA meetings were attended. At baseline, < 50% of clinician-rated and <25% of client self-reported measures were completed. The completion of clinician-rated (OR = 1.18, CI:1.15–1.2, p < 0.001) and client self-reported (OR = 1.05, CI:1.03–1.07, p < 0.001) measures significantly increased across a 12-month period.
Conclusion: These preliminary findings suggest that a TA strategy can support the integration of a comprehensive assessment battery delivered by CSC clinicians. Similar TA strategies may assist CSC programs as they adapt and move toward providing data-driven care in an effort to improve quality of care and client outcomes.
References
1. Wright A, Browne J, Mueser KT, Cather C. Evidence-based psychosocial treatment for individuals with early psychosis. Child and Adolescent Psychiatric Clinics of North America. 2020;29:211–23. Available from: https://www.sciencedirect.com/science/article/pii/S1056499319300902
2. Heinssen RK. Early Psychosis Intervention Network (EPINET): A Learning Healthcare System for Early Serious Mental Illness [Internet]. National Institute of Mental Health (NIMH). Available from: https://www.nimh.nih.gov/funding/grant-writing-and-application-process/concept-clearances/2015/early-psychosis-intervention-network-epinet-a-learning-healthcare-system-for-early-serious-mental-illness
3. Scott K, Lewis CC. Using Measurement-Based Care to Enhance Any Treatment. Cogn Behav Pract. 2015;22:49–59.
Disclosures of Interest: None declared
The role of school leaders and teachers in cultivating a trauma-informed school climate
Authors
Dr. Antonio Garcia - University of Kentucky (UK) College of Social Work and UK Center on Trauma and Children
Dr. Ginny Sprang - College of Medicine, Department of Psychiatry and UK Center on Trauma and Children
Dr. Tracy Clemans - University of Kentucky Center on Trauma and Children
Background: Schools are in a unique position to offer services to mitigate the effects of trauma. However, there is a lack of consensus and empirical support regarding the primary ingredients necessary for a successful trauma-informed approach, making implementation of trauma-informed practices within schools challenging. The primary aim of the current study was to examine trauma-informed practices among school personnel who participated in a trauma-informed learning collaborative virtually and/or face-to-face in Kentucky. The secondary aim was to evaluate the impact of leadership engagement on trauma-informed practices in school settings.
Methods: Secondary analysis of data (N = 205) collected from school personnel were conducted. Participants completed cloud-based surveys at baseline (prior to launch of learning) and at the end of it (Time 2). Surveys assessed 1) the degree to which the school is secondary trauma- informed using the Secondary Traumatic Stress Informed Organizational Assessment (STSI-OA), 2) progress toward becoming a trauma sensitive school via the Trauma Sensitive Schools Checklist (TSSC), and 3) Senior Leader Engagement with a one-item measure: “How engaged is senior leadership in trauma informed care?” A generalized linear mixed model was used to analyze the data to account for the non-independence of repeat observations from the same subject, controlling for gender, age, education, and years worked in education.
Results: Results showed that significant increases in total TSSC scores were conferred to leaders fully implementing TIPE by Time 2. Increases from Level 1 (Time 1) to Levels 2 and 3 (Time 2) were also observed.
Conclusion: Findings highlight the effectiveness of TIPE in preparing school leaders to cultivate a trauma-informed climate. Future research is needed to 1) replicate training and assess impact in other contexts, and 2) determine why and under what conditions TIPE achieved its intended outcomes.
References
1. Crosby, T.S., & Vanderhaar, J. (2019). Trauma-informed practices in schools across two decades: An interdisciplinary review of research. Review of Research in Education, 43(1), 422–452. https://doi.org/10.3102/0091732X18821123
2. Kingston, Mattson, S. A., Dymnicki, A., Spier, E., Fitzgerald, M., Shipman, K., Goodrum, S., Woodward, W., Witt, J., Hill, K. G., & Elliott, D. (2018). Building schools’ readiness to implement a comprehensive approach to school safety. Clinical Child and Family Psychology Review, 21(4), 433–449. https://doi.org/10.1007/s10567-018-0264-7
3. Langley, A.K., Nadeem, E., Kataoka, S.H., et al. (2010). Evidence-based mental health programs in schools: barriers and facilitators of successful implementation. School Mental Health, 2(3):105-113.
4. Robey, N. Margoiles, S, Sutherland, L., Rupp, C., Black, C., Hill, T., & Baker, C. (2021). Understanding staff and system-level contextual factors relevant to trauma-informed care implementation, Psychological Trauma: Theory Research, PracTIPE, and Policy, 13(2), 249-257.
Disclosures of Interest: None declared
Facilitators and barriers to implementing the diabetes prevention program across the UC system
Authors
Dr. Maryam Gholami - Herbert Wertheim School of Public Health and Human Longevity Science, University of California San Diego
Dr. Tamra Loeb - David Geffen School of Medicine at UCLA, Los Angeles, CA; Veterans Health Administration, Greater Los Angeles Healthcare System, National Clinician Scholars Program, University of California Los Angeles
Ms. Kate Ramm - David Geffen School of Medicine, University of California Los Angeles
Ms. Kelly Shedd – University of California Irvine Health
Ms. Samantha Soetenga – Recreation, University of California Los Angeles
Ms. Sarah Alkhatib - UCI MPH Candidate
Dr. Nicholas Jackson - David Geffen School of Medicine, University of California Los Angeles
Dr. Obidiugwu Duru - David Geffen School of Medicine, University of California Los Angeles
Dr. Carol Mangione - David Geffen School of Medicine, University of California Los Angeles
Dr. Alison Hamilton - Department of Veterans Affairs, HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System, Los Angeles, CA; David Geffen School of Medicine, University of California Los Angeles
Dr. Tannaz Moin - David Geffen School of Medicine, University of California Los Angeles
Background: National efforts to disseminate the Diabetes Prevention Program (DPP) are ongoing but engagement among at-risk individuals remains low. The University of California (UC) system is one of the largest universities to adopt DPP as a system-wide initiative. We assessed barriers and facilitators to implementation of the DPP across UC.
Methods: Guided by the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) evaluation framework, we conducted 53 in-depth interviews with UC DPP leaders, coordinators, and participants to assess facilitators and barriers to DPP implementation across UC. Transcripts were analyzed using rapid qualitative analysis and emergent themes categorized by RE-AIM dimension.
Results: Among UC DPP participants (n = 28), common facilitators included DPP session characteristics (i.e., sessions before/after work, workplace proximity pre-COVID, and virtual delivery post-COVID) and sense of community (e.g., social support, accountability). Common barriers were timing of sessions (participation before/after work was not an option for some) and intrapersonal factors (e.g., motivation or Zoom “fatigue” post-COVID). Stakeholders (n = 17) reported UC funding and collaboration within and across campuses, including access to shared resources, coaches, and cross campus networks, as key facilitators. Challenges varied by campus, but included consistent session space and coach availability, timing of sessions, advertising/outreach, and workload or competing priorities. For Stakeholder leaders(n = 8), having the program embedded within the UC system provided access to support, funding, and evidence-based expertise for program management, which were key facilitators. Barriers included the structure and amount of funding, availability of resources, personnel issues (e.g., expertise and competing priorities), which varied by campus.
Conclusion: Although the UC DPP is currently implemented in all UC campuses, efforts must be made to leverage system-wide multilevel facilitators and address barriers to ensure long-term maintenance and success. Dissemination of successful university-based models may help enhance the reach of DPP across large segments of the US population.
References
1. Knowler, W. C., Barrett-Connor, E., Fowler, S. E., Hamman, R. F., Lachin, J. M., Walker, E. A., Nathan, D. M., & Diabetes Prevention Program Research Group (2002). Reduction in the incidence of type 2 diabetes with lifestyle intervention or metformin. The New England journal of medicine, 346(6), 393–403. https://doi.org/10.1056/NEJMoa012512
2. Gholami, M., Jackson, N. J., Chung, U., Duru, O. K., Shedd, K., Soetenga, S., Loeb, T., Elashoff, D., Hamilton, A. B., Mangione, C. M., Slusser, W., & Moin, T. (2021). Evaluation of the University of California Diabetes Prevention Program (UC DPP) Initiative. BMC public health, 21(1), 1775. https://doi.org/10.1186/s12889-021-11731-7
3. Glasgow, R. E., Harden, S. M., Gaglio, B., Rabin, B., Smith, M. L., Porter, G. C., Ory, M. G., & Estabrooks, P. A. (2019). RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review. Frontiers in public health, 7, 64. https://doi.org/10.3389/fpubh.2019.00064
Disclosures of Interest: None declared
Acceptability of reproductive goals assessment in the public mental health setting: Patient and provider perspectives
Authors
Dr. Nichole Goodsmith - VA HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System; Department of Psychiatry and Biobehavioral Sciences, David Geffen School of Medicine at UCLA
Dr. Emily Dossett - LAC + USC Medical Center and Keck School of Medicine, University of Southern California
Dr. Rebecca Gitlin - Los Angeles County Department of Mental Health
Dr. Karissa Fenwick - VA HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System
Dr. Alison Hamilton - Department of Veterans Affairs, HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System, Los Angeles, CA; David Geffen School of Medicine at UCLA, Los Angeles, CA
Dr. Kristina Cordasco - Department of Veterans Affairs
Background: Reproductive goals assessment facilitates brief, patient-centered conversations around pregnancy intention, and is acceptable for use in diverse primary care settings. However, pregnancy can be a sensitive topic for women with mental illness, who may face reproductive stigma and coercion. We assessed the acceptability of reproductive goals assessment in public mental health (MH) clinics to inform tailoring for, and implementation in, these settings.
Methods: We partnered with an urban county MH agency to conduct qualitative interviews with 22 reproductive-age patients (English-speaking, primarily Black or Latina) and 36 providers (psychiatrists, therapists, case managers, nurses) from public MH clinics. Topics included acceptability of reproductive goals assessment and feedback on two widely used tools: PATH (Pregnancy Attitudes, Timing, and How important is pregnancy prevention) and OKQ (One Key Question®). We used rapid qualitative analysis to summarize interview transcripts and identify preliminary themes using matrices.
Results: Most patients said they would appreciate the opportunity to discuss reproductive goals with MH providers. A minority expressed discomfort or ambivalence, suggesting providers ask permission or allow the patient to raise the topic. Providers were generally enthusiastic about reproductive goals assessment, saying it would “open the door” to sometimes challenging conversations, support communication of patients’ goals, and facilitate medication planning. Additional themes included need for framing to contextualize such personal questions, need to build rapport before asking them, and the challenge of balancing competing priorities. Many participants found both PATH and OKQ prompts acceptable; some preferred the “conversational” and “open-ended” PATH phrasing.
Conclusions: Patients and providers supported implementing reproductive goals assessment in public MH settings. Areas of discomfort highlight the sensitivity of these topics for some women with chronic mental illness and suggest opportunities to tailor language, framing, and provider training. Provider- and patient-facing implementation strategies will be needed to enhance implementation outcomes, particularly acceptability and feasibility.
References
1. Averill, J. B. (2002). Matrix analysis as a complementary analytic strategy in qualitative inquiry. Qualitative Health Research, 12(6), 855–866. https://doi.org/10.1177/104973230201200611
2. Callegari, L. S., Aiken, A. R. A., Dehlendorf, C., Cason, P., & Borrero, S. (2017). Addressing potential pitfalls of reproductive life planning with patient-centered counseling. American Journal of Obstetrics and Gynecology, 216(2), 129–134. https://doi.org/10.1016/j.ajog.2016.10.004
3. Dolman, C., Jones, I., & Howard, L. M. (2013). Pre-conception to parenting: A systematic review and meta-synthesis of the qualitative literature on motherhood for women with severe mental illness. Archives of Women’s Mental Health, 16(3), 173–196. https://doi.org/10.1007/s00737-013-0336-0
4. Garbers, S., Falletta, K. A., Srinivasulu, S., Almonte, Y., Baum, R., Bermudez, D., Coriano, M., Iglehart, K., Mota, C., Rodriguez, L., Taveras, J., Tobier, N., & Grosso, A. (2020). “If You Don’t Ask, I’m Not Going to Tell You”: Using Community-Based Participatory Research to Inform Pregnancy Intention Screening Processes for Black and Latina Women in Primary Care. Women’s Health Issues, 30(1), 25–34. https://doi.org/10.1016/j.whi.2019.08.004
5. Hamilton, A. B., & Finley, E. P. (2019). Qualitative methods in implementation research: An introduction. Psychiatry Research, 280, 112516. https://doi.org/10.1016/j.psychres.2019.112516
6. Hipp, S. L., Chung-Do, J., & McFarlane, E. (2019). Systematic Review of Interventions for Reproductive Life Planning. Journal of Obstetric, Gynecologic, and Neonatal Nursing: JOGNN, 48(2), 131–139. https://doi.org/10.1016/j.jogn.2018.12.007
7. Manze, M. G., Calixte, C., Romero, D. R., Roberts, L., Perlman, M., Langston, A., & Jones, H. E. (2020). Physician perspectives on routine pregnancy intention screening and counseling in primary care. Contraception, 101(2), 91–96. https://doi.org/10.1016/j.contraception.2019.11.004
8. Zatloff, J. P., von Esenwein, S. A., Philip, Z., & Ward, M. C. (2020). Navigating a Complex Health System: The Perceptions of Psychiatric Residents in Addressing Sexual and Reproductive Health of Women with Severe Mental Illness. Academic Psychiatry, 44(4), 403–407. https://doi.org/10.1007/s40596-020-01197-x
Disclosures of Interest: None declared
Extending the concept of adverse events to implementation science trials using causal diagrams
Authors
Dr. Charles Goss - Washington University School of Medicine
Dr. Elvin Geng - Washington University School of Medicine
Dr. Aaloke Mody - Washington University School of Medicine
Dr. Lindsey Filiatreau - Washington University School of Medicine
Background: Use of traditional standards for monitoring drug trials is a poor fit for experimental tests of implementation strategies. Principles to justify and guide monitoring of unintended potentially untoward effects of implementation strategies are needed.
Methods: We encode causal assumptions in a directed acyclic graph to highlight the linkages between implementation strategies and potential harms in implementation trials. We express effects of implementation strategies on organizational and individual effects, encode plausible mechanisms by which a strategy could negatively impact intervention integrity, implementation outcomes, and use of other interventions. Finally, we draw inferences and conclusions implied by the causal diagram.
Results: Our causal diagram surfaces three insights for considering adverse events trials of implementation strategies. First, adverse drug events can be potentially affected by an implementation strategy when the strategy worsens the quality of current use. Second, negative impacts on the use of other medications due to a new implementation strategy and mediated by other organizational behaviors may be candidates for monitoring. Third, the risk and benefit of a novel implementation strategy must account for the benefits of additional reach and the harms from failing to use an evidence-based clinical intervention.
Conclusion: We conclude that the mechanism of experimental conditions on patient harms are distinct in implementation studies as compared to traditional drug trials. These new relationships demand additional conceptualization of potential adverse effects that could be considered due to the strategy. Moreover, identifying previously unrecognized effects resulting from other components of the delivery system is critical. Using causal diagrams to identify the causal relationships between strategies and clinical outcomes clarifies the nature of benefits and harms as well as makes clear opportunities to develop standards and measurement.
References
1. Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003). Why Don’t We See More Translation of Health Promotion Research to Practice? Rethinking the Efficacy-to-Effectiveness Transition. American Journal of Public Health, 93(8), 1261-1267. https://doi.org/10.2105/ajph.93.8.1261
2. Greenland, S., Pearl, J., & Robins, J. M. (1999). Causal diagrams for epidemiologic research. Epidemiology, 10(1), 37-48. https://doi.org/10.1097/00001648-199901000-00008
3. Hernan, M. A. (2004). A definition of causal effect for epidemiological research. Journal of Epidemiology & Community Health, 58(4), 265-271.
4. Talbot, G. H. (2008). Efficacy as an Important Facet of “Safety” in Clinical Trials: How Can We Do Our Best for Our Patients? Clinical Infectious Diseases, 47(S3), S180-S185. https://doi.org/10.1086/591401
5. Wolfenden, L., Foy, R., Presseau, J., Grimshaw, J. M., Ivers, N. M., Powell, B. J., Taljaard, M., Wiggers, J., Sutherland, R., Nathan, N., Williams, C. M., Kingsland, M., Milat, A., Hodder, R. K., & Yoong, S. L.. (2021). Designing and undertaking randomised implementation trials: guide for researchers. BMJ, m3721. https://doi.org/10.1136/bmj.m3721
Disclosures of Interest: None declared
Use of the Clinical Sustainability Assessment Tool (CSAT) to examine sustainability of Long-Acting HIV treatment in Europe in Cabotegravir and Rilpivirine Implementation Study in European Locations (CARISEL)
Authors
Dr. Cassidy Gutner - ViiV Healthcare
Ms. Rekha Trehan - ViiV Healthcare
Dr. Samia Dakhia - ViiV Healthcare
Mrs. Beatriz Hernandez - ViiV Healthcare
Dr. Yazdan Yazdanpanah - 4GH Bichat - Claude Bernard-Paris cedex 18-France-C
Dr. Linos Vandekerckhove - UZ Gent
Dr. Leila Belkhir - 6Cliniques Universitaires Saint-Luc
Ms. Emma Low – Evidera
Dr. Owen Cooper – Evidera
Ms. Savita Bkhshi Anand – Evidera
Dr. Maggie Czarnogorski - ViiV Healthcare
Background: Cabotegravir (CAB) and Rilpivirine (RPV) is the first complete long-acting (LA) regimen for the maintenance of HIV-1 virologic suppression1,2, presenting a paradigm shift from oral therapy. CARISEL is a hybrid implementation-effectiveness trial examining strategies to support CAB + RPV LA implementation every two-months in Belgium, France, Germany, the Netherlands, and Spain. Organizational capacity for implementation sustainability was examined at Month 12.
Methods: 62 staff study participants (SSPs) across 18 sites completed the CSAT3 and qualitative interviews in their local language. All SSPs could make implementation strategy modifications during the study.
Results: Overall, the mean CSAT score was high (M = 5.36). Average domain scores ranged from 5.02-5.58, with highest scores observed for ‘Engaged Staff & Leadership’ (M = 5.58, SD = 0.06) and ‘Implementation and Training’ (M = 5.58, SD = 0.48), and lowest for the ‘Engaged Stakeholders’ domain, which also had the largest SD (SD = 0.92). The lowest mean score was observed for item 4 of the latter domain (‘engaging other medical teams and community partnerships;’ M = 3.61). SSPs from France (55%) and Belgium (46%) rated this item a 3 or lower. 50% of the SSPs interviewed discussed participating in CAB + RPV LA information-sharing networks, with those in the Netherlands and Spain appearing the least engaged in these networks. Themes around clinic processes and staff training emerged as important to sustainment. Additional space within clinics, possible administration in local settings, and improvements in medication delivery process were also identified as important for long-term sustainment of CAB + RPV LA post-CARISEL.
Conclusion: CSAT and qualitative interviews indicate strong domains related to sustainment with some expected room for improvement to support sustainability post-CARISEL. Sustainment support will vary by context, with supportive frontline staff, adequate training, and having processes to guide implementation being key for success. To our knowledge, this is the first study to administer the CSAT in 5 languages across 18 HIV clinics in Europe.
References
1. U.S. Department of Health and Human Services. Guidelines for the Use of Antiretroviral Agents in Adults and Adolescents with HIV. 2021. Available at: https://aidsinfo.nih.gov/guidelines/html/1/adult-and-adolescent-arv/15/virologic-failure. Accessed September 10, 2021.
2. Saag, M. S., Gandhi, R. T., Hoy, J. F., Landovitz, R. J., Thompson, M. A., Sax, P. E., … & Volberding, P. A. (2020). Antiretroviral drugs for treatment and prevention of HIV infection in adults: 2020 recommendations of the International Antiviral Society–USA panel. Jama, 324(16), 1651-1669.
3. Malone, S., Prewitt, K., Hackett, R., Lin, J. C., McKay, V., Walsh-Bailey, C., & Luke, D. A. (2021). The Clinical Sustainability Assessment Tool: measuring organizational capacity to promote sustainability in healthcare. Implementation Science communications, 2(1), 1-12.
Disclosures of Interest: Cassidy Gutner is an employee of ViiV Healthcare. Samia Dakhia reports having a conflict of interest with ViiV Healthcare.
Evidence-based mental health interventions for justice-involved individuals with serious mental illness: A Systematic Review
Authors
Dr. Maji Hailemariam - Michigan State University
Dr. Tatiana Bustos - Michigan State University
Dr. Barrett Montgomery - Michigan State University
Mr. Gashaye Melaku Tefera - University of Missouri
Mr. Garrett Brown - Michigan State University
Dr. Rosemary Adaji - Michigan State University
Mr. Brandon Taylor - NHS
Ms. Hiywote Eshetu - Michigan State University
Ms. Clara Barajas - Michigan State University
Mr. Rolando Barajas - NIH
Ms. Vanessa Najjar - Michigan State University
Ms. Christian Jackson - Michigan State University
Dr. Julia Felton - Michigan State University
Dr. Jennifer Johnson - Michigan State University
Background. Currently, there are more people with mental illness receiving care in the criminal justice system than in community mental health facilities. Various evidence-based interventions (EBIs) are implemented to reduce the number of individuals with mental illness in jails and prisons. Yet, several barriers to implementing EBIs with justice involved populations persist.
Methods. A systematic review was conducted to summarize the mental health treatment literature for justice-involved individuals with depression and/or other serious mental illness. Guided by the PRISMA, the review aimed to identify and describe 1) existing mental health EBIs; 2) intervention outcomes and characteristics; 3) settings/contexts where EBIs were delivered and 4) barriers and facilitating factors for implementing EBIs using the CFIR framework. Initial searches were conducted using PsychINFO, Embase, ProQuest, PubMed, and Web of Science in June 2020 and updated searches are underway. Eligibility criteria included: use of an EBI targeting justice involved populations, description of the EBI, peer reviewed, and written in English. No restrictions were placed on setting or dates. A team of 10 reviewers carried out screening and review of articles.
Results. A total of total of 17 eligible studies were identified. EBIs were categorized as cognitive/behavioral, community-based, interpersonal, psychoeducational, or court-based. All studies reported treatment outcomes, with only 8 studies reporting both treatment and implementation outcomes. Implementation outcomes included details on sustainment gains, feasibility, acceptability, and cost-effectiveness of EBIs. Majority of reported barriers were related to inner and outer context settings, while reported facilitators were related to inner settings.
Discussion. A need for implementation research with justice-involved populations is apparent. Our findings call for improved reporting of evidence-based EBIs for justice-involved populations. A careful reporting of implementation outcomes, barriers and facilitators that promote successful implementation of EBIs is crucial.
References
1. Cohen, T. R., Mujica, C. A., Gardner, M. E., Hwang, M., & Karmacharya, R. (2020). Mental Health Units in Correctional Facilities in the United States. Harvard Review of Psychiatry, 28(4), 255-270.
2. Constantine, R., Andel, R., Petrila, J., Becker, M., Robst, J., Teague, G., . . . Howe, A. (2010). Characteristics and experiences of adults with a serious mental illness who were involved in the criminal justice system. Psychiatric Services, 61(5), 451-457.
3. Diamond, P. M., Wang, E. W., Holzer, C. E., & Thomas, C. (2001). The prevalence of mental illness in prison. Administration and Policy in Mental Health and Mental Health Services Research, 29(1), 21-40.
4. Draine, J., & Herman, D. B. (2007). Critical time intervention for reentry from prison for persons with mental illness. Psychiatric Services, 58(12), 1577-1581.
5. Ford, E. (2015). First-episode psychosis in the criminal justice system: identifying a critical intercept for early intervention. Harvard Review of Psychiatry, 23(3), 167-175.
6. Hopkin, G., Evans-Lacko, S., Forrester, A., Shaw, J., & Thornicroft, G. (2018). Interventions at the transition from prison to the community for prisoners with mental illness: a systematic review. Administration and Policy in Mental Health and Mental Health Services Research, 45(4), 623-634.
Disclosures of Interest: None declared
Applying a rapid qualitative approach to inform the scale up of an integrated primary care program to improve child health in Togo, West Africa
Authors
Ms. Jessica Haughton - Integrate Health
Dr. Désiré Dabla - Integrate Health
Dr. Elissa Faro - Carver College of Medicine, University of Iowa
Ms. Fiona Rowles - Integrate Health
Ms. Essodinam Miziou - Integrate Health
Dr. Kevin Fiori - Department of Pediatrics & Department of Family and Social Medicine, Albert Einstein College of Medicine
Background: Despite significant improvements in child survival globally, high mortality rates persist in low-income countries like Togo.(El Bcheraoui et al., 2020; UNICEF et al., 2019) There is insufficient research on how to scale evidence-based interventions and understanding determinants of effective implementation at scale is necessary to reduce preventable child deaths.(Reñosa et al., 2020; Theobald et al., 2018) Formative evaluation (FE) can guide this process by identifying implementation barriers.(Elwy et al., 2020; Stetler et al., 2006) This study utilized FE to inform the scale up of the Integrated Primary Care Program (IPCP) in Togo.
Methods: The IPCP is a tested approach that optimizes deployment of trained, salaried Community Health Workers (CHWs) in coordination with public health centers to effectively deliver evidence-based interventions including the Integrated Management of Childhood Illness.(Fiori et al., 2021; Gera et al., 2016). Trained qualitative researchers conducted interviews (n = 35) with CHWs, health center staff, community leaders, and implementing partners in a district where the IPCP was launched 24 months prior. Interview guides were developed using the Consolidated Framework for Implementation Research and data were analyzed using a rapid qualitative approach.(Beebe, 2001; Damschroder et al., 2009)
Results: While the IPCP has contributed to increased use of services, barriers to care remain, including medication stockouts, transportation challenges, and sociocultural factors (e.g., home deliveries viewed as positive). Communication is strong among implementing partners, CHWs, and health center staff, however community members lack information on covered services and how to access ambulance services. IPCP implementers felt that the trainings provided at launch prepared them well for program implementation, however they expressed interest in more opportunities for specialized training.
Conclusion: Rapid qualitative approaches can be instrumental in FE to inform the implementation and scale-up of evidence-based programs. Data collected from stakeholders with diverse roles in program implementation can produce actionable guidance to scale.
References
1. Beebe, J. (2001). Rapid Assessment Process: An Introduction. AltaMira Press.
2. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009, Aug 7). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci, 4, 50. https://doi.org/10.1186/1748-5908-4-50
3. El Bcheraoui, C., Mimche, H., Miangotar, Y., Krish, V. S., Ziegeweid, F., Krohn, K. J., Ekat, M. H., Nansseu, J. R., Dimbuene, Z. T., Olsen, H. E., Tine, R. C. K., Odell, C. M., Troeger, C. E., Kassebaum, N. J., Farag, T., Hay, S. I., & Mokdad, A. H. (2020, Mar). Burden of disease in francophone Africa, 1990-2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet Glob Health, 8(3), e341-e351. https://doi.org/10.1016/s2214-109x(20)30024-3
4. Elwy, A. R., Wasan, A. D., Gillman, A. G., Johnston, K. L., Dodds, N., McFarland, C., & Greco, C. M. (2020, Jan). Using formative evaluation methods to improve clinical implementation efforts: Description and an example. Psychiatry Res, 283, 112532. https://doi.org/10.1016/j.psychres.2019.112532
5. Fiori, K. P., Lauria, M. E., Singer, A. W., Jones, H. E., Belli, H. M., Aylward, P. T., Agoro, S., Gbeleou, S., Sowu, E., Grunitzky-Bekele, M., Singham Goodwin, A., Morrison, M., Ekouevi, D. K., & Hirschhorn, L. R. (2021, Sep). An Integrated Primary Care Initiative for Child Health in Northern Togo. Pediatrics, 148(3). https://doi.org/10.1542/peds.2020-035493
6. Gera, T., Shah, D., Garner, P., Richardson, M., & Sachdev, H. S. (2016, Jun 22). Integrated management of childhood illness (IMCI) strategy for children under five. Cochrane Database Syst Rev(6), Cd010123. https://doi.org/10.1002/14651858.CD010123.pub2
7. Reñosa, M. D., Dalglish, S., Bärnighausen, K., & McMahon, S. (2020). Key challenges of health care workers in implementing the integrated management of childhood illnesses (IMCI) program: a scoping review. Glob Health Action, 13(1), 1732669. https://doi.org/10.1080/16549716.2020.1732669
8. Stetler, C. B., Legro, M. W., Wallace, C. M., Bowman, C., Guihan, M., Hagedorn, H., Kimmel, B., Sharp, N. D., & Smith, J. L. (2006, Feb). The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med, 21 Suppl 2(Suppl 2), S1-8. https://doi.org/10.1111/j.1525-1497.2006.00355.x
9. Theobald, S., Brandes, N., Gyapong, M., El-Saharty, S., Proctor, E., Diaz, T., Wanji, S., Elloker, S., Raven, J., Elsey, H., Bharal, S., Pelletier, D., & Peters, D. H. (2018, Nov 17). Implementation research: new imperatives and opportunities in global health. Lancet, 392(10160), 2214-2228. https://doi.org/10.1016/s0140-6736(18)32205-0
10. UNICEF, World Health Organization, World Bank Group, & and United Nations. (2019). Levels and trends in child mortality 2019. https://www.unicef.org/reports/levels-and-trends-child-mortality-report-2019
Disclosures of Interest: None declared
Evaluation of a pilot implementation of a digital cognitive-behavioral therapy platform for county mental health services
Authors
Ms. Rosa Hernandez-Ramos - University of California – Irvine
Ms. Kristina Palomares - University of California-Irvine
Mr. Eduardo Ceballos-Corro - University of California-Irvine
Dr. Stephen M. Schueller - University of California-Irvine
Dr. Nicole Stadnick - University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center, United States
Dr. Elizabeth V. Eikey– University of California San Diego
Dr. Margaret Schneider - University of California-Irvine
Dr. Kai Zheng - University of California-Irvine
Dr. Dana B. Mukamel - University of California-Irvine
Dr. Dara H. Sorkin - University of California-Irvine
Background: The mental health workforce is insufficient to meet current service utilization needs, especially with regards to geographic distribution and racial/ethnic diversity.1 Digital mental health tools have potential to increase access by overcoming availability, geographic, or linguistic barriers.2 As part of the state-wide Help@Hand project, Marin County sought to explore the potential of myStrength, a digital cognitive-behavioral therapy platform, to enhance well-being and social connectedness among isolated older adults. We conducted a formative evaluation of this pilot implementation to address feasibility.
Methods: A pilot implementation explored the deployment of myStrength amongst English (n = 20) and Spanish-speaking (n = 18) isolated older adults who received support from a nurse intern (English-speaking participants) or promotore (Spanish-speaking participants). Participants were recruited through sharing information about the program to established community-partnerships and a network of Promotores. We used RE-AIM3 to evaluate the pilot using data from archival records, qualitative interviews, and pre-post questionnaires.
Results: Reach: Respondents were predominantly female (93.0%), Latinx (72.0%) and scored high on loneliness (70%). Effectiveness: Participants experienced a significant decrease in loneliness on pre- (M = 6.0, SD = 1.7) to 8-weeks post-intervention questionnaires (M = 5.5, SD = 2.1), t(21) = 3.04, p < .01. Adoption: 52% of participants used myStrength daily or several times a day. Implementation: Promotores and nurse interns supported several aspects of participants’ use including accessing technology, onboarding, and gaining digital literacy skills.
Conclusion: The use of myStrength showed moderate promise, including reaching the target population, reducing loneliness, and adoption by participants. However, promotores and nurse interns had to invest significant time in various activities to support those with lower digital literacy skills. As such, although digital mental health holds the potential to address needs, their use with underserved populations, particularly those with lower levels of digital literacy, may require upfront and ongoing human support.
References
1. Coffman, J., Bates, T., Geyn, I., Spetz, J., Healhforce Center at UCSF (2018). California’s current and future behavioral health workforce. Healthforce Center at UCSF. Accessed on April 25, 2022. https://healthforce.ucsf.edu/sites/healthforce.ucsf.edu/files/publication-pdf/California%E2%80%99s%20Current%20and%20Future%20Behavioral%20Health%20Workforce.pdf
2. Schueller, S. M., Muñoz, R. F., & Mohr, D. C. (2013). Realizing the potential of behavioral intervention technologies. Current Directions in Psychological Science, 22(6), 478-483.
3. Glasgow, R. E., Harden, S. M., Gaglio, B., Rabin, B., Smith, M. L., Porter, G. C., … & Estabrooks, P. A. (2019). RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Frontiers in public health, 7, 64.
Disclosure of Interest: Rosa Hernandez-Ramos reported that this work was funded by Help@Hand (agreement number 417-ITS-UCI-2019), a project overseen by the California Mental Health Service Authority (CalMHSA). CalMHSA reviewed the abstract for confidentiality. The information or content and conclusions presented here are those of the authors and should not be construed as the official position or policy of, nor should any endorsements be inferred by, the participating Help@Hand Counties and/or CalMHSA.
Stephen Schueller is a member of the Scientific Advisory Board for Headspace, and also reports having received consulting payments from Otsuka Pharmaceuticals and K Health (Trusst).
Associations between intervention activities and implementation of a comprehensive school tobacco policy at Danish vocational schools: A repeated cross-sectional study
Authors
Ms. Anneke Hjort - Health Promotion Research, Steno Diabetes Center Copenhagen – Copenhagen University Hospital
Ms. Mirte Kuipers - Department of Public and Occupational Health, Amsterdam UMC Location University of Amsterdam
Ms. Maria Stage - Prevention, The Danish Cancer Society
Prof. Charlotta Pisinger - Center for Clinical Research and Prevention, Capital Region and Prevention, The Danish Heart Foundation
Ms. Charlotte Demant Klinker - Health Promotion Research, Steno Diabetes Center Copenhagen – Copenhagen University Hospital
Background: School tobacco policies (STPs) are often not well-implemented (Galanti et al., 2014; Schreuders, 2020), which may explain the limited evidence of effectiveness. This study examines which intervention activities promotes implementation of a comprehensive STP.
Methods: The Smoke-Free Vocational Schools intervention is designed to stimulate implementation of a comprehensive STP into routine practice and took place across seven Danish schools in 2018-2020.
We applied a repeated cross-sectional survey design to quantify associations between intervention activities and STP implementation at two time points (T1 and T2). Implementation outcomes were assessed as four domains (adherence, dose, quality of delivery and participant responsiveness) and a total score across domains. Intervention activities that targeted either students (e.g. new school-break facilities) or staff/managers (e.g. a workshop prior to implementation) were included as independent variables. Associations were analyzed separately for students and staff/managers using generalized linear mixed models, adjusting for contextual factors.
Results: At T1, n = 419/staff managers and n = 1222 students participated in the study; at T2, n = 452 and n = 1497. At both staff/manager- and student level, we found positive associations between activities and STP implementation, which were consistent over time. E.g. at student level, smoke-free signage was associated with adherence at both T1 and T2 (OR = 1.45, 95%CI = 1.24-165 at T1).
Conclusion: We found a consistent pattern over time, comprising positive associations between intervention activities and STP implementation, including which activities promotes implementation. These findings have implications for practice.
Further , we developed a pragmatic and reproduceable implementation outcomes measure that reflect the success of STPs, which can inform future studies. Study strengths are that implementation is treated as an dependent variable and that we consider the implementation process timing and level of analysis (Proctor et al., 2011; Proctor & Brownson, 2018).
References
1. Galanti, M. R., Coppo, A., Jonsson, E., Bremberg, S., & Faggiano, F. (2014). Anti-tobacco policy in schools: Upcoming preventive strategy or prevention myth? A review of 31 studies. Tobacco Control, 23 (4), 295. https://doi.org/10.1136/tobaccocontrol-2012-050846
2. Proctor, E. K., & Brownson, R. C. (2018). Chapter 14: Measurement Issues in Dissemination and Implementation Research. I Dissemination and Implementation Research in Health: Translating Science to Practice (Second edition). Oxford University Press. https://doi.org/10.1093/acprof:oso/9780199751877.001.0001
3. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7
Schreuders, M. (2020). Smoke-free school policies: Understanding their implementation and impact by using the realist approach. ISBN: 9789402820362
Disclosures of Interest: None declared
Family Law Stakeholder Perspectives on Conflict and Intimate Partner Violence: A Qualitative Study to Inform Evidence Based Practice Implementation
Authors
Ms. Holly Huber - Indiana University Bloomington
Mx. Jax Witzig - University of Illinois at Chicago
Ms. Catalina Ordorica - University of Illinois at Chicago
Ms. Emily Potter - University of Illinois at Chicago
Dr. Brittany Rudd - University of Illinois at Chicago
Background: Family law settings (e.g., courts handling divorce and child paternity cases) make difficult decisions about parenting arrangements, which can be further complicated when the case involves high levels of inter-parental conflict (IPC) or intimate partner violence (IPV; Rudd & Beidas, 2021). The use of systems to screen and refer parties to services are known as triage models in family law settings and may be particularly important for these high-risk cases, who have greater service needs. However, definitions of IPC are inconsistent (Archer-Kuhn, 2018) and definitions of IPV vary by state (Stark et al., 2019). To support the implementation of triage in family law settings, the current study explored one family law system to 1) elucidate court stakeholders’ definitions of and methods of identifying IPC and IPV, and 2) consider whether stakeholders distinguish the two in meaningful ways. Method: Court stakeholders (e.g., judges, administrators, lawyers, community service providers; n = 16) from the same Midwestern county were interviewed to better understand their definitions, determinations, and distinguishment of IPC and IPV in family law cases. Rapid qualitative analysis methods, including matrix analysis, were used to collect, organize, and analyze the data (Averill, 2002).
Results: Results indicate that stakeholders largely do not determine that a case involves IPC until well after the case begins, IPV definitions vary widely in scope but primarily focus on physical violence, empirically supported methods to identify IPV survivors are infrequently used, and stakeholders disagree about how to distinguish between IPC and IPV. Conclusions: These results demonstrate the need to implement triage models that screen family law cases early for IPC and IPV. To facilitate implementation, educational strategies will be needed to teach family law stakeholders the difference between these two constructs, and state statutes should be revised to reflect the existence of non-physical types of abuse.
References
1. Archer-Kuhn, B. (2018). Domestic violence and high conflict are not the same: A gendered analysis. Journal of Social Welfare and Family Law, 40(2), 216–233. https://doi.org/10.1080/09649069.2018.1444446
2. Averill, J.B. (2002). Matrix analysis as a complimentary analytic strategy in qualitative inquiry. Qualitative Health Research, 12, 855-866. https://doi.org/10.1177/104973230201200611
3. Stark, D. P., Choplin, J. M., & Wellard, S. E. (2019). Properly accounting for domestic violence in child custody cases: An evidence-based analysis and reform proposal. Michigan Journal of Gender & Law, 26(1), 1–119.
Disclosures of Interest: None declared
Developing an implementation blueprint to scale-up an adolescent contraception care intervention in a pediatric hospital setting
Authors
Ms. Kathryn Hyzak - College of Social Work, The Ohio State University
Dr. Alicia Bunger - College of Social Work, The Ohio State University
Dr. Anna Kerlek - Department of Psychiatry and Behavioral Health, Nationwide Children's Hospital and The Ohio State University College of Medicine
Dr. Stephanie Lauden - Division of Hospital Medicine, Nationwide Children's Hospital; Department of Pediatrics, The Ohio State University College of Medicine
Dr. Sam Dudley - Division of Hospital Medicine, Nationwide Children's Hospital; Department of Pediatrics, The Ohio State University College of Medicine
Dr. Elise Berlan - Department of Pediatrics, The Ohio State University College of Medicine; Division of Adolescent Medicine, Nationwide Children's Hospital
Background: Implementation blueprints are comprehensive plans that describe implementation goals, strategies, and timeframes. There is limited attention to how formal blueprints are developed and refined (1). Guided by the Consolidated Framework for Implementation Research (CFIR) (2) and Expert Recommendations for Implementing Change (ERIC) (3), we describe a participatory approach to developing and tailoring a formal blueprint for scaling a contraception care intervention for adolescents hospitalized in psychiatric units within a pediatric hospital.
Methods: In Stage 1 (Preparation), we assembled a research advisory board with 27 implementation stakeholders. Through a facilitated planning activity, stakeholders identified anticipated implementation barriers, which we mapped to implementation strategies using the CFIR/ERIC matching tool. In Stage 2 (Implementation), stakeholders recorded implementation efforts between May 2021 to January 2022 using activity logs which we coded using the ERIC taxonomy. In Stage 3 (Synthesizing), we specified and operationalized strategies from Stages 1 and 2 to develop an implementation blueprint.
Results: At Stage 1, we identified seven strategies for addressing anticipated implementation barriers. At Stage 2, stakeholders logged 118 implementation activities which mapped to 29 strategies within ERIC categories, including training/educating stakeholders (N = 42); using evaluative/iterative strategies (N = 36); developing stakeholder interrelationships (N = 18); changing infrastructure (N = 16), adapting/tailoring to the context (N = 4); supporting clinicians (N = 1); and providing interactive assistance (N = 1). At Stage 3, the implementation blueprint consisted of six strategies for implementation preparation (identify and prepare champions, educational meetings, change record systems for quality monitoring, clinician implementation team meetings, network weaving, and blueprint refinement), and five strategies for implementation (support clinicians, promote adaptability, reexamine implementation, audit and feedback, and facilitate relay of clinical data).
Conclusion: This method combines prospective implementation planning with detailed observation of activities in order to generate implementation blueprints. These blueprints specify key implementation strategies to enhance transparency, replication, and scale-up to other settings.
References
1. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(1), 50. https://doi.org/10.1186/1748-5908-4-50
2. Lewis, C. C., Scott, K., & Marriott, B. R. (2018). A methodology for generating a tailored implementation blueprint: An exemplar from a youth residential setting. Implementation Science, 13(1), 68. https://doi.org/10.1186/s13012-018-0761-6
3. Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015). A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10(1), 21. https://doi.org/10.1186/s13012-015-0209-1
Disclosures of Interest: Elise Berlan receives salary support from research grants to Abigail Wexner Research Institute at Nationwide Children’s Hospital from Merck, Inc. and Organon. Elise Berlan also received a past honoraria as a Nexplanon Clinical Trainer from Merck, Inc. Anna Kerlek receives salary support from research grants to Abigail Wexner Research Institute at Nationwide Children’s Hospital from Merck, Inc. and Organon. Organon provided funding for this study to Nationwide Children's Hospital (prime), and Ohio State University investigators are subcontracted to support this study.
Formative evaluation of barriers and facilitators to implementing opioid overdose education and naloxone distribution in Veterans Health Administration homelessness and housing programs
Authors
Dr. Sarah Javier - VA Center for Innovation to Implementation (Ci2i), VA Palo Alto Healthcare System & Stanford University School of Medicine
Ms. Hannah Cheng - VA Center for Innovation to Implementation (Ci2i), VA Palo Alto Healthcare System & Stanford University School of Medicine
Ms. Angela Kyrish - VA Center for Healthcare Organization and Implementation Research (CHOIR), VA Bedford Healthcare System
Mr. Kenneth Bruemmer - Veterans Health Administration Homeless Programs Office
Dr. Thomas Byrne - VA Center for Healthcare Outcomes and Implementation Research (CHOIR), VA Bedford Healthcare System & School of Social Work, Boston University
Dr. Amanda Midboe - VA Center for Innovation to Implementation (Ci2i), VA Palo Alto Healthcare System, Stanford University School of Medicine, & Division of Health Policy and Management, School of Medicine, University of California, Davis
Background: Individuals who are homeless or unstably housed face increased risk of opioid overdose and death. Opioid overdose education and naloxone distribution (OEND) is a federally sanctioned, evidence-based strategy to prevent opioid-related overdoses. OEND comprises overdose education and distribution of naloxone, a lifesaving opioid antagonist, to individuals at risk for overdose. Yet, Veterans in the Veterans Health Administration (VA) who are homeless or unstably housed receive OEND at low rates. This study aims to describe factors influencing implementation of OEND in the Department of Housing and Urban Development-VA Supportive Housing (HUD-VASH) program, a permanent supportive housing program for formerly homeless Veterans.
Methods: In a pre-implementation formative evaluation, we used a qualitative research design and semi-structured interviews with HUD-VASH supervisors and staff to assess factors influencing OEND implementation at two HUD-VASH sites. Interview questions were guided by the Dynamic Sustainability Framework. We analyzed all interview data using rapid thematic analysis.
Results: We interviewed 30 supervisors and staff across two HUD-VASH sites between September and December 2021. Five themes emerged as being consequential to OEND implementation and its sustainment in HUD-VASH: (1) Perceived value of OEND for HUD-VASH Veterans; (2) Knowledge about OEND, including clarifying misperceptions about staff’s role in OEND implementation; (3) Beliefs that Veterans may be hesitant to discuss substance use and OEND; (4) Perceived lack of access to resources that would support implementation (e.g., naloxone prescribers); and (5) Factors that would help sustain OEND implementation (e.g., leadership support, data showing decreases in overdoses as a result of implementation).
Conclusion: Our findings highlight factors that influence implementation of OEND among individuals who are homeless or unstably housed. Implementation strategies that bolster the perceived benefit of OEND for individuals at risk for opioid overdose, increase OEND knowledge, and reduce stigma and access-barriers are likely to enhance implementation and sustainment of OEND.
References
1. Chambers, D.A., Glasgow, R.E. & Stange, K.C. (2013). The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implementation Science, 8, 117. https://doi.org/10.1186/1748-5908-8-117
2. Gale, R. C., Wu, J., Erhardt, T., Bounthavong, M., Reardon, C. M., Damschroder, L. J., & Midboe, A. M. (2019). Comparison of rapid vs. in-depth qualitative analytic methods from a process evaluation of academic detailing in the Veterans Health Administration. Implementation Science, 14(11), 1-12. doi: 10.1186/s13012-019-0853-y
3. Midboe, A. M., Byrne, T., Smelson, D., Jasuja, G., McInnes, K., & Troszak, L. K. (2019). The Opioid Epidemic In Veterans Who Were Homeless Or Unstably Housed. Health Affairs, 38(8), 1289–1297. https://doi.org/10.1377/hlthaff.2019.00281
Disclosures of Interest: None declared
Identification and mitigation of unintended consequences: What implementation science can learn from the fields of bioethics and health equity
Authors
Ms. Christina Johnson - University of Pennsylvania
Dr. Rinad Beidas - University of Pennsylvania
Ms. Isabelle Kaminer - University of Pennsylvania
Dr. Rachel C Shelton - Columbia University Mailman School of Public Health
Dr. Katelin Hoskins - University of Pennsylvania
Background: Unintended consequences (UCs) are positive or negative spillover effects of well-intentioned interventions. Given the risks of reifying or creating health inequities posed by implementing interventions at scale, negative UCs – especially equity-related UCs – are highly relevant for implementation research. The Standards for Reporting Implementation Studies statement1 and the World Health Organization’s Ethics in Implementation Research Facilitator’s Guide2 respectively call on implementation researchers to report on and attend to possible UCs in their work. However, the concept of UCs requires further characterization within Implementation Science; moreover, consolidated guidance is needed for (1) identifying UCs, (2) balancing implementation benefits and UC harms, and (3) potentially mitigating UCs.3, 4 To address gaps in the literature on equity-related UCs and related ethical considerations, this study aims to gain a comprehensive understanding of how experts from the fields of bioethics, health equity, and implementation science conceptualize UCs within implementation research.
Methods: We are purposively sampling scholars who identify as bioethicists, health equity experts, and implementation scientists (n = 8 each; 24 total). Participants complete semi-structured qualitative interviews, and are asked about: (1) definitions of, identification and monitoring of, and obligations to mitigate UCs; and (2) ethical and equity considerations related to UCs in implementation studies. Data analysis is being conducted concurrently with data collection in NVivo software using reflexive thematic analysis.5
Results: Data collection and analysis are ongoing and will be completed by July and August 2022, respectively.
Conclusion: Findings from this study will promote a more comprehensive understanding of UCs and close important gaps in the literature. Additionally, findings will provide an evidence base to inform future implementation study designs with considerations of health equity and ethics. Finally, this work will advance the field of implementation science by providing guidance to identify, evaluate, and mitigate UCs, in efforts to generate more just and equitable outcomes.
References
1. Pinnock, H., Barwick, M., Carpenter, C. R., Eldridge, S., Grandes, G., Griffiths, C. J., Rycroft-Malone, J., Meissner, P., Murray, E., Patel, A., Sheikh, A., & Taylor, S. J. C. (2017). Standards for Reporting Implementation Studies (StaRI) statement. BMJ, i6795. https://doi.org/10.1136/bmj.i6795
2. World Health Organization. (2019). Ethics in implementation research facilitator's guide. https://apps.who.int/iris/bitstream/handle/10665/325608/9789241515375-eng.pdf?ua = 1
3. Baumann, A. A., & Cabassa, L. J. (2020). Reframing Implementation Science to address inequities in healthcare delivery. BMC Health Services Research, 20. https://doi.org/10.1186/s12913-020-4975-3
4. Shelton, R. C., Adsul, P., Oh, A., Moise, N., & Griffith, D. M. (2021). Application of an antiracism lens in the field of Implementation Science (IS): Recommendations for reframing implementation research with a focus on justice and racial equity. Implementation Research and Practice, 2. https://doi.org/10.1177/26334895211049482
5. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
Disclosure of Interest: Dr. Beidas reports receiving grants from the National Institutes of Health, Patient Centered Outcomes Research Institute, the US Centers for Disease Control and Prevention, and the National Psoriasis Foundation outside of the submitted work. Dr. Beidas is principal at Implementation Science & Practice, LLC. Dr. Beidas receives royalties from Oxford University Press. Dr. Beidas receives consulting fees from United Behavioral Health and OptumLabs. Dr. Beidas serves on the advisory boards for Optum Behavioral Health, AIM Youth Mental Health Foundation, and the Klingenstein Third Generation Foundation outside of the submitted work.
Using the consolidated framework for implementation research to identify and describe factors critical for the implementation of outpatient pharmacy programs
Authors
Dr. Michelle Keller - Division of General Internal Medicine-Health Services Research, Department of Medicine, Cedars-Sinai Medical Center; Department of Health Policy and Management, UCLA Fielding School of Public Health
Mr. Nabeel Qureshi - Division of General Internal Medicine-Health Services Research, Department of Medicine, Cedars-Sinai Medical Center; Pardee RANDGraduate School
Background: Pharmacist-led programs have been integrated into primary and specialty care clinics in a variety of ways, for example, to improve diabetes outcomes via patient education and counseling. Yet factors important to the implementation of different outpatient pharmacy models have not been well elucidated (Bacci et al., 2019). The objective of this study was to identify patient-, provider-and health system-level barriers and enablers to implementing pharmacy-led programs in the outpatient setting.
Methods: We used semi-structured interviews with key informants (n = 14) working in various roles throughout a large health system, including ambulatory clinical pharmacists (n = 6), pharmacy managers (n = 3), medical directors and physician leaders (n = 2), and operations and quality managers (n = 3). We coded the interviews using a codebook derived from the Consolidated Framework for Implementation Research (CFIR), which details various internal and external factors important for implementation (Damschroder et al., 2009; Keith, Crosson, O’Malley, Cromp, & Taylor, 2017).
Results: Our findings demonstrate that the following factors are critical for implementing successful ambulatory care pharmacy programs: (1) external policies and incentives from payors such as incentives tied to quality measures and the inability for pharmacists to bill health insurers directly for their services (2) embedding pharmacists in physician offices to increase visibility and connection to providers (implementation design), (3) reimbursement models which generate revenue for the health system (implementation design), and (4) a strong organizational culture of pharmacy support (inner setting).
Conclusion: Findings from this qualitative study of key informants within one health system level reveal that external policies and incentives play a significant role in shaping health system programs. Ensuring that quality metrics used in Accountable Care Organizations, Medicare Advantage, and other Medicare programs address issues of most import to older adults will be critical to supporting pharmacy programs. Our findings will also be used to co-design a polypharmacy clinic in the health system, leveraging enablers and addressing identified barriers.
References
1. Bacci, J. L., Bigham, K. A., Dillon-Sumner, L., Ferreri, S., Frail, C. K., Hamada, C. Y., . . . Curran, G. M. (2019). Community pharmacist patient care services: A systematic reviewof approaches used for implementation and evaluation. Journal of the American College of Clinical Pharmacy, 2(4), 423-432. doi:https://doi.org/10.1002/jac5.1136
2. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. ImplementationScience, 4, 50. doi:10.1186/1748-5908-4-50
3. Keith, R. E., Crosson, J. C., O’Malley, A. S., Cromp, D., & Taylor, E. F. (2017). Using the Consolidated Framework for Implementation Research (CFIR) to produce actionable findings: a rapid-cycle evaluation approach to improving implementation. Implementation Science, 12(1), 15. doi:10.1186/s13012-017-0550-7
Disclosures of Interest: None declared
Development of the Whole School, Whole Community, Whole Child policy and practice blueprints
Authors
Dr. Jessica Koslouski - University of Connecticut
Dr. Sandra Chafouleas - University of Connecticut
Dr. Emily Iovino - University of Connecticut
Dr. Marlene Schwartz - University of Connecticut
Background: The CDC’s Whole School, Whole Community, Whole Child (WSCC) model advocates coordination and integration of school efforts that have historically been siloed. Despite support for a WSCC approach, uptake has been slowed by a lack of guidance on putting the model into action. This guidance must be driven by attention to implementation science along with focus on equitable whole child outcomes. In this presentation, we describe the process for developing school action planning resources, the WSCC Policy and Practice Blueprints.
Methods: The WSCC Policy and Practice Blueprints were developed over a 4-year period using an iterative design process that included cycles of research, development, and consultation with school leaders and implementation experts. Guided by Backwards Design (Wiggins & McTighe, 2005), we identified the desired outcomes of the blueprinting process, acceptable evidence of those outcomes, and then planned learning activities to facilitate those outcomes. We reviewed literature on implementation, coordination, and integration and identified theoretical frameworks to guide these efforts. We also reviewed available WSCC resources and sought school personnel feedback on developed materials.
Results: We identified desired outcomes of the WSCC Policy and Practice Blueprint process to include school- or district-based action plans for implementing equitable, coordinated, and integrated WSCC initiatives. We designed policy and practice blueprints that include a series of learning activities in which school personnel explore their context, evaluate directions, and establish actions for strengthening whole child efforts. These activities are guided by theories of implementation (Sustain Collaborative, 2021), coordination (Nadler & Tushman, 1980), and integration (Wiggins & McTighe, 2005), and culminate in producing the desired evidence of an action plan to drive adoption of WSCC.
Conclusion: These guidance blueprints may support more successful adoption of the WSCC model, with next steps to include testing of the full blueprints and identifying any additional supports needed for successful uptake of WSCC.
References
1. Nadler, D. & Tushman, M. (1980). A model for diagnosing organizational behavior. Organizational Dynamics, 9(2): 35–51. https://doi.org/10.1016/0090-2616(80)90039-X
2. Sustain Collaborative (2021). Sustain Collaborative framework. Authors. Retreived April 27, 2022 from www.sustaincollaborative.org
3. Wiggins, G., & McTighe, J. (2005). Understanding by design: Expanded 2nd edition. ASCD.
Disclosures of Interest: None declared
A randomized trial examining implementation and intervention effectiveness of a multifaceted implementation strategy compared to a single strategy for improving adherence to the guideline for prevention of mental ill-health among school personnel
Authors
Dr. Lydia Kwak - Karolinska Institutet
Dr. Anna Toropova - Karolinska Institutet
Dr. Christina Björklund - Karolinska Institutet
Prof. Gunnar Bergström - Karolinska Institutet and Högskolan i Gävle
Prof. Liselotte Schäfer Elinder - Karolinska Institutet and Stockholm Region
Dr. Kjerstin Stigmar - Lund University
Mr. Andreas Rödlund - Karolinska Institutet
Dr. Charlotte Wåhlin - Linköping University
Prof. Irene Jensen - Karolinska Institutet
Background: Common mental disorders are prevalent in school personnel in Sweden as well as in many other countries (Gray et al., 2017; Arvidsson et al., 2019). This is due to a number of social and organizational risk factors in the school work environment such as high workload levels, student discipline problems, and lack of support from school management. However, there is limited knowledge about the systematic approach to managing these factors within schools in order to prevent mental ill-health among school personnel (Kwak et al., 2019). This study aimed to compare the effectiveness of a single (an educational meeting) vs. a multifaceted implementation strategy (an educational meeting, an ongoing training in the form of workshops, implementation teams and Plan-Do-Study-Act cycles) for implementing the guideline recommendations for the prevention of mental ill-health in schools. Implementation effectiveness was investigated in relation to the degree to which the guideline recommendations are implemented. Intervention effectiveness was studied with regards to social and organizational risk factors for mental ill-health among school personnel.
Method: A cluster-randomized controlled trial with a 6- and 12-months follow-up was conducted among school personnel in 19 Swedish schools. Implementation effectiveness outcome measure was adherence to guideline recommendations. Intervention effectiveness outcome measure was exhaustion (primary outcome) as well a number of secondary outcomes, e.g. self-reported stress and health, psychosocial safety climate.
Generalized Linear Mixed Models were used as the main analytical approach.
Results: Small improvements in guideline adherence were observed for both groups, however, there were no statistically significant differences between baseline, 6- and 12-months follow-up. Preliminary results on intervention effectiveness demonstrate statistically significant differences for some of the outcomes, including stress and psycho-social safety climate.
Conclusions: The study fills the knowledge gaps in school-based implementation research. Psychometric properties of a self-reported outcome measure of adherence need to be further improved. Implementation mechanisms should be explored to understand how the implementation strategies worked.
References
1. Gray, C., Wilcox, G., & Nordstokke, D. (2017). Teacher mental health, school climate, inclusive education and student learning: A review. Canadian Psychology/psychologie canadienne, 58(3), 203.
2. Arvidsson, I., Leo, U., Larsson, A., Håkansson, C., Persson, R., & Björk, J. (2019). Burnout among school teachers: quantitative and qualitative results from a follow-up study in southern Sweden. BMC Public Health, 19(1), 1-13.
3. Kwak, L., Lornudd, C., Björklund, C., Bergström, G., Nybergh, L., Elinder, L. S., … & Jensen, I. (2019). Implementation of the Swedish Guideline for Prevention of Mental ill-health at the Workplace: study protocol of a cluster randomized controlled trial, using multifaceted implementation strategies in schools. BMC Public Health, 19(1), 1-19.
Disclosures of Interest: None declared
Centering community-academic partnerships and key stakeholders to facilitate scaling up infant achievements for community implementation
Authors
Dr. Rebecca Landa - Kennedy Krieger Institute
Dr. Ebony Holliday - Kennedy Krieger Institute
Background: Participation in early intervention can positively impact developmental trajectories and maximize outcomes for young children at-risk for social-communication delays.1-4 Initial efficacy of the Infant Achievements (IA) intervention was established in a randomized controlled trial of caregiver/infant dyads. Caregivers in our IA group received coaching in evidence-based practices to promote early language skills in their infants. IA caregivers demonstrated significantly greater increases in implementation fidelity from pre- to post-intervention (β=10.6, p = 0.004); their infants exhibited greater increases in social communication behaviors. These promising results motivated us to scale up and spread IA into a sustainable community-based program.
Methods: We use the Exploration, Preparation, Intervention, Sustainment (EPIS) Model5-6 to inform our research. A community advisory board (CAB) was developed to explore the inner/outer contextual factors for scaling IA. CAB members completed the Hexagon Tool7 to assess IA’s fit and feasibility across six implementation and program indicators. We then scaled out IA to an outpatient clinic sample (population fixed, different delivery system). A CAB workgroup (community-academic partnership) was formed to build infrastructure, and support bridging our contextual factors for scale up. Community focus groups are being conducted with stakeholders to explore buy-in and support the tailoring of IA.
Results: The CAB facilitated meetings/interviews with 33 additional stakeholders. A stakeholder mapping and analysis matrix yielded individuals with high support/high influence (n = 20), and high support/low influence (n = 13). Additional social networking analysis (NodeXL) is being conducted to examine interrelationships and linkages among stakeholders. The CAB completed Hexagon Tool ratings at Time 1 and 2 (1 = low/5 = high; M = 3.1). Our preliminary scaling out of IA with 10 clinic families supported implementation feasibility (71% of 136 appointments were attended). Focus group data are in progress (completed by 6/1/22).
Conclusion: Our Exploration phase activities effectively position us to continue scaling IA for uptake in local communities. Next steps include capacity building with new delivery systems.
References
1. Feil, E. G., Baggett, K., Davis, B., Landry, S., Sheeber, L., Leve, C., & Johnson, U. (2020). Randomized control trial of an internet-based parenting intervention for mothers of infants. Early Childhood Research Quarterly, 50, 36-44. https://doi.org/10.1016/j.ecresq.2018.11.003
2. Leung, C. Y., Hernandez, M. W., & Suskind, D. L. (2020). Enriching home language environments among families from low-SES backgrounds: A randomized controlled trial of a home visiting curriculum. Early Childhood Research Quarterly, 50, 24-35. https://doi.org/10.1016/j.ecresq.2018.12.005
3. McGillion, M. M., Pine, J. M., Herbert, J. S., & Matthews, D. (2017). A randomized controlled trial to test the effect of promoting caregiver contingent talk language development in infants from diverse socioeconomic status backgrounds. The Journal of Child Psychology and Psychiatry, 58(10), 1122-1131. https://doi.org/10.1111/jcpp.12725
4. Rowe, M. L., & Leech, K. A. (2019). A parent intervention with a growth mindset approach improves children’s early gesture and vocabulary development. Developmental Science, 22, 1-10. https://doi.org/10.1111/desc.12792
5. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health, 38, 4-23. https://doi.org/10.1007/s10488-010-0327-7
6. Moullin, J. C., Dickson, K.S., Stadnick, N.A., Rabin, B., & Aarons, G.A. (2019). Systemic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implementation Science, 14(1), 1-16. https://doi.org/10.1186/s13012-018-0842-6
7. Metz, A., & Loulson, L. (2018). The Hexagon tool: Exploring context. National Implementation Research Network.
Disclosures of Interest: None declared
Implementation outcomes for the national implementation of a mental health innovation for Nurse Family Partnership
Authors
Ms. Alasia Ledford - University of North Carolina at Chapel Hill School of Nursing
Dr. Jennifer Leeman - University of North Carolina at Chapel Hill School of Nursing
Ms. Sharon Sprinkle - Nurse-Family Partnership National Service Office
Mrs. Mariarosa Gasbarro - Prevention Research Center, University of Colorado Anschutz Medical Campus
Mr. Michael Knudtson - Prevention Research Center, University of Colorado Anschutz Medical Campus
Ms. Elizabeth Bernhardt - University of North Carolina at Chapel Hill School of Nursing
Dr. Paula Zeanah - College of Nursing and Allied Health Professions and Picard Center for Child Development and Lifelong Learning, University of Louisiana at Lafayette
Ms. Georgette McMichael - Nurse-Family Partnership National Service Office
Mrs. Allison Mosqueda - Nurse- Family Partnership, Invest in Kids
Dr. Linda Beeber - University of North Carolina at Chapel Hill School of Nursing
Background- Perinatal mood disorders affect the health and well-being of nearly half of low-income mothers. To address this disparity, a research/practice partnership designed the Mental Health Innovation (MHI) for Nurse Family Partnership (NFP). Four implementation strategies compose the MHI: online training modules, clinical resources, team meeting modules, and virtual consultation.
Methods- Guided by the Reach-Effectiveness-Adoption-Implementation-Maintenance (RE-AIM) framework, implementation outcomes were evaluated via a convergent, mixed methods observational design. Data were collected through key informant interviews, focus groups, and surveys as well as NFP’s national data management system. Quantitative data were analyzed using counts and summary statistics. Qualitative themes were generated through content analysis.
Results- The MHI reached 264 agencies across 40 states serving 51,398 low-income mothers (31.2% African American, 29.9% Latina). Nurses and their supervisors generally valued the implementation strategies but were more likely to adopt some strategies than others. Most nurses (n = 2,100) adopted each of the five online modules (60%-76%). Less than half of nurse supervisors surveyed (n = 125) adopted team meeting modules. Of 110 teams invited to participate in virtual consultation, 40.9% (n = 45) participated and of those 33 (73.3%) attended all six sessions. Supervisors and nurses identified multiple factors that influenced their adoption of the four implementation strategies (e.g., time requirement, ease of use, and practical application).
Conclusion- Findings from the evaluation were applied to further improve implementation strategies. Future evaluation is needed to determine how well the refined strategies work and whether they impact nurse home visitors’ capacity to address the mental health needs of low-income mothers.
References
1. Glasgow, R. E., Harden, S. M., Gaglio, B., Rabin, B., Smith, M. L., Porter, G. C., Ory, M. G., & Estabrooks, P. A. (2019). RE-AIM Planning and Evaluation Framework: Adapting to new science and practice with a 20-year review. Frontiers in Public Health, 7, 64. https://doi.org/10.3389/fpubh.2019.00064
2. Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: the RE-AIM framework. American Journal of Public Health, 89(9), 1322–1327. https://doi.org/10.2105/ajph.89.9.1322
3. Kitzman, H., Olds, D. L., Knudtson, M. D., Cole, R., Anson, E., Smith, J. A., Fishbein, D., DiClemente, R., Wingood, G., Caliendo, A. M., Hopfer, C., Miller, T., & Conti, G. (2019). Prenatal and infancy nurse home visiting and 18-year outcomes of a randomized trial. Pediatrics, 144(6), e20183876. https://doi.org/10.1542/peds.2018-387
4. Li, Y., Long, Z., Cao, D., & Cao, F. (2017). Social support and depression across the perinatal period: A longitudinal study. Journal of Clinical Nursing, 26(17-18), 2776–2783. https://doi.org/10.1111/jocn.13817
5. Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015). A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10, 21. https://doi.org/10.1186/s13012-015-0209-
6. Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013). Implementation strategies: recommendations for specifying and reporting. Implementation Science, 8(1), 139. https://doi.org/10.1186/1748-5908-8-139
Disclosures of Interest: None declared
Organizational climate and readiness for culturally adapted early intervention: A qualitative study
Authors
Dr. James Lee - University of Kansas
Ms. Adriana Kaori Terol - Department of Special Education, University of Illinois
Ms. Christy Yoon - Department of Special Education, University of Illinois
Dr. Brian Boyd - University of Kansas, Juniper Gardens Children's Project
Background: Evidence-based interventions (EBI) has been emphasized in education literature for the past few decades to address the research-to-practice gaps (Cook & Odom, 2013). The dissemination and implementation of these EBI in natural environments by practitioners often face barriers (Fixsen et al., 2013), such as in early intervention. Although optimal outcomes require close examination of how to adapt existing interventions and optimize their delivery to best suit the needs of the targeted population (Rieth et al., 2018), little is known about how early interventionists and their organizations are equipped to provide early intervention services to culturally and linguistically diverse families.
Methods: We are conducting a qualitative study with the purpose to gain a better understanding of early intervention providers’ perceptions on working with culturally and linguistically diverse young children with disabilities and their families. As this work is in progress, we have individually interviewed five participants via Zoom so far and plan to conduct at least 10 more interviews. Each interview was recorded and transcribed to be analyzed using thematic analysis.
Results: The preliminary data suggest that these professionals experience difficulties related to working with culturally and linguistically diverse families partly due to (a) limited training and awareness, (b) limited organizational support, and (c) policies not being lenient enough. Some of the strategies that were suggested by the participants include: (a) allowing time for more professional development on cultural awareness and adaptation, (b) providing content-related training to interpreters who work with families, and (c) using local resources to alleviate disconnect between providers and families.
Conclusion: Although cultural adaptation in early intervention may alleviate challenges brought to culturally and linguistically diverse families, we still know very little about this topic. These data will be helpful in designing implementation research that will address the gap in both research and practice.
References
1. Cook, B. G., & Odom, S. L. (2013). Evidence-based practices and Implementation Science in special education. Exceptional Children, 79(2), 135-144.
2. Fixsen, D., Blase, K., Metz, A., & Van Dyke, M. (2013). Statewide implementation of evidence-based programs. Exceptional Children, 79(2), 213-230.
3. Rieth, S. R., Stahmer, A. C., & Brookman-Frazee, L. (2018). A community collaborative approach to scaling-up evidence-based practices: Moving parent-implemented interventions from research to practice. In Handbook of parent-implemented interventions for very young children with autism (pp. 441-458). Springer, Cham.
Disclosures of Interest: None declared
Validity and reliability of the Spanish version of the Implementation Leadership Scale
Authors
Dr. Marta Llarena - Biocruces Bizkaia Health Research Institute, Network for Research on Chronicity, Primary Care, and Health Promotion (RICAPPS), Carlos III Institute of Health
Dr. Alvaro Sanchez - Primary Care Research Unit of Bizkaia, Deputy Directorate of Healthcare Assistance, Biocruces Bizkaia Health Research Institute, Basque Healthcare Service - Osakidetza, RICAPPS Network, Carlos III Institute of Health
Dr. Marta Montejo - Rontegi-Barakaldo Primary Care Center, University of the Basque Country, UPV/EHU, Biocruces Bizkaia Health Research Institute, Network for Research on Chronicity, Primary Care, and Health Promotion (RICAPPS), Carlos III Institute of Health,
Dr. Natalia Paniagua - Pediatric Emergency Department, Cruces University Hospital, Biocruces Bizkaia Health Research Institute, Network for Research on Chronicity, Primary Care, and Health Promotion (RICAPPS), Carlos III Institute of Health
Mr. Mikel Rueda-Etxebarria - Biocruces Bizkaia Health Research Institute
Dr. Javier Benito - Pediatric Emergency Department, Cruces University Hospital, Biocruces Bizkaia Health Research Institute, University of the Basque Country, UPV/EHU, RICAPPS Network, Carlos III Institute of Health
Background: There is a need for pragmatic and reliable measures regarding sound factors that affect Evidence Based Practices (EBP) adoption and implementation in different languages and cultural environments. The Implementation Leadership Scale (ILS) is a brief and efficient measurement tool of strategic leadership for EBP implementation. The objective of this study was to assess the psychometric properties of the Spanish version of ILS.
Methods: Translation of the original ILS into Spanish consisted of forward translation, panel meeting, backward translation, pre-testing and final version agreement. Scale face and content validity were assessed. Psychometric properties were examined in 144 primary care professionals (family physicians, pediatricians, practice and pediatric nurses) involved in several implementation/improvement research projects. ILS factorial structure was tested by Confirmatory Factor Analysis (CFA). Reliability was assessed by internal consistency analysis. Pearson correlation between ILS and the Organizational Support dimension of the Organizational Readiness for Knowledge Translation (OR4KT) questionnaire in the subsample of pediatricians and pediatric nurses (n = 52) was estimated for convergent validity analysis.
Results: CFA results indicated that the original four-factor model fit the data well (x2 = 112.43; df = 50; p < .001). All standardized first-order and second-order factor loadings were significant. Fit indexes showed acceptable figures (GFI = 0.89; CFI = 0.98; RMSEA = 0.09; SRMR =0.037). Cronbach’s alpha coefficient for the four dimensions of ILS ranged from 0.92 to 0.98, while the reliability estimated for the total scale was 0.97. Results of convergent validity revealed moderate correlation (r = 0.62) between ILS and OR4KT’s Organizational Support dimension.
Conclusion: CFA results indicated that the four-factor model for the 12-item Spanish version of ILS is well adapted and consistent with the original version of the tool. The availability of ILS will allow Spanish-speaking researchers to assess and advance understanding regarding the implementation leadership’s construct as a predictor of organizational implementation context.
References
1. Aarons, G. A., Ehrhart, M. G., & Farahnak, L. R. (2014). The Implementation Leadership Scale (ILS): development of a brief measure of unit level implementation leadership. Implementation Science : IS, 9(1), 45. https://doi.org/10.1186/1748-5908-9-45
2. Lyon, A. R., Cook, C. R., Brown, E. C., Locke, J., Davis, C., Ehrhart, M., & Aarons, G. A. (2018). Assessing organizational implementation context in the education sector: confirmatory factor analysis of measures of implementation leadership, climate, and citizenship. Implementation Science : IS, 13(1), 5. https://doi.org/10.1186/s13012-017-0705-6
3. Aarons, G. A., & Sommerfeld, D. H. (2012). Leadership, innovation climate, and attitudes toward evidence-based practice during a statewide implementation. Journal of the American Academy of Child and Adolescent Psychiatry, 51(4), 423–431. https://doi.org/10.1016/j.jaac.2012.01.018
4. Grandes, G., Bully, P., Martinez, C., & Gagnon, M. P. (2017). Validity and reliability of the Spanish version of the Organizational Readiness for Knowledge Translation (OR4KT) questionnaire. Implementation Science: IS, 12(1), 128. https://doi.org/10.1186/s13012-017-0664-y
Disclosures of Interest: None declared
The impact of COVID-19 on the RE-AIM dimensions of the University of California Diabetes Prevention Program Initiative
Authors
Dr. Tamra Loeb - UCLA David Geffen School of
Medicine
Dr. Maryam Gholami - Herbert Wertheim School of Public Health and Human Longevity Science, University of California San Diego
Ms. Kate Ramm - UCLA David Geffen School of Medicine
Ms. Sarah Alkhatib - UCI MPH Candidate
Ms. Kelly Shedd - UCI Health
Ms. Samantha Soetenga - UCLA Recreation
Dr. Nicholas Jackson - UCLA David Geffen School of Medicine
Dr. Obidiugwu Duru - UCLA David Geffen School of Medicine
Dr. Carol Mangione - UCLA David Geffen School of Medicine
Dr. Alison Hamilton - UCLA David Geffen School of Medicine
Dr. Tannaz Moin - UCLA David Geffen School of Medicine
Background: The University of California’s Diabetes Prevention Program (UC DPP) Initiative was implemented across all 10 UC campuses in 2018. The COVID-19 pandemic required changes to program delivery, including transitioning from in-person to virtual delivery (i.e., Zoom). Our goal was to assess the impacts of COVID-19 on UC DPP Initiative across UC.
Methods: We conducted qualitative interviews with 53 UC DPP leaders, coordinators, and participants to assess pandemic impacts on the reach, effectiveness, adoption, implementation, and maintenance of UC DPP. Transcripts were analyzed using rapid qualitative analysis and emergent themes were categorized using RE-AIM dimensions.
Results: Among UC DPP participants (n = 28), COVID-19 primarily impacted perceptions of UC DPP effectiveness and implementation. Some participants perceived program effectiveness to be negatively impacted, given their preference for in-person sessions, which they felt provided more engagement, peer support and accountability. Implementation challenges included problems with virtual format (e.g., “Zoom fatigue”); however, benefits were also noted (e.g., increased flexibility, maintenance of connections during campus closures). UC DPP coordinators (n = 17) perceived COVID-19 as positively impacting UC DPP reach, since virtual platforms increased reach among those who could not participate in-person, and negatively impacting effectiveness due to reduced engagement and loss of peer support. UC leaders (n = 8) highlighted COVID-19’s positive impacts on reach (e.g., increased availability, accessibility) and its negative impacts on effectiveness (e.g., less interaction with the virtual platform).
Conclusion: Perceptions of the impact of COVID-19 pandemic on reach, effectiveness, and implementation of the UC DPP varied across stakeholder groups, although all stakeholders noted a potential negative impact of COVID-19 on program effectiveness. Unanticipated program adaptations, including virtual delivery, present potential benefits as well as perceived drawbacks across RE-AIM dimensions. Understanding the impact of these pandemic related changes can help maximize RE-AIM and inform future strategies for UC DPP delivery.
References
1. Gholami, M., Jackson, N. J., Chung, U., Duru, O. K., Shedd, K., Soetenga, S., Loeb, T., Elashoff, D., Hamilton, A. B., Mangione, C. M., Slusser, W., & Moin, T. (2021). Evaluation of the University of California Diabetes Prevention Program (UC DPP) Initiative. BMC public health, 21(1), 1775. https://doi.org/10.1186/s12889-021-11731-7
2. Glasgow, R. E., Harden, S. M., Gaglio, B., Rabin, B., Smith, M. L., Porter, G. C., Ory, M. G., & Estabrooks, P. A. (2019). RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review. Frontiers in public health, 7, 64. https://doi.org/10.3389/fpubh.2019.00064
3. Knowler, W. C., Barrett-Connor, E., Fowler, S. E., Hamman, R. F., Lachin, J. M., Walker, E. A., Nathan, D. M., & Diabetes Prevention Program Research Group (2002). Reduction in the incidence of type 2 diabetes with lifestyle intervention or metformin. The New England journal of medicine, 346(6), 393–403. https://doi.org/10.1056/NEJMoa012512
Disclosures of Interest: None declared
Economic, socio-cultural, and political implementation determinants that count: findings from a meta-review on policies promoting healthy lifestyle
Authors
Prof. Aleksandra Luszczynska - SWPS University of Social Sciences and Humanities
Ms. Karolina Lobczowska - SWPS University of Social Sciences and Humanities
Dr. Anna Banik - SWPS University of Social Sciences and Humanities
Dr. Sarah Forberger - Leibniz Institute for Prevention Research and Epidemiology – BIPS
Dr. Krzysztof Kaczmarek - Medical University of Silesia in Katowice
Prof. Thomas Kubiak - Johannes Gutenberg University Mainz
Dr. Agnieszka Neumann-Podczaska - Poznan University of Medical Sciences
Prof. Piotr Romaniuk - Medical University of Silesia in Katowice
Ms. Marie Scheidmeir - Johannes Gutenberg University Mainz
Mr. Daniel Scheller - University Hospital Ulm
Prof. Juergen Steinacker - University Hospital Ulm
Ms. Janine Wendt - University Hospital Ulm
Prof. Hajo Zeeb - Leibniz Institute for Prevention Research and Epidemiology – BIPS
Background: Applying the Context and Implementation of Complex Interventions (CICI) framework (Pfadenhauer et al., 2017), this meta-review explored social inequality context-related policy implementation determinants (Nilsen, 2015) from seven domains: geographical, epidemiological, sociocultural, economic, ethics-related, political, and legal. In particular, we aimed at identifying which context-related determinants are consistently indicated in the implementation process of policies targeting healthy diet, physical activity, and/or sedentary behavior.
Methods: Data from nine databases and nine international stakeholder repositories (e.g. the World Health Organization) were analyzed. Context-related determinants were considered strongly supported when referenced in ≥ 60% of the reviews/stakeholder documents. The ROBIS tool was applied to assess the quality-related risk of bias. The meta-review was preregistered in the PROSPERO database (#CRD42019133341). The review was conducted as one of the tasks undertaken by the Policy Evaluation Network consortium (Lakerveld et al., 2020).
Results: Across 42 documents included in the review combing published reviews (k = 25), and stakeholder documents (k = 17), six implementation determinants from three context domains received strong support:: economic resources at the macro level (66.7% of analyzed documents) and meso/micro levels (71.4%); sociocultural context determinants at the meso/micro level, references to knowledge/beliefs/abilities of target groups (69.0%) and implementers (73.8%); and political context determinants (interrelated policies supported in 71.4% of analyzed reviews/documents; policies within organizations, 69.0%). Limited support was found for the ethics domain. Legal and epidemiological domains were supported in case of specific settings or the target behaviors (e.g., the legal domain was supported in case of implementation of healthy diet-related policies whereas geographical domain was supported in case of physical activity-promoting policies).
Conclusion: The results may help researchers and practitioners to prioritize the contextual factors that should be considered when planning the implementation of policies promoting healthy diet and physical activity.
References
1. Lakerveld, J., Woods, C., Hebestreit, A., Brenner, H., Flechtner-Mors, M., Harrington, J. M., Kamphuis, C. B. M., Laxy, M., Luszczynska, A., Mazzocchi, M., Murrin, C., Poelman, M. P., Steenhuis, I., Roos, G., Steinacker, J. M., Stock, C., van Lenthe, F. J., Zeeb, H., Zukowska, J., Ahrens, W. (2020). Advancing the evidence base for public policies impacting on dietary behaviour, physical activity and sedentary behaviour in Europe: The Policy Evaluation Network promoting a multidisciplinary approach. Food Policy, 96, 101873. https://doi.org/10.1016/j.foodpol.2020.101873.
2. Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation Science, 10, 53. https://doi.org/10.1186/s13012-015-0242-0.
3. Pfadenhauer, L.M., Gerhardus, A., Mozygemba, K., Lysdal, K. B., Booth, A., Hofmann, B., Wahlster, P., Polus, S., Burns, S., Brereton, L., & Rehfuess, E. (2017). Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions (CICI) framework. Implementation Science, 12, 21. https://doi.org/10.1186/s13012-017-0552-5
Disclosures of Interest: None declared
Patterns of program sustainability capacity among national DPP delivery organizations: A latent profile analysis
Authors
Ms. Lillian Madrigal - Emory University School of Public Health
Dr. Regine Haardörfer - Emory University School of Public Health
Dr. Michelle Kegler - Emory University
Dr. Mary Beth Weber - Emory University School of Public Health
Dr. Linelle Blais - Emory University School of Public Health
Dr. Cam Escoffery - Emory University School of Public Health
Background. The National Diabetes Prevention Program (DPP), a lifestyle intervention to delay the onset of diabetes, has been rigorously tested, adapted, and scaled nation-wide (Albright, 2012; Ely et al., 2017). Over 2000 registered National DPP organizations are implementing the program; however, not much is known about their capacity for sustainability. We explored patterns of their sustainability capacity using the Program Sustainability Assessment Tool (PSAT) in order to understand their sustainability strengths and weaknesses and associated organizational characteristics.
Methods. This study analyzed organization characteristics and PSAT data from a 2021 cross-sectional online survey with National DPP Staff. Latent profile analysis (LPA) was employed to explore patterns of sustainability capacity. LPA identifies latent subpopulations within a population based on a certain set of continuous variables into mutually exclusive classes (Spurk et al., 2020). To estimate associations between derived latent classes and organization characteristics multivariable multinomial logistic regression was conducted. Multiple variable linear regression with the PSAT score as the outcome was used to compare against the LPA model results.
Results. Our analysis included the 440 program implementers with a calculable PSAT score. Two-class through eight-class LPA models were run. Fit statistics for all models indicated good model fit and entropy. The LPA did not find distinct patterns, but resulted in levels of sustainability capacity across all eight PSAT domains. All organizations, despite capacity level, tend to have the same areas of strength (Program Evaluation and Adaptation) and relative weakness (Funding Stability and Partnerships). Higher PSAT scores were associated with the number of staff, virtual delivery modes, grant funding sources, and specific organization types.
Conclusion: The results of the LPA and regression models provide evidence to support the use of the PSAT score to identify organization sustainability capacity reliably and insight into which organizational variables are important to consider for sustainable implementation.
References
1. Albright, A. (2012). The national diabetes prevention program: from research to reality. Diabetes care & education newsletter, 33(4), 4.
2. Ely, E. K., Gruss, S. M., Luman, E. T., Gregg, E. W., Ali, M. K., Nhim, K., Rolka, D. B., & Albright, A. L. (2017). A National Effort to Prevent Type 2 Diabetes: Participant-Level Evaluation of CDC's National Diabetes Prevention Program. Diabetes Care, 40(10), 1331-1341. https://doi.org/10.2337/dc16-2099
3. Spurk, D., Hirschi, A., Wang, M., Valero, D., & Kauffeld, S. (2020). Latent profile analysis: A review and “how to” guide of its application within vocational behavior research. Journal of Vocational Behavior, 120, 103445. https://doi.org/10.1016/j.jvb.2020.103445
Disclosures of Interest: None declared
A research to practice collaboration supporting widespread implementation of evidence-based disclosure support in the veterans health administration
Authors
Ms. Elizabeth Maguire - VA Center for Healthcare Organization and Implementation Research (CHOIR), VA Bedford Healthcare System
Dr. Gavin West - VHA Clinical Services; VA Salt Lake City
Ms. Beth Ann Petrakis - VA Center for Healthcare Organization and Implementation Research (CHOIR), VA Bedford Healthcare System
Ms. Angela Kyrish - VA Center for Healthcare Organization and Implementation Research (CHOIR), VA Bedford Healthcare System
Ms. Zenith Rai - VA Center for Healthcare Organization and Implementation Research (CHOIR), VA Bedford Healthcare System
Dr. A. Rani Elwy - US Department of Veterans Affairs
Background: We are partnering with the VA Clinical Episode Review Team (CERT), an operational program to support the identification and disclosure of large-scale adverse events on a facility-by-facility basis through implementation of our evidence-based disclosure toolkit. 1 Guided by High Reliability Organization (HRO) principles 2 and the QUERI Implementation Roadmap,3 our ongoing evaluation of disclosure processes enterprise-wide seeks to improve patient safety culture nationally, create tools to support this culture, and increase reporting and disclosures when needed.
Methods: Our concurrent mixed methods involves two parts. After bi-weekly CERT meetings, VA employees complete surveys to identify unmet needs. Interviews with stakeholders with and without disclosure experience in the past 5 years examines key components of HRO. Directed content analysis is applied to interview transcripts. Lessons learned are compiled into de-identified infographics shared each week with CERT, to inform ongoing disclosure implementation.
Results: Data from 16 surveys and 25 interviews highlight strengths of psychologically safe culture, strong CERT leadership, focus on continuous learning to improve disclosure and teamwork with subject matter experts (HRO principles). To address pre-implementation barriers (lack of clarity around when/how to report events), we focus on teamwork and communication and psychological safety by regularly presenting national calls to highlight the non-punitive nature and field questions about the CERT process. To address implementation barriers, we focus on improvement of processes using standard tools and continuous learning to clarify disclosure processes and expectations, including creating a CERT overview, templates for presenting events, and communications scripts for Veterans. All tools are available broadly and continually updated to promote sustainment.
Conclusion: The strong leadership and psychologically safe culture of CERT were identified as strengths in both surveys and interviews. In order to sustain and widely implement disclosure processes across all health systems, tools are needed to reach facilities who are not yet engaged.
References
1. Elwy, A. R., Maguire, E.M., McCullough, M., et. al. (2021). A large-scale adverse event disclosure support program generated through embedded research in the Veterans Health Administration. Healthcare: The Journal of Delivery Science and Innovation. 8 (S1):100496. doi: 10.1016/j.hjdsi.2020.100496
2. Veazie, S., Peterson, K., Bourne, D., et. al. (2019). Evidence Brief: Implementation of High Reliability Organization Principles. Washington, DC: Evidence Synthesis Program, Health Services Research and Development Service, Office of Research and Development, Department of Veterans Affairs. VA ESP Project #09-199. https://www.hsrd.research.va.gov/publications/esp/reports.cfm Accessed April 24, 2022.
3. Kilbourne, A.M., Goodrich, D.E., Miake-Lye, I. et, al. (2019). Quality Enhancement Research Initiative Implementation Roadmap: Toward sustainability of evidence-based practices in a Learning Health System. Medical Care. 57 (S10, S3), S286-S293. doi: 10.1097/MLR.0000000000001144
Disclosures of Interest: None declared
Hire within or contract out? Staffing strategies for implementing interventions with peer recovery supporters in child welfare
Authors
Mr. Jared Martin – College of Education & Human Ecology, The Ohio State University
Mrs. Fawn Gadel - Public Children Services Association of Ohio
Dr. Emmeline Chuang – School of Social Welfare, University of California, Berkeley
Dr. Kathryn Lancaster – College of Public Health, The Ohio State University
Dr. Alicia Bunger – College of Social Work, The Ohio State University
Background: Peer recovery supporters (PRS) with lived child welfare (CW) experience and sustained recovery can improve service engagement and timeliness for families involved with CW and affected by parental substance use disorders (SUD; Acri et al., 2021). However, implementing PRS can be challenging due to concerns about hiring individuals with prior felony or child maltreatment records, risk for relapse, and potential for PRS to violate parent boundaries and CW policies (Huebner et al., 2018). This study examined two distinct implementation strategies (hire PRS staff in-house vs. contract-out) and their association with implementation outcomes (program reach and timeliness).
Method: This study used a multiple-case study design with 17 counties implementing START (Sobriety, Treatment, and Recovery Teams), an evidence-informed intervention that uses PRS to help bridge CW and SUD treatment providers (Huebner et al., 2012). Staffing strategies were identified using group interviews (n = 48) with 104 CWS and behavioral health (BH) professionals, and 5 expert panel meetings with community partners. County-level implementation outcomes came from administrative data on 352 families participating in START.
Results: Most CW agencies (n = 15; 88%) contracted-out PRS positions to BH partners; because the majority of CW agencies had prior relationships with these partners, only 7 used separate contracts for PRS. CW agencies that contracted reported higher program reach (93.87% of families received a PRS visit vs 83.57% for in-house PRS). Timeliness (of first PRS contact or SUD treatment access) did not differ. CW stakeholders preferred contracting PRS positions because they perceived that BH organizations had more expertise hiring, supervising, and supporting PRS; however, contracted partners identified difficulties recruiting PRS and a time-consuming PRS certification process.
Conclusions: Contracting out PRS positions might address internal implementation challenges in CW and enhance the feasibility and reach of these PRS interventions. However, success likely depends on strong, trust-based relationships with local BH partners.
References
1. Acri, M., Falek, I., Bunn, M., Domineuez, J., Gopalan, G., & Chacko, A. (2021). Peer models for families involved in the child welfare system: A systematic review. Journal of Public Child Welfare, 1-23. https://doi.org/10.1080/15548732.2021.1996503
2. Huebner, R.A., Hall, M.T., Smead, E., Willauer, T., & Posze, L. (2018). Peer mentoring services, opportunities, and outcomes for child welfare families with substance use disorders. Children and Youth Services Review, 84, 239-246. https://doi.org/10.1016/j.childyouth.2017.12.005
3. Huebner, R. A., Willauer, T., & Posze, L. (2012). The impact of sobriety treatment and recovery teams (START) on family outcomes. Families in Society: The Journal of Contemporary Social Services, 93(3), 196-203. https://doi.org/10.1606/1044-3894.4223
Disclosures of Interest: None declared
Putting equity into practice: Assessing the utility and feasibility of an equity impact assessment for implementation resource development
Authors
Ms. Capri McDonald - The Impact Center at FPG Child Development Institute, University of North Carolina at Chapel Hill
Ms. Sherra Lawrence - The Impact Center at FPG Child Development Institute, University of North Carolina at Chapel Hill
Ms. Jessica Reed - Cornerstone
Ms. Alana Gilbert - UNC Gillings School of Global Public Health
Introduction: Implementation support tools and resources are key to driving the implementation and scale-up of evidence-based interventions, however, there are limited tool development processes that embed an equity lens. The ICTP Tools and Resource Equity Impact Assessment (EIA) was developed to guide teams to use equity best practices and prioritize equitable implementation and dissemination of tools and resources. In this study, we examined the usability and feasibility of the EIA by identifying facilitators and barriers to using equity best practices in tool and resource design. We also identified what worked well with the use of the EIA and improvements that can be made to the tool.
Method: The evaluation team conducted a focus group (n = 5) with an internal project team. Participants were asked to retrospectively reflect on the facilitators and barriers to using the EIA equity best practices in their tool development process. The focus group participants were also asked to respond to questions about the utility and feasibility of the EIA tool. Emerging themes and recommendations were shared with the project team.
Results: Focus group participants reported that the focus group was a driver for incorporating more equity best practices in future tool development processes. They also found the tool useful and provided feedback on how to make it easier to use such as shortening it and making it available in a digital format. Broader takeaways from the focus group were the importance of cultivating an internal culture around the use of equity best practices and incorporating equity best practices into the planning and scoping phases of projects to ensure adequate time, talent, and resources for implementation.
Conclusion: This study has implications for how organizations incorporate equity key principles into implementation strategies in the broader Implementation Science and Practice field.
References
1. Annie E. Casey Foundation. (2021). Equity vs. Equality and Other Racial Justice Definitions. The Annie E. Casey Foundation.
2. Baumann, A. A., & Cabassa, L. J. (2020). Reframing implementation science to address inequities in healthcare delivery. BMC Health Services Research, 20(1), 1-9.
3. Facilitating Power. (2019). Spectrum of Family & Community Engagement for Educational Equity.
4. Keleher, T. (2009). Racial equity impact assessment. Race Forward.
5. King County. (2016). 2015 Equity Impact Review Process.
6. National Academies of Sciences, Engineering, and Medicine. (2019). Fostering healthy mental, emotional, and behavioral development in children and youth: A national agenda.
Disclosures of Interest: None declared
Associations between teamwork and implementation outcomes in multidisciplinary cross-sector teams implementing a mental health screening and referral protocol
Authors
Dr. Elizabeth McGuier - University of Pittsburgh School of Medicine
Dr. Gregory Aarons - University of California - San Diego
Dr. Kara Byrne - University of Utah
Dr. Kristine Campbell - University of Utah
Dr. Brooks Keeshin - University of Utah
Dr. Scott Rothenberger - University of Pittsburgh School of Medicine
Dr. Laurie Weingart - Tepper School of Business, Carnegie Mellon University
Dr. Eduardo Salas - Rice University
Dr. David Kolko - University of Pittsburgh School of Medicine
Background: Teams play a central role in implementation of new practices in settings providing team-based care. However, the implementation science literature has paid little attention to potentially important team-level constructs. Aspects of teamwork, including team interdependence, team functioning, and performance, may affect implementation processes and outcomes. This cross-sectional study tests associations between teamwork and implementation outcomes in a statewide initiative to implement a standardized mental health screening/referral protocol in Child Advocacy Centers (CACs).
Methods: Multidisciplinary team members (N = 433) from 21 CACs completed measures of team interdependence, affective, behavioral, and cognitive team functioning, and team performance and rated the acceptability, appropriateness, and feasibility of the screening/referral protocol. The implementation outcomes of days to adoption and reach were assessed with administrative data. Associations between team constructs and implementation outcomes were tested with linear mixed models and regression analyses.
Results: Team task interdependence was positively associated with implementation climate (B = .87) and reach (B = .78), and outcome interdependence was negatively correlated with days to adoption (r = -.52). Task and outcome interdependence were not associated with acceptability, appropriateness, or feasibility of the screening/referral protocol. Affective team functioning was associated with greater acceptability (B = .26), appropriateness (B = .24), and feasibility (B = .25). Behavioral and cognitive team functioning were not significantly associated with any implementation outcomes in multivariable models. Team performance was positively associated with acceptability (B = .10), appropriateness (B = .10), and feasibility (B = .09) as well as implementation climate (B = .59); it was not associated with days to adoption or reach.
Conclusion: We found some associations of team interdependence, functioning, and performance with individual- and center-level implementation outcomes. Implementation strategies targeting teamwork, especially task interdependence, affective functioning, and performance, may improve implementation outcomes in team-based service settings.
References
1. Courtright, S. H., Thurgood, G. R., Stewart, G. L., & Pierotti, A. J. (2015). Structural interdependence in teams: An integrative framework and meta-analysis. Journal of Applied Psychology, 100(6), 1825–1846. https://doi.org/10.1037/apl0000027
2. Herbert, J. L., & Bromfield, L. (2019). Multi-disciplinary teams responding to child abuse: Common features and assumptions. Children and Youth Services Review, 106, 104467. https://doi.org/10.1016/j.childyouth.2019.104467
3. Hülsheger, U. R., Anderson, N., & Salgado, J. F. (2009). Team-level predictors of innovation at work: A comprehensive meta-analysis spanning three decades of research. Journal of Applied Psychology, 94(5), 1128–1145. https://doi.org/10.1037/a0015978
Disclosures of Interest: None declared
A mixed methods implementation evaluation across a consortium of implementation trials focused on increasing access to medication for opioid use disorder
Authors
Dr. Amanda Midboe - Center for Innovation to Implementation (Ci2i), VA Palo Alto Healthcare System, Menlo Park, CA and Stanford University School of Medicine, Stanford, CA
Dr. Karen Drummond - Department of Psychiatry, University of Arkansas for Medical Sciences, North Little Rock, AR and VA HSR&D Center for Mental Healthcare & Outcomes Research, Central Arkansas Veterans Healthcare System (CAVHS), North Little Rock, AR
Ms. Taryn Perez - Center for Innovation to Implementation (Ci2i), VA Palo Alto Healthcare System, Menlo Park, CA
Dr. Sarah Daniels - Center for Innovation to Implementation (Ci2i), VA Palo Alto Healthcare System, Menlo Park, CA
Dr. Shari Rogal - Department of Medicine, University of Pittsburgh, Pittsburgh, PA and Center for Health Equity Research and Promotion, VA Pittsburgh Healthcare System, University Drive (151C), Pittsburgh, PA
Dr. William Becker - Pain Research, Informatics, Multimorbidities and Education (PRIME) Center, VA Connecticut Healthcare System, West Haven, CT, Yale School of Medicine, New Haven, CT
Background: To address rising opioid overdose deaths in the US,1 improving access to treatment of opioid use disorder (OUD) is critical. Within the Veterans Health Administration (VA), implementation experts with experience in addiction and pain formed a consortium to increase access to evidence-based medication (MOUD) and treatment for Veterans living with OUD.2 These experts were organized into five project teams focused on outpatient primary and specialty care, as well as inpatient care settings across the US. The project teams used an Implementation Facilitation (IF) strategy tailored to local context.
Methods: An Implementation Core (IC) team, led by an implementation scientist, trained and coordinated work across the IF teams, standardized measurement, and supported mixed-method data collection activities. The quantitative methods included use of a secure, web-based REDCap platform for obtaining implementation strategy data (Expert Recommendations for Implementing Change survey3,4 and IF activity data, including for time-driven activity-based costing).5 Qualitative methods included monthly Periodic Reflections6 with all project teams and open-ended descriptions of IF strategies within REDCap logs. We relied on rapid qualitative coding, compiled into matrices guided by the CFIR.
Results: Quantitative methods identified unique and common implementation strategies as well as the cost of external facilitation team efforts. Qualitative methods provide more in-depth understanding of contextual barriers and facilitators that IF logging alone. Barrier-related themes were workload burden and staff turnover, and challenges presented by the COVID-19 pandemic. Enabler-related themes were leadership support, cross-project IF team support, and strong site champion support.
Conclusion: The coordinated, mixed-methods approach led by the IC provided comprehensive findings that can inform future implementation work focused on treatment for patients living with OUD in integrated healthcare settings. We will provide details on use of IF and other implementation strategies, as well as cost estimates, including identifying barriers and enablers and IF adaptations.
References
1. Hedegaard H, Miniño AM, Spencer MR, Warner M. Drug overdose deaths in the United States, 1999–2020. NCHS Data Brief, no 428. Hyattsville, MD: National Center for Health Statistics. 2021. DOI: https://dx.doi.org/10.15620/cdc:112340.
2. VA Quality Enhancement Research Initiative (QUERI). (2019). Consortium to Disseminate and Understand Implementation of Opioid Use Disorder (OUD) Treatment. US Department of Veterans Affairs. https://www.queri.research.va.gov/centers/CONDUIT.pdf
3. Waltz, T. J., Powell, B. J., Chinman, M. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., … & Kirchner, J. E. (2014). Expert recommendations for implementing change (ERIC): protocol for a mixed methods study. Implementation Science, 9(1), 1-12.
4. Rogal, S. S., Yakovchenko, V., Waltz, T. J., Powell, B. J., Kirchner, J. E., Proctor, E. K., … & Chinman, M. J. (2017). The association between implementation strategy use and the uptake of hepatitis C treatment in a national sample. Implementation Science, 12(1), 1-13.
5. Cidav, Z., Mandell, D., Pyne, J., Beidas, R., Curran, G., & Marcus, S. (2020). A pragmatic method for costing implementation strategies using time-driven activity-based costing. Implementation Science, 15(1), 1-15.
6. Finley, E. P., Huynh, A. K., Farmer, M. M., Bean-Mayberry, B., Moin, T., Oishi, S. M., … & Hamilton, A. B. (2018). Periodic reflections: a method of guided discussions for documenting implementation phenomena. BMC medical research methodology, 18(1), 1-15.
Disclosures of Interest: None declared
An evaluation of a hybrid Type III implementation trial of an integrated multidisciplinary clinic to address unsafe use of opioids for veterans living with chronic pain
Authors
Dr. Amanda Midboe - Center for Innovation to Implementation (Ci2i), VA Palo Alto Healthcare System, Menlo Park, CA and Stanford University School of Medicine, Stanford, CA
Ms. Taryn Perez - Center for Innovation to Implementation (Ci2i), VA Palo Alto Healthcare System, Menlo Park, CA
Ms. Shayna Cave - Center for Innovation to Implementation (Ci2i), VA Palo Alto Healthcare System, Menlo Park, CA
Dr. Sara Edmond - Pain Research, Informatics, Multimorbidities and Education (PRIME) Center, VA Connecticut Healthcare System, West Haven, CT, Yale School of Medicine, New Haven, CT
Dr. William Becker - Pain Research, Informatics, Multimorbidities and Education (PRIME) Center, VA Connecticut Healthcare System, West Haven, CT, Yale School of Medicine, New Haven, CT
Background: While opioid overdose deaths continue to climb1, chronic pain remains prevalent with approximately 20% of Americans living with this disorder2. Preventing overdose deaths and supporting the unique needs of patients who are living with chronic pain requires not only evidence-based treatments options but improving access to them through implementation science. The Opioid Reassessment Clinic (ORC) model3 relies on a multidisciplinary team applying evidence-based practices in the treatment of those living with chronic pain and complex psychiatric comorbidities. The ORC model is integrated within primary care, with the objective of providing longitudinal co-management of pain and opioid use disorder while promoting engagement in non-pharmacological pain treatments (NPTs). As part of a hybrid type III trial, we evaluated the effect of Implementation Facilitation (IF)4 on the implementation of an ORC at three Veterans Health Administration (VA) sites.
Methods: This mixed-methods evaluation relied on the Consolidated Framework for Implementation Research (CFIR) and the RE-AIM framework to evaluate the use of Implementation Facilitation across 18 months to implement an ORC at 3 VA sites in different regions of the United States.
Results: Initial analyses reveal that 532 patients living with chronic pain and complex psychiatric comorbidities (81-91% of patients) were treated across 3 sites. A total of 127 providers adopted the ORC or were trained on the model. Reductions in morphine equivalent daily doses (MEDD) have ranged from 34% to 75% decrease in MEDD at sites. Prescribing of buprenorphine, a safer alternative to full agonist opioids, has increased significantly alongside referrals to NPTs. A formative qualitative evaluation revealed several relevant CFIR-related determinants for tailoring of IF and informing future dissemination efforts.
Conclusion: At three VA implementation sites, Implementation Facilitation was an effective implementation strategy to improve care for veterans living with chronic pain and complex psychiatric comorbidities.
References
1. Dahlhamer, J., Lucas, J., Zelaya, C., Nahin, R., Mackey, S., DeBar, L., … & Helmick, C. (2018). Prevalence of chronic pain and high-impact chronic pain among adults—United States, 2016. Morbidity and Mortality Weekly Report, 67(36), 1001.
2. Hedegaard H, Miniño AM, Spencer MR, Warner M. Drug overdose deaths in the United States, 1999–2020. NCHS Data Brief, no 428. Hyattsville, MD: National Center for Health Statistics. 2021. DOI: https://dx.doi.org/10.15620/cdc:112340.
3. Becker, W. C., Edmond, S. N., Cervone, D. J., Manhapra, A., Sellinger, J. J., Moore, B. A., & Edens, E. L. (2018). Evaluation of an integrated, multidisciplinary program to address unsafe use of opioids prescribed for pain. Pain Medicine, 19(7), 1419-1424. DOI: https://doi.org/10.1093/pm/pnx041.
4. Ritchie, M. J., Kirchner, J. E., Parker, L. E., Curran, G. M., Fortney, J. C., Pitcock, J. A., … & Kilbourne, A. M. (2015). Evaluation of an implementation facilitation strategy for settings that experience significant implementation barriers. Implementation Science, 10(1), 1-3. DOI: https://doi.org/10.1186/1748-5908-10-S1-A46.
Disclosures of Interest: None declared
Instrumental variables for implementation science: Efficacy in the context of an implementation strategy
Authors
Dr. Aaloke Mody - Washington University School of Medicine
Dr. Lindsey Filiatreau - Washington University School of Medicine
Dr. Charles Goss - Washington University School of Medicine
Dr. Byron Powell - Brown School, Washington University in St. Louis, St. Louis, MO, United States
Dr. Elvin Geng - Washington University School of Medicine
Background: Implementation science makes clear distinctions between implementation strategies (IS) and efficacy of an evidence-based intervention (EBI), but this dichotomization fails to recognize that the causal effect of an EBI is closely intertwined with the IS leading to EBI uptake.
Methods: We explore the use of instrumental variable methods as a critical tool in implementation science study design to simultaneously and causally assess three relevant quantities within the same intervention context: 1) the effect of an IS on implementation outcomes (e.g., uptake), 2) effect of EBI uptake on patient outcomes due to the IS, and 3) overall effectiveness of the IS (i.e.,∼implementation*efficacy). We illustrate these concepts using a real-world example assessing implementation of new HIV guidelines for same-day treatment initiation in Zambia.
Results: Causal questions relevant to implementation science are answered at each stage of an IV analysis. The first stage assesses the effect of new guidelines (i.e., the IS) on same-day treatment initiation (i.e., the EBI). The second stage leverages the IS as an IV to estimate the complier average causal effect (CACE) of the EBI on patient outcomes (i.e., viral suppression). The assumptions underlying the CACE formalizes that the causal effect of same-day ART initiation may differ when it is due to implementation of new guidelines compared to different circumstances or implementation strategies; thus, it makes explicit that efficacy is context-dependent and specific to the IS. Lastly, the overall effectiveness of implementing new guidelines on viral suppression is assessed.
Discussion: Leveraging IV methods for implementation science studies helps to conceptualize the dependencies between implementation strategies, EBIs, and patient outcomes and highlights that the causal effect of an EBI may be specific to the context of the implementation strategy used to promote uptake.
References
1. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217-26.
2. Barnighausen T, Oldenburg C, Tugwell P, Bommer C, Ebert C, Barreto M, et al. Quasi-experimental study designs series-paper 7: assessing the assumptions. J Clin Epidemiol. 2017;89:53-66.
3. Mody A, Sikazwe I, Namwase AS, Wa Mwanza M, Savory T, Mwila A, et al. Effects of implementing universal and rapid HIV treatment on initiation of antiretroviral therapy and retention in care in Zambia: a natural experiment using regression discontinuity. Lancet HIV. 2021;8(12):e755-e65.
Disclosures of Interest: None declared
Application of Normalization Process Theory to evaluate a virtual training program for clinicians about screening and caring for children with Autism Spectrum Disorder
Authors
Ms. Belinda O'Hagan - Boston Medical Center
Dr. Marilyn Augustyn - Boston Medical Center
Mx. Rachel Amgott - Boston Medical Center
Mx. Julie White - Boston University
Mx. Ilana Hardesty - Boston University
Mx. Candice Bangham - Boston University
Mx. Amy Ursitti - Boston Medical Center
Mx. Sarah Foster - Boston Medical Center
Mx. Alana Chandler - Boston Medical Center
Dr. Jacey Greece - Boston University
Background: The diagnostic process for autism spectrum disorder (ASD) involves multiple steps and requires multi-sector collaboration (Sheldrick et al., 2016). The national evidence-based Extension for Community Health Outcomes (ECHO)® model was used to virtually train 21 primary care clinicians about ASD screening and care (Arora et al., 2007). Monthly training sessions (n = 12) were held between November 2020 and October 2021.
Methods: A mixed-methods evaluation was conducted using the Normalization Process Theory (NPT) to assess knowledge, skill, and self-efficacy outcomes (May & Finch, 2009). Pre-, mid-, and post-test surveys assessed shifts in these outcomes along with clinician burnout due to the pandemic, training satisfaction, and sense of community. Findings from the surveys along with the NPT informed development of a semi-structure interview guide. Interviews (n = 10) occurred three months post-program for practice-based and sustainability outcomes. NPT allowed examination of internal factors to implementation of the training objectives and external factors influencing adoption and sustainability.
Results: Qualitative findings included increased (1) use and confidence in administering ASD screeners, (2) confidence in educating families about ASD (e.g., using the word “autism”), and (3) familiarity with relevant resources. Participants found the training to be more impactful toward their individual practice as clinicians, rather than systemic changes on a clinic level. Barriers to implementation of skills (i.e., access to specialists, education services) were identified as well as areas for further training and technical assistance, especially given the stressors imposed by COVID-19.
Conclusion: The NPT allows a focus on clinicians’ individual practice and clinic protocols with a framework to guide recommendations considering the pandemic context. Given the increased need for developmental evaluations imposed by pandemic-related factors, the NPT is a valuable tool to examine the implementation of an evidence-based model to enhance the workforce and ensure equitable care of patients who may be on the autism spectrum.
References
1. Arora, S., Geppert, C. M. A., Kalishman, S., Dion, D., Pullara, F., Bjeletich, B., Simpson, G., Alverson, D. C., Moore, L. B., Kuhl, D., & Scaletti, J. V. (2007). Academic health center management of chronic diseases through knowledge networks: Project ECHO. Academic Medicine : Journal of the Association of American Medical Colleges, 82(2), 10.1097/ACM.0b013e31802d8f68. https://doi.org/10.1097/ACM.0b013e31802d8f68
2. May, C., & Finch, T. (2009). Implementing, embedding, and integrating practices: An outline of normalization process theory. Sociology, 43(3), 535–554.
3. Sheldrick, R. C., Breuer, D. J., Hassan, R., Chan, K., Polk, D. E., & Benneyan, J. (2016). A system dynamics model of clinical decision thresholds for the detection of developmental-behavioral disorders. Implementation Science, 11(1), 156. https://doi.org/10.1186/s13012-016-0517-0
Disclosures of Interest: None declared
Implementation of health information technology for secondary cancer prevention in primary care: A scoping review
Authors
Ms. Constance Owens - National Cancer Institute’s Consortium for Cancer Implementation Science, National Institutes of Health
Dr. Jinying Chen - National Cancer Institute’s Consortium for Cancer Implementation Science, National Institutes of Health
Dr. Ran Xu - National Cancer Institute’s Consortium for Cancer Implementation Science, National Institutes of Health
Dr. Heather Angier - National Cancer Institute’s Consortium for Cancer Implementation Science, National Institutes of Health
Dr. Amy Huebschmann - National Cancer Institute’s Consortium for Cancer Implementation Science, National Institutes of Health
Dr. Mayuko Ito Fukunaga - National Cancer Institute’s Consortium for Cancer Implementation Science, National Institutes of Health
Dr. Krisda Chaiyachati - National Cancer Institute’s Consortium for Cancer Implementation Science, National Institutes of Health
Dr. Katharine Rendle - National Cancer Institute’s Consortium for Cancer Implementation Science, National Institutes of Health
Dr. Kim Robien - National Cancer Institute’s Consortium for Cancer Implementation Science, National Institutes of Health
Dr. Lisa DiMartino - National Cancer Institute’s Consortium for Cancer Implementation Science, National Institutes of Health
Dr. Daniel Amante - National Cancer Institute’s Consortium for Cancer Implementation Science, National Institutes of Health
Dr. Jamie Faro - National Cancer Institute’s Consortium for Cancer Implementation Science, National Institutes of Health
Dr. Maura Kepper - National Cancer Institute’s Consortium for Cancer Implementation Science, National Institutes of Health
Dr. Alex Ramsey - National Cancer Institute’s Consortium for Cancer Implementation Science, National Institutes of Health
Dr. Eric Bressman - National Cancer Institute’s Consortium for Cancer Implementation Science, National Institutes of Health
Dr. Rachel Gold - National Cancer Institute’s Consortium for Cancer Implementation Science, National Institutes of Health
Background: A substantial percentage of the population is not up to date on guideline-recommended cancer screenings, and this worsened during the COVID-19 pandemic .1,2 This scoping review assessed the state of science on how health information technology (HIT) supports guideline-concordant secondary cancer prevention in primary care, and how to support the implementation of effective HIT-based interventions.
Methods: Following scoping review guidelines3-5 we searched MEDLINE, CINAHL Plus, Web of Science, and IEEE Xplore for U.S.-based references from 2015-2021 that featured HIT targeting breast, colorectal, and cervical cancer screening in primary care. All references were dual-screened using a review criteria checklist. Data extracted was guided by the RE-AIM framework,6 Integrated Technology Implementation Model,7 Expert Recommendations for Implementing Change (ERIC) compilation,8 and Implementation Strategy reporting domains.9
Results: A total of 101 references met inclusion criteria. Most involved electronic health record (EHR)-based HIT tools (85%). The most common HIT function was clinical decision support (68%), primarily used during panel management (55%) and at the point of care (39%). Although less than 25% of included references reported HIT effectiveness for breast and cervical cancer screening, significant improvements in cancer screening were reported when HIT was used; most evidence supported HIT targeting colorectal cancer screening (over 50%). However, less than 25% of included references reported reach, adoption, or maintenance of HIT. Barriers / facilitators to HIT adoption primarily related to the inner and outer context of tool use. Implementation strategies for effective HIT adoption were reported in less than 25% of included references.
Conclusion: Substantial evidence gaps remain regarding the effectiveness of HIT tools targeting guideline-concordant breast and cervical cancer screening. Even less is known about how to enhance the adoption of such technologies proven effective in primary care. Research in both areas is needed to ensure that HIT’s potential benefits to population health are achieved.
References
1. Joseph, D. A., King, J. B., Dowling, N. F., Thomas, C. C., & Richardson, L. C. (2020). Vital Signs: Colorectal Cancer Screening Test Use - United States, 2018. MMWR. Morbidity and mortality weekly report, 69(10), 253–259. https://doi.org/10.15585/mmwr.mm6910a1
2. Mitchell E. P. (2020). Declines in Cancer Screening During COVID-19 Pandemic. Journal of the National Medical Association, 112(6), 563–564. https://doi.org/10.1016/j.jnma.2020.12.004
3. Arksey H., O’Malley L. Scoping Studies: Towards a Methodological Framework. International Journal of Social Research Methodology, 8:19–32.
4. Levac, D., Colquhoun, H., & O'Brien, K. K. (2010). Scoping studies: advancing the methodology. Implementation science : IS, 5, 69. https://doi.org/10.1186/1748-5908-5-69
5. Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M., Horsley, T., Weeks, L., Hempel, S., Akl, E. A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M. G., Garritty, C., Lewin, S., … Straus, S. E. (2018). PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Annals of internal medicine, 169(7), 467–473. https://doi.org/10.7326/M18-0850
6. Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: the RE-AIM framework. American journal of public health, 89(9), 1322–1327. https://doi.org/10.2105/ajph.89.9.1322
7. Schoville, R. R., & Titler, M. G. (2015). Guiding healthcare technology implementation: a new integrated technology implementation model. Computers, informatics, nursing : CIN, 33(3), 99–E1. https://doi.org/10.1097/CIN.0000000000000130
8. Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015). A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation science : IS, 10, 21. https://doi.org/10.1186/s13012-015-0209-1
9. Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013). Implementation strategies: recommendations for specifying and reporting. Implementation science : IS, 8, 139. https://doi.org/10.1186/1748-5908-8-139
Disclosures of Interest: Katharine Rendle received grant funding to her institution from AstraZeneca, is a paid scientific advisor to Merck, and has grant funding awarded to her institution by Pfizer.
Supporting healthy health and human service systems: Understanding organizational commitment and turnover within child welfare workforces
Authors
Ms. Rebecca Phillips - College of Social Work, The Ohio State University
Dr. Alicia Bunger – College of Social Work, The Ohio State University
Ms. Sarah Parmenter - College of Social Work, The Ohio State University
Background: Occupational stress and subsequent negative turnover (e.g., not related to promotion or retirement) are prevalent and severe issues within health and human service systems (Mor Barak et al., 2001), particularly amongst child welfare workforces (Graham et al., 2014). However, previous attempts to address occupational stress and workforce turnover have predominantly featured individual-level strategies, which have been implemented primarily through top-down methods. The minimally effective interventions currently available reflects a lack of consensus regarding the occupational feature levels and specific targets that should be involved when addressing workforce stress, organizational commitment, and related turnover (Garcia et al., 2016).
Methods: Ohio is one of nine states with a state-supervised county-administered child welfare system, wherein specific policies and practices may range widely between agencies. A multi-level mixed methods approach was therefore used, as solely quantitative data may fail to accurately capture work-related psychosocial influences. Specific research methods included: (1) assessing the local context through quantitative surveys and qualitative focus groups with Ohio CW staff; and analyzing the evidence available in current literature; (2) analyzing administrative and secondary data regarding salient features of the CW occupational context (e.g., agency, county).
Results: Multivariate multiple regression, ANCOVA, and mixed-method analyses revealed that in addition to individual-level occupational influences (e.g., role characteristics), systemic features of the work-related environment (e.g., agency structures and staffing, demographic indicators, family functioning) can significantly affect employment responses (e.g., organizational commitment, turnover; p < .01).
Conclusion: Employee feedback that is inclusive of all professional levels and roles is particularly critical, both for determining significant retention-related influences, as well as identifying contextual resources and needs relevant to solution planning and implementation. Additionally, systemic features of the occupational environment must be carefully considered for accurate identification of evidence-based workforce improvement strategies appropriate and feasible for specific implementation contexts.
References
1. Beer, O. W. J., Phillips, R., Stepney, L., & Quinn, C. R. (2020). The feasibility of mindfulness training to reduce stress among social workers: A conceptual paper. The British Journal of Social Work, 50(1), 243-263.
2. Beidas, R. S., Wolk, C. L. B., Walsh, L. M., Evans, A. C., Hurford, M. O., & Barg, F. K. (2014). A complementary marriage of perspectives: Understanding organizational social context using mixed methods. Implementation Science, 9(175), 1-15.
3. Choi, G. Y. (2011). Organizational impacts on the secondary traumatic stress of social workers assisting family violence or sexual assault survivors. Administration in Social Work, 35(3), 225-242. doi:10.1080/03643107.2011.575333
4. Dorch, E., McCarthy, M. L., & Denofrio, D. (2008). Calculating child welfare separation, replacement, and training costs. Social Work in Public Health, 23(6), 39-54.
5. Egan, M., Bambra, C., Thomas, S., Petticrew, M., Whitehead, M., & Thomson, H. (2007). The psychosocial and health effects of workplace reorganisation : A systematic review of organisational-level interventions that aim to increase employee control. Journal of Epidemiology and Community Health, 61(11), 945–954.
6. Garcia, A. R., Kim, M., & DeNard, C. (2016). Context matters: The state of racial disparities in mental health services among youth reported to child welfare in 1999 and 2009. Children and Youth Services Review, 66, 101-108.
7. Graham, J. R., Bradshaw, C., Surood, S. and Kline, T. J. B. (2014). Predicting social workers subjective well-being. Human Service Organizations Management, Leadership & Governance, 38(4), 405-417.
8. Inauen, A., Jenny, G. J., & Bauer, G. F. (2012). Design principles for data-and change-oriented organisational analysis in workplace health promotion. Health Promotion International, 27(2), 275-283.
9. Kimberly, J., & Cook, J. M. (2008). Organizational measurement and the implementation of innovations in mental health services. Administration and Policy in Mental Health, 35(1-2), 11– 20.
10. Mor Barak, M. E., Nissly, J. A., & Levin, A. (2001). Antecedents to retention and turnover among child welfare, social work, and other human service employees: What can welearn from past research? A review and meta-analysis. Social Service Review, 75(4), 625-661.
11. Saldaña, J. (2016). The coding manual for qualitative researchers. Thousand Oaks, CA: Sage Publications.
12. Stalker, C., & Harvey, C. (2003). Professional Burnout in Social Service Organizations: A Review of Theory, Research, and Prevention (pp. 1–56, Report). Wilfrid.
13. Taris, T. W. (2006). Is there a relationship between burnout and objective performance? A critical review of 16 studies. Work & Stress, 20(4), 316-334.
14. Wännström, I., Peterson, U., Åsberg, M., Nygren, Å., & Gustavsson, J. P. (2009). Psychometric properties of scales in the General Nordic Questionnaire for Psychological and Social Factors at Work (QPSNordic): Confirmatory factor analysis and prediction of certified long-term sickness absence. Scandinavian Journal of Psychology, 50(3), 231-244.
15. Wiltsey-Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of new programs and innovations: A review of the empirical literature and recommendations for future research. Implementation Science 7(17), 1-19.
Disclosures of Interest: None declared
Using rapid qualitative analysis, D&I Science, and participatory research to disseminate training in religious and spiritual competencies across mental health professions
Authors
Dr. James Pittman - University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
Dr. Joseph Currier - University of South Alabama
Ms. Chloe Tenega - University of California San Diego
Ms. Megan Blanton - University of South Alabama
Ms. Ruiyan Hu - University of California San Diego
Dr. Cassandra Vieten - University of California San Diego
Background: Each year, 20% of U.S. adults meet criteria for a mental health (MH) disorder. Many seeking MH care cope via beliefs, practices, and relationships that have been shaped by religion and/or spirituality (R/S; Pargament, 2013). Research affirms the role of R/S in MH (Koenig et al., 2012), but limited knowledge and skills related to R/S leads most clinicians to neglect clients’ R/S, reducing care quality and utilization (Oxhandler et al., 2015). The goal of this project is to identify barriers and potential solutions from stakeholders to collaboratively develop a roadmap to facilitate implementation of R/S competencies in MH professional training programs.
Methods: This is a qualitative participatory action project. We used the Consolidated Framework for Implementation Research (CFIR; Damschroder et al., 2009) to develop an interview guide for focus groups and individual interviews. We conducted 8 focus groups with a diverse group of stakeholders. We used a rapid qualitative analytic approach (Hamilton, 2013) to analyze the data, which we then used to inform the question set for individual interviews with 30 stakeholders. Finally, we are using the results from the focus groups and individual interviews to structure the agenda for a 2-day stakeholder meeting to develop a systems change roadmap.
Results: Barriers included practical considerations (e.g. not enough space in the curriculum) and concerns about the R/S topic overall (e.g. R/S is controversial). Participants identified development of training and practice guidelines to be adopted by professional associations, and multiple resources for training settings as potential solutions. Comprehensive results and the roadmap will be presented and discussed.
Conclusion: This project will inform our ongoing efforts to make evidence-based R/S competencies standard in all MH training programs nationally. Rapid qualitative analysis was instrumental in completing this phase of the project in 6 months.
References
1. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A Consolidated Framework for advancing implementation science. Implementation Science, 4(1). https://doi.org/10.1186/1748-5908-4-50
2. Hamilton, A. (2013). Qualitative methods in rapid turn-around health services research. Health services. research & development cyberseminar.
3. Koenig, H. G. (2012). Handbook of Religion and Health. Oxford University Press.
4. Oxhandler, H. K., Parrish, D. E., Torres, L. R., & Achenbaum, W. A. (2015). The integration of clients’ religion and spirituality in Social Work Practice: A national survey. Social Work, 60(3), 228–237. https://doi.org/10.1093/sw/swv018
5. Pargament, K. I. (Ed.). (2013). Searching for the sacred: Toward a nonreductionistic theory of spirituality. In K. I. Pargament, J. J. Exline, & J. W. Jones (Eds.), APA handbook of psychology, religion, and spirituality (Vol. 1): Context, theory, and research(pp. 257–273). American Psychological Association. https://doi.org/10.1037/14045-014
Disclosures of Interest: None declared
Development and initial results of the Practical, Robust Implementation and Sustainability Model (PRISM) survey to quantitatively assess multilevel contextual factors
Authors
Dr. James Pittman - VA Center of Excellence for Stress and Mental Health & UC San Diego
Dr. Laurie Lindamer - VA Center of Excellence for Stress and Mental Health, VA San Diego Healthcare System, UC San Diego Dept of Psychiatry
Dr. Erin Almklov - VA Center of Excellence for Stress and Mental Health
Mr. John Gault - VA Center of Excellence for Stress and Mental Health
Mr. Brian Blanco - VA San Diego Healthcare System
Mrs. Kelli Cain - Herbert Wertheim School of Public Health and Human Longevity Science, University of California San Diego, La Jolla, CA,
Dr. Borsika Rabin - UC San Diego Dissemination and Implementation Science Center, UC San Diego Herbert Wertheim School of Public Health and Human Longevity Science
Background: PRISM (Feldstein & Glasgow, 2008) is a contextually expanded version of the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework and includes consideration for multilevel factors from the perspective of stakeholders. Our study uses a cluster-randomized, stepped-wedge design to assess the implementation and impact of an electronic self-report system on Veteran care (Pittman et al., 2021). PRISM guides the implementation and evaluation of this system. Qualitative measures have primarily been used to assess PRISM contextual domains (McCreight et al., 2019), but quantitative measures are needed to assess context longitudinally. This study describes the development of a set of PRISM survey items and presents the data collected during the pre-implementation phase of the project across 8 VA sites.
Methods: Survey items were developed using an iterative expert process where initial items were developed for each PRISM context domain considering the key barriers and facilitators of the implementation of eScreening followed by a multi-round review by members of the RE-AIM National Workgroup and pilot testing followed by further refinements. The final PRISM survey consisted of 29 items measuring 6 domains with response options on a 5-point Likert scale.
Results: Thirty participants completed the survey during pre-implementation of the study. Sites varied on staffing levels, patient volume, and rurality. Responses on all domains were favorable with the highest overall scores for program organizational perspective (mean(SD) = 4.4(0.3)) and program patient perspective: (mean(SD) = 4.3(0.2)) and the lowest scores for patient characteristics (mean(SD) = 3.6(0.6)) and infrastructure (mean(SD) = 3.7(0.4)). Complete results from the initial survey will be presented when analyses are complete.
Conclusion: The development of this PRISM survey will allow us to evaluate context longitudinally and can be adapted and used widely in other research programs.
References
1. Feldstein, A. C., & Glasgow, R. E. (2008). A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. The Joint Commission Journal on Quality and Patient Safety, 34(4), 228-243.
2. McCreight, M. S., Rabin, B. A., Glasgow, R. E., Ayele, R. A., Leonard, C. A., Gilmartin, H. M., … & Battaglia, C. T. (2019). Using the Practical, Robust Implementation and Sustainability Model (PRISM) to qualitatively assess multilevel contextual factors to help plan, implement, evaluate, and disseminate health services programs. Translational Behavioral Medicine, 9(6), 1002-1011.
3. Pittman, J. O., Lindamer, L., Afari, N., Depp, C., Villodas, M., Hamilton, A., … & Rabin, B. (2021). Implementing eScreening for suicide prevention in VA post-9/11 transition programs using a stepped-wedge, mixed-method, hybrid type-II implementation trial: a study protocol. Implementation science communications, 2(1), 1-13.
Disclosures of Interest: None declared
Is training enough? Examining drivers of provider knowledge of evidence-based engagement practices
Authors
Ms. Kendal Reeder – University of California Los Angeles
Ms. Hyun Seon Park – University of California Los Angeles
Dr. Kimberly Becker - University of South Carolina
Dr. Bruce Chorpita – University of California Los Angeles
Background: The impact of evidence-based practices (EBPs) for youth mental health is limited by low service engagement.1 However, providers report using few evidence-based engagement practices (EBEPs) when working with youth.2 To better understand this science-to-practice gap, this study aimed to characterize provider knowledge of EBEPs, examine provider-level characteristics that relate to EBEP knowledge, and test strategies for improving EBEP knowledge.
Methods: Data came from a cluster-randomized trial examining a coordinated system for implementing EBEPs in publicly-funded school-based services.3 Teams of providers were randomized to one of two conditions. The coordinated condition received didactic training in EBEPs and learned a coordinated process for implementing them based on the Knowledge-to-Action Framework.4 The control condition received only didactic training. Providers (N = 93) completed three measures: a knowledge test assessing EBEP knowledge before and after training, the Evidence-Based Practice Attitude Scale,5 and self-reported training in EBPs for youth mental health. Pearson correlation coefficients were used to examine the relationship between EBEP knowledge with EBP training and attitudes toward EBPs. One-way ANOVA was used to compare mean EBEP knowledge change between conditions.
Results: The mean pre-training EBEP knowledge score was 6.36 (SD = 1.97) out of 14. Knowledge of EBEPs was not significantly correlated with number of EBP trainings (r = .11, p = .29) or attitudes toward EBPs (r = .20, p = .06). The mean increase in EBEP knowledge was higher (F(1,91) = 54.4, p < .001) for the coordinated condition (N = 55, mean = 3.15) than the control condition (N = 38, mean = .29).
Conclusion: Training in EBPs for youth mental health was not correlated with knowledge on how to effectively engage youth in mental health services, suggesting that provider ability to act on the evidence base may be focally limited to their specific training experiences. A coordinated process for implementing EBEPs using existing supports may be a promising approach to enhancing provider knowledge and use of EBEPs, thus promoting the public health impact of EBPs for youth mental health.
References
1. Kazdin, A. E., & Blase, S. L. (2011). Rebooting psychotherapy research and practice to reduce the burden of mental illness. Perspectives on psychological science, 6(1), 21-37.
2. Becker, K. D., Dickerson, K., Boustani, M. M., & Chorpita, B. F. (2021). Knowing what to do and when to do it: Mental health professionals and the evidence base for treatment engagement. Administration and Policy in Mental Health and Mental Health Services Research, 48(2), 201-218.
3. Becker, K. D., Park, A. L., Boustani, M. M., & Chorpita, B. F. (2019). A pilot study to examine the feasibility and acceptability of a coordinated intervention design to address treatment engagement challenges in school mental health services. Journal of School Psychology, 76, 78-88.
4. Graham, I. D., Logan, J., Harrison, M. B., Straus, S. E., Tetroe, J., Caswell, W., & Robinson, N. (2006). Lost in knowledge translation: time for a map?. Journal of Continuing Education in the Health Professions, 26(1), 13-24.
5. Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research, 6(2), 61-74.
Disclosures of Interest: None declared
A use case for designing feasible and equitable multi-level implementation strategies in behavioral cardiology: Promoting depression screening and treatment in cardiac patients
Authors
Dr. Katja Reuter - SUNY Upstate Medical University, New York, USA
Dr. Andrea T. Duran - Columbia University Irving Medical Center, New York, USA
Dr. Kirali Genao - Columbia University Irving Medical Center, New York, USA
Ms. Emily Callanan - Columbia University Irving Medical Center, New York, USA
Ms. Diane E. Cannone - Columbia University Irving Medical Center, New York, USA
Dr. Nancy Chang - Columbia University Irving Medical Center, New York, USA
Prof. Elsa-Grace Giardina - Columbia University Irving Medical Center, New York, USA
Dr. Jessica Singer - Columbia University Irving Medical Center, New York, USA
Dr. Amy Slutzky - SUNY Upstate Medical University, New York, USA
Dr. Siqin Ye - Columbia University Irving Medical Center, New York, USA
Prof. Nathalie Moise - Columbia University Irving Medical Center, New York, USA
Background: The prognostic association between depression and coronary heart disease (CHD) is an area of great interest in behavioral cardiology1-5. Despite guidelines recommending depression screening and treatment in cardiac patients6, only 10-30% of depressed CHD patients receive optimal treatment. This study applied behavioral and implementation science methods to (i) identify generalizable, multi-level barriers to depression screening and treatment in CHD patients and (ii) develop a theory-informed, multi-level implementation strategy for promoting guideline adoption.
Methods: We conducted a narrative review of barriers to depression screening and treatment (i.e., medications, exercise, cardiac rehabilitation, and/or therapy) in CHD patients. Informed by the Behavior Change Wheel (BCW)7,8 and Expert Recommendations for Implementing Change (ERIC)9, we defined multi-level target behaviors, characterized determinants (capability, opportunity, motivation), and mapped barriers to feasible, acceptable, and equitable intervention functions and behavior change techniques (BCTs) to develop a multi-level implementation strategy.
Results: We identified implementation barriers at the system/provider-level (Capability: knowledge, time, sociocultural resources, decision process issues; Opportunity: screening/workflow integration; Motivation: ownership, qualification, reinforcement) and patient-level (Capability: knowledge; Opportunity: mobility, finances, peer support; Motivation: screening validity and symptom denial). Acceptable, feasible, and equitable intervention functions included education, persuasion, environmental restructuring, and enablement. ERIC strategies included implementation team meetings/local technical assistance. The final multi-component strategy (iHeart DepCare) for promoting depression screening/treatment delivered BCTs via problem-solving meetings with clinic staff (system); educational/motivational videos and electronic health record reminders/decisional support (provider); and a multi-modal shared decision-making (eSDM) tool with screening functionality, psychoeducation, patient activation [around depression and health behaviors), and patient preference-driven treatment selection support the results of which are automatically delivered to providers.
Conclusion: We provide a blueprint and use case for applying implementation science to behavioral cardiology, applying the BCW framework to develop a multi-level implementation strategy to impact multiple behaviors. Implementation is currently being tested in clinical practice.
References
1. Rozanski, A. (2014). Behavioral cardiology: current advances and future directions. Journal of the American College of Cardiology, 64(1), 100–110.
2. Lett, H. S., Blumenthal, J. A., Babyak, M. A., Sherwood, A., Strauman, T., Robins, C., & Newman, M. F. (2004). Depression as a risk factor for coronary artery disease: evidence, mechanisms, and treatment. Psychosomatic Medicine, 66(3), 305–315.
3. Burg, M. M., Edmondson, D., Shimbo, D., Shaffer, J., Kronish, I. M., Whang, W., Alcántara, C., Schwartz, J. E., Muntner, P., & Davidson, K. W. (2013). The “perfect storm” and acute coronary syndrome onset: do psychosocial factors play a role? Progress in Cardiovascular Diseases, 55(6), 601–610.
4. Carney, R. M., Blumenthal, J. A., Catellier, D., Freedland, K. E., Berkman, L. F., Watkins, L. L., Czajkowski, S. M., Hayano, J., & Jaffe, A. S. (2003). Depression as a risk factor for mortality after acute myocardial infarction. The American Journal of Cardiology, 92(11), 1277–1281.
5. Peltzer, S., Müller, H., Köstler, U., Schulz-Nieswandt, F., Jessen, F., Albus, C., & CoRe-Net study group. (2020). Detection and treatment of mental disorders in patients with coronary heart disease (MenDis-CHD): A cross-sectional study. PloS One, 15(12), e0243800.
6. Lichtman, J. H., Bigger, J. T., Jr, Blumenthal, J. A., Frasure-Smith, N., Kaufmann, P. G., Lespérance, F., Mark, D. B., Sheps, D. S., Taylor, C. B., Froelicher, E. S., American Heart Association Prevention Committee of the Council on Cardiovascular Nursing, American Heart Association Council on Clinical Cardiology, American Heart Association Council on Epidemiology and Prevention, American Heart Association Interdisciplinary Council on Quality of Care and Outcomes Research, & American Psychiatric Association. (2008). Depression and coronary heart disease: recommendations for screening, referral, and treatment: a science advisory from the American Heart Association Prevention Committee of the Council on Cardiovascular Nursing, Council on Clinical Cardiology, Council on Epidemiology and Prevention, and Interdisciplinary Council on Quality of Care and Outcomes Research: endorsed by the American Psychiatric Association. Circulation, 118(17), 1768–1775.
7. Michie, S., van Stralen, M. M., & West, R. (2011). The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implementation Science: IS, 6, 42.
8. Michie S Atkins. (2014). The behaviour change wheel: a guide to designing interventions. Silverback Publishing.
9. Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015). A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science : IS, 10, 21. https://doi.org/10.1186/s13012-015-0209-1
Disclosures of Interest: None declared
Bringing implementation research to family law through legal-academic partnership: Results from a contextual inquiry of one county court system
Authors
Dr. Brittany Rudd - University of Illinois at Chicago
Mx. Jax Witzig - University of Illinois at Chicago
Ms. Holly Huber - Indiana University Bloomington
Ms. Catalina Ordorica - University of Illinois at Chicago
Ms. Emily Potter - University of Illinois at Chicago
Ms. Maria Granger - Superior Court for Floyd County, Indiana
Background: Family law settings serve families during the process of divorce or separation. Despite evidence that screening families for legal and psychosocial needs and connecting to appropriate services during their cases improves outcomes (a process known as “triaging”), few family law settings have implemented triage models (Rudd & Beidas, 2021). Supported by a grant from the Indiana Supreme Court, a legal-academic partnership was formed in 2020 to support the development and implementation of family law triaging in Indiana. This presentation will review the development of this partnership and results from an analysis of the barriers to and facilitators of family law triaging.
Methods: Stakeholder mapping was used to identify key stakeholders (i.e., judges, administrators, lawyers, community service providers). Stakeholders were then recruited (n = 18) for individual interviews regarding their perceptions of the barriers to and facilitators of family law triaging implementation. Interview data were collected using summary templates and analyzed using matrix analysis (Averill, 2002). The Exploration, Preparation, Implementation and Sustainment model (EPIS; Aarons et al., 2011) informed the matrix.
Results: There was convergence across stakeholders that triaging was appropriate for family law. Stakeholders identified 25 facilitators and 16 barriers to triage model implementation. Half of stakeholders identified existing court infrastructure (EPIS: organizational characteristics) that could facilitate implementation. The most frequently cited barriers clustered in the inner context. For example, 44% of stakeholders reported lack of buy-in (EPIS: individual characteristics) and limited court resources (EPIS: organizational characteristics), especially personnel to support triaging, as key barriers to implementation.
Conclusions: Our study identified numerous determinants of success to triage model implementation in family law settings. While existing court infrastructure provides a foundation for triaging, stakeholders highlighted a clear need for hiring and training what we now call a Family Law Navigator, who will take a leadership role in family law triaging.
References
1. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and policy in mental health, 38(1), 4–23. https://doi.org/10.1007/s10488-010-0327-7
2. Averill, J. B. (2002). Matrix Analysis as a Complementary Analytic Strategy in Qualitative Inquiry. Qualitative Health Research, 12(6), 855–866. https://doi.org/10.1177/104973230201200611
3. Rudd, B.N.,& Beidas, R.S. (2021). Reducing the scientific bench to judicial bench research-to-practice gap: Applications of implementation science to family law research and practice. Family Court Review. Advance online publication. https://doi.org/10.1111/fcre.12606
Disclosures of Interest: None declared
Barriers to engagement and adaptation strategies for socioculturally sensitive delivery of an evidence-based parent-mediated intervention for children with ASD: Clinician and family perspectives
Authors
Ms. Asha Rudrabhatla - Emory University; Marcus Autism Center
Ms. Natalie Brane - Emory University; Marcus Autism Center
Dr. Liz Greenfield - Emory University; Marcus Autism Center
Ms. Karen Guerra - Emory University; Marcus Autism Center
Dr. Nicole Hendrix - Emory University School of Medicine
Ms. Kadie Ulven - Emory University; Marcus Autism Center
Dr. Katherine Pickard - Emory University; Marcus Autism Center
Background: Parent-mediated interventions (PMIs) foster social-communication skills in children with Autism Spectrum Disorder (ASD), and parent responsivity1. However, there remain barriers to parent engagement when PMIs are implemented within historically underserved community settings2. Limited research has examined the structural barriers experienced by families of diverse sociocultural backgrounds when attempting to engage in PMIs. Further, it is unclear how PMIs for ASD are adapted during delivery to increase their sociocultural relevance.
Methods: Participants included n = 137 caregivers of a child with ASD (14-42 months) receiving Project ImPACT and clinicians included n = 3 speech language pathologists and n = 2 psychologists delivering Project ImPACT as part of routine outpatient services. Prior to participation in Project ImPACT, caregivers completed demographic information and the Barriers to Treatment Participation Scale (BTPS). After each Project ImPACT session, clinicians rated session-level adaptations using an augmenting and reducing framework4. Clinicians also participated in a semi-structured interview, analyzed via thematic analysis. Independent t-tests examined whether perceived barriers and session adaptations differed by a child’s racial, ethnic, and sociocultural background.
Results: BTPS scores did not differ by race, ethnicity, or insurance type. However, clinicians made more adaptations per session for Hispanic/Latinx-identifying families than Non- Hispanic/Latinx-identifying families (F(38) = -1.775, p = .042). Further, clinicians made more adaptations per session for primarily Spanish-speaking families than primarily English-speaking families: F(40) = -2.366, p = .011. Qualitative interviews suggest that clinicians adapt Project ImPACT for socioculturally diverse families to accommodate the language preferences of bilingual families, target goals of priority to families, involve extended family members, and validate family concerns about psychosocial stressors (e.g., discrimination/bias).
Conclusion: Findings suggest that clinicians make more adaptions to Project ImPACT for diverse families to promote engagement by increasing responsivity to sociocultural values and psychosocial stressors.
References
1. Nevill, R. E., Lecavalier, L., & Stratis, E. A. (2018). Meta-analysis of parent-mediated interventions for young children with autism spectrum disorder. Autism, 22(2), 84-98.
2. Kasari, C., Gulsrud, A., Paparella, T., Hellemann, G., & Berry, K. (2015). Randomized comparative efficacy study of parent-mediated interventions for toddlers with autism. Journal of Consulting and Clinical Psychology, 83(3), 554.
3. Castro, F. G., Barrera Jr, M., & Holleran Steiker, L. K. (2010). Issues and challenges in the design of culturally adapted evidence-based interventions. Annual review of Clinical Psychology, 6, 213-239.
4. Kim, J. J., Brookman-Frazee, L., Barnett, M. L., Tran, M., Kuckertz, M., Yu, S., & Lau, A. S. (2020). How community therapists describe adapting evidence-based practices in sessions for youth: Augmenting to improve fit and reach. Journal of Community Psychology, 48(4), 1238-1257.
Disclosures of Interest: None declared
Community colleges as a space for increasing equity in access to evidence-based mental health: Understanding current access, barriers and facilitators for Michigan community college students
Authors
Ms. Amy Rusch - University of Michigan
Dr. Seo Youn Choi - University of Michigan
Ms. Alex Ammann - University of Michigan
Ms. Kendall Mosher - Univeristy of Michigan
Dr. Sara Abelson - Hope Center for College, Community, and Justice
Dr. Shawna Smith - University of Michigan
Background: Nearly half of community college (CC) students report clinically significant symptoms of a mental health disorder, yet fewer than half of these students receive services. Treatment gaps are largest for racial/ethnic minorities and low-income students, and have been exacerbated by COVID-19. To better understand CC student mental health needs, Mental Health Improvement through Community Colleges (MHICC) aims to improve CC student access to evidence-based mental health treatments for depression/anxiety by assessing current gaps in mental health access and identifying barriers and facilitators to service utilization.
Methods: A landscape analysis was conducted of 28 CCs and 3 tribal colleges in Michigan to ascertain the availability and accessibility of mental health resources for students provided through CC and community settings. Following completion of the landscape analysis, interviews were conducted with CC mental health stakeholders (e.g., counselors, administrators) to better understand the barriers and facilitators to students CC mental health resource provision, access, and use.
Results: N = 31 Michigan CCs were included in the landscape analysis. The modal number of mental CC health professionals was two. N = 16 (52%) CC websites referenced providing community referrals and N = 14 (45%) mentioned teletherapy. N = 15 (48%) CCs participated in a follow-up stakeholder interview. Mental health stakeholders highlighted several common barriers to providing mental health support, including inability to reach students; overextended staff; and limited funding for providing mental health services. Key facilitators identified as potentially improving access were administration buy-in/championing, and collaboration from faculty, especially in identifying student need.
Conclusion: MHICC strives to increase awareness of CCs as integral to collegiate mental health and highlight opportunities for dissemination and implementation specific to this sector. As CCs drive much social mobility and serve diverse populations, improving access to high-quality mental health resources and treatments for CC students, through CCs and surrounding communities, holds promise for increasing equitable mental health access and outcomes.
References
1. Eisenberg, D., Golberstein, E., & Gollust, S. E. (2007). Help-seeking and access to mental health care in a university student population. Medical Care, 45(7), 594–601. https://doi.org/10.1097/MLR.0b013e31803bb4c1
2. Ellis, A. R., Konrad, T. R., Thomas, K. C., & Morrissey, J. P. (2009). County-level estimates of mental health professional supply in the United States. Psychiatric Services, 60(10), 1315–1322. https://doi.org/10.1176/ps.2009.60.10.1315
3. Lipson, S. K., Kern, A., Eisenberg, D., & Breland-Noble, A. M. (2018). Mental health disparities among college students of color. Journal of Adolescent Health, 63(3), 348–356. https://doi.org/10.1016/j.jadohealth.2018.04.014
Disclosures of Interest: None declared
Stakeholder perspective on the necessary conditions for successful implementation of parenting interventions in Botswana: What works?
Authors
Dr. Hlengiwe Sacolo Gwebu - University of Cape Town
Mr. Tendai Mutembedza - University of Cape Town
Ms. Jacqueline Kilby - University College London
Ms. Mary T. Mosenke - Ministry of Local Government and Rural Development, Department of Social Protection, Family Welfare Services Division
Mrs. Jeldau Rieff - Stepping Stones International
Ms. Nomsa Monare - Stepping Stones International
Mrs. Lisa Jamu - Stepping Stones International
Dr. Styn Jamu - Stepping Stones International
Dr. Jamie Lachman - University of Oxford
Prof. Lucie Cluver - University of Oxford
Prof. Catherine Ward - University of Cape Town
Background: Parent support programmes play a major role in the prevention of child abuse by encouraging positive parenting practices that promote safety and well-being for children and families. As the uptake of parenting programmes intensifies in LMICs; governments, policymakers, and programme implementers grapple with multiple decisions including how to efficiently transport, adapt and implement parenting interventions across settings.
Methods: In a series of interviews with programme implementers and stakeholders from 13 organisations in Botswana, we reflected on real world experiences, challenges, and best practices in the implementation of parent support programmes, using the Exploration, Preparation, Implementation, Sustainment (EPIS) framework (Moullin, J. C., et al. 2019). Purposive and snowball sampling were used to recruit key informants from governmental and non-governmental institutions. Out of 20 potential participants recruited, 17 were able to participate in the study. Data collection was carried out online, via Zoom video conferencing, at a time and space convenient to participants.
Results: Findings highlight several factors concerning the successful uptake and implementation of parenting programmes in Botswana including (1) an enabling environment such as policies, regulations and guidelines that support the implementation of parenting interventions, (2) innovation factors including capacity building, programme adaptation and remote programme delivery; (3) bridging factors such as working with skilled organisations and (4) inner context factors including positive work environments, ongoing training and support to programme staff.
Conclusion: The results highlight practical insights required by policymakers, and practitioners for successful implementation of parenting interventions in Botswana and other similar settings.
References
1. Moullin, J. C., Dickson, K. S., Stadnick, N. A., Rabin, B., & Aarons, G. A. (2019). Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implementation Science, 14(1), 1-16.
Disclosures of Interest: Lucie Cluver (0000-0002-0418-835X), Catherine Ward (0000-0001-8727-4175), and Jamie Lachman (0000-0001-9475-9218) were involved in the development of the PLH programs: they do not draw any income from programme. Yulia Shenderovich (0000-0002-0254-3397) and Jamie Lachman worked on the PLH trials in South Africa and based their doctoral work on these. Further, Jamie Lachman is the former Executive Director of Clowns Without Borders South Africa. Jamie Lachman also receives income as a master trainer for PLH programs.
Longitudinal evaluation of an integrated case management pilot for high-risk, high-need patients: a qualitative study using the Health Equity Implementation Framework (HEIF) to assess implementation
Authors
Ms. Nadia Safaeinili - University of California Berkeley
Dr. Mark Fleming - University of California Berkeley
Dr. Amanda Brewster - University of California Berkeley
Background: Health systems increasingly use case management programs to integrate social and medical services to support health equity. Limited evidence exists about key components of integrated case management program implementation, especially from a health equity perspective. This longitudinal qualitative study applied a health equity implementation framework to examine implementation of a multidisciplinary team-based case management pilot serving high-risk, high-need MediCal patients
Methods: We conducted 86 semi-structured phone interviews with patients (n = 31), case managers (n = 41), and county administrators (n = 14) across two time points using purposive sampling to identify a representative sample. Interviews were transcribed and coded using an inductive-deductive approach informed by the Health Equity Implementation Framework (HEIF) to identify facilitators and barriers to equitable implementation.
Results: Supportive innovation characteristics included prevention-focused nursing leadership during pilot design and development of a predictive algorithm incorporating socioeconomic factors to determine pilot eligibility. Encounters between patients and case managers centered around a hierarchically flat, multidisciplinary team of case managers leveraging varied expertise. Provider factors surfaced the importance of frontline and supervisory staff with diverse backgrounds and lived experience. Challenges included the invisible emotional labor of case management work. Patient factors highlighted the interdependent nature of patient needs, from emotional support to stable housing. In the inner and outer context, initial challenges included shifting health system values to prioritize preventative care and service integration. Strong inter- and intra-organizational relationships were essential to implementation success and equitable resource access. Finally, national recognition of structural and social influences on health through Centers from Medicare and Medicaid Services funding provided needed resources for pilot implementation.
Conclusion: Factors central to equitable implementation included equity-focused leadership, multidisciplinary teams with lived experience, eligibility criteria attentive to social factors, strong inter- and intra-organizational partnerships, and sufficient resources. Challenges included an initial mismatch in value alignment within the organization, and case manager administrative and emotional burden.
References
1. Chuang, E., Pourat, N., Haley, L. A., O’Masta, B., Albertson, E., & Lu, C. (2020). Integrating Health And Human Services In California’s Whole Person Care Medicaid 1115 Waiver Demonstration. Health Affairs, 39(4), 639–648. https://doi.org/10.1377/hlthaff.2019.01617
2. Woodward, E. N., Matthieu, M. M., Uchendu, U. S., Rogal, S., & Kirchner, J. E. (2019). The health equity implementation framework: Proposal and preliminary study of hepatitis C virus treatment. Implementation Science, 14(1), 26. https://doi.org/10.1186/s13012-019-0861-y
3. Woodward, E. N., Singh, R. S., Ndebele-Ngwenya, P., Melgar Castillo, A., Dickson, K. S., & Kirchner, J. E. (2021). A more practical guide to incorporating health equity domains in implementation determinant frameworks. Implementation Science Communications, 2(1), 61. https://doi.org/10.1186/s43058-021-00146-5
Disclosures of Interest: None declared
GuideMe: Guideline evaluation and implementation mechanisms in school health services. Protocol for a hybrid factorial trial
Authors
Dr. Åse Sagatun - Regional Centre for Child and Adolescent Mental Health. Eastern and Southern Norway
Dr. Solveig Holen - Regional Centre for Child and Adolescent Mental Health. Eastern and Southern Norway
Dr. Annette Jeneson - Regional Centre for Child and Adolescent Mental Health. Eastern and Southern Norway
Ms. Malene Brekke - Regional Centre for Child and Adolescent Mental Health. Eastern and Southern Norway
Ms. Kristin Waldum-Grevbo - Regional Centre for Child and Adolescent Mental Health. Eastern and Southern Norway
Dr. Hege Sjølie - VID Specialized University
Dr. Stine Ekornes - Regional Centre for Child and Youth and Child Welfare, NTNUal Health
Dr. Thomas Engell - Regional Centre for Child and Adolescent Mental Health
Background: A national guideline with best practice recommendations for school health services in Norway was launched in 2017. To promote healthy life skills and identify students in need of follow-up, the guideline strongly recommends individual consultations with all 8th graders and increased cooperation with schools (1). Implementation of guidelines and practice change is a challenge in many sectors (2). We therefore co-created an implementation tool (SchoolHealth) together with practitioners, students, and other stakeholders (3) consisting of three discrete implementation strategies: 1. “Digital dialogue and administration tool” (audit and feedback); 2. “Dialogue support” (ongoing consultation); 3. “School cooperation” (dissemination). Our overall objectives are to help the service implement the guideline recommendations, reach their goals, and increase knowledge about effective implementation mechanisms.
Methods: We will evaluate and optimize SchoolHealth by studying the three strategies’ effectiveness on guideline fidelity and the extent to which fidelity helps reach guideline goals. Using a hybrid evaluation design with a factorial experiment (4,5), forty schools will be randomly assigned to eight different combinations of the strategies. This multifactorial design allows for the testing of interactions as well as main effects due to equal distribution of all factors within each main effect (5). Students, school nurses, and school personnel will complete qualitative interviews and questionnaires at baseline, after consultation, and 6-months post consultation. Data collection will start autumn 2022. Sustainment and scale-up of the optimized SchoolHealth strategies using national e-health infrastructure are prepared.
Results: Our design allows us to identify effective combinations of strategies and test several implementation mechanisms. Identifying successful strategies for implementing the current guideline can support adolescents in a life phase vital for future health and wellbeing.
Conclusion: The study will yield valuable causal knowledge about implementation strategies and mechanisms, which can support service innovation and implementation of evidence-based practice.
References
1. Norwegian Directorate of Health. (2017). The National guideline for health promotion and preventive work in the child and youth health centres and school health service. https://www.helsedirektoratet.no/retningslinjer/helsestasjons-og-skolehelsetjenesten 30.04.2022
2. Correa, V. C., Lugo-Agudelo, L. H., Aguirre-Acevedo, D. C., Contreras, J. A. P., Borrero, A. M. P., Patiño-Lugo, D. F., & Valencia, D. A. C. (2020). Individual, health system, and contextual barriers and facilitators for the implementation of clinical practice guidelines: a systematic metareview. Health research policy and systems, 18(1), 1-11
3. Sagatun Å, Kvarme LG, Misvær N, Myhre M, Valla L, Holen S. (2021) Evaluating a Web-Based Health-Promoting Dialogue Tool in School Health Services: Feasibility and User Experiences. J Sch Nurs. 37(5):363-373.
4. Curran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012). Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical care, 50(3), 217–226.
5. Collins, L.M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer.
Disclosures of Interest: None declared
One state’s approach to a needs assessment and gaps analysis for expanding prevention services in child welfare
Authors
Dr. Rafaella Sale - Virginia Commonwealth University
Ms. Natalie Finn - Virginia Commonwealth University
Ms. Gabriela Aisenberg - Virginia Commonwealth University
Ms. Navneet Kaur - Virginia Commonwealth University
Dr. Michael Southam-Gerow - Virginia Commonwealth University
Background: The Needs Assessment Gaps Analysis-Virginia (NAGA-Va) project was developed through an academic-policy partnership between university-affiliated researchers and state government leaders. The Center for Evidence-based Partnerships in Virginia is a technical assistance provider that works to leverage one state’s vision for expanding evidence-based services via the Family First Prevention Services Act. NAGA-Va represents the Center’s initial attempt to characterize the service landscape to the extent possible within the pre-implementation phase of prevention services expansion.
Methods: Six studies were designed to address state partners’ questions relating the needs of the families in their localities and the accessibility of existing services. A contextual analysis included a review of archival records from multiple agencies and past implementation initiatives. An agency leadership stakeholder survey (n = 177) assessed organizational readiness for program implementation, knowledge of evidence-based interventions (EBIs) present in the state. Individual, in-depth interviews (n = 40) were also conducted with a different sample of state and local government employees. A series of eleven listening forums were held to collect qualitative information from local service referral brokers, agency leaders, private providers, caseworkers, supervisors, and community members (n = 176). Lastly, products included continuously updating knowledge banks that a) compile data to characterize current local service arrays, and b) converge provider databases to map workforce capacity statewide.
Results: Themes emerged to reveals barriers to EBI adoption at the service coordination layer, including a fractured system initially established to streamline referrals and severe workforce capacity concerns. Survey data demonstrated operational leader confidence and high self-reported EBI implementation readiness. Additional findings from NAGA-Va products will be described.
Conclusion: A multi-method, multi-layer measurement approach determined a series of implementation barriers that require attention. Pros and cons of a partnership’s approach to assessing the service landscape will be presented. Attendee feedback will be encouraged and facilitated.
References
1. Cervantes, P. E., Seag, D. E., Nelson, K. L., Purtle, J., Hoagwood, K. E., & Horwitz, S. M. (2021). Academic-Policy Partnerships in Evidence-Based Practice Implementation and Policy Maker Use of Child Mental Health Research. Psychiatric Services, 72(9), 1076-1079.
2. Chorpita, B. F., & Daleiden, E. L. (2018). Coordinated strategic action: Aspiring to wisdom in mental health service systems. Clinical Psychology: Science and Practice, 25(4), e12264.
3. Fagan, A. A., Bumbarger, B. K., Barth, R. P., Bradshaw, C. P., Cooper, B. R., Supplee, L. H., & Walker, D. K. (2019). Scaling up evidence-based interventions in US public systems to prevent behavioral health problems: Challenges and opportunities. Prevention Science, 20(8), 1147-1168.
Disclosures of Interest: None declared
Multi-method ethnography to evaluate community-engaged implementation research during the COVID-19 pandemic
Authors
Ms. Linda Salgin - San Ysidro Health, San Diego, CA, USA & Joint Doctoral Program in Public Health, San Diego State University/University of California San Diego, USA
Mrs. Kelli Cain - Herbert Wertheim School of Public Health and Human Longevity Science, University of California San Diego, La Jolla, CA,
Dr. Borsika Rabin - Herbert Wertheim School of Public Health and Human Longevity Science, University of California San Diego & UC San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
Mr. Paul Watson - The Global Action Research Center, San Diego, CA
Dr. William Oswald - The Global Action Research Center, San Diego, CA
Dr. Bonnie Kaiser - Department of Anthropology and Global Health Program, University of California San Diego, La Jolla, CA
Mr. Lawrence Ayers - Department of Obstetrics, Gynecology, and Reproductive Sciences, University of California San Diego, La Jolla, CA
Ms. Crystal Yi - Department of Urban Studies and Planning, University of California San Diego, La Jolla, CA
Ms. Alexander Alegre - Department of Urban Studies and Planning, University of California San Diego, La Jolla, CA,
Ms. Jessica Ni - UC San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center, University of California San Diego, La Jolla, CA
Ms. Allyn Reyes - Department of Urban Studies and Planning, University of California San Diego, La Jolla, CA
Ms. Kasey Yu - UC San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center, University of California San Diego, La Jolla, CA
Dr. Shelia Broyles - Department of Pediatrics, University of California San Diego, La Jolla, CA & UC San Diego Altman Clinical and Translational Research Institute Community Engagement Division, University of California San Diego, La Jolla, CA
Dr. Robert Tukey - Superfund Research Center & Department of Pharmacology, University of California San Diego, La Jolla, CA
Dr. Louise Laurent - Department of Obstetrics, Gynecology, and Reproductive Sciences, University of California San Diego, La Jolla, CA
Dr. Nicole Stadnick - University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center & Department of Psychiatry & Child and Adolescent Services Research Center
Background: Consistent and meaningful engagement of diverse partners is at the heart of successful program development and implementation (Angier et al., 2013; Pinto et al., 2021; Ryan et al., 2022). Community Advisory Boards (CABs) are frequently used to engage diverse partners to inform research projects. However, formal evaluation of the quality and degree of partner engagement within CABs is needed, especially when conducted virtually (Thayer et al., 2021). We describe a multi-method ethnographic approach to assessing partner engagement in two CABs that convened virtually during the COVID-19 pandemic.
Methods: Two projects focused on increasing equitable COVID-19 testing, vaccination, and clinical trial participation for underserved communities involved remote CAB meetings. CAB members co-created a Theory of Change to elucidate necessary conditions and actions to reduce COVID-19 disparities. Thirty-three partners representing 17 community groups participated in 15 sessions across the two CABs. Ethnographic documentation forms were collaboratively developed to assess multiple aspects of engagement (e.g., time spent speaking, types of interactions). Trained documenters observed CAB sub-groups and participated in debriefing meetings for quality assurance and process refinement. Documentation data were analyzed using content and rapid thematic analysis, descriptive statistics were summarized and findings were triangulated.
Results: A total of 4,540 interactions were identified across 15 meetings. The most frequent interaction was providing information (44%), followed by responding (37-38%). Speaking time varied by subgroups depending on the meeting topic. Rapid thematic analysis of stakeholder interactions identified three main categories: Theory of Change (e.g., providing input during activities), Meeting Logistics (e.g., bandwidth issues and translation), and Other (e.g., meeting reflections). Debriefing sessions led to several methodological refinements.
Conclusions: Assessing partner engagement through multiple methods allowed for nuanced ethnographic data collection that refined our local work with CAB members and contributes to the needed literature and pragmatic resources for evaluating community engagement in implementation research.
References
1. Angier, H., Wiggins, N., Gregg, J., Gold, R., & DeVoe, J. (2013). Increasing the relevance of research to underserved communities: lessons learned from a retreat to engage community health workers with researchers. Journal of Health Care for the Poor and Underserved, 24(2), 840–849. https://doi.org/10.1353/hpu.2013.0086
2. Pinto, R. M., Park, S. (Ethan), Miles, R., & Ong, P. N. (2021). Community engagement in dissemination and implementation models: A narrative review. Implementation Research and Practice, 2, https://doi.org/10.1177/2633489520985305
3. Ryan, G., Gilbert, P. A., Ashida, S., Charlton, M. E., Scherer, A., & Askelson, N. M. (2022). Challenges to Adolescent HPV Vaccination and Implementation of Evidence-Based Interventions to Promote Vaccine Uptake During the COVID-19 Pandemic: “HPV Is Probably Not at the Top of Our List.” Preventing Chronic Disease, 19(E15), https://doi.org/10.5888/pcd19.210378
4. Thayer, E. K., Pam, M., Al Achkar, M., Mentch, L., Brown, G., Kazmerski, T. M., & Godfrey, E. (2021). Best Practices for Virtual Engagement of Patient-Centered Outcomes Research Teams During and After the COVID-19 Pandemic: Qualitative Study. Journal of Participatory Medicine, 13(1), https://doi.org/10.2196/24966.
Disclosures of Interest: None declared
Adapting interoperable clinical decision support tools for chronic pain in primary care: A multidisciplinary system design workshop
Authors
Dr. Ramzi Salloum - University of Florida
Dr. Lori Bilello - University of Florida
Ms. Cara McDonnell - University of Florida
Ms. Christina Guerrier - University of Florida
Ms. Laura Gonzalez Paz - University of Florida
Dr. Francisco Martinez-Wittinghan - University of Florida
Dr. Maria Gutierrez - University of Florida
Dr. Ghania Masri - University of Florida
Dr. Ross Jones - University of Florida
Dr. Bryn Rhodes - Alphora
Dr. Laura Marcial - RTI International
Dr. Robert Hurley - Wake Forest School of Medicine
Ms. Julie Diiulio - Applied Decision Science
Ms. Laura Militello - Applied Decision Science
Dr. Chris Harle - University of Florida
Background: Chronic pain affects 50-100 million Americans and costs more than $600 billion annually. Yet, primary care clinicians and patients often lack the information to inform choosing pain treatments that balance risks and benefits. We conducted a multidisciplinary system design workshop to identify adaptations to two existing interoperable clinical decision support (CDS) tools for chronic pain, MyPAIN and PainManager.
Methods: MyPAIN is a patient-facing application that integrates with the patient portal and collects patient-reported pain symptoms, treatment information, and goals. PainManager is a clinician-facing application that displays patient-reported information from MyPAIN and other clinical information via a dashboard. To identify adaptations to these applications, we conducted a system design workshop involving members of the research team and 14 participants representing local clinical IT staff, primary care clinicians, pain specialist clinicians, and patient representatives. The 3.5-hour workshop was held via videoconference and included breakout and large-group discussions. The workshop was audio-recorded, and research team members collected detailed field notes, which were independently analyzed by four researchers, and themes identified by consensus.
Results: Patients provided positive feedback about the questionnaires in MyPAIN and expressed interest in feedback about their pain based on the data they report. Providers expressed the importance of ensuring that pain-relevant information is comprehensively displayed in PainManager (right information). Patients and providers expressed a desire to see historical trends of pain or related outcomes (right format). Patients reported that having the flexibility to access MyPAIN from home was a desirable feature (right time). All participants expressed that success of the applications depends on both care team member and patient engagement (right person).
Conclusion: A multidisciplinary design workshop identified several potential adaptations to MyPAIN and PainManager, including improvements related to the right format, information, and person. These themes will inform future cognitive interviews to further specify user-centered adaptions prior to implementation.
References
1. HL7 FHIR Foundation. Clinical decision support for chronic pain management and shared decision-making IG [Internet]. 2020 [updated 2021 Dec 20; cited 2022 Mar 6]. Available from: https://build.fhir.org/ig/cqframework/cds4cpm/index.html
2. Osheroff JA. Improving medication use and outcomes with clinical decision support. Chicago: HIMSS Publishing. 2009.
3. Salloum RG, Bilello L, Bian J, et al. Study protocol for a type III hybrid effectiveness-implementation trial to evaluate scaling interoperable clinical decision support for patient-centered pain management in primary care. Under review.
Disclosures of Interest: None declared
Implementation of cardiac rehabilitation – focus on core components from the perspective of professionals
Authors
Ms. Sari Saukkonen - South-Eastern Finland University of Applied Sciences
Dr. Maarit Karhula - Social Insurance Institution of Finland, Kela
Ms. Hennariikka Heinijoki - Social Insurance Institution of Finland, Kela
Dr. Riitta Seppänen-Järvelä - Social Insurance Institution of Finland, Kela
Background: Cardiac rehabilitation should be based on scientific research, specific expertise, and competence. This paper investigates the implementation of intervention in terms of what the core components of rehabilitation are from the perspective of service providers (multi-professional team). Core components refer to the essential functions, principles, and intervention activities considered necessary to produce desired outcomes.
The study is carried out in the field of rehabilitation in Finland. Regulated by legislation, Finnish Social Insurance Institution provides various rehabilitation interventions. Rehabilitation services executed by local service providers are guided by service descriptions defining the features of interventions.
Methods: The study applies the framework of Wierenga et al. (2012, 2013), combining the Consolidated Framework for Implementation Research (CFIR) and RE-AIM framework. It also applies a multi-method and multifaceted approach that provides information on the implementation, functionality, and outcomes.
Data sets are collected through electronic surveys and group interviews. In order to investigate the core components, we use vignettes in the interviews to trigger discussion and as a means to reveal the core components of the intervention. A vignette is a short story (case description) about a hypothetical person, and it is presented to participants during an interview. According to the mixed methods convergent design, the survey data and interviews are analysed separately before merging.
Results: The core components are identified and compared with service descriptions based on mixed methods integrated data analysis.
Conclusion: The study sheds light on the core components from within the rehabilitation intervention. Taking into account the perspectives of the professionals, it contributes to the understanding of the ‘black box’ of the implementation of intervention.
References
1. Blase, K. & Fixsen, D. (2013). Core Intervention Components: Identifying and Operationalizing What Makes Programs Work. ASPE research brief, office of the assistant secretary for planning and evaluation, Office of Human Services Policy. 1-21.
2. Jesus, T. S., Papadimitriou, C., Bright, F. A., Kayes, N. M., Pinho, C. S. & Cott, C. A. (2022). Person-centered rehabilitation model: Framing the concept and practice of person-centered adult physical rehabilitation based on a scoping review and thematic analysis of the literature. Archives of Physical Medicine and Rehabilitation, 1, 106-120. https://doi.org/10.1016/j.apmr.2021.05.005
3. Morris, J. H., Bernhardsson, S., Bird, M-L., Connell, L., Lynch, E., Jarvis, K., Kayes, N. M., Miller, K., Mudge, S. & Fisher, R. (2020). Implementation in rehabilitation: a roadmap for practitioners and researchers. Disability and Rehabilitation, 42(22), 3265-3274. https://doi.org/10.1080/09638288.2019.1587013
4. Wierenga,D., Engbers, L. H., Empelen, P. V., Duijts, S., Hildebrandt, V. H. & Mechelen, W. V. (2013). What is actually measured in process evaluations for worksite health promotion programs: a systematic review. BMC Public Health, 13, 1190. https://doi.org/10.1186/1471-2458-13-119
5. Winnige, P., Vysoky, R., Dosbaba, F. & Batali L. (2021). Cardiac rehabilitation and its essential role in the secondary prevention of cardiovascular diseases. World Journal of Clinical Cases, 9(8), 1761-1784. https://doi.org/10.12998/wjcc.v9.i8.1761
6. Åkerblad, L., Seppänen-Järvelä, R., & Haapakoski L. (2020). Integrative strategies in mixed methods research. Journal of Mixed Methods Research, 15(2), 1-19: https://doi.org/10.1177/1558689820957125
Disclosures of Interest: None declared
The influence of outer-context determinants on the implementation of school-based innovations to support gender and sexual minority adolescent mental health
Authors
Dr. Daniel Shattuck - Pacific Institute for Research and Evaluation
Dr. Cathleen Willging - Center Director, Senior Research Scientist Pacific Institute for Research and Evaluation
Dr. Jeffry Peterson - Pacific Institute for Research and Evaluation
Dr. Bonnie Richard - Pacific Institute for Research and Evaluation
Mr. Adrien Lawyer - Transgender Resource Center of New Mexico
Dr. Mary Ramos - University of New Mexico Health Sciences Center
Background: Schools are critical for supporting gender and sexual minority (GSM) youth well-being (Johns et al., 2019). Implementation of GSM-supportive interventions can stave off experiences of victimization, discrimination, and stigma that lead to adverse mental health outcomes like anxiety, depression, and suicidality (Ancheta et al., 2021; Kaczkowski et al., 2022). However, schools are subject to a wide range of factors that shift and subsume internal priorities. In the context of a cluster randomized controlled trial to study the implementation of six GSM-supportive evidence-informed practices (EIPs) in New Mexico high schools, we analyzed emergent outer-context determinants that affected EIP implementation.
Methods: We compiled 18 comprehensive school reports based on qualitative data organized by the Exploration, Preparation, Implementation, and Sustainment Framework phases to reflect EIP implementation at baseline and then across three years. Data included annual individual and small group qualitative interviews with school professionals, most of whom were members of Implementation Resource Teams charged with assessment, planning, and implementation. We analyzed data using deductive and inductive coding techniques.
Results: Analysis yielded three categories of outer-context determinants that created challenges and opportunities for implementation: 1) structurally-based social barriers expressed through competing ideologies of religion, social conservatism, and treating all students the same; 2) local, state, and national policy and political discourse that created new urgencies and leverage points for addressing GSM student well-being; and 3) crises events (e.g., school shootings, COVID-19 pandemic) that deprioritized EIPs and disrupted school-based supports, yet also catalyzed collaborations to deepen capacity for suicide prevention initiatives attentive to GSM student needs.
Conclusion: Longitudinal qualitative research is useful for documenting outer-context determinants that shape implementation environments, elucidating their origins and implications for the uptake of new practices to promote mental health equity for marginalized populations. Such determinants are rarely static, and demand targeted planning, resources, and commitment to address.
References
1. Ancheta, A. J., Bruzzese, J.-M., & Hughes, T. L. (2021). The Impact of Positive School Climate on Suicidality and Mental Health Among LGBTQ Adolescents: A Systematic Review. The Journal of School Nursing, 37(2), 75–86.
2. Johns, M. M., Poteat, V. P., Horn, S. S., & Kosciw, J. (2019). Strengthening our schools to promote resilience and health among LGBTQ youth: Emerging evidence and research priorities from The State of LGBTQ Youth Health and Wellbeing Symposium. LGBT health, 6(4), 146-155.
3. Kaczkowski, W., Li, J., Cooper, A. C., & Robin, L. (2022). Examining the relationship between LGBTQ-supportive school health policies and practices and psychosocial health outcomes of lesbian, gay, bisexual, and heterosexual students. LGBT health, 9(1), 43-53.
Disclosures of Interest: None declared
Primary aim results of a clustered SMART for developing a school-level, adaptive implementation strategy to support CBT delivery at high schools in Michigan
Authors
Dr. Shawna Smith - University of Michigan
Dr. Daniel Almirall - University of Michigan
Dr. Seo Youn Choi - University of Michigan
Dr. Elizabeth Koschmann - TRAILS at Tides Center
Ms. Amy Rusch - University of Michigan
Dr. Emily Bilek - University of Michigan
Ms. Annalise Lane - Univeristy of Michigan
Dr. James Abelson - University of Michigan
Dr. Daniel Eisenberg - UCLA Fielding School of Public Health
Dr. Joseph Himle - University of Michigan
Dr. Kate Fitzgerald - Columbia University Department of Psychiatry
Ms. Celeste Liebrecht - University of Michigan
Dr. Amy Kilbourne - University of Michigan
Background: Schools increasingly provide mental health services to students, but often lack access to implementation support for evidence-based practice (EBP) delivery. Given heterogeneity in implementation barriers across schools, development of adaptive implementation strategies (AISs) to guide implementation strategies in schools may support EBP scale-up.
Methods: A clustered, sequential, multiple-assignment, randomized trial (SMART) was used to inform development of a school-level AIS for supporting school professional (SP)-delivered cognitive behavioral therapy (CBT). All schools were provided with initial low-level implementation support and then were randomized to add Coaching or not. After eight weeks, schools were assessed for response based on to-date CBT delivery and/or perceived organizational barriers. “Slower-responder” schools were re-randomized to add Facilitation or not; responders continued with current support. The primary aim hypothesis was that SPs at schools receiving the REP + Coaching + Facilitation AIS would deliver more CBT than SPs at schools receiving REP alone. Secondary aims compared four embedded implementation strategies on CBT sessions, including by type (group, brief and full individual). Analyses used a marginal, weighted least squares approach developed for clustered SMARTs.
Results: Of 94 Michigan high schools (N = 169 SPs), 83 (88%) were slower-responders. Contrary to our hypothesis, there was no evidence of a significant difference in CBT sessions delivered between REP + Coaching + Facilitation and REP alone (121.1 vs. 111.4 average total CBT sessions; p = 0.68). In secondary analyses, the adaptive REP + Facilitation stratey resulted in the highest average CBT delivery (154 sessions) and the non-adaptive REP + Coaching the lowest (94.5 sessions).
Conclusion: In terms of SP-reported average CBT delivery, findings suggest that the most effective strategy is the AIS that (i) begins with REP, (ii) augments with Facilitation for slower-responder schools, and (iii) continues REP for responder schools. Future exploratory analyses will examine the impact of these strategies on fidelity of CBT delivery and change in student mental health outcomes.
References
1. Kilbourne, A. M., Almirall, D., Eisenberg, D., Waxmonsky, J., Goodrich, D. E., Fortney, J. C., Kirchner, J. E., Solberg, L. I., Main, D., Bauer, M. S., Kyle, J., Murphy, S. A., Nord, K. M., & Thomas, M. R. (2014). Protocol: Adaptive Implementation of Effective Programs Trial (ADEPT): cluster randomized SMART trial comparing a standard versus enhanced implementation strategy to improve outcomes of a mood disorders program. Implementation Science, 9(1), 132. https://doi.org/10.1186/s13012-014-0132-x
2. Kilbourne, A. M., Goodrich, D. E., Lai, Z., Almirall, D., Nord, K. M., Bowersox, N. W., & Abraham, K. M. (2015). Re-Engaging Veterans with Serious Mental Illness into Care: Preliminary Results from a National Randomized Trial of an Enhanced versus Standard Implementation Strategy. Psychiatric Services (Washington, D.C.), 66(1), 90–93. https://doi.org/10.1176/appi.ps.201300497
3. Kilbourne, A. M., Neumann, M. S., Pincus, H. A., Bauer, M. S., & Stall, R. (2007). Implementing evidence-based interventions in health care: Application of the replicating effective programs framework. Implementation Science, 2(1), 42. https://doi.org/10.1186/1748-5908-2-42
4. Kilbourne, A. M., Smith, S. N., Choi, S. Y., Koschmann, E., Liebrecht, C., Rusch, A., Abelson, J. L., Eisenberg, D., Himle, J. A., Fitzgerald, K., & Almirall, D. (2018). Adaptive School-based Implementation of CBT (ASIC): Clustered-SMART for building an optimized adaptive implementation intervention to improve uptake of mental health interventions in schools. Implementation Science, 13(1), 119. https://doi.org/10.1186/s13012-018-0808-8
5. Koschmann, E., Abelson, J. L., Kilbourne, A. M., Smith, S. N., Fitzgerald, K., & Pasternak, A. (2019). Implementing evidence-based mental health practices in schools: Feasibility of a coaching strategy. The Journal of Mental Health Training, Education and Practice, 14(4), 212–231. https://doi.org/10.1108/JMHTEP-05-2018-0028
6. Langley, A. K., Nadeem, E., Kataoka, S. H., Stein, B. D., & Jaycox, L. H. (2010). Evidence-Based Mental Health Programs in Schools: Barriers and Facilitators of Successful Implementation. School Mental Health, 2(3), 105–113. https://doi.org/10.1007/s12310-010-9038-1
7. NeCamp, T., Kilbourne, A., & Almirall, D. (2017). Comparing cluster-level dynamic treatment regimens using sequential, multiple assignment, randomized trials: Regression estimation and sample size considerations. Statistical Methods in Medical Research, 26(4), 1572–1589. https://doi.org/10.1177/0962280217708654
Disclosures of Interest: None declared
Using the CFIR and the RE-AIM CONSORT guidelines to evaluate implementation of a motivational intervention in university psychology courses.
Authors
Dr. Raechel Soicher - Oregon State University
Dr. Kathryn Becker-Blease - Oregon State University
Background: The research-practice gap refers to the failure of empirical effectiveness research to translate meaningfully into practical applications. In higher education research, this is evident in the low use or uptake of evidence-based practices in college classrooms. To help address the research-practice gap, we adapted multiple frameworks and validated measures of implementation outcomes to identify the facilitators to and barriers of implementing a motivational intervention in university-level general psychology courses (Hulleman et al., 2010).
Methods: This was a mixed methods study that examined (1) students’ perceptions of the appropriateness, feasibility, and acceptability (Weiner et al., 2017) of a motivational intervention used in their Introductory Psychology courses, (2) instructor’s perceptions of the facilitators and barriers associated with implementing the intervention, (3) an evaluation of the implementation using the Consolidated Framework for Implementation Research (CFIR), and (4) a reporting of the implementation study using the RE-AIM CONSORT guidelines from Glasgow et al. (2018).
Results: Most students agreed or strongly agreed that the motivational intervention was acceptable and appropriate for use in their Introductory Psychology courses.
Most students agreed, but none strongly agreed, that the intervention was easy to complete (i.e., feasible). Semi-structured interviews with 10 of the Introductory Psychology instructors identified both barriers and facilitators to the implementation along the CFIR dimensions of Intervention Characteristics, Inner Setting, Individual Characteristics, and Implementation Process. RE-AIM analysis identified issues with the student consent process that likely impacted our ability to generalize from the results.
Conclusions: Overall, we were able to successfully adapt multiple Implementation Science frameworks to our unique setting (pedagogy in higher education) to identify both facilitators of and barriers to the implementation of a motivational intervention. Specifically, our RE-AIM analysis highlighted challenges of classroom-based research that have not been previously discussed in the literature. Lastly, our research contributes to the motivational intervention research in higher education by being a novel investigation into specific implementation processes.
References
1. Glasgow, R. E., Huebschmann, A. G., & Brownson, R. C. (2018). Expanding the CONSORT figure: Increasing transparency in reporting on external validity. American Journal of Preventive Medicine, 55(3), 422–430. https://doi.org/10.1016/j.amepre.2018.04.044
2. Hulleman, C. S., Godes, O., Hendricks, B. L., & Harackiewicz, J. M. (2010). Enhancing interest and performance with a utility value intervention. Journal of Educational Psychology, 102(4), 880–895. https://doi.org/10.1037/a0019506
3. Weiner, B. J., Lewis, C. C., Stanick, C., Powell, B. J., Dorsey, C. N., Clary, A. S., Boynton, M. H., & Halko, H. (2017). Psychometric assessment of three newly developed implementation outcome measures. Implementation Science, 12(1), 108. https://doi.org/10.1186/s13012-017-0635-3
Disclosures of Interest: None declared
Improving adherence to postpartum family planning guidance: A hybrid Type 2 implementation and effectiveness study
Authors
Dr. Sarita Sonalkar - University of Pennsylvania
Dr. Ernest Maya - University of Ghana School of Public Health
Dr. Chris Guure - University of Ghana School of Public Health
Ms. Arden McAllister - University of Pennsylvania
Ms. Dzifa Adimle Puplampu - University of Ghana School of Public Health
Dr. Robert Gallop - University of Pennsylvania
Dr. Roseline Doe - World Health Organization, Ghana
Dr. Courtney A. Schreiber - University of Pennsylvania
Dr. Mary Eluned Gaffield - World Health Organization
Background: Postpartum family planning reduces morbidity and mortality (DaVanzo et al., 2007, 2008), but up to 62% of birthing people have an unmet need for contraception, primarily due to implementation challenges (Cleland et al., 2015; Rossier et al., 2015). We evaluated the implementation and effectiveness of the Postpartum Family Planning Package (PPFP), a multifaceted implementation strategy combining the World Health Organization mobile family planning app (Sonalkar et al., 2021), provider education, and counselling restructuring to promote individualized family planning counselling prior to hospital discharge after delivery.
Methods: Guided by the Consolidated Framework for Implementation Research (Damschroder et al., 2009), we conducted a stepped-wedge randomized controlled trial of the PPFP in 3 hospitals in Accra, Ghana. Inpatient postpartum encounters were observed during one random week of each month over a 12-month period, with random implementation of the intervention at one site every 3 months. The primary implementation outcome was penetration, or proportion of encounters in which a provider discussed all guideline-appropriate contraceptive methods; the primary effectiveness outcome was family planning method used by 6 weeks postpartum.
Results: We evaluated 945 pre- and 1151 post-intervention encounters from October 2020 to September 2021. All guideline-appropriate family planning methods were discussed in 366/945 (38.7%) pre- and 724/1151(62.9%) post-intervention encounters (p < 0.001). Clients decided upon a method in 173/945 (18.3%) of pre-, and 570/1151 (49.5%) of post-intervention encounters (p < 0.001). A method was received prior to discharge in 51/945 (5.4%) pre- and 86/1151 (7.4%) post-intervention participants (p = 0.056). By 6 weeks, method use increased to 60/945 (6.3%) in pre- and 112/1151 (9.7%) in post-intervention participants (p = 0.005).
Conclusion: Use of a hospital-based implementation strategy combining a provider-facing mobile app with provider education and individualized counseling had high penetration and was effective in promoting client decision to uptake contraceptive methods postpartum. Future research should examine strategies to improve chosen method provision and receipt.
References
1. Cleland, J., Shah, I. H., & Benova, L. (2015). A Fresh Look at the Level of Unmet Need for Family Planning in the Postpartum Period, Its Causes and Program Implications. Int Perspect Sex Reprod Health, 41(3), 155-162. https://doi.org/10.1363/4115515
2. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci,4, 50. https://doi.org/10.1186/1748-5908-4-50
3. DaVanzo, J., Hale, L., Razzaque, A., & Rahman, M. (2007). Effects of interpregnancy interval and outcome of the preceding pregnancy on pregnancy outcomes in Matlab, Bangladesh. BJOG, 114(9), 1079-1087. https://doi.org/10.1111/j.1471-0528.2007.01338.x
4. DaVanzo, J., Hale, L., Razzaque, A., & Rahman, M. (2008). The effects of pregnancy spacing on infant and child mortality in Matlab, Bangladesh: how they vary by the type of pregnancy outcome that began the interval. Popul Stud (Camb), 62(2), 131-154. https://doi.org/10.1080/00324720802022089
5. Rossier, C., Bradley, S. E., Ross, J., & Winfrey, W. (2015). Reassessing Unmet Need for Family Planning in the Postpartum Period. Stud Fam Plann, 46(4), 355-367. https://doi.org/10.1111/j.1728-4465.2015.00037.x
6. Sonalkar, S., Maya, E., Adanu, R., Samba, A., Mumuni, K., McAllister, A., Fishman, J., Schurr, D., Schreiber, C. A., Kolev, S., Doe, R., & Eluned Gaffield, M. (2021). Pilot monitoring and evaluation of the WHO postpartum family planning compendium mobile application: An in-depth, qualitative study. Int J Gynaecol Obstet, 153(3), 508-513. https://doi.org/10.1002/ijgo.13631
Disclosures of Interest: Sarita Sonalkar reports receiving research funding from Evofem, Myovant, and NIH.
Implementation facilitation efforts on intimate partner violence screening and subsequent psychosocial service utilization
Authors
Ms. Kelly Stolzmann - VA Center for Healthcare Organization and Implementation Research, VA Boston Healthcare System
Dr. Christopher Miller - VA Boston Healthcare System and Harvard Medical School
Ms. Julianne Brady - VA Center for Healthcare Organization and Implementation Research, VA Boston Healthcare System
Ms. Omonyele Adjognon - Center for Healthcare Organization and Implementation Research, VA Boston Healthcare System & Boston University
Dr. Melissa Dichter - Center for Health Equity Research and Promotion, Corporal Crescenz VA Medical Center & Department of Social Work, Temple University
Dr. Katherine Iverson - National Center for PTSD, VA Boston Healthcare System
Background: Intimate partner violence (IPV) against women is a population health issue in the United States. Screening increases IPV detection and, when paired with appropriate interventions, can mitigate IPV’s health effects [1]. The Veterans Health Administration (VHA) initiated implementation facilitation (IF) to roll out IPV screening programs in primary care [2]. IF consists of personalized, interactive support that can include trainings, education, and technical assistance. VHA recommends that women primary care patients are screened annually and that IPV + women are offered referrals for psychosocial services [3]. This study examines IPV screening rates and post-screening psychosocial service use at sites receiving IF.
Methods: A cluster randomized, stepped wedge, Hybrid Type II design was used in this study. IF occurred in two waves across nine sites. In the current report, we examine rates of IPV screening and post-screening psychosocial service uptake for the first wave of sites (n = 5, treating n = 4,075 women eligible for IPV screening during the IF study period). We examined medical records to identify IPV screening and psychosocial service utilization (e.g., social work and psychology) in the 60 days post-screening.
Results: Across the five sites, 47% of eligible women had an IPV screen (1,904/4,075) during the IF study period. Compared to those eligible but not screened, women who were screened for IPV were 1.37 times more likely to have a post-screening psychosocial visit, after adjusting for utilization pre-screening (p < .0001). IPV + women were 1.80 times more likely to have a psychosocial visit compared to IPV- women, adjusting for psychosocial utilization pre-screening (p = .0007).
Conclusions: These results suggest IF may impact the rate of IPV screening as well as post-screening psychosocial service utilization. Increased psychosocial service use following positive screens provides initial evidence that IPV screening programs are being implemented with fidelity (i.e., women who experience IPV are being effectively connected with potentially life-saving follow-up interventions).
References
1. Gerber MR, Iverson KM, Dichter ME, Klap R, Latta RE. Women veterans and intimate partner violence: current state of knowledge and future directions. J Women's Health. 2014;23(4):302–309. doi: 10.1089/jwh.2013.4513.
2. Iverson KM, Dichter ME, Stolzmann K, Adjognon OL, Lew RA, Bruce LE, Gerber MR, Portnoy GA, Miller CJ. Assessing the Veterans Health Administration's response to intimate partner violence among women: protocol for a randomized hybrid type 2 implementation-effectiveness trial. Implement Sci. 2020 May 7;15(1):29. doi: 10.1186/s13012-020-0969-0.
3. Veterans Health Administration. VHA Directive 1198. Intimate Partner Violence Assistance Program; 2018.
Disclosures of Interest: None declared
The effect of learning collaboratives on service, client, and implementation outcomes in healthcare settings: A systematic review
Authors
Ms. Diondra Straiton - Michigan State University
Ms. Jessie Greatorex - Michigan State University
Dr. Brooke Ingersoll - Michigan State University
Background: Though previous reviews have evaluated some outcomes for learning collaboratives/quality improvement collaboratives (Nadeem et al., 2013; Schouten et al., 2008; Wells et al., 2018), to date, no systematic review about has summarized the effect of learning collaboratives across all service, client, and implementation outcomes defined by Proctor and colleagues (2009). We evaluated how learning collaboratives affect each of these 15 outcomes.
Methods: We searched 6 major databases for studies of learning collaboratives. We included studies published prior to January 12, 2020 in peer-reviewed academic journals that took place in a healthcare setting. 2152 references were reviewed. The final sample included 86 articles across 81 studies. We also extracted learning collaborative components (e.g., multidisciplinary teams) and processes (e.g., training period before the learning collaborative).
Results: Learning collaboratives resulted in favorable improvement across all domains. Service outcomes (reported in 54.7% of articles) were slightly favorable, with 63.8% of those articles finding strong improvement, 19.1% having mostly mixed results, and 17.0% demonstrating little or no improvement. Client outcomes (reported in 45.3% of articles) were strongly favorable, with 76.9% of those resulting in strong improvement, 12.8% resulting in mostly mixed outcomes, and 10.3% resulting in demonstrating little or no improvement. Implementation outcomes (reported in 47.7% of articles) were slightly favorable, with 63.4% of those articles resulting in strong improvement, 19.5% resulting in mostly mixed results, and 17.1% resulting in demonstrating little or no improvement. Articles that included ongoing consultation (74.4% of articles) resulted in better improvement than those that did not. Results for each outcome (e.g., sustainability) will also be presented.
Conclusion: To reduce the research-to-practice gap, healthcare organizations should prioritize continuous quality improvement activities. Though learning collaboratives are effective, more cost-effective implementation strategies should also be considered, such as ongoing consultation – a component used in most effective learning collaboratives.
References
1. Nadeem, E., Olin, S. S., Hill, L. C., Hoagwood, K. E., & Horwitz, S. M. (2013). Understanding the components of quality improvement collaboratives: A systematic literature review. The Milbank Quarterly, 91(2), 354–394. https://doi.org/10.1111/milq.12016
2. Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009). Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36(1), 24–34. http://dx.doi.org.proxy2.cl.msu.edu/10.1007/s10488-008-0197-4
3. Schouten, L. M. T., Hulscher, M. E. J. L., Everdingen, J. J. E. van, Huijsman, R., & Grol, R. P. T. M. (2008). Evidence for the impact of quality improvement collaboratives: Systematic review. BMJ : British Medical Journal, 336(7659), 1491. Research Library. https://doi.org/10.1136/bmj.39570.749884.BE
4. Wells, S., Tamir, O., Gray, J., Naidoo, D., Bekhit, M., & Goldmann, D. (2018). Are quality improvement collaboratives effective? A systematic review. BMJ Quality & Safety, 27(3), 226–240. https://doi.org/10.1136/bmjqs-2017-006926
Disclosures of Interest: None declared
“No one ever asked us": From ideas to implementation with polio frontline workers in Peshawar, Pakistan
Authors
Ms. Marium Sultan - Johns Hopkins Bloomberg School of Public Health
Prof. Svea Closser - Johns Hopkins Bloomberg School of Public Health
Dr. Erin Finley - University of Texas Health Science Center at San Antonio; VA Greater Los Angeles
Background: Our intervention was conducted with an international health project aimed at improving polio vaccine coverage rates in Pakistan, where polio eradication efforts are currently under threat (Independent Monitoring Board of the Global Polio Eradication Initiative, 2019). At the community level, programs and initiatives for vaccination are usually determined by regional or national level planners. Frontline workers (FLWs) are critical partners in ensuring campaign success, but often feel disempowered and disengaged from their work, which is too frequently dangerous, low-status, and underpaid (Maes et al., 2014). This project aimed to motivate and empower FLWs of the Global Polio Eradication Initiative (GPEI) by working with them to develop and refine ideas for program improvement, while also engaging local and regional leadership in adopting and supporting recommended improvements.
Methods: Our team created an implementation science methodology, called “IMPACT”, based around the principles of Human-Centered Design (Adam et al., 2020), and inspired by the Veteran’s Administration Diffusion of Excellence Program (VHA Best Practices: Exploring the Diffusion of Excellence Initiative, 2016). The first step of our process had FLWs ideate ways to break down the biggest barriers they face while conducting their work. Shortlisted teams participated in a pitch competition, and winning ideas were selected by local polio leadership. We continued to work with FLWs and polio leadership through the implementation process, moving their ideas from conception to reality.
Results: Over the course of two rounds of IMPACT, we received 181 idea submissions, and 9 were chosen for implementation. Ideas spanned multiple topics, such as community relations, operational efficiency, integration with other health services, skills training, and greater FLW recognition. We heard, across management levels, that the process positively impacted FLWs motivation and confidence. FLWs expressed wanting more opportunities to present their suggestions for change to leadership.
Conclusion: IMPACT resulted in meaningful engagement and collaboration from both male supervisors and low-status female workers in a highly gender-stratified setting. We hope our methodology will guide other organizations to solicit and implement locally tailored innovations from groups that have been rarely heard.
References
1. Adam, M. B., Minyenya-Njuguna, J., Karuri Kamiru, W., Mbugua, S., Makobu, N. W., & Donelson, A. J. (2020). Implementation research and human-centred design: How theory driven human-centred design can sustain trust in complex health systems, support measurement and drive sustained community health volunteer engagement. Health Policy and Planning, 35(Supplement_2), ii150–ii162. https://doi.org/10.1093/heapol/czaa129
2. Independent Monitoring Board of the Global Polio Eradication Initiative. (2019). The Art of Survival: The Polio Virus Continues to Exploit Human Frailties (No. 17). Independent Monitoring Board of the Global Polio Eradication Initiative.
3. Maes, K., Closser, S., & Kalofonos, I. (2014). Listening to Community Health Workers: How Ethnographic Research Can Inform Positive Relationships Among Community Health Workers, Health Institutions, and Communities. American Journal of Public Health, 104(5), e5–e9. https://doi.org/10.2105/AJPH.2014.301907
4. VHA Best Practices: Exploring the Diffusion of Excellence Initiative, 114th Congress, 2 (2016).
Disclosures of Interest: None declared
A comparison of continuous quality improvement and fidelity-based learning collaboratives for cognitive processing therapy
Authors
Mrs. Kera Swanson - Stanford University & National Center for PTSD
Ms. Freya Whittaker - Stanford University and National Center for PTSD
Dr. Heidi La Bash - Stanford University and National Center for PTSD
Dr. Rachel Liebman - Ryerson University, Canada
Dr. Christopher Miller - VA Boston Healthcare System and Harvard Medical School
Dr. Rachel Haine-Shlagel - Department of Child & Family Development, San Diego State University
Dr. Shannon Wiltsey Stirman - Stanford University & National Center for PTSD
Dr. Candice Monson - Ryerson University
Background: Learning collaboratives (LCs) are effective in addressing implementation barriers across healthcare settings 1,2. Intensive, in-person LCs are not always feasible 3. This research characterizes the areas of focus and elements used to support sustainability of Cognitive Processing Therapy (CPT) within the context of a comparison of two virtual, lighter touch LCs based on the Breakthrough Collaborative Model 4.
Methods: Participating clinicians (N = 150) were assigned to either fidelity-based (FID) or Continuous Quality Improvement (CQI) LCs, aimed at improving CPT delivery. The FID LC focused on addressing challenges to the CPT protocol in a fidelity consistent way. The CQI LC focused on implementing CQI processes to address challenges to CPT use. LCs facilitators completed facilitation checklists (N = 331), summarizing the LC activities. Clinicians from both conditions also completed monthly surveys assessing their satisfaction with the LC that month.
Results: In the CQI groups, the most common themes and plans focused on: Maintaining fidelity to the CPT protocol (30%), addressing patient-related barriers (27.7%), and improving clinician-related barriers (17.6%). As expected, during the FID LC, the most discussed theme was maintaining fidelity to the CPT protocol (78.6%). The CQI condition (x̄ = 2.97; SD = 1.50) had higher rates of satisfaction with the LC compared to the FID condition (x̄ = 2.69; SD = 1.60), t(518) = 2.02, p < .001. However, there were no significant differences among conditions on the amount of consultation received being sufficient in addressing barriers to clinician’s CPT delivery t(448) = 2.05, p > .05.
Conclusion: Focusing on maintaining fidelity to the CPT protocol was the most common theme discussed during the LCs. However, the method in which these themes were addressed had significant differences in the clinician’s satisfaction of the LCs. These findings expand our understanding around topics helpful in improving clinician skills and invite future studies to further clarify the mechanisms that drive successful CQI efforts.
References
1. Cavaleri, M. A., Gopalan, G., McKay, M. M., Messam, T., Velez, E., & Elwyn, L. (2010). The effect of a learning collaborative to improve engagement in child mental health services. Children and Youth Services Review, 32(2), 281-285. ps://doi.org/10.1016/j.childyouth.2009.09.007
2. Schouten, L. M., Hulscher, M. E., van Everdingen, J. J., Huijsman, R., & Grol, R. P. (2008). Evidence for the impact of quality improvement collaboratives: Systematic review. BMJ, 336, 1491–1494. https://doi.org/10.1136/bmj.39570. 749884.BE
3. Kizilcec, R. F. (2013, June). Collaborative learning in geographically distributed and in-person groups. In AIED 2013 Workshops Proceedings Volume (Vol. 67).
4. Kilo, C. M. (1998). A Framework for Collaborative Improvement: Lessons from the Institute for Healthcare Improvement's Breakthrough Series. Quality management in health care, 6(4), 1-13.
Disclosures of Interest: None declared
Development of implementation strategies to overcome barriers when implementing a combined lifestyle intervention for community-dwelling older people in community-care settings
Authors
Mrs. Patricia Van Der Laag - Julius Center for Health Sciences and Primary Care, Nursing Science, University Medical Center Utrecht, University Utrecht, Utrecht, The Netherlands
Dr. Berber Dorhout - Research Group Innovation of Human Movement Care, Research Centre for Healthy and Sustainable Living, Utrecht University of Applied Sciences, Utrecht, The Netherlands
Mr. Aaron Heeren - Julius Center for Health Sciences and Primary Care, Nursing Science, University Medical Center Utrecht, University Utrecht, Utrecht, The Netherlands
Dr. Di-Janne Barten - Research Group Innovation of Human Movement Care, Research Centre for Healthy and Sustainable Living, Utrecht University of Applied Sciences, Utrecht, The Netherlands
Prof. Cindy Veenhof - Department of Rehabilitation, Physical Therapy Science & Sports, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
Prof. Lisette Schoonhoven - Julius Center for Health Sciences and Primary Care, Nursing Science, University Medical Center Utrecht, University Utrecht, Utrecht, The Netherlands
Background: ProMuscle is a combined lifestyle intervention that has shown to be effective in improving muscle mass, muscle strength, and physical functioning in community-dwelling older adults. Potentially, it could facilitate older people in maintaining their functional independence.
To increase the likelihood of successful implementation of ProMuscle, this study aims to develop appropriate implementation strategies targeting previously identified barriers to implement ProMuscle in community-care.
Methods: A theory-informed approach was adopted to develop appropriate implementation strategies, consisting of four subsequent steps. First, previously identified barriers for implementation were categorized into the constructs of the Consolidated Framework for Implementation Research (CFIR), including the underlying theoretical constructs. Second, the CFIR-ERIC matching Tool linked barriers to implementation strategies. Behavioral change strategies were added from literature. Third, evidence for implementation strategies was sought in literature. Fourth, in co-creation with involved healthcare professionals and implementation experts, implementation strategies were operationalized to practical implementation activities following the guidance of Proctor. Lastly, an implementation plan that can be tailored to individuals’ context was developed, prioritizing implementation activities over time.
Results: A total of 654 barriers were categorized to the CFIR framework. The majority of barriers were related to the CFIR domain outer setting. Subsequently, the identified barriers were linked to 37 unique strategies. As many strategies affected multiple barriers, strategies were assigned in eight overarching themes: assessing the context, network internally, network externally, costs, education, process, champions, content of the intervention, and behavioral change of the end-users.
Co-creation sessions with professionals and implementation-experts resulted in tangible implementation actions, processed into an online implementation toolbox that supports healthcare professionals chronologically during the implementation process.
Conclusion: The theory-informed approach in combination with co-creation led to the development of practical multicomponent implementation strategies to implement ProMuscle. Next step is to evaluate the implementation strategies including the implementation toolbox regarding the implementation of ProMuscle in community-care.
References
1. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(1). https://doi.org/10.1186/1748-5908-4-50
2. Proctor, E. K., Powell, B. J., & Mcmillen, J. C. (2013). Implementation strategies: recommendations for specifying and reporting. http://www.implementationscience.com/content/8/1/139
3. van Dongen, E. J. I., Haveman-Nies, A., Doets, E. L., Dorhout, B. G., & de Groot, L. C. P. G. M. (2020). Effectiveness of a Diet and Resistance Exercise Intervention on Muscle Health in Older Adults: ProMuscle in Practice. Journal of the American Medical Directors Association, 21(8), 1065-1072.e3. https://doi.org/10.1016/j.jamda.2019.11.026
4. Waltz, T. J., Powell, B. J., Chinman, M. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., Damschroder, L. J., & Kirchner, J. A. E. (2014). Expert recommendations for implementing change (ERIC): Protocol for a mixed methods study. Implementation Science, 9(1). https://doi.org/10.1186/1748-5908-9-39
Disclosures of Interest: None declared
Spanish translation of the Consolidated Framework for Implementation Research (CFIR) and the Expert Recommendations for Implementing Change (ERIC) taxonomy
Authors
Dr. Amelia Van Pelt - University of Pennsylvania
Dr. Alejandra Paniagua-Avila - Department of Epidemiology, Mailman School of Public Health, Columbia University
Dr. Amanda Sanchez - Department of Psychiatry, Perelman School of Medicine at the University of Pennsylvania; Penn Medicine Nudge Unit, University of Pennsylvania Health System, Philadelphia, PA, United States
Ms. Stephanie Sila - University of Pennsylvania School of Veterinary Medicine
Dr. Elizabeth Lowenthal - Department of Pediatrics, Perelman School of Medicine at the University of Pennsylvania
Dr. Byron Powell - Brown School, Washington University, St. Louis
Dr. Rinad Beidas - University of Pennsylvania
Background: Despite high-need for increasing adoption of evidence-based practices, limited implementation research occurs in low- and middle-income countries (LMICs). To address this disparity, the World Health Organization (WHO) proposed strengthening research capacity and assessing implementation frameworks for relevance to LMICs. The Consolidated Framework for Implementation Research (CFIR) and Expert Recommendations for Implementation Change (ERIC) taxonomy are commonly used frameworks, but only English and German versions exist. To build capacity for the international community, linguistic translation is needed. Addressing the second most-spoken language in the world, this research aims to develop a Spanish translation of the CFIR and ERIC taxonomy.
Methods: Utilizing the WHO guidelines for translation, we are translating the CFIR (version 2.0 and additions for LMICs) and ERIC taxonomy into Spanish through a systematic process: 1) forward translation into Spanish by a native Spanish-speaking implementation scientist, 2) back-translation into English by a bilingual global health researcher, and 3) piloting via group discussions with researchers working with Spanish-speaking populations not conducting implementation research (i.e., target users). To achieve a generalizable translation, we are recruiting a multi-cultural group of Spanish-speaking researchers. After each step, we are meeting to discuss the translation process and refine deliverables.
Results: This work has a target completion of summer 2022. This research will yield Spanish-translated versions of the CFIR 2.0 and ERIC taxonomy, with accompanying codebooks. Additionally, this work will describe the WHO “gold-standard” approach and identify challenges in the translation processes (e.g., scientific nuances).
Conclusions: The translation of these frameworks will provide tools to conduct implementation science among Spanish-speaking populations, thus advancing the development of a global implementation science community. Use of the translations will be monitored by downloads and citations. Successful translation will demonstrate feasibility and impact of systematic translation of frameworks, which can serve as a model for future studies involving additional languages.
References
1. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implement Sci, 4(50).
2. Eberhard, D. M., Simons, G. F., & Fennig, C. D. (Eds.). (2021). Ethnologue: Languages of the World. SIL International.
3. Guillemin, F., Bombardier, C., & Beaton, D. (1993). Cross-cultural adaptation of health-related quality of life measures: Literature review and proposed guidelines. J Clin Epidemiol, 46, 1417-32.
4. Kemp, C. G., Weiner, B. J., Sherr, K. H., Kupfer, L. E., Cherutich, P. K., Wilson, D., Geng, E. H., & Wasserheit, J. N. (2018). Implementation science for integration of HIV and non-communicable disease services in sub-Saharan Africa: A systematic review. AIDS, 32 Suppl 1, S93-S105.
5. Kirk, M. A., Kelley, C., Yankey, N., Birken, S. A., Abadie, B., & Damschroder L. (2016). A systematic review of the use of the Consolidated Framework for Implementation Research. Implement Sci, 11, 72.
6. Means, A. R., Kemp, C. G., Gwayi-Chore, M. C., Gimbel, S., Soi, C., Sherr, K., Wagenarr, B. H., Wasserheit, J. N., & Weiner, B. J. (2020). Evaluating and optimizing the consolidated framework for implementation research (CFIR) for use in low- and middle-income countries: A systematic review. Implement Sci, 15(17).
7. Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Procter, E. K., & Kirchner, J. E. (2015). A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci, 10(21).
8. Regauer, V., Seckler, E., Campbell, C., Phillips, A., Rotter, T., Bauer, P., & Muuler, M. (2021). German translation and pre-testing of Consolidated Framework for Implementation Research (CFIR) and Expert Recommendations for Implementing Change (ERIC). Implement Sci Commun, 2 (120).
9. van Widenfelt, B. M., Treffers, P. D., de Beurs, E., Siebelink, B. M., & Koudijs, E. (2005). Translation and cross-cultural adaptation of assessment instruments used in psychological research with children and families. Clin Child Fam Psychol Rev, 8, 135-47.
10. World Health Organization. (2020). Seven Approaches to Investing in Implementation Research in Low- and Middle Income Countries.
11. Yapa, H. M., & Barnighausen, T. (2018). Implementation science in resource-poor countries and communities. Implement Sci, 13(154).
Disclosures of Interest: None declared
Guiding and evaluating implementation strategies targeting improving interprofessional collaborative practices in health and social care professionals in the community regarding fall prevention.
Authors
Mrs. Meike van Scherpenseel - Research Group Innovation of Human Movement Care, HU University of Applied Sciences Utrecht, The Netherlands
Mrs. Rixt Zuidema - Research group Proactive Care for Elderly People Living at Home, HU University of Applied Sciences Utrecht, The Netherlands
Mrs. Saskia te Velde - Research Group Innovation of Human Movement Care, HU University of Applied Sciences Utrecht, The Netherlands
Background: Providing fall prevention practices (FPP) for community-dwelling older adults requires many health and social care services, due to the multifactorial nature of falls. Therefore, interprofessional collaboration practices (ICP) among health and social care professionals in communities are essential. This study aims to provide 1) insight into underlying structures that develop within local interprofessional networks and 2) practical guidance to improve and evaluate ICP.
Methods: First, multiple interprofessional teams of health and social care professionals involved in FPP in three communities in The Netherlands were established. Then, based on prior identified barriers for the implementation of FPP, implementation strategies focusing on improving ICP were selected. Next, practical actions were established to improve ICP, mainly aiming at the development of an interprofessional network. To provide insight in this network development over time, a Social Network Analysis (SNA) survey was distributed at baseline and after six months, which measures, among other things, the number of links between professionals. Evaluation of the process of improving ICP will be guided by The Conceptual Framework ICP.
Results: Actions to improve ICP included: 1) engagement in local interprofessional teams; 2) following an interprofessional training; 3) establishing referral pathways and 4) creating a health and social care map. At baseline, 26 professionals completed the SNA. Findings suggest that physical therapists and primary care and community nurses play a central role in local FPP. Data on the repeated SNA survey and the process evaluation is currently being collected.
Conclusion: In co-creation with local professionals, we established a set of four actions that are considered to be essential for fostering ICP. The SNA can be applied to gain insight in the development of networks and ICP over time. To clarify underlying processes in the network, evaluation with professionals is necessary, for which the Conceptual Framework ICP may be used.
References
1. Baxter, P., Markel-Reid, M. An interprofessional team approach to fall prevention for older home care clients ‘at risk’ of falling: health care providers share their experiences. International Journal of Integrated Care, 9(2), 1-12. https://doi:10.5334/ijic.317
2. Waltz, T. J., Powell, B. J., Fernández, M. E., Abadie, B., Damschroder, L. J. (2019). Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions. Implementation Science, 14(42). https://doi.org/10.1186/s13012-019-0892-4
3. Valente, T. W., Palinkas, L. A., Czaja, S., Chu, K-H, Brown, C. H. (2015), Social Network Analysis for Program. PLoS ONE, 10(6), e0131712. https://doi:10.1371/journal.pone.0131712
4. Smit, L. C., Dikken, J., Schuurmans, M. J., de Wit, N. J., Bleijenberg, N. (2020). Value of social network analysis for developing and evaluating complex healthcare interventions: a scoping review. BMJ Open, 10(11),e039681. https://doi:10.1136/bmjopen-2020-039681
5. Stutsky, B. J., Spence Laschinger, H. K. (2014). Development and Testing of a Conceptual Framework for Interprofessional Collaborative Practice. Health, Interprofessional Practice & Education 2(2):eP1066. https://doi.org/10.7710/2159-1253.1066
Disclosures of Interest: None declared
Development of a D&I capacity building logic model to evaluate UC San Diego ACTRI Dissemination and Implementation Science Center (DISC)
Authors
Ms. Clare Viglione - UC San Diego ACTRI Dissemination and Implementation Science Center,
Ms. Olivia Fang - UC San Diego ACTRI Dissemination and Implementation Science Center
Ms. Laura Sheckter - UC San Diego ACTRI Dissemination and Implementation Science Center
Dr. Nicole Stadnick - University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
Dr. Lauren Brookman-Frazee - UC San Diego Dissemination and Implementation Science Center & UC San Diego Department of Psychiatry & Child and Adolescent Services Research Center
Dr. Gregory Aarons - Department of Psychiatry, University of California, San Diego; Child & Adolescent Services Research Center; University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
Dr. Borsika Rabin - University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
Background: The UC San Diego ACTRI Dissemination and Implementation Science Center (DISC) is a leader for dissemination and implementation science through training, technical assistance, community engagement, and research advancement. The DISC developed a program-wide logic model integrating domains from the WUNDIR capacity building model and the Translational Science Benefits Model (TSBM) to inform the annual evaluation of member engagement and impact.
Methods: The DISC Logic Model (DLM) served as the framework for evaluation capturing information about Scientific Activities, Outputs, and Impact. The survey was distributed online to members with two reminders and a raffle in 2021 and 2022. The 2022 survey included items from the TSBM.
Results: The final DLM included four Inputs (e.g., Financial, Infrastructural), seven Scientific Activities (e.g., DIS Professional Networking), 20 Scientific Outputs (e.g., grants submitted), and areas of Impact organized into the TSBM Societal Benefits domains. In 2021, 98% (n = 111) of member respondents participated in at least one Scientific Activity with the most popular being ‘Monthly Journal Club’ (47%) and ‘Annual Methods Workshop’ (39%). In 2022, 94% (n = 95) participated in a Scientific Activity including ‘Monthly Journal Club’ (39%) and ‘DISC website resources’ (35%). With respect to Scientific Outputs, 56% endorsed at least one (e.g., grant or paper preparation) in 2021 and 38% endorsed an output in 2022. In terms of Impact, 14% reported measuring Translational Science Benefits with ‘Community & Public Health’ measures (e.g., healthcare accessibility) being most popular (92%).
Conclusion: The DLM facilitated a comprehensive evaluation of our center. Member engagement was high with nearly all respondents participating in at least one Scientific Activity. Technical assistance offerings such as Journal Club were most popular in 2021, while asynchronous online resources increased in popularity in 2022. Actionable steps include prioritizing support with dissemination (e.g., writing workshops) and resources focused on population- and societal-level measurement.
References
1. Brownson, R. C., Proctor E. K., Luke, D. A., Baumann, A. A., Staub, M., Brown, M. T., & Johnson, M. (2017). Building capacity for dissemination and implementation research: one university's experience. Implementation Science, 12(1), 104. https://doi.org/10.1186/s13012-017-0634-4
2. Davis, R., & D’Lima, D. (2020). Building capacity in dissemination and Implementation Science: a systematic review of the academic literature on teaching and training initiatives.Implementation Science, 15(1), 1-26.Chicago
3. Brownson, R. C., Proctor E. K., Luke, D. A., Baumann, A. A., Staub, M., Brown, M. T., & Johnson, M. (2017). Building capacity for dissemination and implementation research: one university's experience. Implementation Science, 12(1), 104. https://doi.org/10.1186/s13012-017-0634-4
Disclosures of Interest: None declared
Does all roads lead to Rome? A coincidence analysis illustrating different paths from implementation outcomes to improved organizational occupational health
Authors
Prof. Ulrica von Thiele Schwarz - Mälardalen University
Dr. Susanne Tafvelin - Umeå University
Dr. Marta Roczniewska - Karolinska Institutet
Dr. Ole Sørensen - National Research Centre for the Working Environment (NFA), Copenhagen, Denmark
Prof. Karina Nielsen - Sheffield University
Dr. Johan Simonsen Abildgaard - Copenhagen Business School
Dr. Kasper Edwards - Technical University of Denmark
Background: In occupational health, organizational interventions (OOHIs) are recommended but the scientific support for such interventions is mixed. One reason may be that current evaluations fail to sufficiently consider the way these interventions are implemented (Nielsen et al., 2021). The aim of this presentation is to explain under what conditions the implementation of an OOHI leads to desired outcomes. We investigate whether perceived appropriateness of the intervention, implementation team intensity and time investment were necessary and/or sufficient conditions for achieving desired intervention outcomes.
Method: The case is an OOHI implemented at 64 preschools in Denmark (n = 1800). Intervention effectiveness has previously been evaluated using a quasi-experimental design (Framke & Sørensen (2015). The role of appropriateness (staff ratings of relevance), intensity (implementation team ratings) and time investment (hours spent on intervention and implementation activities among employees) for change in work environment satisfaction (WES) and core task quality (CTQ, staff ratings at T1 and T2) and sick leave (register data) was analyzed with Coincidence Analysis (CNA) (Whitaker et a., 2020). CNA combines case and variable-oriented approaches, allowing for investigation of the impact of configurations of factors and the different ways in which factors can be combined to produce the same result.
Results: Preliminary analyses indicate that appropriateness is a sufficient but not necessary conditions for improved CTQ and WES and that time investment is sufficient but not necessary for decreased sickness absence. Implementation team intensity can be either high or low and still lead to outcomes, depending on how it is combined with time invested.
Conclusion: In logic models, implementation outcomes are generally considered roads to intervention outcomes. This study illuminates a need to advance understanding of how and why implementation outcomes can be a road to some intervention outcomes, but not others, or only in combination with other implementation outcomes.
References
1. Framke, E., & Sørensen, O. H. (2015). Implementation of a participatory organisational-level occupational health intervention-focusing on the primary task. International Journal of Human Factors and Ergonomics, 3(3-4), 254-270.
2. Nielsen, K., Antino, M., Rodríguez-Muñoz, A., & Sanz-Vergel, A. (2021). Is it me or us? The impact of individual and collective participation on work engagement and burnout in a cluster-randomized organisational intervention. Work & Stress, 35(4), 374-397.
3. Whitaker, R. G., Sperber, N., Baumgartner, M., Thiem, A., Cragun, D., Damschroder, L., … & Birken, S. (2020). Coincidence analysis: a new method for causal inference in implementation science. Implementation Science, 15(1), 1-10.
Disclosures of Interest: None declared
Multisector stakeholders’ perceptions of the sustainability of a culturally-adapted, evidence-based intervention for South Asians at risk for cardiovascular disease
Authors
Dr. Milkie Vu - Northwestern University Feinberg School of Medicine
Ms. Nicola Lancki - Northwestern University Feinberg School of Medicine
Dr. C. Hendricks Brown - Northwestern University Feinberg School of Medicine
Dr. Namratha Kandula - Northwestern University Feinberg School of Medicine
Background: Sustaining evidence-based interventions (EBIs) for communities experiencing health disparities is central to health equity and requires meaningful engagement with multisector stakeholders [1,2]. Little is known about stakeholders’ perceptions of EBI sustainability and whether perceptions vary based on stakeholders’ contexts. Using community-based participatory research, we previously developed the South Asian Healthy Lifestyle Intervention (SAHELI), a culturally-adapted EBI targeting diet, physical activity, and stress management among South Asian adults at risk for cardiovascular disease [3]. SAHELI is being tested in a type 1 hybrid effectiveness-implementation study and includes a mixed-methods exploration of stakeholders’ perceptions of intervention sustainability.
Methods: In 2022, we conducted a cross-sectional survey with 18 SAHELI stakeholders from different settings (e.g., community organizations, public health department, health system, and academic university). The survey was adapted from the Sustainment Measurement System Scale [4] and the Program Sustainability Assessment Tool [5] and included subscales related to sustainability. Subscale scores were mean of non-missing items and ranged from 1-5; higher scores indicated greater agreement with factors influencing SAHELI sustainability.
Results: Sixteen participants (88% response rate) completed the survey; of those, 56% were from community organizations, 6% from the public health department, 19% from the health system, and 19% from the academic university. Across settings, participants reported highest subscale scores for “responsiveness to community values” (M = 4.9, SD = 0.2) and “responsiveness to community needs” (M = 4.9, SD = 0.3). On average, participants reported lowest subscale scores for “funding and financial support” (M = 3.5, SD = 1.5) and “infrastructure and capacity to support sustainment” (M = 4.2, SD = 1.1); those from the public health department and health system also reported lower scores for these subscales compared to those from other settings.
Discussion: Findings highlight the needs to explore multiple aspects of EBI sustainability and differences in sustainability perceptions based on stakeholders’ contexts. Future qualitative interviews will investigate stakeholders’ perspectives on different strategies to maintain sustainability.
References
1. Brownson, R. C., Kumanyika, S. K., Kreuter, M. W., & Haire-Joshu, D. (2021). Implementation science should give higher priority to health equity. Implementation Science, 16(1), 1-16.
2. Shelton, R. C., Chambers, D. A., & Glasgow, R. E. (2020). An extension of RE-AIM to enhance sustainability: addressing dynamic context and promoting health equity over time. Frontiers in Public Health, 8, 134.
3. Kandula, N. R., Bernard, V., Dave, S., Ehrlich-Jones, L., Counard, C., Shah, N., … & Siddique, J. (2020). The South Asian Healthy Lifestyle Intervention (SAHELI) trial: Protocol for a mixed-methods, hybrid effectiveness implementation trial for reducing cardiovascular risk in South Asians in the United States. Contemporary clinical trials, 92, 105995.
4. Palinkas, L. A., Chou, C. P., Spear, S. E., Mendon, S. J., Villamar, J., & Brown, C. H. (2020). Measurement of sustainment of prevention programs and initiatives: the sustainment measurement system scale. Implementation Science, 15(1), 1-15.
5. Luke, D. A., Calhoun, A., Robichaux, C. B., Elliott, M. B., & Moreland-Russell, S. (2014). Peer reviewed: The program sustainability assessment tool: A new instrument for public health programs. Preventing chronic disease, 11.
Disclosures of Interest: None declared
Implementation quality of a participatory worker health intervention was associated with changes in workers’ perceived health climate and supervisor support
Authors
Mr. Ryan Walsh - Washington University School of Medicine
Mrs. Jaime Strickland - Washington University School of Medicine
Mrs. Anna Kinghorn - Washington University School of Medicine
Mr. Sam Biver - Washington University School of Medicine
Mr. Ryan Colvin - Washington University School of Medicine
Dr. Ann Marie Dale - Washington University School of Medicine
Dr. Bradley Evanoff - Washington University School of Medicine
Background: Implementation quality and efficacy of worker health behavior change programs are associated. We assessed these complex associations in Working For You, a participatory worker weight loss program for low-income hospital workers. We aimed to 1) specify associations between implementation quality and efficacy and 2) the responsiveness of efficacy measures to implementation quality measures.
Methods: We recruited workers and a design team to develop and implement workplace health programs over 24 months. We assessed program implementation and quality with a modified Process Evaluation Rating Sheet (PERS) and study log metrics. We assessed perceived health climate with the Multifaceted Organizational Health Climate Assessment (MOHCA) and perceived coworker and supervisor support with the Job Content Questionnaire (JCQ) every 6 months. We assessed correlations between implementation quality and efficacy at the workgroup-level. We stratified implementation quality (i.e., low, medium, and high) based on PERS scores and assessed associations between PERS levels and MOHCA and JCQ scores with hierarchical linear models at the worker-level.
Results: 407 workers from 11 workgroups participated. PERS scores and the number of programs generated were associated (rs = 0.82, p < 0.05). Workgroup MOHCA scores and PERS scores at 12-months were associated (rs = 0.18-0.21, p < 0.05). Workers in groups with high PERS levels experienced changes in MOHCA scores at 12- and 18-months (ΔMOHCA = 0.31-0.57, p < 0.05) and changes in JCQ supervisor support scores at 12-months (ΔJCQ = 0.63, p < 0.05). Workers in groups with low and medium PERS levels had no changes in JCQ and MOHCA scores. No group maintained changes by 24-months.
Conclusion: Implementation quality was associated with efficacy, but the association faded over time. Using the PERS enabled us to highlight that 1) maintaining high implementation quality led to meaningful changes in efficacy measures and 2) efficacy measures had the strongest association with PERS metrics. Use of the PERS may enhance studies of worker health behavior change programs.
References
1. Stein, R. I., Strickland, J. R., Tabak, R. G., Dale, A. M., Colditz, G. A., & Evanoff, B. A. (2019). Design of a randomized trial testing a multi-level weight-control intervention to reduce obesity and related health conditions in low-income workers. Contemporary Clinical Trials, 79, 89-97.
2. Strickland, J. R., Kinghorn, A. M., Evanoff, B. A., & Dale, A. M. (2019). Implementation of the healthy workplace participatory program in a retail setting: A feasibility study and framework for evaluation. International Journal of Environmental Research and Public Health, 16(4), 590.
3. Wolfenden, L., Goldman, S., Stacey, F. G., Grady, A., Kingsland, M., Williams, C. M., … & Yoong, S. L. (2018). Strategies to improve the implementation of workplace-based policies or practices targeting tobacco, alcohol, diet, physical activity and obesity. Cochrane Database of Systematic Reviews, (11).
Disclosures of Interest: None declared
Provider-level results from two randomized pilot feasibility studies of a digital health tool to promote health behavior change
Authors
Ms. Callie Walsh Bailey - Brown School, Washington University, St. Louis, MO, United States
Dr. Ross Brownson - Washington University in St. Louis
Ms. Zoe Miller - Brown School, Washington University, St. Louis, MO, United States
Dr. Jane Garbutt - Washington University School of Medicine
Dr. Lisa de las Fuentes - Washington University School of Medicine
Dr. Russell Glasgow - University of Colorado Anschutz Medical Campus
Dr. Maura Kepper - Washington University in St. Louis
Background: Obesity among adolescents and young adults (AYA) is a major public health concern and puts AYA at risk for poor cardiovascular health. PREVENT is a digital health tool designed to support providers in delivering tailored, evidence-based behavior change recommendations and community and digital resource information to support cardiovascular health among AYA (Kepper et al., 2021.) We report provider-level findings from two randomized pilot feasibility studies of PREVENT.
Methods: We trained healthcare providers (N = 10) across four clinics who delivered PREVENT to 41 patients randomized to intervention (vs. usual care). We collected mixed methods data via direct observation and post-intervention surveys and interviews. We assessed implementation outcomes (acceptability, appropriateness, and feasibility measures scored on 5-point scales; Weiner et al., 2017), barriers and facilitators (guided by the Consolidated Framework for Implementation Research; Damschroder et al., 2009), and recommendations for adaptations and implementation strategies to promote ongoing use of PREVENT.
Results: On average, providers spent 7 minutes using PREVENT with patients. Nine providers completed surveys, eight participated in interviews. Mean implementation outcome scores were: acceptability = 3.90, appropriateness = 3.92, and feasibility = 4.14. Providers indicated PREVENT was easy to use and the health information was easy to understand. Providers found PREVENT’s specific, evidence-based physical activity and nutrition recommendations and community resource map offered advantage over current practices. Lack of integration with the electronic health record was a key barrier to using PREVENT. Providers noted need for additional training and support staff to continue using PREVENT after the study.
Conclusion: Findings suggest that PREVENT was well-received, yet may be improved by better alignment with current systems and workflows. Future work is needed to optimize PREVENT integration and to design an implementation strategy package to support sustained use. Lessons learned can inform other efforts to adapt and implement digital health tools to support healthcare providers to promote cardiovascular health.
References
1. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation science, 4(1), 1-15.
2. Kepper, M. M., Walsh-Bailey, C., Brownson, R. C., Kwan, B. M., Morrato, E. H., Garbutt, J., De Las Fuentes, L., Glasgow, R. E., Lopetegui, M. A., & Foraker, R. (2021). Development of a health information technology tool for behavior change to address obesity and prevent chronic disease among adolescents: Designing for dissemination and sustainment using the ORBIT model. Frontiers in Digital Health, 3, 23.
3. Weiner, B. J., Lewis, C. C., Stanick, C., Powell, B. J., Dorsey, C. N., Clary, A. S., Boynton, M. H., & Halko, H. (2017). Psychometric assessment of three newly developed implementation outcome measures. Implementation Science, 12(1), 1-12.
Disclosures of Interest: None declared
It is important to be evidence-based in supporting interventions: What do we know about the evaluation and effectiveness of technical assistance
Authors
Dr. Abraham Wandersman - Wandersman Center
Dr. Victoria Scott - University of North Carolina–Charlotte
Ms. Zara Jillani – University of North Carolina–Charlotte
Ms. Adele Malpert - Southern Methodist University
Ms. Jennifer Kolodny-Goetz - Wandersman Center
Background: The interactive systems framework for dissemination and implementation (ISF) emphasizes the importance of providing capacity building support (training, technical assistance-TA)] to delivery systems (e.g., clinics, schools) involved in implementing evidence-based interventions (EBIs). TA is a widely used individualized capacity-building strategy for supporting EBI implementation and quality improvement efforts. Until now, no reviews have systematically examined the evaluation of TA across implementation contexts and capacity building aims. This study draws on two decades of peer-reviewed publications to summarize the evidence on the evaluation and effectiveness of TA.
Methods: Guided by Arksey and O’Malley’s six-stage methodological framework, we used a scoping review methodology to map research on TA evaluation in five databases: Business Source Complete, CINAHL, ERIC, PsycINFO, and PubMed of peer-reviewed research published between 2000-2020.
Results: 125 evaluation research studies met the study criteria. Findings include: publications have increased, signaling a growth in the recognition and reporting of TA; TA is being implemented across diverse settings, often serving socially vulnerable and under-resourced populations; most evaluation research studies involved summative evaluations, with TA outcomes mostly reported at the organizational level; Only 5% of the studies examined sustainability of TA outcomes; lack of consistent standards regarding the definition of TA and of relevant TA evaluation categories (e.g., cadence of contact, directionality).
Conclusions: Advances in the science and practice of TA hinge on understanding what aspects of TA are effective, when, how, for whom. Addressing these core questions requires i) a standard definition for TA, ii) more robust and rigorous evaluation research designs that involve comparison groups and assessment of direct, indirect, and longitudinal outcomes, iii) increased use of reliable and objective TA measures, and iv) development of reporting standards. This review is foundational for improving the role of evidence-based support strategies in implementation science –including a key component of the ISF framework.
References
1. Dunst CJ, Annas K, Wikie H, Hamby D. (2019) Scoping review of the core elements of Technical assistance and frameworks. World Journal of Education. 9:109-122.
Katz J, Wandersman A. (2016) Technical assistance to enhance prevention capacity: A research synthesis of the evidence base. Prevention Science. 17:417–428
1. Wandersman A, Duffy J, Flaspohler P, Noonan R, Lubell K, Stillman L, Blachman M, Dunville R, & Saul J. (2008) Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. American Journal of Community Psychology. 2008;1(3):171-81.
Disclosures of Interest: None declared
The North Carolina partnerships to increase colorectal cancer screening: An evaluation of a CDC-funded colorectal cancer control program
Authors
Ms. Mary Wangen - University of North Carolina- Chapel Hill
Ms. Dina Alabsew - University of North Carolina- Chapel Hill
Ms. Molly Black - The American Cancer Society
Dr. Alison Brenner - University of North Carolina- Chapel Hill
Ms. Heather Dolinger - University of North Carolina- Chapel Hill
Ms. Alexis Hoyt - The American Cancer Society
Ms. Jean MacKay - The American Cancer Society
Ms. Debi Nelson - The North Carolina Division
of Public Health, Department of Health and Human Services, Cancer Prevention and Control Branch
Ms. Jennifer Park - The North Carolina Division of Public Health, Department of Health and Human Services, Cancer Prevention and Control Branch
Dr. Catherine Roweder - University of North Carolina at Chapel Hill
Dr. Renee Ferrari - University of North Carolina at Chapel Hill
Background: The North Carolina Partnerships to Increase Colorectal Cancer Screening (NC PICCS) is a collaboration between the NC Division of Public Health Department of Health and Human Services Cancer Prevention and Control Branch (CPCB), The American Cancer Society (ACS), and the University of North Carolina (UNC). NC PICCS is a CDC-funded Colorectal Cancer Control Program designed to increase colorectal cancer (CRC) screening rates by working with clinics to implement evidence-based interventions (EBIs) and providing program-eligible patients with funding for diagnostic colonoscopies. We report evaluation results from year one (July 1, 2020 – June 30, 2021).
Methods: Clinics attended a two-day Quality Improvement (QI) Learning Collaborative and monthly QI and technical assistance calls facilitated by ACS and CPCB. Clinics formed QI teams and used QI tools to select and implement EBIs. Guided by CDC’s Framework for Program Evaluation, UNC assessed program engagement, quality improvement capacity, improvements in CRC screening processes, implementation of EBIs, and changes in screening rates. Data sources included baseline and readiness assessments, an annual survey, pre- and post-focus groups, monthly surveys, attendance during trainings and calls, completed QI tools, and pre-, post-, and monthly screening rates.
Results: Two clinics from one Federally Qualified Health System participated in year one. Both clinics completed all QI tools: aim statements, process maps, gap analyses, and PDSA cycles. Clinics chose to implement patient and provider reminders, provider assessment and feedback, and EBIs that reduce structural barriers. Attendance and engagement in program activities were high, with representatives from each clinic at all trainings and calls. Screening rates steadily improved from 7.6 to 33% in Clinic A and 33.2 to 48% in Clinic B. Focus group results highlighted successes, challenges, and recommendations.
Conclusion: Results of this mixed methods evaluation support the NC PICCS program’s impact on clinics’ improved capacity to implement EBIs and increase CRC screening rates.
References
1. Division of Cancer Prevention and Control, Centers for Disease Control and Prevention (2022). Colorectal Cancer Control Program. https://www.cdc.gov/cancer/crccp/index.htm
Centers for Disease Control and Prevention, Program Performance and Evaluation Office. (2017). A Framework for Program Evaluation. https://www.cdc.gov/evaluation/framework/index.htm
2. Rohweder, C., Wangen, M., Black, M., Dolinger, H., Wolf, M., O'Reilly, C., Brandt, H., & Leeman, J. (2019). Understanding quality improvement collaboratives through an Implementation Science lens. Preventive medicine, 129S, 105859. https://doi.org/10.1016/j.ypmed.2019.105859
Disclosures of Interest: None declared
Piloting a community pharmacy-based colorectal cancer screening program (PharmFIT): Findings from stakeholder-engaged implementation planning and process mapping
Authors
Mr. Austin Waters - University of North Carolina at Chapel Hill
Ms. Mary Wangen - University of North Carolina at Chapel Hill
Ms. Olufeyisayo Odebunmi - University of North Carolina at Chapel Hill
Dr. Catherine Roweder - University of North Carolina at Chapel Hill
Dr. Renee Ferrari - University of North Carolina at Chapel Hill
Dr. Stephanie Wheeler - University of North Carolina at Chapel Hill
Dr. Parth Shah - Fred Hutchinson Cancer Research Center
Dr. Alison Brenner - University of North Carolina at Chapel Hill
Background: Colorectal cancer (CRC) is the second-leading cause of cancer death in North Carolina; however, current screening rates remain suboptimal. Community pharmacies are generally more accessible than clinic-based care and may be a promising source of cancer screening. The PharmFIT pilot objective is to implement CRC screening by distributing Fecal Immunochemical Test (FIT) kits in the community pharmacy setting. The objective of this abstract is to report on the findings of stakeholder-engaged process mapping, intended to identify how the implementation of PharmFIT can be tailored to each community pharmacy.
Methods: In collaboration with two pharmacist-primary care provider dyads, we conducted process mapping prior to initiating the PharmFIT intervention. The purpose of developing process maps was to inform the implementation of each pharmacy-located FIT kit distribution pilot. The findings were used to adapt and tailor pilot processes to address anticipated implementation barriers.
Results: During process mapping activities, a variety of potential implementation barriers were identified including ordering and billing for FIT kits and care coordination for FIT + patients. Barriers were discussed during process mapping to identify potential solutions that were later matched to ERIC implementation strategies. Several hour-long implementation planning meetings, including process mapping, resulted in robust stakeholder informed implementation guidance and process maps. The maps include swim lane diagrams, identification of task responsibilities, and materials and documentation needed for implementation. Through this stakeholder-engaged activity, each pharmacy-primary care dyad was able to tailor their approach to implementing PharmFIT to their pharmacy setting.
Conclusions: Overall, both pharmacy-primary care dyads were highly engaged in the process mapping methodology. The outcome was an implementation plan that is anticipated to be highly feasible and appropriate given the context of pharmacy dispensing and clinical workflows. Our findings illustrate the value of process mapping in selecting and refining implementation strategies in preparing for a larger PharmFIT trial.
References
1. Holle, L. M., Levine, J., Buckley, T., White, C. M., White, C., & Hadfield, M. J. (2020). Pharmacist intervention in colorectal cancer screening initiative. Journal of the American Pharmacists Association, 60(4), e109-e116. doi:https://doi.org/10.1016/j.japh.2020.02.014
2. Joseph, D. A., King, J. B., Richards, T. B., Thomas, C. C., & Richardson, L. C. (2018). Use of Colorectal Cancer Screening Tests by State. Preventing chronic disease, 15, E80-E80. doi:10.5888/pcd15.170535
3. Kononowech, J., Landis-Lewis, Z., Carpenter, J., Ersek, M., Hogikyan, R., Levy, C., . . . Sales, A. E. (2020). Visual process maps to support implementation efforts: a case example. Implementation Science Communications, 1(1), 105. doi:10.1186/s43058-020-00094-6
Disclosures of Interest: None declared
The feasibility of implementing Recovery Management Checkups (RMC) in an FQHC setting: Finding out what it takes to move a well-established evidence-based practice into real-world practice
Authors
Dr. Dennis Watson - Chestnut Health Systems
Dr. Christy Scott - Chestnut Health Systems
Dr. Mike Dennis - Chestnut Health Systems
Dr. Christine Grella - Chestnut Health Systems
Background: The Recovery Management Checkup (RMC) intervention has been demonstrated over the course of more than 20 years to improve substance use disorder (SUD) treatment linkage and associated outcomes. Recognizing that SUD is a chronic, cyclical disease, RMC utilizes an initial linkage meeting and brief quarterly checkups to motivate and support clients. Despite positive outcomes from multiple clinical trials, RMC has not been fully replicated in real-world practice. This presentation will discuss actions taken to lay the groundwork for a hybrid effectiveness-implementation trial of RMC in collaboration with a Federally Qualified Health Center (FQHC). This work comes on the heels of a randomized control trial that demonstrated the effectiveness of RMC in four FQHCs. The primary goals of this next stage of work are to (a) identify what is required to implement RMC into FQHC practice and (b) establish feasibility of RMC outside of a highly controlled clinical trial.
Methods: Our approach is guided by the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework. We have completed intervention adaptation in collaboration with FQHC staff and are running a small feasibility study (n = 60 patients) assessing the ability of FQHC staff to deliver the adapted RMC.
Results: The presentation will discuss findings as they relate to the first three phases of EPIS. We will specifically describe our initial discussions with the FQHC that informed their choice to participate in this work (exploration), how we identified adaptations needed to the RMC intervention and chose our implementation strategy (preparation), and the feasibility of having the adapted RMC delivered by FQHC staff (implementation).
Conclusion: We will demonstrate how information learned has informed choices regarding the design of the future proposed hybrid study and how our general process can benefit researchers working with other well-established interventions that have not yet been fully translated to practice.
References
1. Dennis, M. L., & Scott, C. K. (2012). Four-year outcomes from the Early Re-Intervention (ERI) experiment using recovery management checkups (RMCs). Drug and alcohol dependence, 121(1-2), 10-17.
2. Scott, C. K., Dennis, M. L., & Foss, M. A. (2005). Utilizing recovery management checkups to shorten the cycle of relapse, treatment reentry, and recovery. Drug and Alcohol Dependence, 78(3), 325-338.
3. Scott, C.K., Dennis, M. L., Grella, C. E., Watson, D. P., & Davis, J. P. (2022). Recovery management check-ups for alcohol and other drug use treatment in primary care (RMCPC): A randomized controlled trial [Manuscript submitted for publication].
Disclosures of Interest: None declared
Advancing new approaches to implementation process measurement: Integration of the Exploration, Preparation, Implementation, Sustainment framework and the Stages of Implementation Completion
Authors
Mr. Dylan Randall Wong - Oregon Social Learning Center; Department of Psychology, University of South Carolina
Dr. Lisa Saldana - Oregon Social Learning Center
Dr. Gregory Aarons - Department of Psychiatry, University of California, San Diego; Child & Adolescent Services Research Center; University of California San Diego Altman Clinical and Translational Research Institute Dissemination and Implementation Science Center
Background: Implementation theories, models, and frameworks (TMFs) are increasingly used to guide implementation. A growing range of measures also has been developed to support implementation efforts. However, many of these measures are developed ad-hoc and independently of TMFs. While frameworks such as Exploration, Preparation, Implementation, Sustainment (EPIS; Aarons et al., 2011; Moullin et al., 2019) and the Consolidated Framework for Implementation Research (CFIR) have recently emphasized the inclusion of measures of the implementation constructs identified by those frameworks, implementation determinant measures have generally been prioritized over implementation process measures. In this study, we aim to integrate a process measure – the Stages of Implementation Completion (SIC; Chamberlain et al., 2011) – with EPIS by (1) synthesizing EPIS as a process framework with SIC as a process measure and (2) identifying the links between EPIS determinants and SIC process activities.
Methods: EPIS phase definitions from two foundational EPIS papers were extracted and parsed for key markers and activities that delineated the phase. SIC activity definitions were drawn from the Universal SIC codebook and matched to EPIS phase definitions, allowing for SIC activities to be sorted into EPIS phases. Example definitions of SIC activities in relation to their functional purpose within the EPIS implementation process were generated.
Results: Results provide practical guidance on using SIC to provide more nuanced assessment of moving through the EPIS phases. This analysis yielded outcomes mapping SIC activities/stages to EPIS phases and EPIS determinants to SIC activities. This study advances implementation science by better articulation of implementation process and underscores important future directions for framework-measure integration.
Conclusion: This study illustrates a systematic approach to understanding the implementation process and the relevant determinants to consider during implementation. Users of EPIS and SIC – and of other TMFs and process measures – might benefit from such integration efforts.
References
1. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23. https://doi.org/10.1007/s10488-010-0327-7
2. Moullin, J. C., Dickson, K. S., Stadnick, N. A., Rabin, B., & Aarons, G. A. (2019). Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implementation Science, 14(1), 1. https://doi.org/10.1186/s13012-018-0842-6
3. Chamberlain, P., Brown, C. H., & Saldana, L. (2011). Observational measure of implementation progress in community based settings: The Stages of implementation completion (SIC). Implementation Science, 6(1), 116. https://doi.org/10.1186/1748-5908-6-116
Disclosure of Interest: Dr. Saldana has a potential conflict as the developer of the SIC that is available for purchase.
“Finding a seat at the table”: Complex relationships among veteran treatment courts, community organizations, and the VA in the MISSION-CJ intervention
Authors
Ms. Vera Yakovchenko - Center for Health Equity Research and Promotion (CHERP) VA Pittsburgh Healthcare System
Ms. Kathryn Bruzios - VA Center for Healthcare Organization and Implementation Research (CHOIR), VA Bedford Healthcare System
Ms. Jessica Blue-Howells - Veterans Justice Programs
Dr. David Smelson - VA Center for Healthcare Organization and Implementation Research (CHOIR), VA Bedford Healthcare System
Dr. Megan Mccullough - VA Center for Healthcare Organization and Implementation Research (CHOIR), VA Bedford Healthcare System
Background: Veteran Treatment Courts (VTC) are an alternative to incarceration and provide intensive, supervised probation, drug testing, and treatment through a multidisciplinary team. This team involves Department of Veterans Affairs (VA) social workers assigned to each of the 500 VTCs across the country. However, VTCs do not have a universal way to operate, including in addressing co-occurring disorders (COD). We took a multi-partner approach to study pre-implementation factors related to VTCs adopting an evidence-based COD intervention called MISSION-CJ (Maintaining Independence and Sobriety through Systems Integration, Outreach and Networking-Criminal Justice).
Methods: We conducted qualitative interviews, guided by the Dynamic Sustainability Framework, with VA, VTC, and community organization staff between April 2021 and April 2022.
Results: We interviewed 24 individuals across 5 courts (7 VA social workers, 4 VA leaders, 3 probation officers, 2 judges, 4 VTC staff, and 4 community organization staff). The interorganizational complexities discussed, challenged the perception of MISSION-CJ fit and early adoption by partners. The MISSION-CJ intervention team faced challenges negotiating for access as a partner in VTCs. Key issues involved identifying gatekeepers, attempting to broker “a seat at the table,” managing communication, and sharing information within and across partners. Once “at the table,” staffing and coordination issues hindered implementation planning and preparation. While each VTC and/or community organization promoted their mentor program, existing mentors were not equipped to act as peers and deliver MISSION-CJ. Attempts to hire VA peers were hampered due to partners’ competing goals unless working through local VA opinion leader. Technical challenges in virtually training staff across settings reduced intervention credibility and delayed implementation.
Conclusion: This study contributes to an innovative and growing area of implementation science on multi-partnered work. Future work includes repeat interviews to observe changes in perceptions over the course of MISSION-CJ implementations effort.
References
1. Finlay, A. K., Smelson, D., Sawh, L., McGuire, J., Rosenthal, J., Blue-Howells, J., … & Harris, A. H. (2016). US Department of Veterans Affairs Veterans Justice Outreach program: Connecting justice-involved veterans with mental health and substance use disorder treatment. Criminal Justice Policy Review, 27(2), 203-222.
2. Pinals, D. A., Gaba, A., Clary, K. M., Barber, J., Reiss, J., & Smelson, D. (2019). Implementation of MISSION–Criminal Justice in a treatment court: Preliminary outcomes among individuals with co-occurring disorders. Psychiatric Services, 70(11), 1044-1048.
3. Yakovchenko, V., McCullough, M. B., Smith, J. L., Gabrielian, S., Byrne, T., Bruzios, K. E., … & Smelson, D. A. (2021). Implementing a complex psychosocial intervention for unstably housed Veterans: A realist-informed evaluation case study. Implementation Research and Practice, 2, 26334895211049483.
Disclosures of Interest: None declared
The PRIDI (Pragmatic, Rapid, and Iterative Dissemination & Implementation) tool to inform and evaluate implementation fit to shifting landscapes
Authors
Dr. Reza Yousefi Nooraie - University Of Rochester
Ms. Melhaney Reichelt - University of R
Dr. Kevin Fiscella - University Of Rochester
Dr. Bethany M Kwan - University of Colorado School of Medicine
Dr. Sarah A Birken - Wake Forest School of Medicine
Dr. Rachel C Shelton - Columbia University Mailman School of Public Health
Dr. Alden Lai - New York University
Background: The COVID-19 pandemic accelerated the need for rapid dissemination and implementation (D&I) of evidence-based interventions, amidst dynamic landscapes (the nature of the problem, available interventions, public perception, resources). Successful implementation requires attention to and evaluation of the dynamic landscapes (the nature of health problem, contexts, and resources) in which the implementation process is embedded. Previously, we developed the Pragmatic, Rapid, and Iterative D&I (PRIDI) framework1 to guide an iterative process of optimizing fit between implementation and dynamic landscapes. The purpose of this presentation is to describe a tool for applying PRIDI components to an implementation initiative.
Methods and Results: We developed a process tool to assess and optimize fit between implementation activities/processes (PRIDI’s inner circle: setting goals and targets, determining interventions and strategies, identifying and engaging implementation actors and stakeholders, evaluation of the processes and outcomes) and shifting landscapes (outer circle: the evolving problem & existing solutions, resources & capacities, people & contexts). The tool helps users to determine whether they need to assess fit dynamically, how to form a fit assessment/optimization team, how to ask guiding questions that focuses on the fit between implementation activities and landscapes, and how to evaluate the fitting process. We will present the process guide to 3-5 implementation science experts and 5-7 members of the CDC Vaccine Confidence Network (who are involved in the implementation of interventions to address COVID19 vaccine hesitancy), in a series of cognitive interviews2. We will report interviewee reflections on the relational nature of the fit between inner and outer circles of PRIDI, and conceptual and practical considerations in the fitting process.
Discussion: The PRIDI tool and process guide can provide a structured framework to inform and evaluate iterative planning and evaluation3 to optimize fit throughout implementation process.
References
1. Yousefi Nooraie, R., Shelton, R. C., Fiscella, K., Kwan, B. M., & McMahon, J. M. (2021). The pragmatic, rapid, and iterative dissemination and implementation (PRIDI) cycle: adapting to the dynamic nature of public health emergencies (and beyond). Health Research Policy and Systems, 19(1), 1-10.
2. Ryan, K., Gannon-Slater, N., & Culbertson, M. J. (2012). Improving survey methods with cognitive interviews in small-and medium-scale evaluations. American Journal of Evaluation, 33(3), 414-430.
3. Glasgow, R. E., Battaglia, C., McCreight, M., Ayele, R. A., & Rabin, B. A. (2020). Making implementation science more rapid: use of the RE-AIM framework for mid-course adaptations across five health services research projects in the veterans health administration. Frontiers in Public Health, 8, 194.
Disclosures of Interest: None declared
Author Index
A. Rani Elwy
Aaloke Mody
Aaron Heeren
Aaron Luneke
Aaron Lyon
Aaron Seaman
Abraham Wandersman
Adam Kinney
Adele Malpert
Adina Lieberman
Adriana Kaori Terol
Adrien Lawyer
Agnieszka Neumann-Podczaska
Ahmad Firas Khalid
Alana Chandler
Alana Gilbert
Alasia Ledford
Alayna Park
Alden Lai
Alejandra Arce
Alejandra Paniagua-Avila
Aleksandra Luszczynska
Alethea Desrosiers
Alex Ammann
Alex Collie
Alex Dopp
Alex Kandah
Alex Ramsey
Alexander Alegre
Alexis Hoyt
Alexis Huynh
Ali Rowhani-Rabhar
Alicia Bunger
Alisa Lincoln
Alison Brenner
Alison Hamilton
Alithia Zamantakis
Allison Jobin
Allison Metz
Allison Mosqueda
Allyn Reyes
Allyson Cogan
Allyson Varley
Althea Hart
Alvaro Sanchez
Alyson Codner
Amanda Brewster
Amanda Farr
Amanda Jensen-Doss
Amanda Midboe
Amanda Sanchez
Amber Calloway
Amberlee Venti
Amelia Palumbo
Amelia Van Pelt
Amy Doyle
Amy Drahota
Amy Huebschmann
Amy Kilbourne
Amy Law
Amy Miller
Amy Rusch
Amy Slutzky
Amy Ursitti
Ana Pachicano
Ana Stefancic
Andel Nicasio
Andrea Chalem
Andrea Graham
Andrea T. Duran
Andreas Rödlund
Andrew Clapperton
Andrew Quanbeck
Andrew Rosser
Andria Eisman
Ane-Marthe Skar
Angela Garza Mcwethy
Angela Klipsch
Angela Kyrish
Angela Pollard
Angelina Ruiz
Anjuli Wagner
Ann Marie Dale
Anna Banik
Anna Giannicci
Anna Kerlek
Anna Kinghorn
Anna Lau
Anna Toropova
Annalise Lane
Anne Mauricio
Anne Mbwayo
Anne Sales
Anneke Hjort
Annette Jeneson
Anouk Driessen
Anthony P. Mannarino
Antonio Garcia
Arden McAllister
Arianna Means
Artur Queiroz
Aruni Bhatnagar
Åse Sagatun
Asha Rudrabhatla
Ashelsha Onawale
Ashley Helle
Ashley Knapp
Ashlinn Quinn
Aubrey R. Dueweke
Aubyn Stahmer
Augustine Wasonga
Austin Waters
Barbara Caplan
Barrett Montgomery
Bart Cillekens
Beatriz Hernandez
Belinda O'Hagan
Berber Dorhout
Beth Ann Petrakis
Bethany Hipple Walters
Bethany M Kwan
Bevanne Bean-Mayberry
Beverly Funderburk
Bianca Albers
Bianca Brijnath
Blaine Garman-McClaine
Bo Kim
Boaz Kipkorir
Bonnie Kaiser
Bonnie Richard
Borsika Rabin
Bradley Evanoff
Bradley H Wagenaar
Brandon Taylor
Branson Fox
Brennan Keiser
Brian Blanco
Brian Boyd
Brian Bumbarger
Brian Mittman
Brian Mustanksi
Brian Pace
Brian Smith
Briana Last
Briana Lott
Brigitte Vachon
Brittany Cooper
Brittany Rudd
Brooke Ingersoll
Brooks Keeshin
Bruce Chorpita
Bryan Garner
Bryan Weiner
Bryn Rhodes
Bryony Stokes
Byron Powell
C. Amanda Schweizer
C. Hendricks Brown
Cady Berkel
Callie Walsh Bailey
Calvin Lai
Cam Escoffery
Candice Bangham
Candice Monson
Capri McDonald
Cara Lewis
Cara McDonnell
Carol Mangione
Carolyn Minor
Carolyn Schafer
Carrier Comeau
Caryn Blitz
Casey Chandler
Cassandra Vieten
Cassidy Gutner
Castillo Gisell
Catalina Ordorica
Catherine Chanfreau-Coffinier
Catherine Corbin
Catherine Kiptinness
Catherine Roweder
Catherine Ward
Cathleen Willging
Celeste Liebrecht
Chang(Lucy) Liu
Charles Goss
Charles Henderson
Charlotta Pisinger
Charlotte Demant Klinker
Charlotte Wåhlin
Chloe Tenega
Chris Gray
Chris Guure
Chris Harle
Christiaan Vis
Christian Helfrich
Christian Jackson
Christina Björklund
Christina Guerrier
Christina Harvey
Christina Johnson
Christine Grella
Christine Koffkey
Christine Stewart
Christopher Kemp
Christopher Miller
Christy Scott
Christy Yoon
Chynna Mills
Ciara Oliver
Cindy Veenhof
Claire Gwayi-Chore
Claire Than
Clara Barajas
Clara Johnson
Clare Viglione
Claudia Der-Martirosian
Clayton Cook
Colby Chlebowski
Collette Ncube
Constance Owens
Courtney A. Schreiber
Crystal Yi
Cynthia Weaver
Cyrilla Amanya
Daisy Le
Dallas Elgin
Dan Wu
Dana B. Mukamel
Dana Devine
Dana Hagele
Dani Schenk
Daniel Almirall
Daniel Amante
Daniel Blumenthal
Daniel Cheron
Daniel Edwards
Daniel Eisenberg
Daniel Scheller
Daniel Shattuck
Daniela Tuda
Daniella Kanyer
Danielle Adams
Danielle Fettes
Danielle Mazza
Danielle Stern
Daphne Lew
Dara H. Sorkin
David Adkins
David Atkins
David Hutton
David Kolko
David Mandell
David Rosengren
David Smelson
David Sommerfeld
Dawne Vogt
DeAujZhane Coley
Debbie Innes-Gomberg
Debi Nelson
Deborah Cohen
Dennis Li
Dennis Watson
Désiré Dabla
Devon Minch
Devynne Diaz
Diane Carney
Diane E. Cannone
Di-Janne Barten
Dina Alabsew
Diondra Straiton
Don Lapierre
Donna Potter
Donna Shelley
Donoria Evans
Douglas Luke
Doyanne Darnell
Duncan Mortimer
Dylan Randall Wong
Dzifa Adimle Puplampu
Ebony Holliday
Eduardo Ceballos-Corro
Eduardo Salas
Elaina Monague
Elena Kuo
Eleni Sofouli
Elijah Sosa
Elisabeth Vesnaver
Elisabeth Williams
Elise Berlan
Elissa Faro
Elissa Gomez
Elizabeth Bernhardt
Elizabeth Casline
Elizabeth Koschmann
Elizabeth Lane
Elizabeth Lowenthal
Elizabeth Maguire
Elizabeth McGuier
Elizabeth R. Fraser
Elizabeth Rangel
Elizabeth Spitzer
Elizabeth V. Eikey
Elizabeth Yano
Ella Baumgarten
Ellen Rubinstein
Elsa-Grace Giardina
Elvin Geng
Emily Becker-Haimes
Emily Bilek
Emily Callanan
Emily Dossett
Emily Feinberg
Emily Gibson
Emily Iovino
Emily Lattie
Emily Potter
Emily Treichler
Emily Velandia
Emma Low
Emmah Owidi
Emmeline Chuang
Eric Bressman
Erica Doering
Erika Crable
Erin Almklov
Erin Finley
Erin McRee
Erlend Høen Laukvik
Ernest Maya
Erwin Ista
Essodinam Miziou
Eva Woodward
Fabrizia Giannotta
Faiza Rab
Fatima Mabrouk
Fatima Mozaffari
Fawn Gadel
Faye Curran
Femke Van Nassau
Fiona Riordan
Fiona Rowles
Florence Momplaisir
Francisco Martinez-Wittinghan
Francisco Villaruell
Freda Liu
Freya Whittaker
Gabriela Aisenberg
Gabriela Becerra
Gail D'Souza
Gakuo Maina
Galina Portnoy
Garrett Brown
Gashaye Melaku Tefera
Gavin West
Gemmae Fix
Geoffrey Curran
Georgette McMichael
Gera Welker
Ghania Masri
Gila Neta
Gillian Dysart
Ginny Sprang
Glenndl Miguel
Golda Ginsburg
Gordon Kordas
Grace Woodard
Gracelyn Cruden
Gregory Aarons
Gregory Simon
Gregory Young
Gunnar Bergström
Gwen Lawson
Hajo Zeeb
Hannah Cheng
Hannah Espeleta
Hannah Frank
Hannah Gelman
Hannah Samuels
Hannah Sebald
Heather Angier
Heather Cook
Heather Dolinger
Heather Kane
Heather Pane Seifert
Heather Reisinger
Heather Schacht Reisinger
Hege Sjølie
Heidi La Bash
Henna Hasson
Hennariikka Heinijoki
Hiba Abousleiman
Hidde Van der Ploeg
Hikabasa Halwiindi
Hiywote Eshetu
Hlengiwe Sacolo Gwebu
Holle Schaper
Holly Ann Russell
Holly Huber
Holly Lanham
Hyun Seon Park
Ian Muse
Ilana Hardesty
Ingrid Eshun-Wilson
Ingrid Eshun-Wilsonova
Irene Blair
Irene Jensen
Iruma Bello
Isabelle Kaminer
Jacey Greece
Jacob Szeszulski
Jacqueline Kilby
Jacquie Brown
Jaime Strickland
James Abelson
James Ford
James Lee
James Merle
James Pittman
Jamie Faro
Jamie Jaramillo
Jamie Lachman
Jane Ann McCullough
Jane Garbutt
Jane Silovsky
Janet Welsh
Janine Wendt
Janna Martin
Jansey Lagdamen
Jared Martin
Jason Chapman
Javier Benito
Jax Witzig
JD Smith
Jean MacKay
Jeanine Lane
Jeanne Ringle
Jeff Stone
Jeffrey Smith
Jeffry Peterson
Jeldau Rieff
Jemma Hawkins
Jenna Rudo-Stern
Jennifer Boggs
Jennifer Elston-Lafata
Jennifer Freeman
Jennifer Hardee
Jennifer Johnson
Jennifer Kolodny-Goetz
Jennifer Leeman
Jennifer Park
Jennifer Sharpe Potter
Jennifer Sullivan
Jennifer Velloza
Jeremy Grimshaw
Jeri Forster
Jerica Knox
Jessica Blue-Howells
Jessica Coifman
Jessica Fishman
Jessica Haughton
Jessica Koslouski
Jessica Montoya
Jessica Ni
Jessica Reed
Jessica Singer
Jessica Suhrheinrich
Jessica Tschida
Jessica Young
Jessie Greatorex
Jessie Sanders
Jill Ehrenreich-May
Jill Locke
Jillian Abasta
Jinying Chen
Jiyoung Song
JoAnn Kirchner
Joanna Jain
Joanna Kim
Joanna Moullin
Joanne Enticott
JoAnne Malloy
Joanne Otis
Jochebed Gayles
Jodie Buntain-Ricklefs
Johan Simonsen Abildgaard
John Gault
John Griffith
John Lestino
Jonathan Purtle
Jorge Castro
Joseph Currier
Joseph Gardella
Joseph Himle
Joyce H. L. Lui
Juan Villamar
Judith A. Cohen
Judith Jelsma
Judy Fridline
Juergen Steinacker
Julia Cox
Julia D'Ambrosio
Julia Felton
Julia Moore
Julian Brunner
Julianne Brady
Julie Chin
Julie Diiulio
Julie Richards
Julie Rutledge
Julie White
Jure Baloh
Justin Kenardy
Justin Presseau
Jyoti Mathad
Kadie Ulven
Kai Zheng
Kane Carlock
Kanika Arora
Kara Byrne
Kara Izzo
Karen Bearss
Karen Drummond
Karen Guerra
Karey O'Hara
Karina Egeland
Karina Nielsen
Karissa Fenwick
Karla Ramirez
Karolina Lobczowska
Kasey Yu
Kasper Edwards
Kate Comtois
Kate Fitzgerald
Kate Guastaferro
Kate Lovero
Kate Ramm
Kate Whetten
Katelin Hoskins
Katharine Rendle
Katherine Dollar
Katherine Hirchak
Katherine Iverson
Katherine Pickard
Kathleen Brady
Kathryn Becker-Blease
Kathryn Broderick
Kathryn Bruzios
Kathryn Hyzak
Kathryn Lancaster
Kathy Francoeur
Katie McCormick
Katja Reuter
Kaylee Kruzan
Kelli Cain
Kelli Scott
Kelly Edyburn
Kelly Shedd
Kelly Stolzmann
Kendal Reeder
Kendall Mosher
Kenneth Bruemmer
Kenneth Ngure
Kenneth Ruggiero
Kera Mallard-Swanson
Kera Swanson
Kevin Fiori
Kevin Fiscella
Kim Robien
Kimberly Arnold
Kimberly Becker
Kimberly Garner
Kimberlye Dean
Kirali Genao
Kjerstin Stigmar
Krisda Chaiyachati
Kristen Benito
Kristen Higgins
Kristin Vick
Kristin Waldum-Grevbo
Kristina Cordasco
Kristina Oishi
Kristina Palomares
Kristine Campbell
Kristoffer Pettersson
Krzysztof Kaczmarek
Kyle Rubini
Larissa Gaias
Laura Beres
Laura Gonzalez Paz
Laura Marcial
Laura Militello
Laura Sheckter
Laura Taylor
LauraEllen Ashcraft
Lauren Bochicchio
Lauren Brookman-Frazee
Lauren Clack
Lauren Milgram
Lauren Tessier
Laurie Lindamer
Laurie Weingart
Lawrence Ayers
Lawrence Palinkas
Lea Parker
Leah Bartley
Leah Buhrmann
Leigh Ridings
Leila Belkhir
Leila Kahwati
Leopoldo J. Cabassa
Leti van Bodegom
Lillian Madrigal
Linda Beeber
Linda Salgin
Lindsey Filiatreau
Lindsey Knowles
Lindsey Zimmerman
Linelle Blais
Linos Vandekerckhove
Lisa Brenner
Lisa de las Fuentes
Lisa DiMartino
Lisa Dixon
Lisa Gary
Lisa Hirschhorn
Lisa Jamu
Lisa Saldana
Lisa Shulman
Lisa Spees
Liselotte Schäfer Elinder
Lisette Schoonhoven
Liz Greenfield
Liza Tomczuk
Lizeth Alonso
Lori Bilello
Louis Brown
Louise Laurent
Luana Marques
Lucie Cluver
Luke Wolfenden
Lydia Kwak
Lyndal Trevena
Lynne Carter
M. Kit Delgado
Maarit Karhula
Maddison North
Madeline Fagen
Madisen Brown
Maggie Czarnogorski
Maji Hailemariam
Malene Brekke
Maoliosa Donald
Marais Pletsch
Marc Atkins
Marc Germain
Marco Reid
Margaret Crane
Margaret Schneider
Margaret Sullivan
Margo Fudge
Maria Fernandez
Maria Granger
Maria Gutierrez
Maria Monroe-DeVita
Maria Stage
Marian Betz
Mariarosa Gasbarro
Marie Scheidmeir
Marilyn Augustyn
Marisa Sklar
Marium Sultan
Mark Campbell
Mark Ehrhart
Mark Fleming
Mark Greaves
Mark Hawes
Marlena Shin
Marlene Schwartz
Marta Llarena
Marta Montejo
Marta Roczniewska
Martin Lee
Mary Beth Weber
Mary Eluned Gaffield
Mary Lisa Penilla
Mary Mallory
Mary Ramos
Mary T. Mosenke
Mary Wangen
Maryam Gholami
Marylou Gilbert
Mathilde Endsjø
Matthew McCoy
Matthew Nielsen
Matthew Ponticiello
Maura Kepper
Maximilian Labrecque
Mayuko Ito Fukunaga
Meagan Brown
Mechelle Sanders
Meg Small
Megan Blanton
Megan Gerber
Megan Mccullough
Megan Stanton
Megan Wainwright
Meghan Lane-Fall
Meike van Scherpenseel
Melanie Pellecchia
Melhaney Reichelt
Melina Melgarejo
Melissa Dichter
Melissa Farmer
Melissa Mugambi
Mellicent O'Brien Blythe
Meredith Crews
Michael A. de Arellano
Michael Kidd
Michael Knudtson
Michael McDonell
Michael Parchman
Michael Pullmann
Michael Southam-Gerow
Michael Stoolmiller
Michael Tanana
Michaela Harris
Michel Wensing
Michele Staton
Michelle Kegler
Michelle Keller
Michelle Upham
Mike Dennis
Mikel Rueda-Etxebarria
Mildred Johnson
Milkie Vu
Mirte Kuipers
Mojdeh Motamedi
Molly Black
Molly Davis
Mona Ritchie
Monica Matthieu
Monica Perez-Jolles
Musu Jambai
Mylien Duong
Myra Piat
Nabeel Qureshi
Nadia Safaeinili
Nadia Sam-Agudu
Nadina Peters
Nalani Thomas
Nallely Vega
Namratha Kandula
Nancy Chang
Nancy Gonzales
Nancy Jacquelyn Pérez-Flores
Nandana Parakh
Nanette Benbow
Naomi Goldstein
Natalia Paniagua
Natalie Brane
Natalie Dallard
Natalie Finn
Nataly Barragan
Nate Williams
Nathalie Moise
Navneet Kaur
Nazanin Bahraini
Neena Smith-Bankhead
Nicholas Jackson
Nichole Goodsmith
Nickola Pallin
Nicola Lancki
Nicole Errett
Nicole Gumport
Nicole Hendrix
Nicole Litvitskiy
Nicole Stadnick
Njoroge Mwathi
Noah Triplett
Noelle Le Tourneau
Nolan Hill
Nomsa Monare
Nora Braathu
Norman Shields
Obidiugwu Duru
Oladunni Oluwoye
Ole Sørensen
Olivia Brauer
Olivia Bravoco
Olivia Fang
Olufeyisayo Odebunmi
Omonyele Adjognon
Owen Cooper
Palmira dos Santos
Pamela Kohler
Parth Shah
Patrice Enders
Patricia Chamberlain
Patricia Elliott
Patricia Schetter
Patricia Van Der Laag
Patrick Kierkegaard
Paul MacPherson
Paul Watson
Paula Zeanah
Per Nilsen
Pernilla Liedgren
Peter Mogere
Peter Taber
Philip Kendall
Piotr Romaniuk
Piper Block
Predrag Klasnja
Puja Chebrolu
R. Christopher Sheldrick
Rachael Kenney
Rachel Amgott
Rachel C Shelton
Rachel Gold
Rachel Haine-Shlagel
Rachel Kim
Rachel Liebman
Rachel Sturke
Radhika Sundararajan
Raechel Soicher
Rafaella Sale
Ramzi Salloum
Ran Xu
Rana Charide
Randi Hovden Borge
Rashed AlRasheed
Reanne Rahim
Rebecca Esliker
Rebecca Gitlin
Rebecca Landa
Rebecca Lengnick-Hall
Rebecca Parrish
Rebecca Phillips
Rebecca Roppolo
Regan W. Stewart
Regine Haardörfer
Rekha Trehan
Rena Kreimer
Renee Boothroyd
Renee Ferrari
Reza Yousefi Nooraie
Rianne Van der Kleij - Van der Sluis
Richard Armenta
Richard MacDonagh
Riitta Seppänen-Järvelä
Rinad Beidas
Rita Dykstra
Rixt Zuidema
Rob DeRubeis
Robert Franks
Robert Gallop
Robert Gross
Robert Hurley
Robert Klocko
Robert Lew
Robert Tukey
Robin Jenkins
Rochelle Hanson
Rohit Ramaswamy
Roisín Slevin
Rolando Barajas
Rosa Hernandez-Ramos
Rosaura Orengo Aguayo
Roseline Doe
Rosemary Adaji
Rosemary Meza
Rosemary Reyes
Ross Brownson
Ross Jones
Roy Njiru
Ruiyan Hu
Russell Glasgow
Ryan Colvin
Ryan Singh
Ryan Thompson
Ryan Walsh
Saira Bashir
Salim Sohani
Saloni Dev
Sam Biver
Sam Dudley
Samantha Chakraborty
Samantha Connolly
Samantha Hernandez
Samantha Soetenga
Samia Dakhia
Samina Iqbal
Samira Ali
Samreen Fathima
Samuel Packard
Sandra Chafouleas
Sandra Diehl
Sapana Patel
Sapna Mendon-Plasek
Sara Abelson
Sara Becker
Sara Edmond
Sara Fernandez-Marcote
Sara Parent
Sarah A Birken
Sarah Alkhatib
Sarah Chilenski
Sarah Daniels
Sarah Fielman
Sarah Forberger
Sarah Foster
Sarah Javier
Sarah Kaye
Sarah Masyuko
Sarah Parmenter
Sarah Shafir
Sarah Walker
Sari Saukkonen
Sarita Sonalkar
Saskia te Velde
Savita Bkhshi Anand
Scott Roesch
Scott Rothenberger
Seo Youn Choi
Shane Orvis
Shannon Chaplo
Shannon Dorsey
Shannon Pagdon
Shannon Sweeney
Shannon Wiltsey Stirman
Shari Rogal
Sharon Kiche
Sharon Reid
Sharon Sprinkle
Shaun Treweek
Shawna Smith
Shay Cannedy
Shayna Cave
Sheena McHugh
Sheila O'Brien
Sheldon Stokes
Shelia Broyles
Shelley Crawford
Sheree Schwartz
Sherra Lawrence
Simone Schriger
Siqin Ye
Sirina Tiwari
Sita Lujintanon
Sobia Khan
Sohayla Elhusseini
Solveig Holen
Sonja Schoenwald
Soo Park
Soo Youn
Soohyun Hwang
Srinath Adusumalli
Stefan Baral
Stephanie Clone
Stephanie Doupnik
Stephanie Lauden
Stephanie Sila
Stephanie Taylor
Stephanie Wheeler
Stephen M. Schueller
Stephen Phillippi
Steven Marcus
Steven Zeliadt
Stine Ekornes
Styn Jamu
Susan Frayne
Susan Schmidt
Susana Rivera
Susanna Didrickson
Susanne Tafvelin
Svea Closser
Sydney Keller
Sydney Seaton
Syed Aajmain
Sylvia Lin
Taffeta Wood
Tail Al-Bakri
Tamra Loeb
Tamra Williams
Tannaz Moin
Tanya Olmos-Ochoa
Tara Karns-Wright
Taren Swindle
Taryn Perez
Tasoula Masina
Tatiana Baquero-Devis
Tatiana Bustos
Tatiana Davidson
Taylor Loskot
Taylor Randall
Tendai Mutembedza
Teraisa Mullaney
Teresa Lind
Terrie Butler-Foster
Tessa Concepcion
Tessa Matson
Theresa S. Betancourt
Thomas Byrne
Thomas Engell
Thomas Kubiak
Thomas Olino
Tiffany Lagare
Timothy Walker
Todd Wagner
Tommi Gaines
Tonya Hazelton
Torrey Creed
Tracy Clemans
Ulrica von Thiele Schwarz
Ursula Whiteside
Vaishali Kulkarni
Valentina Gestaldo
Vanesa Mora Ringle
Vanessa Najjar
Vanessa Trenton
Vaughan Collins
Vera Costa
Vera Yakovchenko
Victoria Scott
Vikram Patel
Virginia Mckay
Whitney Irie
Wieneke Mokkink
Will Aldridge
William Becker
William Bridel
William Martinez
William Osbourne-Sorrell
William Oscar Fleming
William Oswald
Wouter Keijser
Yanchen Zhang
Yazdan Yazdanpanah
Yesenia Mejia
Yue Yu
Zara Jillani
Zenith Rai
Zoe Alley
Zoe Miller
Zoë Wangstrom
