Abstract
Background
Few “intervention agnostic” strategies have been developed that can be applied to the broad array of evidence-based practices (EBPs) in schools. This paper describes two studies that reflect the initial iterative redesign phases of an effective leadership-focused implementation strategy—Leadership and Organizational Change for Implementation (LOCI)—to ensure its acceptability, feasibility, contextual appropriateness, and usability when used in elementary schools. Our redesigned strategy—Helping Educational Leaders Mobilize Evidence (HELM)—is designed to improve principals’ use of strategic implementation leadership to support the adoption and high-fidelity delivery of a universal EBP to improve student outcomes.
Method
In Study 1, focus groups were conducted (n = 6) with 54 district administrators, principals, and teachers. Stakeholders provided input on the appropriateness of original LOCI components to maximize relevance and utility in schools. Transcripts were coded using conventional content analysis. Key themes referencing low appropriateness were summarized to inform LOCI adaptations. We then held a National Expert Summit (Study 2) with 15 research and practice experts. Participants provided feedback via a nominal group process (NGP; n = 6 groups) and hackathon (n = 4 groups). The research team rated each NGP suggestion for how actionable, impactful/effective, and feasible it was. We also coded hackathon notes for novel ideas or alignment with LOCI components.
Results
Study 1 suggestions included modifications to LOCI content and delivery. Study 2's NGP results revealed most recommendations to be actionable, impactful/effective, and feasible. Hackathon results surfaced two novel ideas (distributed leadership teams and leaders’ knowledge to support educators EBP use) and several areas of alignment with LOCI components.
Conclusion
Use of these iterative methods informed the redesign of LOCI and the development of HELM. Because it was collaboratively constructed, HELM has the potential to be an effective implementation strategy to support the use of universal EBP in schools.
Plain Language Summary
Our research team designed a strategy (HELM) for school principals to improve the support they provide to staff to implement practices proven to work in research for improving student outcomes. We designed HELM by conducting focus groups with school district administrators, principals, and teachers. Participants were asked for their feedback on how to adapt an existing leadership strategy (LOCI) to the school context. After collecting this feedback, we held a meeting with 15 research and practice experts. During this meeting, the group of experts reviewed the focus group feedback and decided how to incorporate it into the design of the HELM strategy. We believe that collecting this feedback and involving research and practice experts in interpreting and integrating participant feedback into the HELM strategy will make HELM a more effective strategy for supporting school principals’ in implementing supports in their schools.
Introduction
A growing body of research suggests that organizational factors are particularly critical in the successful implementation of evidence-based practices (EBPs), defined as practices and programs shown by high-quality research to have meaningful effects on student outcomes (Aarons et al., 2012; Beidas et al., 2013, 2014, 2015; Bonham et al., 2014; Cook & Odom, 2013; Locke et al., 2019; Williams et al., 2019). Implementation leadership (i.e., specific leader behaviors that support EBP use; Aarons, Ehrhart, et al., 2014) and implementation climate (i.e., shared perceptions among implementers of the extent to which EBP use is expected, supported, and rewarded within the organization; Ehrhart et al., 2014) are malleable organizational factors that predict EBP use (Aarons, Ehrhart, et al., 2014, 2015, 2017; Beidas & Kendall, 2010; Brookman-Frazee & Stahmer, 2018; Damschroder et al., 2009; Locke et al., 2019; Williams et al., 2018, 2020). Implementation leadership behaviors are theorized to affect implementation outcomes by creating a positive implementation climate that subsequently affects teachers’ implementation behaviors (Aarons, Farahnak, et al., 2014; Williams et al., 2020), including intervention fidelity, the degree to which an EBP is implemented as designed (Proctor et al., 2011). Considering the resources invested in school organizational change (Darling-Hammond et al., 2009), new implementation strategies, defined as methods used to enhance the adoption, implementation, and sustainment of an EBP (Powell et al., 2015; Proctor et al., 2013), need to target school leaders, such as principals, who have a large role in decision-making and are most proximal to implementation efforts.
One organizational implementation strategy, Leadership and Organizational Change for Implementation (LOCI), targets general and implementation-specific leadership to improve implementation climate (Aarons et al., 2015, 2017). LOCI is a data-driven intervention with eight core components: (1) assessment and feedback (360° surveys on leadership and climate are administered to the leader and their staff, with feedback to the leader); (2) leadership development plan (leader's goals for improving their and others’ leadership and the implementation climate of their organization); (3) initial training (a two-day didactic and interactive training that covers effective implementation leadership); (4) individual coaching (weekly coaching calls to review progress and update the development plan); (5) group coaching (monthly calls with trainees to review progress and share strategies); (6) organizational strategy development (monthly meetings with first-level and upper-level leaders as well as other employees to develop an organizational implementation strategy and work on alignment between levels); (7) follow-up training (two one-day sessions to review content, feedback, and update leadership development plans); and (8) graduation (review leaders’ final feedback and celebrate progress). LOCI has been successfully tested in mental health and substance use settings, resulting in leaders reporting higher perceived utility for managing organizational change, higher utility for implementing EBPs, and clinicians reporting improved implementation leadership and climate for targeted EBPs (Aarons et al., 2015, 2017; Skar et al., 2022). LOCI also has recently been used in mental health and school programs to support therapists and teachers to deliver EBPs for autistic youth (Brookman-Frazee & Stahmer, 2018); however, LOCI was not systematically adapted or iteratively redesigned for use in schools, which is a unique implementation context. The perpetual shifts in staff, students, operating policies and procedures, and emergent crises that plague public schools suggest the need for LOCI redesign for it to support high-fidelity EBP use to achieve positive outcomes for students.
Despite considerable research documenting the criticality of building-level leadership—and principal leadership in particular—to implementation efforts designed to improve student social, emotional, behavioral, and academic outcomes (Hallinger & Heck, 1996; McIntosh et al., 2016; Pinkelman et al., 2015), little research has developed and evaluated strategies to support leaders’ implementation of effective programs and practices in schools. Although implementation leadership has high relevance to universal social, emotional, and behavioral programming in schools, redesign of LOCI is needed to attend to the specific organizational structures, priorities, and personnel present in schools considering LOCI was developed and tested outside of educational settings. Factors that affect the implementation process (staff turnover, emergent crises in schools, policies, etc.) differ in schools in comparison to community agencies (Brookman-Frazee et al., 2020). While LOCI has recently been used in mental health and school programs to support therapists and teachers to deliver EBPs for autistic youth (Brookman-Frazee & Stahmer, 2018), it was not systematically adapted to specifically address implementation determinants most proximal to EBP implementation in educational settings. The activities in the current study (e.g., usability testing) are driven by our theory of change (see Figure 1), which details how the redesigned Helping Educational Leaders Mobilize Evidence (HELM) implementation strategy will impact proximal mechanisms (e.g., leader behaviors) and distal outcomes (e.g., student behavior).
HELM Theory of Change
We used the Discover, Design, Build, and Test Framework (DDBT) to guide the iterative redesign of the LOCI implementation strategy. The DDBT framework combines user-centered design and implementation science to enhance the usability, contextual fit, and uptake of complex interventions, including implementation strategies (Lyon et al., 2019). The Discover Phase engages stakeholders to determine: (1) context requirements, so the redesign of the implementation strategy is useful and usable in elementary schools; and (2) required changes to ensure usability and contextual appropriateness. The Design/Build Phase uses the information gleaned from the Discover phase to iteratively build, evaluate, and refine a redesigned implementation strategy, and the Test Phase involves feasibility testing with a larger number of end-users and in their actual milieu.
To maximize the relevance and utility of HELM in schools, two studies were designed to gather expert input on the appropriateness (i.e., suitability/fit) of the original LOCI components to inform systematic adaptation. In both studies, the original components of LOCI (described above) were presented to stakeholders to elicit their feedback. In Study 1, focus groups were conducted with principals, district administrators, and teachers. Inclusion of these three stakeholder groups ensured input from the perspectives of individuals who (a) engage in strategic leadership (principals), (b) oversee strategic leadership (district administrators), and (c) are the targets of strategic leadership (teachers). Study 1 sought to answer the following research question:
What modifications are necessary to render LOCI components appropriate for use with elementary school principals and teachers?
What modifications are necessary to ensure that HELM's theory of change is clear, comprehensive, and accurate?
What additional refinements or additions to LOCI intervention components are needed to enhance impact on its proximal (e.g., implementation climate and citizenship) and distal (e.g., fidelity, penetration) mechanisms of change?
Informed by the results of the first study (e.g., key themes referencing low appropriateness), Study 2 involved an “Expert Summit” in which research and practice experts took part in a structured information-generation and prioritization process (the Nominal Group Process [NGP]) and hackathon (both methods described below) to gather input about LOCI and the proposed adaptations, evaluate the assumptions of the theory of change (e.g., the links between implementation leadership and climate), and finalize adaptations to enhance their effectiveness, while simultaneously ensuring appropriateness and feasibility in elementary school settings. Summit participants represented a breadth of experience in: (1) principal leadership; (2) organizational leadership training and professional development; (3) implementation science; (4) evidence-based social, emotional, and behavioral practices; and (5) constraints and opportunities in educational settings. Study 2 sought to answer the following research questions:
Study 1
Method
Participants
Fifty-nine district administrators, principals, and teachers were recruited from two school districts in one Pacific Northwestern and one Midwestern state in the USA (see Table 1). Fifty-four (91.5%) participants provided consent and participated in the focus groups. Of the five participants who were recruited and did not participate, three district administrators were unavailable during the scheduled focus groups and one district administrator and one principal did not provide consent. The final sample included 14 district administrators, 21 principals, and 19 teachers (see Table 2).
Study 1 Sample Counts by State, Role, and Recruitment Status
Study 1 Participant Demographics
Procedures
Following the Institutional Review Board and Protection of Human Subjects approval, we worked with our district partners to identify potential participants, met with them to describe the study, and obtained informed consent. As part of the DDBT Discover Phase, we scheduled a convenient time to conduct separate virtual focus groups for each set of participants to identify redesign targets to be an efficient and contextually appropriate version for schools. Two weeks prior, participants received written materials describing LOCI and its components to review before attending the focus groups. Focus groups were facilitated by the first (female), eighth, and last authors (both male) who were academic researchers at the time of the study and all of whom were formally trained in qualitative methods, including focus group facilitation (same for Study 2). Facilitators presented a review of the LOCI components and led a discussion of the questions detailed in the semi-structured interview protocol (see below). Consistent with recommendations from the literature, focus groups lasted approximately 90-min (Morgan, 1997; Vaughn et al., 1996) and included 7–12 participants each. The groups were audio recorded for transcription. Participants were offered $50 as an incentive for their time.
Two systematic and comprehensive semi-structured focus group protocols were developed: one for district administrators/principals and another for teachers. Each protocol emphasized: (1) the relevance and utility of the original LOCI intervention and its overarching objectives in schools; (2) the feasibility and appropriateness of each LOCI component; (3) strategies or modifications for ensuring the appropriateness, impact, and scalability of the HELM adaptation in schools (e.g., identifying formal and informal school implementation leaders); and (4) alignment and misalignment between the LOCI participation expectations and standard professional development processes for principals and teachers. District administrators and principals provided feedback on all original LOCI components; teachers provided feedback on components most relevant to their role (360° assessment, leadership development plan, and distributed leadership teams).
Data Analysis
Because the purpose of Study 1 was to explore the full range of possible adaptations to LOCI and to improve the feasibility and contextual appropriateness of HELM, a conventional content analysis approach (Hsieh & Shannon, 2005) was used in which meaning is derived from the content of verbal communications, but no a priori codes were identified prior to reviewing transcripts. Focus group recordings were transcribed, anonymized, and saved as electronic documents for the coding team to review. The first author, who has extensive training in and experience leading qualitative research, led the development of the coding scheme using a rigorous, systematic, transparent, and iterative approach. A codebook was iteratively developed via close reading of the initial transcripts (i.e., inductive approach; Bradley et al., 2008), code generation, and group meetings. The codebook included operational definitions of each code, examples of the code from the data, and guidance on when to use and not use the code (see Tables 3 and 4). After a stable set of codes was reached, four different raters coded each transcript independently and resolved disagreements through consensus dialogue (DeSantis & Ugarriza, 2000; Hill et al., 2005). No member checking was done. Inter-rater reliability was calculated using percent agreement (98% across all transcripts).
Study 1 District Administrator and Principal Codebook
Study 1 Teacher Codebook
Results
HELM Content Modifications
Focus group participants across all roles focused primarily on content modification related to the original LOCI 360° assessment and coaching. Participants strongly encouraged the research team to clearly communicate the “why” of the 360° assessment to ensure respondents provided an accurate representation of the school. One district administrator shared, I think … it's really important for the people who are completing the survey to know … what is the purpose of the goal for this survey, how will it be used, because the responses that people give will depend on if they really understand the goal or the intended outcome. I think, too, maybe just starting with the why with your staff, too, and I think as a staff member, when you feel really supported, you want to reciprocate that. If our principal [is] looking for feedback, I would definitely want to put in the time and thoughtfulness to respond to that. But if you start with why we’re doing this and what we’re hoping to gain … that's probably helpful for staff. I would not want this to get confused with principal evaluation … because I think most of us, when we think of 360° assessments, it's very much in an evaluative framework. Is it going to affect my compensation? Is it going to affect my career?
Principals also discussed ways to increase the effectiveness of HELM coaching. School leaders noted the importance of HELM coaches having experience as a principal to ensure they understand principals’ roles, responsibilities, and day-to-day activities, which would build buy-in and trust. One principal explained, At least for me, [it] is someone [who has] done the job. I approach coaching differently when I know they have walked in my shoes … because I approach feedback from someone who's done the job way differently than someone who hasn’t.
Finally, principals and district administrators suggested aligning HELM content with existing systems, structures, practices, and policies. One principal noted that alignment with existing evaluation systems would be, “…really nice because if you can make it as much about what you’re already doing and tying it to things that you’re trying to implement so that you can move forward with whatever feedback you receive would be helpful.” A district administrator shared a similar sentiment, … if this was built into a principal meeting or a structure we already have versus … in isolation, it makes sense and I understand it, and when I think of this in the context of … all of the moving parts, then it feels like it's overwhelming or it's big. And so I would think we could layer it into principal professional development or say improvement planning, as an element of that. There's existing structures that this could be woven into versus an add on.
HELM Delivery Modifications
Several actionable suggestions emerged related to HELM delivery. The most prominent theme was to continuously emphasize the “why” behind HELM. Principals felt the “why” would paint the “big picture” and make clear “… how does this relate to my current state of being an administrator.” Another principal pointed out that re-orienting school leader teams to the “why” could help them know, “…it's going to be additional work, but can it be work that helps me now …” Taking a broader view, one district administrator shared, “I guess I’m trying to emphasize that it really needs to be systemic, that people understand the why behind the importance of this and what better outcomes we can get if we do implement things in this fashion.”
Principals also proposed several suggestions to increase the flexibility of coaching to meet the ebbs and flows of the principalship across time. Several principals suggested flexibility in the frequency of coaching sessions, with one explaining, “So, the frequency changes if you’re going through an implementation cycle, there might be … at one point in the year you want it to be weekly or monthly and then later on, [coaching] might be more sporad[ic].” Flexibility in the duration of coaching sessions also was discussed. One principal reflected, …weekly 15-minute coaching … That seems a lot when I just think about the year and how it flows, like I’ve done coaching before where we’ve done it every other week. And at times it feels really helpful around certain projects and at other times it's kind of like we don’t have much to check in about.
Discussion of Modifications Informed by Study 1
The research team made the first-round of iterative changes to LOCI to develop HELM for use in schools based on the recommendations from the focus groups. Specifically, the purpose and non-evaluative nature of the 360° assessment was communicated to all participants in various formats and settings (e.g., recruitment materials, participant disclosure form, HELM training). HELM coaching also was modified to ensure that (a) HELM coaches had experience as a principal, (b) one of the first two coaching sessions happened in-person, and (c) the first two coaching sessions were partially focused on building strong relationships with principals. In terms of delivery, each HELM component and learning target was cross-walked with the principal evaluation framework of the National Educational Leadership Standards (NELS), the professional standards that define the nature and quality of the principal profession that guide how principals are prepared, hired, developed, supervised, and evaluated in their profession in the United States (US). The NELS was used in the development of HELM to ensure alignment between the duties and responsibilities of the principalship in US public schools.
Additionally, HELM recruitment, training, and coaching materials were all modified to underscore the “why” behind HELM as an implementation strategy. Altogether, Study 1 allowed HELM to modify integral components (i.e., 360° assessments, coaching) to ensure its feasibility, flexibility, and usability with principals and distributed leadership teams in elementary schools.
Study 2
Methods
Participants
Eighteen implementation research and practice experts, who were known contacts of study team members or snowball sampled from those contacts, were invited via email to participate in Study 2. Of these, 15 (83%) provided consent and participated in the Expert Summit. Two participants did not respond to recruitment emails, and one was unavailable during the scheduled time. Participants identified themselves as experts in implementation research (n = 5), practice (n = 8), and research and practice (n = 2). Five attendees had previous experience with LOCI as a coach or trainer. See Table 5 for sample demographic information.
Study 2 Participant Demographics
Procedures
We convened a one-day Expert Summit over Zoom. The summit provided an opportunity for focused expert feedback and discussion about the overall project and proposed adaptations to LOCI to maximize its effectiveness. After reviewing the overall purpose and objectives of the project (documents were sent to participants three weeks prior to the summit; see pre-summit procedures below), we presented the following topics: (1) implementation leadership and climate; (2) EBPs in the school setting; and (3) principal leadership to support social, emotional, and behavioral-focused EBP implementation. Two breakout sessions with three groups in each were held to gather specific qualitative input from participants regarding: (1) HELM's theory of change; and (2) adaptations to enhance LOCI content/activities for each of the proposed mechanisms of change (e.g., implementation leadership and climate), including strategies through which principals can activate and leverage distributed leadership structures; and (3) recommendations to maximize the impact, feasibility, and contextual fit of LOCI within standard professional development structures/sequences (e.g., length, number of consultations, combination of general and implementation-specific leadership content) in authentic elementary settings. An additional breakout session with four groups was used to generate novel ideas that might be integrated into the HELM strategy. Participants with varying expertise were evenly distributed in each group across the breakout sessions. Summit participants received $500.
The hackathon methodology was created to support the rapid development of information technology advances, though it has been widely adopted to promote innovation across various fields (Rys, 2023). Standard hackathons are time-limited (e.g., 24 h), focused on a specific challenge or theme, and result in a demonstration of the proposed innovation (Komssi et al., 2015). The third breakout session used a modified hackathon such that four groups with 4–5 participants in each were prompted to develop an innovation that focused on elementary school leaders, supported school leaders to create the conditions for successful implementation of any EBP that enhances student outcomes, and was feasible for use in authentic educational contexts. All groups were given a one-page document that described a hackathon, success criteria (i.e., impact, innovation, pragmatic), and what needed to be accomplished. Each group had 45-min to brainstorm and develop a pitch for their proposed innovation, which was then shared with the larger group.
NGP 1 Codes
NGP 2 Codes
Hackathon Codes
Results
NGP 1: HELM Theory of Change
Participants of the first NGP were asked to provide feedback on HELM's theory of change. Feedback centered on two types of modifications to HELM content and/or delivery that were not surfaced in study 1. First, participants suggested that HELM content better reflect how distributed leadership teams could be organized and leveraged to support EBP implementation. There was a strong emphasis on learning more about implementation specific leadership behaviors. One participant prompted the research team to, “Take into account how formal and informal leaders can impact the process. Everyone needs to be on the same page.” This feedback aligned with another participant who wondered, “How do we capture the whole team (each member has a different role – so how do you integrate that and show it works together/not siloed)?” Second, several participants felt that aligning personnel across different levels of influence (e.g., district, school) would result in a unified team approach that would maximize the effectiveness of HELM components. One participant called for, “more alignment processes to create a team throughout this process. Recognize and break down the levels as much as you can. Intentional alignment process to allow school community to function as a team.” Similarly, another participant suggested that HELM, “foster authentic collab[oration] between administrators and teachers, first level leader [and] upper management.”
NGP 2: HELM Components
Participants shared several pieces of feedback to improve the feasibility and contextual appropriateness of the following HELM components.
Hackathon
Most of the brainstorming and ideas generated by hackathon groups aligned with existing LOCI components. For example, most groups (n = 3) invoked diverse learning approaches and materials. One group prioritized, “…using a coherent framework and human-centered design principles through which the learning framework is designed that allows people to acquire specific competencies linked to … improving … leadership.” Another group focused on, “Self-assessment: where are [leaders] in their learning journey to acquire competencies consistent with the coherent framework?” Relatedly, one group's innovation to “Create a database for implementation data akin to student information systems that are ubiquitous. What is happening? Where is it happening? Who is it happening to/with?” was aligned with LOCI's extensive collection and use of implementation-related data.
Hackathon results also surfaced two innovative ideas that led to HELM modifications. First, one group reflected on distributed leadership teams, pondering, “How can I intentionally build a distributed leadership model so that [leader] turnover is less disruptive to the school as a whole?” Prior feedback led to suggestions about modifying HELM content to include distributed leadership teams, not about how to develop one. The second piece of feedback emphasized the knowledge leaders need to support educators to effectively implement EBPs. Specifically, one group wanted to, “Ensur[e] that leaders have the on the ground knowledge for what it takes in order to support those that are implementing the EBPs.”
Discussion of Modifications Informed by Study 2
We made a second round of iterative changes to LOCI to further develop HELM for use in schools based on the feedback from Study 2. First, it was necessary to streamline the HELM training to fit within the school calendar year and constraints of competing responsibilities (e.g., availability of teachers and administrators to attend training). The Theory of Change discussion suggested implementation specific leadership behaviors with an emphasis on distributed leadership was critical and more proximal to our target mechanisms. Therefore, we removed the transformational and transactional leadership model as part of the training. Second, a worksheet was developed to help school leaders determine how to develop or improve their existing distributed leadership team. Further, the LDP was adapted to log, track, and link individual team member's goals to the shared team goal. The coaching manual also was edited to reference distributed leadership teams throughout. In response to calls for multilevel alignment, the decision was made to invite district administrators to the HELM training to provide opportunities for shared learning that might promote teamwork across levels of influence. Relatedly, school leaders were required to attend the EBP training to ensure they had the knowledge necessary to support their staffs’ EBP implementation. Changes also were made to the 360° assessment such that the proposed fourth time point of data collection was eliminated, and the language in selected instruments (i.e., School Implementation Leadership Scale and School Implementation Climate Scale) was edited to “our principal” or “this school.” Finally, more flexibility was built into HELM coaching to meet school leaders’ constantly changing needs and availability. For example, a suggested dosage of coaching (i.e., time) was recommended, but scheduling of that time was determined by the coach and school leader. The culmination of Studies 1 and 2 led to a modification blueprint; differences between LOCI and HELM are highlighted in Table 9.
Differences Between LOCI and HELM
Discussion
This study applied the DDBT Framework to systematically and iteratively redesign an organizational-focused implementation strategy in partnership with community stakeholders to enhance its acceptability, appropriateness, feasibility, and usability in elementary schools. The collaborative redesign process included different methods (i.e., focus groups, NGP, and hackathon) and the culmination of Studies 1 and 2 generated a modification blueprint that ultimately informed the redesign of LOCI to develop HELM. The prototype of HELM is streamlined in comparison to LOCI to fit within the school calendar year and includes specific features for use in schools (e.g., former principals as coaches, distributed leadership teams, alignment with NELS). Because it was collaboratively constructed across iterations, HELM has the potential to be an effective organizational-level implementation strategy to support the use of universal EBP in schools.
The results of this study point to the importance of incorporating stakeholder feedback in all phases of redesign as a methodology to ensure the end product (i.e., HELM) is applicable to support EBP implementation in a different setting (i.e., schools) than originally designed. Stakeholder feedback in Study 1 resulted in setting-specific considerations that led to initial LOCI modifications (shortening and streamlining LOCI components such as assessment and coaching frequency to fit within the school timeline; enlisting former principals with experience with EBP implementation in schools as coaches). The principalship is complex and in the US, requires adherence to NELS. Without contextualization into what the principalship requires, HELM appears to be another “task” or thing to do for principals. But stakeholder feedback highlighted that one of the core components of HELM—coaching—could benefit from having former principals who deeply understand the role, responsibilities, and regulations that guide the profession as coaches. This ensures HELM activities align with what principals already do as part of the job and may enhance usability, buy-in, and feasibility. Subsequently, the NGP and hackathon yielded additional LOCI modifications (forming and activating distributed leadership, creation of user-friendly tools and worksheets, surface-level changes to the assessment battery) to enhance the contextual appropriateness and support the development of HELM. Co-creation with local stakeholders resulted in school-facing tools (i.e., LDP) that were relevant to the implementation context and usable among end-users. Ultimately, the collaborative redesign process allowed greater alignment with HELM components and school needs and priorities and should, in theory, increase adoption and implementation. These findings are consistent with recent calls for more tailorable or customizable implementation in complex settings (Munson et al., 2022) and other efforts to collaboratively redesign implementation strategies outside the education sector (Deatrick et al., 2021; Harkness et al., 2022). Given there are a number of usability challenges for implementation strategies when applied to novel settings, it is critical to ensure redesign solutions address barriers in the implementation context (Lyon et al., 2020).
Study 2 employed an Expert Summit that comprised an NGP and a hackathon. The hackathon was a novel methodology for participants to come together and generate ideas and potential solutions to a problem (e.g., how to feasibly use HELM in schools). The use of a hackathon allowed Summit participants to engage in structured creative and innovative thinking for increased and distributed participation (Yarmohammadian et al., 2021). Researchers ought to consider employing hackathons in future redesign work to promote collaborative teaming and idea generation that can inform redesign solutions to usability challenges and/or foreground aspects of the existing intervention that align with innovations emerging from the hackathon (e.g., the implementation database suggested by one hackathon group is a technological extension of implementation leadership and climate survey data).
Limitations and Future Directions
Several limitations are noted. First, recruitment was geographically limited to two USA regions; the inclusion of other geographically diverse school districts may provide additional insights that would lead to further adaptations. Future studies should expand the recruitment catchment area to ensure HELM is universally applicable and usable. Second, special educators were not recruited. Because special educators also implement Tier 1 programs and often are siloed within schools, their feedback may be instrumental to further refining HELM. Third, data collection occurred during the COVID-19 pandemic and all data collection activities were conducted via Zoom. While we had a set of agreements during the conduct of each activity (e.g., one person speaking at a time), Zoom does not allow natural conversations to emerge and thereby limits the potential collaborations or idea generation that may have otherwise occurred.
Conclusion
An approach combining human-centered design with implementation science has strong utility to ensure implementation strategies are intentionally redesigned for use in settings different from that for which they were developed. Using different methodologies to engage with local stakeholders with lived experience and expertise about the implementation context should ensure an end-product that is usable. Future research should conduct usability testing on HELM to continue to iterate on practical and logistical considerations to ensure its usefulness in supporting school based EBP implementation, and pilot testing to better understand whether HELM activates its theorized mechanisms of change.
Footnotes
Declaration of Conflicting Interests
Dr. Lyon is an Associate Editor for Implementation Research and Practice. As such, he had no part in the peer review process. All other authors declare that there is no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Ethics Approval
All procedures were approved by the University of Washington IRB (Study No. STUDY00010282).
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Institute of Education Sciences (R305A200023) and National Institute of Mental Health (P50MH115837). The content is solely the responsibility of the authors and does not necessarily represent the view of the funder.
Informed Consent
This study was determined to be IRB exempt.
