Community-based implementation strategy use and satisfaction: A mixed-methods approach to using the ERIC compilation for organizations serving children on the autism spectrum

Abstract

Background

Despite efforts to use standardized taxonomy and research reporting, documenting implementation strategies utilized in community settings remains challenging. This case study demonstrates a practical approach to gather use of and satisfaction with implementation strategies utilized within community-based sites to understand community providers’ perspectives of implementing an early intensive behavioral intervention (EIBI) for children on the autism spectrum across different settings.

Methods

Using a sequential explanatory mixed-methods design, survey and interview data were collected from directors/supervisors and direct providers (n = 26) across three sites (one university and two community-based replication sites). The Implementation Strategies and Satisfaction Survey (ISSS) was administered to identify staff-reported implementation strategy use and satisfaction. Informed by quantitative results, follow-up semi-structured interviews were conducted with a subsample (n = 13) to further understand staff experiences with endorsed implementation strategies and elicit recommendations for future efforts.

Results

Survey results were used to demonstrate frequencies of implementation strategies endorsed by site and role. Overall, staff felt satisfied with implementation strategies used within their agencies. Content analysis of qualitative data revealed three salient themes related to implementation strategy use—context, communication, and successes and challenges—providing in-depth detail on how strategies were utilized, and strategy effectiveness based on community providers’ experiences. Recommendations were also elicited to improve strategy use within “broader” community settings.

Conclusions

The project demonstrated a practical approach to identifying and evaluating implementation strategies used within sites delivering autism services. Reporting implementation strategies using the ISSS can provide insight into community providers’ perspectives and satisfaction with agency implementation strategy use that can generate more relevant and responsive strategies to address barriers in community settings.

Plain language abstract

Examining community providers’ preferences and experiences with implementation strategies used to facilitate evidence-based practice uptake can broaden our understanding of what, how, and why implementation strategies work in community-based settings ( Chaudoir et al., 2013; Leeman et al., 2017; Proctor et al., 2013). Such efforts have great potential to tailor implementation strategies to address barriers/facilitators typically found in community-based settings.

This case study demonstrates a practical approach using mixed methodology to: (a) gather self-reported use of and satisfaction with implementation strategies to understand community providers’ perspectives of implementation strategy success. Using a new survey, the Implementation Strategies and Satisfaction Survey (ISSS) conjoined with interviews, the study demonstrated a practical approach using standardized language to report strategies used in one university-based site and two community-based replication sites that deliver an early intensive behavioral intervention (EIBI) for children on the autism spectrum. This paper contributes to one of the five priorities to enhance public health impact—improve tracking and reporting of implementation strategies utilized when translating research into practice ( Dingfelder & Mandell, 2011; Powell et al., 2019; Stahmer et al., 2019). This approach emphasizes the importance of understanding context (e.g., community organizations providing services to children on the autism spectrum) to develop strategies that work better for EIBI implementation and scale-up. Understanding community provider's preferences and experiences with implementation strategies can support use of implementation strategies that better fit usual care contexts, with the ultimate goal of improving implementation practice in community-based settings.

Implementation science (IS) serves an important role in facilitating the successful uptake of evidence-based practices (EBPs)—programs, practices, and policies proven to be efficacious, effective and/or informed by research—across settings (APA, 2006). Critical to EBP uptake, implementation strategies refer to the actions (or “systematic processes”) utilized to facilitate the implementation and sustainment of practices in usual care settings, thereby promoting healthcare practice changes for broader impact (Bauer et al., 2015; Brownson et al., 2017; Proctor et al., 2013). These strategies influence the success of EBP use and have significant implications for the effectiveness and quality of implementation- and individual-level outcomes. However, ongoing challenges in implementation research and practice limit our understanding of what, how, and why implementation strategies work.

Two challenges include a lack of clarity in conceptualizing and reporting on implementation strategies. Many studies have either not labelled strategies with standardized language or poorly described implementation strategies utilized in their study (Huynh et al., 2018; McKibbon et al., 2010; Perry et al., 2019; Powell et al., 2012, 2015). These inconsistencies have led to poor understanding of the causal pathways of implementation strategies on EBP implementation within a given setting (Lewis et al., 2020; Powell et al., 2015). Current research recommends the use of a standardized language around implementation strategies to develop a more comprehensive understanding of its impact and role on EBP outcomes (Moullin et al., 2020; Powell et al., 2015, 2019).

Efforts to provide conceptual clarity of implementation strategies with definitions have been made by a panel of IS experts through a modified Delphi process (Powell et al., 2015). The Expert Recommendations for Implementing Change (ERIC) study resulted in a compilation of 73 implementation strategies, refining an earlier compilation of categorized discrete strategies (e.g., strategies collapsed into general categories; Powell et al., 2012). Other efforts have developed reporting standards that provide guidelines for researchers and practitioners to detail implementation strategies utilized in studies and further classify strategies into actions and targets (Leeman et al., 2017; Proctor et al., 2013).

To enhance the potential impact of implementation strategies collectively, implementation researchers and practitioners are called to focus on five priorities including “improving tracking and reporting of implementation strategies” (Powell et al., 2019, p. 2). While standards and guidelines for implementation strategies are useful, tracking implementation strategies using standards remains difficult, particularly in community settings (Bunger et al., 2017; Leeman et al., 2017; Lokker et al., 2015). More recent efforts have yielded measures and/or procedures to evaluate and track implementation strategies in community-based agencies, specifically. For example, innovative, pragmatic approaches to track implementation strategies being used in community agencies have involved surveys, process mapping, coding systems, ethnographic procedures, observational methods, and retrospective mapping with archival records (Boyd et al., 2018; Bunger et al., 2017; Fernandez et al., 2019; Huynh et al., 2018; Lewis et al., 2018; Proctor et al., 2019; Rogal et al., 2020; Walsh-Bailey et al., 2021). These practical approaches have allowed for iterative refinement of implementation strategies reportedly used within community settings; yet, quality of use and the associated effectiveness remains unclear. Further, little to no guidance has been provided on tailoring implementation strategies to address determinants typically found in community settings (Fernandez et al., 2019; Waltz et al., 2019).

Without full knowledge of what, how, and why implementation strategies work in community settings, there is a need to track and document strategies that are effective in practice and aligned with IS recommendations (Chaudoir et al., 2013; Leeman et al., 2017). In alignment with one IS priority area (Powell et al., 2019), this study demonstrates an approach for tracking and reporting implementation strategies utilized to translate autism spectrum disorder (ASD) EBPs into practice among community-based agencies (Dingfelder & Mandell, 2011; Stahmer et al., 2019). Moreover, very little is known about community stakeholders’ preferences for implementation strategies utilized for autism-related services (Proctor et al., 2013; Drahota et al., 2021). Staff characteristics may impact perceptions of EBP adoption and utilization, as well as the overall implementation process (Aarons, 2004; Frambach & Schillewaert, 2002; Rogal et al., 2019). The current study conceptualizes implementation strategies using the ERIC compilation to demonstrate one practical, retrospective mixed-methods approach to (a) report use and satisfaction with implementation strategies utilized in a coordinated network of one university-site and two community replication sites delivering an early intensive behavioral intervention (EIBI) for children on the autism spectrum and (b) explore the impact of staff characteristics on reported implementation strategy use to obtain a greater understanding of the implementation process across settings.

Case study

To illustrate this approach, the paper describes a case study focused on the implementation of the Early Learning Institute (ELI) within one university-based and two community-based replication sites. ASD is a lifelong neurodevelopmental disorder impacting around 1.8% of the U.S. population (Maenner et al., 2020). EBPs, including EIBIs, developed for this population can improve both short- and long-term outcomes for young autistic children (Estes et al., 2015; Landa, 2018). The ELI is an EIBI founded upon applied behavior analysis and designed to help young autistic children develop adaptive and functional skills at an early age by fostering social, communication, and independent living skills. Notably, ELI was developed to expand access to EIBIs for families receiving services through Medicaid (Plavnick et al., 2020). As many children receive services from community-based agencies, effectively translating EIBIs to this context can ensure service consistency and quality, which is crucial for the successful development of young autistic children (Plavnick et al., 2020). Though ELI initially began in a university-based setting in August 2015, staggered implementation was replicated at two community settings over the next 2 years (August 2016, 2017). To read more about the development of ELI, please refer to Plavnick et al. (2020).

Methodology

This case study utilized a sequential explanatory mixed methods (Ivankova et al., 2006; Palinkas et al., 2011; Pluye et al., 2018) structured in two phases (QUAN→QUAL). In phase one, information related to implementation strategy use from staff at the three sites was collected via a survey. In phase two, semi-structured interviews were conducted with a subsample of participants to capture experiences with endorsed implementation strategies and elicit suggestions for improving future implementation efforts in community settings. Both data strands served the function of expansion, where quantitative data evaluated what implementation strategies were used to provide breadth, and qualitative data further elaborated on how strategies were used to provide depth of understanding (Palinkas et al., 2011). This study was part of a larger project assessing implementation outcomes and determinants to the implementation of ELI (Bustos, et al., 2021). The current paper focuses on data related to reported implementation strategy use and community provider experiences.

Qualitative data were quantitized and merged with quantitative results to create a joint display table—a useful method to identify points of convergence and divergence in participant responses (Aarons et al., 2014). The joint display provides ever-coded (total number of transcripts with assigned codes) and frequency (total number of times codes were assigned throughout transcripts) counts, supporting the salience of the emergent themes (Landrum & Garza, 2015).

Quantitative phase

Measure

Implementation Strategies and Satisfaction Survey (ISSS)

The ISSS was developed in 2015 by the third author to obtain information about discrete and general implementation strategy use and satisfaction. Implementation strategies included in the ISSS were based on existing compilations and categorization of strategies (Powell 2012; 2015), as well as additional strategies thought to facilitate sustainment of innovations (cf. Hailemariam et al., 2019). First, participants endorsed (yes/no) whether their site used the general implementation strategies (i.e., Planning, Financial, Restructuring, Implementation Testing, Quality Management, Sustainment). If a general implementation strategy category was endorsed, participants were asked to endorse whether their site used each of the discrete strategies that are subsumed within the general category (e.g., “Revise professional roles” is a discrete implementation strategy within the “Restructuring” general implementation strategy category). Further, participants rated their level of satisfaction with each endorsed strategy (1 = Extremely Dissatisfied; 6 = Extremely Satisfied) (see Appendix I). Additionally, the ISSS was adapted in consultation with the ELI director to remove strategies categorized as “Education Strategies” (n = 16 discrete strategies; Powell et al., 2012) and include items categorized as “Implementation Testing Strategies” (n = 3; Drahota 2015). In early summer, respondents were asked to select implementation strategies used at the agency in the past academic year. Demographic information was collected on provider and organizational characteristics (Table 1).

Table 1.

Participant demographics.

Role	Directors/supervisors	BTs
N	N = 6	N = 20
Site
University (Site 1)	83%	40%
Community (Sites 2 and 3)	17%	60%
Gender
Female	75%	85%
Male	33%	15%
Race/Ethnicity
Non-Latino/White	83%	100%
African American	0%	0%
Hispanic or Latinx	0%	0%
Asian American/Pacific Islander	17%	0%
Highest degree earned
Some college	N/A	5%
Bachelor's degree	0%	65%
Master's degree	67%	30%
Doctorate	33%	0%
Educational training
Psychology	17%	70%
Education	33%	5%
Speech/Language/Communication	0%	5%
Special Education	17%	0%
Behavior Specialist	0%	25%
Other	33%	0%
ASD training in education
Yes	83%	70%
No	17%	30%
Work history
Years current agency M (SD)	1 (1.41)	.45 (.68)
Years experience w/ASD M (SD)	10 (7.82)	2.2 (2.95)
Employment status
Full-time	83%	80%
Part-time	0%	10%
As needed	17%	5%
Temporary	0%	5%

ASD, autism spectrum disorder.

Case study illustration: quantitative methods

Participants

Directors, supervisors, and providers knowledgeable about the EIBI implementation process at each site during the 2018–2019 academic year were recruited to participate (Palinkas et al., 2015). Eligibility criteria included: 18 years or older; employment at one of the sites involved; and proficiency in reading and speaking English. A total of 26 staff members participated in the quantitative phase. Across sites, the sample included 20 direct providers (behavior technicians [BTs]), three board certified behavioral analysts (BCBAs), and three site directors. Participants were later categorized as “BTs” or “Directors/Supervisors” (including BCBAs; Table 1).

Procedures

Michigan State University IRB approved all study procedures (#STUDY00000234). Research personnel recruited site directors, providing information about the study and an opportunity to ask clarifying questions. Once site directors agreed to participate, site staff were recruited through recruitment flyers and presentations. Staff email addresses were collected. Staff were emailed the online consent form to obtain written consent, screening questionnaire, and a link to the 10-min Qualtrics survey. Participants completing the survey received a $10 honorarium.

Data analysis

Implementation strategy endorsement and reported satisfaction was examined by role, then by setting (community vs. university) using descriptive and exploratory statistics. Frequencies of endorsed strategies were transformed to count variables; endorsement scores were calculated for each general and discrete category. For each general category, satisfaction scores were calculated by averaging discrete strategy satisfaction scores that comprised the general category (e.g., “Build coalition” is a discrete strategy within the general “Planning” implementation strategy). Continuous data was normalized using the fractional rank method in SPSS (Solomon & Sawilowsky, 2009). Independent samples t-tests were utilized for mean comparisons by role and setting. Correlational analysis using Spearman's Rho evaluated associations among variables—both continuous and discrete level data (Prion & Haerling, 2014).

Case study illustration: quantitative results

Implementation strategies by role

Results indicate that Directors/Supervisors and BTs most commonly reported use of quality management (100%; 65%), restructuring (100%; 60%) and sustainment (100%; 50%) general strategies (Table 2). Independent samples t-tests of count scores demonstrated significant mean differences by role for all endorsed general strategies (p's < .05), with the exception of implementation testing strategies. Thus, Directors/Supervisors endorsed higher rates of general implementation strategy use within the past year than BTs (Table 3).

Table 2.

Frequencies of endorsed general and discrete implementation strategies and average rate of perceived satisfaction by role.

General and discrete implementation strategies	Director/Supervisor	Satisfaction a M (SD)	BT	Satisfaction a M (SD)
Planning	83.3%	5.37 (.68)	25%	4.81 (.32)
Conduct a local needs assessment	16.7%	6 (.00)	40%	4.50 (.70)
Assess for readiness; Identify barriers	66.7%	5.35 (.95)	100%	4.40 (1.14)
Visit other sites	66.7%	5.25 (.95)	60%	2.67 (1.52)
Develop a formal implementation blueprint	50%	5 (1.00)	40%	4.00 (1.41)
Tailor strategies to overcome barriers	66.7%	5.25 (.95)	60%	4.33 (1.52)
Stage implementation scale-up	33.3%	5.50 (.70)	20%	4.00 (.00)
Model and simulate change	66.7%	5.50 (.57)	80%	4.50 (.57)
Conduct community consensus discussions	66.7%	4.75 (.95)	40%	5.50 (.70)
Involve executive board	16.7%	6.00 (0.00)	40%	4.00 (1.41)
Identify/prepare key team members to implement	66.7%	5.50 (.57)	60%	4.33 (1.15)
Involve patients/consumers and family	50%	5.33 (1.15)	100%	5.00 (.70)
Recruit, designate, and train for leaderships	83.3%	5 (1.00)	60%	5.00 (1.00)
Mandate change	66.7%	5.75 (.50)	60%	4.67 (.57)
Build coalition	66.7%	5.00 (1.15)	40%	5.50 (.70)
Develop resource-sharing agreement	50%	6 (.00)	60%	5.00 (.00)
Obtain formal commitments	67%	5.25 (.95)	40%	5.00 (.00)
Develop academic partnerships	83.3%	5.20 (1.09)	80%	5.00 (.81)
Other	16.7%	6 (.00)	0%	–
Financial strategies	100%	4.5 (.92)	5%	4.22 (.00)
Alter incentive/allowance structures	33%	4.5 (.70)	0%	–
Use capitated payments	16.7%	4.00	0%	–
Penalize	0%	NA	0%	–
Use other payment schemes	16.7%	4.00 (0.00)	0%	–
Reduce or increase patient/consumer fees	0%	–	0%	–
Place a fee for service lists/formularies	0%	–	0%	–
Fund and contract for the clinical innovation	16.7%	5.00 (.000)	50%	3.00 (.00)
Access new funding	50%	5.33 (.57)	50%	4.00 (.00)
Make billing easier	33%	4.50 (.70)	0%	–
Other: Staff incentives	33%	2.50 (2.12)	0%	–
Restructuring strategies	100%	5.36 (1.04)	60%	4.62(.62)
Revise professional Roles	66.7%	5.75 (.50)	50%	5.00 (.00)
Change service sites	16.7%	4.00 (.00)	33%	4.50 (1.00)
Change physical structure and equipment	100%	5.17 (1.32)	75%	4.33 (.86)
Create new clinical teams	83.3%	5.00 (1.41)	50%	4.17 (.75)
Facilitate relay of clinical data to providers	83.3%	5.20 (1.09)	66.7%	3.88 (.83)
Change record systems	100%	5.33 (1.21)	41.7%	4.20 (1.09)
Other: Provide clarity of roles and responsibilities	16.7%	6.00 (.00)	0%	–
Implementation testing strategies	83.3%	5.27 (1.23)	40%	4.53 (.70)
Pilot test the implementation effort then gradually move to system-wide roll out	66.7%	5.00 (1.15)	33.3%	4.57 (1.13)
Model or simulate changes that will be implemented	50%	5.67 (.57)	38.1%	4.38 (1.18)
Implement changes in a cyclical fashion using small tests	66.7%	5.25 (.95)	28.6%	4.50 (.83)
Other: PI checks	16.7%	6 (.00)	0%	–
Quality management strategies	100%	5.13 (1.11)	65%	4.37 (.69)
Develop and organize quality monitoring systems	83.3%	4.60 (1.14)	53.8%	4.00 (1.00)
Develop tools for quality monitoring	100%	4.83 (1.16)	30.8%	4.25 (.95)
Audit and provide feedback	83.3%	4.60 (1.34)	76.9%	4.10 (.99)
Remind clinicians	66.7%	5.25 (1.50)	53.8%	4.43 (.53)
Use advisory boards and work groups	16.7%	6.00 (.00)	7.7%	4.00 (.00)
Obtain and use patient/consumer and family feedback	66.7%	5.25 (.95)	53.8%	4.71 (.95)
Centralize technical assistance	66.7%	5.00 (1.15)	38.5%	4.40 (.54)
Provide clinical supervision	100%	5.17 (.98)	53.8%	3.86 (1.46)
Intervene with patients/consumers to enhance uptake	66.7%	5.25 (1.50)	3.8%	4.75 (.50)
Purposefully reexamine the implementation	66.7%	5.75 (.50)	69.2%	4.00 (.70)
Conduct cyclical small tests of change	83.3%	5.20 (1.09)	30.1%	4.25 (1.50)
Use an improvement/ implementation advisor	50%	5.67 (.57)	61.5%	4.13 (.64)
Use data experts	50%	5.67 (.57)	38.5%	4.00 (.70)
Capture and share local knowledge	66.7%	5.25 (.95)	53.8%	4.14 (.69)
Organize clinician implementation team meetings	83.3%	5.20 (1.09)	38.5%	4.20 (.83)
Other: Develop a shared mission	16.7%	6.00 (.00)	15.4%	3.00 (.00)
Sustainment strategies	100%	5.2 (1.12)	50%	4.5 (.44)
Leader prioritization and support	66.7%	5.50 (.57)	50%	4.40 (.89)
Maintain workforce skills through continued training	100%	4.83 (1.32)	80%	4.50 (.75)
Maintain staff buy-in	66.7%	5.50 (1.00)	40%	4.75 (.50)
Fund and contract for continued use	33.3%	5.00 (1.41)	50%	4.60 (.54)
Systematic adaptation of the new practice or strategy to increase fit/compatibility	83.3%	5.40 (.89)	70%	4.86 (.37)
Mutual adaptation between intervention and site structure	66.7%	6.00 (.00)	70%	4.29 (.75)
Other: Staff orientations	16.7%	6.00 (.00)	10%	3.00 (.00)

^a

Continuous data was normalized using fractional rank method.

BT, behavior technician. Bolded entries indicate the “general strategy” that aggregates all discrete strategies listed below it.

Table 3.

T-test comparisons of count scores for implementation strategies by role.

Strategy type	Director/Supervisor (n= 6)		Behavior technician (n= 20)		t
	M (SD)	SE	M (SD)	SE
General	5.67 (0.82)	0.33	2.80 (1.91)	0.43	5.29**
Planning	10.0 (5.98)	2.44	2.45 (4.85)	1.09	3.18**
Financial	2.00 (1.27)	0.52	0.10 (0.48)	0.10	3.61*
Restructuring	4.67 (1.21)	0.49	1.90 (2.10)	0.47	3.05*
Implementation testing	2.00 (1.55)	0.63	1.05 (1.36)	0.30	1.46
Quality management	10.7 (3.33)	1.36	4.60 (5.11)	1.14	2.71**
Sustainment	4.33 (1.97)	0.80	1.85 (2.25)	0.50	2.43**

*p < .05, **p < .01.

Implementation strategies by setting

For both university- and community-based sites, the most commonly reported general implementation strategies included quality management (21%; 21%), sustainment (19%; 21%), and restructuring strategies (19%; 21%). Financial-based strategies were least endorsed in community sites, while planning strategies were the least endorsed at the university site. See Table 4 for frequencies in each discrete category. Independent samples t-test comparisons of general implementation strategies count scores only indicated significant mean differences in financial-based strategies endorsed, with the university site endorsing more use than community sites (p = .024; Table 5).

Table 4.

Frequencies of endorsed general and discrete implementation strategies and average rate of perceived satisfaction by setting.

General and discrete implementation strategies	Site 1 (University)	Satisfaction a M (SD)	Sites 2 and 3 (Community)	Satisfaction a M (SD)
Planning	38%	4.13 (.84)	38%	4 (.36)
Conduct a local needs assessment	8%	6 (.00)	15%	4.5 (.70)
Assess for readiness; Identify barriers	38%	5 (1.00)	31%	4.5 (1.29)
Visit other sites	31%	4.75 (.95)	23%	3.33 (2.51)
Develop a formal implementation blueprint	23%	4.67 (1.52)	15%	4.5 (.70)
Tailor strategies to overcome barriers	38%	4.80 (1.30)	15%	5 (1.41)
Stage implementation scale-up	15%	5.50 (.70)	8%	4 (0.00)
Model and simulate change	31%	5.25 (.95)	31%	4.75 (.5)
Conduct community consensus discussions	23%	5 (1.00)	23%	5 (1.00)
Involve executive board	15%	4.50 (2.12)	8%	5 (.00)
Identify/ prepare key team members who drive implementation efforts	31%	5 (1.14)	23%	5 (.00)
Involve patients/ consumers & family	31%	5.50 (1.00)	31%	4.75 (.50)
Recruit, designate and train for leaderships	38%	5.40 (.89)	23%	4.33 (.57)
Mandate change	31%	5.50 (.57)	23%	5 (1.00)
Build coalition	23%	5.33 (1.15)	23%	5 (1.00)
Develop resource-sharing agreement	23%	5.67 (.57)	23%	5.33 (.57)
Obtain formal commitments	31%	5.67 (.57)	15%	4.5 (.70)
Develop academic partnerships	38%	5.40 (.89)	31%	4.75 (.95)
Other	8%	6 (.00)	0%	–
Financial strategies	46%	4.26 (1.03)	23%	4 (.00)
Alter incentive/ allowance structures	15%	4.50 (.70)	0%	–
Use capitated payments	8%	4.00 (.00)	0%	–
Penalize	0%	–	0%	–
Use other payment schemes	0%	–	8%	4 (.00)
Reduce or increase patient/consumer fees	0%	–	0%	–
Place a fee for service lists/formularies	0%	–	0%	–
Fund and contract for the clinical innovation	15%	4.000 (1.41)	0%	–
Access new funding	31%	5 (.81)	0%	–
Make billing easier	15%	4.50 (.70)	0%	–
Other: Staff incentives	15%	2.50 (2.12)	0%	–
Restructuring srategies	69%	4.77 (1.09)	69%	4.64 (.87)
Revise professional roles	38%	5.40 (.54)	38%	5.2 (.44)
Change service sites	23%	4.67 (.57)	15%	4 (1.41)
Change physical structure and equipment	69%	4.56 (1.23)	46%	4.83 (.98)
Create new clinical teams	54%	4.71 (1.11)	31%	4.25 (1.25)
Facilitate relay of clinical data to providers	69%	4.67 (1.11)	31%	3.75 (.95)
Change record systems	69%	4.89 (1.16)	15%	4.5 (2.12)
Other: Provide clarity of roles and responsibilities	8%	6 (.00)	0%	–
Implementation testing strategies	62%	4.45 (1.40)	62%	4.86 (.29)
Pilot test the implementation effort then gradually move to system-wide roll out	46%	4.67 (1.50)	38%	4.8 (.44)
Model or simulate changes that will be implemented	46%	4.50 (1.64)	38%	5 (.00)
Implement changes in a cyclical fashion using small tests	46%	4.83 (1.16)	31%	4.75 (.50)
Other: Not specified	8%	6 (.00)	0%	–
Quality management strategies	77%	4.39 (1.24)	69%	4.39 (.76)
Develop and organize quality monitoring systems	54%	4.43 (1.13)	38%	4 (1.00)
Develop tools for quality monitoring	62%	4.63 (1.18)	15%	4.5 (.70)
Audit and provide feedback	54%	4.29 (1.49)	62%	4.25 (.70)
Remind clinicians	54%	4.86 (1.21)	31%	4.5 (.57)
Use advisory boards and work groups	15%	5 (1.41)	0%	–
Obtain and use patient/consumer and family feedback	54%	4.71 (1.11)	31%	5.25 (.50)
Centralize technical assistance	38%	4.80 (1.09)	31%	4.5 (.57)
Provide clinical supervision	69%	4.44 (1.59)	31%	4.5 (1.00)
Intervene with patients/consumers to enhance uptake	38%	5.40 (.89)	23%	4.33 (1.15)
Purposefully reexamine the implementation	54%	4.86 (1.21)	46%	4.17 (.75)
Conduct cyclical small tests of change	38%	5.40 (.89)	31%	4 (1.41)
Use an improvement/ implementation advisor	38%	4.80 (1.09)	46%	4.33 (.81)
Use data experts	31%	4.75 (1.50)	31%	4.5 (.57)
Capture and share local knowledge	38%	4.60 (1.34)	46%	4.5 (.54)
Organize clinician implementation team meetings	54%	4.86 (1.06)	23%	4.33 (1.15)
Other: Develop a shared mission	8%	6 (.00)	15%	3 (.00)
Sustainment strategies	69%	4.88 (1.07)	69%	4.22 (.79)
Leader prioritization and support	46%	5 (1.09)	23%	4.67 (.57)
Maintain workforce skills through continued training	62%	5 (1.06)	46%	4.17 (.75)
Maintain staff buy-in	46%	5.33 (.81)	15%	4.5 (.70)
Fund and contract for continued use	31%	5 (.81)	23%	4.33 (.57)
Systematic adaptation of the new practice or strategy to increase fit/compatibility	54%	5.29 (.75)	38%	4.8 (.44)
Mutual adaptation between intervention and site structure	46%	5 (1.09)	38%	4.8 (1.09)
Other: Staff orientations	8%	6 (.00)	0%	–
Other: Checks for principal investigators (PIs)	0%	–	8%	3 (.00)

^a

Continuous data was normalized using fractional rank method. Bolded entries indicate the “general strategy” that aggregates all discrete strategies listed below it.

Table 5.

T-test comparisons of count scores for implementation strategies by setting.

Strategy type	University-based setting (n = 13)		Community-based setting (n = 13)		t
	M (SD)	SE	M (SD)	SE
General	3.62 (2.33)	0.65	3.31 (1.93)	0.54	0.37
Planning	4.77 (6.65)	1.84	3.62 (5.39)	1.50	0.49
Financial	1.00 (1.35)	0.38	0.08 (0.28)	0.08	2.41*
Restructuring	3.31 (2.46)	0.68	1.77 (1.79)	0.50	1.82
Implementation testing	1.46 (1.45)	0.40	1.08 (1.44)	0.40	0.68
Quality management	7.00 (5.80)	1.60	5.00 (4.98)	1.38	0.95
Sustainment	2.92 (2.78)	0.77	1.92 (1.94)	0.54	1.06

*p < .05, **p < .01.

Satisfaction

Overall, Directors/Supervisors were very satisfied with endorsed general implementation strategies, while BTs felt somewhat satisfied with endorsed general implementation strategies. Directors/supervisors and BTs felt somewhat less satisfied with financial-based strategies than other strategies. Moreover, both university and community settings reported high levels of satisfaction (ranging from somewhat satisfied to very satisfied) with endorsed implementation strategies. Results from two independent samples t-test indicated no significant differences between roles or settings in rates of satisfaction with endorsed implementation strategies (p's>.05; Tables 6 and 7).

Table 6.

T-test comparison of mean satisfaction with implementation strategies endorsed by role.

Strategy type	Director/Supervisor			Behavior Technician			t
	n	M (SD)	SE	n	M(SD)	SE
Planning	5	5.37 (0.68)	0.30799	5	4.82 (0.32)	0.14472	1.640
Financial	3	4.54 (0.92)	0.53471	1	4.22 (0.00)	–
Restructuring	6	5.36 (1.04)	0.42634	12	4.62 (0.62)	0.18039	6.851
Implementation testing	5	5.27 (1.23)	0.55269	8	4.53 (0.70)	0.24983	1.382
Quality management	6	5.13 (1.11)	0.45656	13	4.37 (0.69)	0.19211	1.846
Sustainment	6	5.20 (1.12)	0.46019	10	4.58 (0.44)	0.14026	1.565

*p < .05, **p < .01 Note: Comparisons between satisfaction with financial strategies could not be computed due to insufficient sample size needed for calculations.

Table 7.

T-test comparison of mean satisfaction with implementation strategies endorsed by setting.

Strategy type	University			Community			t
	n	M (SD)	SE	n	M(SD)	SE
Planning	5	5.30 (0.74)	0.33499	5	4.88 (0.32)	0.14532	1.131
Financial	4	4.46 (0.77)	0.38656	0	–	–
Restructuring	9	4.91 (0.95)	0.31818	9	4.82 (0.76)	0.25414	0.213
Implementation testing	8	4.81 (1.25)	0.44216	5	4.82 (0.28)	0.12720	−0.024
Quality management	10	4.64(1.11)	0.35382	9	4.57 (0.62)	0.20855	0.156
Sustainment	8	5.07 (0.98)	0.34806	8	4.56 (0.50)	0.17712	1.283

*p < .05, **p < .01.

Note: Comparisons between satisfaction with financial strategies could not be computed due to insufficient sample size needed for calculations.

Spearman's Rho indicated negative correlations between role and total number of general strategies, planning, financial-based, restructuring, quality, and sustainment strategies (p's < .05), but not implementation-testing strategies (p > .05). Employment status was positively correlated with implementation-testing (r = .415, p = .035). Total number of general implementation strategies endorsed was positively correlated with perceived satisfaction of financial-based, restructuring, quality, sustainment implementation strategies (p's < .05; Table 8). Thus, a higher number of endorsed strategies was related to higher rates of satisfaction, depending on the general category.

Table 8.

Spearman's Rho correlations.

	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19
1. Site	–	.298	−.229	.102	.302	.056	−.104	−.041	−.303	−.337	−.131	−.233	−.001	−.085	.100	−.150	−.011	−.155	−.274
2. Role	.298	–	−.210	−.420*	.127	.009	−.613**	−.521**	−.453	−.792**	−.393	−.522**	−.379	−.329	−.245	−.474*	−.332	−.472*	−.381
3. Duration of employment	−.229	−.210	–	.178	−.319	−.372	−.108	−.182	−.045	.110	−.414	−.263	−.267	−.343	−.303	−.027	−.503*	−.242	−.187
4. Education	.102	−.420*	.178	–	−.156	.005	−.031	.130	−.172	.314	.494	−.152	.337	−.103	.244	−.013	.280	.020	−.104
5. Prior ASD Training	.302	.127	−.319	−.156	–	−.076	−.112	.099	−.114	−.141	.655	.047	−.121	.044	.437	−.175	.026	.125	−.103
6. Employment status	.056	.009	−.372	.005	−.076	–	.070	−.180	.290	.021	.393	.101	.246	.415*	.207	.144	.454	.041	.271
7. Number of general strategies	−.104	−.613**	−.108	−.031	−.112	.070	–	.740**	.500	.545**	.845*	.778**	.484*	.525**	.333	.656**	.659**	.750**	.809**
8. Number of planning strategies	−.041	−.521**	−.182	.130	.099	−.180	.740**	–	.494	.292	.812*	.569**	.536*	.270	.462	.575**	.556*	.655**	.773**
9. Satisfaction with planning strategies	−.303	−.453	−.045	−.172	−.114	.290	.500	.494	–	.171	.800	.025	.745*	.496	.883*	.365	.908**	.117	.966**
10. Number of financial strategies	−.337	−.792**	.110	.314	−.141	.021	.545**	.292	.171	–	−.031	.595**	.125	.277	.056	.402*	.085	.457*	.088
11. Satisfaction with financial strategies	−.131	−.393	−.414	.494	.655	.393	.845*	.812*	.800	−.031	–	.717	.698	.500	.462	.928**	.886*	.971**	.812*
12. Number of restructuring strategies	−.233	−.522**	−.263	−.152	.047	.101	.778**	.569**	.025	.595**	.717	–	−.016	.533**	.173	.675**	.253	.752**	.411
13. Satisfaction with restructuring strategies	−.001	−.379	−.267	.337	−.121	.246	.484*	.536*	.745*	.125	.698	−.016	–	.382	.580*	.305	.693**	.302	.837**
14. Number of implementation testing strategies	−.085	−.329	−.343	−.103	.044	.415*	.525**	.270	.496	.277	.500	.533**	.382	–	.560*	.505**	.447	.413*	.661**
15. Satisfaction with implementation testing strategies	.100	−.245	−.303	.244	.437	.207	.333	.462	.883*	.056	.462	.173	.580*	.560*	–	.368	.796**	.340	.656
16. Number of quality strategies	−.150	−.474*	−.027	−.013	−.175	.144	.656**	.575**	.365	.402*	.928**	.675**	.305	.505**	.368	–	.255	.655**	.691**
17. Satisfaction with quality strategies	−.011	−.332	−.503*	.280	.026	.454	.659**	.556*	.908**	.085	.886*	.253	.693**	.447	.796**	.255	–	.370	.859**
18. Number of sustainment strategies	−.155	−.472*	−.242	.020	.125	.041	.750**	.655**	.117	.457*	.971**	.752**	.302	.413*	.340	.655**	.370	–	.717**
19. Satisfaction with sustainment strategies	−.274	−.381	−.187	−.104	−.103	.271	.809**	.773**	.966**	.088	.812*	.411	.837**	.661**	.656	.691**	.859**	.717**	–

*Correlation is significant at the 0.05 level (two-tailed).

**Correlation is significant at the 0.01 level (two-tailed).

ASD, autism spectrum disorder.

Qualitative phase

Measure

Semi-structured interview

The semi-structured interview protocol was designed to expand on quantitative findings to explain and contextualize participants’ perspectives on and experiences with implementation strategies used within the past year at their sites as well as elicit suggestions for improving implementation in community settings. Examples of prompts included: “What implementation strategies worked well?” and “Would other strategies work better?” To facilitate these discussions, the interviewer provided participants with a copy of the ERIC compilation to ensure common terms/definitions for implementation strategies.

Case study illustration: qualitative methods

Participants

Using purposive sampling, all participants who completed the survey were invited via email to participate in the qualitative phase. Thirteen participants (50%) agreed to participate (nine BTs, four Directors/Supervisors). Qualitative data saturation was determined at the data collection stage using consensus procedures after the completion of 13 interviews. The first and second author agreed that data collected within the interviews were no longer eliciting “new information,” such that any potential codes or themes had already been collected (Saunders et al., 2018).

Procedure

The first author, who had certified qualitative research expertise, conducted individual, semi-structured interviews via Zoom. These interviews took place in confidential locations at the participants’ convenience with verbal consent obtained. Interviews ranged from 21:45 min to 1:07:11 min (M = 46.87, SD = 15.48). All participants received a $15 honorarium for their participation.

Data analysis

Qualitative data was analyzed using inductive coding and content analysis to expand on quantitative findings by providing context for strategy use and satisfaction within each setting (Bernard, 2006; Hsieh & Shannon, 2005; Vaismoradi & Snelgrove, 2019). Interview data was transcribed, de-identified, and verified for accuracy. The first and second authors developed the coding schema by independently reviewing two transcriptions for emergent codes. The coding schema included code definitions and exemplar text to guide the coding process. After the coding schema was finalized, two additional coders were trained to conduct the qualitative coding. All four coders were randomly assigned three transcripts to independently code for reliability procedures. Once consensus was reached and reviews were considered reliable, two pairs of coders were randomly assigned to independently code 6–7 interviews; all transcripts were double-coded. Coding discrepancies were resolved with consensus meetings. Codes and categories revealed underlying meaning and patterns found within the data. Emergent themes were identified by assessing frequency and salience of codes within and across transcripts. All data was analyzed using MAXQDA software. Additionally, the first and second author reviewed the consolidated criteria for reporting qualitative studies (COREQ) to ensure inclusion of key elements (Tong et al., 2007).

Case study: qualitative findings

Content analysis of semi-structured interviews revealed three emergent themes related to implementation strategy use: (1) Context, (2) Communication Strategies, and (3) Successes and Challenges. The research aims and questions guided the development of themes and categories. Themes, categories, and illustrative quotes are displayed in Table 9.

Table 9.

Qualitative codes, definitions, and frequency.

Code	Definition	Quotes	Frequency (ever-coded, total Frequency)
Theme 1: Context	Responses describe and identify the use of specific implementation strategies in further detail, including planning, financial-based, implementation-testing, restructuring, quality management, and sustainment strategies
Planning strategy	Comments include planning strategies used to support the use/delivery of ELI. Examples may include: consistent communication with MEDICAID recipients, developing a formal implementation blueprint, or integrating feedback from preschool teams (outside of ELI) into ELI's design, and building community partnerships.	“having that bridge across the sites um and then also having [director] have personal relationships with our clients”—BT, university setting	9, 30
Financial-based strategy	Responses related to financial based strategies used to support the use or delivery of ELI, often to incentivize the use of new practices or strategies and provide resources for training and ongoing support for clinical staff.	“Once [BTs] complete that training we give them an incremental bump up of their hourly rate and then if they decide to move on we actually uh we give, we pay for the clinical hours that would be required the first week of training so most individuals who become a registered behavior tech have to have at least forty hours of supervised experience. In most settings an individual would have to pay for that; we’re providing that training.”—Director, all settings	4, 9
Restructuring strategy	Responses related to restructuring strategies used to deliver ELI. Responses may include: altering staffing, professional roles, the physical structure of the agency or service setting, intervention equipment, materials and resources, and data systems. One example relates to changing the intervention's environmental setting to be more “natural looking” than a clinic.	“we have ongoing support in terms of restructuring strategies, altering staff, professional roles. We've already gone through that a little bit this year in terms of what the master's students are going to look like next year. So, for example, they received uh a package that provided fellowship and hourly work. Um moving forward next year they'll be in clinic for 20 h a week in assistant behavior analyst roles, so that's changing a bit.”—Director, all settings	4, 9
Implementation- testing strategy	Responses related to the use of implementation-testing strategies used to deliver ELI within the past school year. Comments can include the use of pilot-testing and small tests of change.	“..There are different things that we pilot at different sites, at different times and then see how those work and then we will roll them out to other sites depending on what was revealed in the pilot test”—Supervisor, university setting	1, 1
Quality management strategy	Responses related to quality improvement strategies used to support the delivery of ELI, which may involve support networks or data systems that act to continually evaluate and enhance client's quality of care. One example describes how manuals are always updated and managing data collected in binders for fidelity.	“We're developing and organizing all of the monitoring systems. [Director] is doing that weekly and in terms of implementing what is or is not working and we're making changes as we need to. Um and developing tools for quality monitoring, they're taking data every day so that helps with um tracking that.”—Director, all settings	12, 38
Sustainment strategy	Responses related to sustainment strategies used to support the continued delivery of ELI. Example of responses may emphasize ongoing training of staff to maintain the quality of ELI.	“maintain workforce skills to continue training, we do that every two weeks usually on Fridays were providing training for our staff here at [university location]”—Director, all settings	3, 5
Unsure or unaware of implementation strategies used	Responses describe how participant is not aware of implementation strategies used or does not demonstrate knowledge about the implementation process related to ELI	“… I don’t because we there was never any really like I-I know there are definitely a lot of people in the background um that do a lot of work to to keep it going but we never really talked that much about it. And maybe it was just my cohort that didn’t really know about it I don’t know.”—BT, community setting	5, 10
Theme 2: Communication Strategies	Includes responses related to how information on new initiatives or announcements is communicated to ELI staff. This includes formal and informal strategies used for communication. Codes are separated by the type of information discussed by participants.
Staff departures/issues	Responses related to communication strategy used to update ELI staff on staff departures or issues related to staff members. Responses may include formal or informal strategies, such as email or staff meeting.	“it's usually like one BT told another BT that um the BCBA was gossiping with the lead who then told another BT and then you hear kind of through the grapevine that way.”—BT, university setting	8, 14
New initiatives or announcements	Responses related to communication strategy used to update ELI staff on new initiatives, new policies, or modifications made to the program. Responses may include formal or informal strategies, such as email, word of mouth, or meetings on a needed basis.	“the new initiatives or programming, we usually get it directly from the executive director or the clinical director, which is the same person currently”—Supervisor, all settings	10, 18
Theme 3: Success and Challenges	Includes descriptions of implementation and communication strategies that did and did not work well, and responses that describe suggestions to improve strategy use for community-based settings.
Strategies that were successful	Responses describe strategies that were actively carried out and had positive outcomes. An example of a successful strategy includes, “Having it all mapped out before made it easier to get things implemented quickly” or “making changes when things didn’t work.”	“I think that we have organization of um team meetings, clinician implementation team meetings, I think that has worked exceptionally well. Um, I think maintaining work force skills through continued training is an area of strength right now.”—Supervisor, university setting	8, 20
Challenges with implementation strategies	Responses describe strategies that were actively carried out but did not work well or meet objectives of the program.	“whatever was developed from different work groups wasn't always shared uh across the entire organization or what was shared, we weren't quite sure how it got developed or came to fruition so I think there was just kind of a lack of history or knowledge there for those of us that weren't on specific work groups to understand how things came about”—Supervisor, university setting	10, 19

Context

This theme expanded on and contextualized the use of implementation strategies endorsed on the ISSS. Subcategories were developed based on ISSS general categories: planning, financial, implementation testing, restructuring, quality management, and sustainment strategies. Qualitative data illustrated how strategies were utilized in practice, providing context and insight into variations across roles and sites.

Planning and quality management strategies were most frequently discussed throughout the interviews. Activities described for planning strategies included maintaining relationships among staff through regular “team meetings,” preexisting relationships with implementation sites, and developing and maintaining relationships with clients and families through “family training.” BTs and Directors/Supervisors both discussed the role of preexisting relationships in identifying new implementation sites, and the importance of maintaining consistent communication with families and community partners throughout implementation to ensure buy-in. One Director/Supervisor shared that all sites are “very open to visitors, which is very difficult, but that really helps sell the program…it really is great for our community partnerships.”

Both BTs and Directors/Supervisors discussed the use of various quality management strategies, including collecting and auditing client-level data, receiving feedback from the executive director, accessing new and ongoing staff training, maintaining communication and eliciting feedback from client's families, capturing and sharing local knowledge, and developing and utilizing various monitoring systems. Among BTs, staff training was viewed as the most influential for improving implementation skills across sites: “We would also have training…that were site specific that were more related to individual clients, to talk about what was happening at that specific site and I think that the group trainings were very beneficial because we were learning about the concepts and applying them and practicing them.” Conversely, Directors/Supervisors highlighted efforts to develop new monitoring systems for ongoing feedback: “We're developing and organizing all of the monitoring systems. [Executive director] is doing that weekly and in terms of implementing what is or is not working and we're making changes as we need to. And developing tools for quality monitoring, they're taking data every day so that helps with tracking.”

Financial-based and restructuring strategies were the least discussed throughout the interviews and least endorsed in the ISSS. Nonetheless, qualitative interviews contextualized staff's limited experiences in use, ranging from brief discussions about grants, maintaining a budget for professional development, and utilizing monetary incentives for BTs. Of note, there appeared to be less awareness of these strategies being used in practice. For instance, one BT (university site) stated “I assume there's research funding given to ELI which allows us to do what we do and everything.” In terms of restructuring strategies, staff highlighted daily adaptations made to the physical structure in community sites and transferring files and data to an electronic server for better assessments. One BT shared, “Upon arriving at work we'd set up the room in a way that we think is helpful for children to learn and where children are able to have enough space to be by themselves to learn and not get distracted by anything else.”

Sustainment strategies were described in the context of ongoing staff training and professional development, incentivizing strategies to maintain staff buy-in, and continued adaptations to implementation sites. One Director/Supervisor highlighted the frequency and context of strategies used across sites:“Maintain workforce skills to continue training, we do that every two weeks… we try incentivize training for people… one thing that we're doing for all of our staff we received some gift funding and that with some of our professional development funds that we had we providing a [EBP] training for everybody and we're getting…both of the individuals who came up that curriculum are delivering the training for all of our staff…”

Communication strategies

This theme described how information (e.g., new initiatives, staff departures, announcements, feedback) was communicated to program staff through formal and informal communication strategies. Overall, BTs and Directors/Supervisors across sites described various strategies used for feedback, “Rich opportunities and feedback comes in a variety of ways: written, in person, in the moment, after the fact, you know formal trainings, casual conversations, all kinds of ways,” as well as general information being disseminated through different individuals, “It just really depends on the information, who that comes through.”

Formal communication strategies included monthly performance evaluations, live feedback/supervision, weekly meetings with the executive director, and morning and afternoon announcement meetings. Directors/Supervisors (all sites) also stated that specific information was typically communicated to staff by specific individuals. For example, one director shared, “New initiatives, programming changes, those usually come through our executive director,” while “staff issues usually come through…our HR person.”

Informal communication strategies included communication through word of mouth, emails, phone call or text messages, checking in with Directors/Supervisors, and informal feedback structures. Of note, informal feedback structures were viewed as particularly helpful for improving implementation skills: One BT shared, “Informal feedback was really helpful so…the BCBA would pop in and watch a few things and either give constructive, positive reinforcement…and… some constructive feedback.”

Successes/challenges

This theme described strategies that did and did not work well and provided suggestions to improve strategy use for community settings.

Strategies That Were Successful described successful experiences with implementation strategies within the past year. When asked about which implementation strategies worked well, two Directors/Supervisors (all sites) discussed strategies related to adapting the environment, establishing a positive and nurturing environment for staff, developing tools for quality monitoring, and offering continued training to staff. For example, a Supervisor (university site) stated, “I think maintaining workforce skills through continued training is an area of strength right now.” Six BTs (all sites) discussed staff training, regular communication with the executive director, workgroups, and consistent team meetings as strategies that worked well. One BT (university site) explained that “meetings at the beginning and end of day, and then we do longer meetings on Fridays when we don’t have [clients] and I think having those full team sessions…at the site is incredibly helpful and bouncing ideas off each other” were effective implementation strategies to increase knowledge and skills to deliver the EIBI by learning from one another.

Challenges with Implementation Strategies captured staff experiences with strategies that did not work well or meet intended objectives. Three Directors/Supervisors (all sites) discussed challenges in implementing workgroups, lack of structure for receiving client feedback, not having clearly defined staff roles, and difficulty maintaining communication across various staff levels. For example, a Director (university site) explained, “It's sometimes really hard to get feedback on things like patient and consumer kind of things…it's hard to get any feedback and then what feedback we do get is sometimes hard to decipher.” Related to communication among staff, one Director (all sites) stated, “administration…lost the consistent contact and communication with the clinical group.” Seven BTs (all sites) discussed challenges related to not having enough time or access to meet with supervisors, limited staff training, and communication challenges related to treatment changes or updates made for specific family cases. For instance, one BT (community site) described lack of communication leading to limited preparation: “It wasn't as clear about what was actually going on in the meetings or what the family's really interested in so we would know…about the meetings happening but then all the sudden the families would just come into the room and we weren't as prepared.”

Staff also identified opportunities for improvement and provided suggestions to improve strategy use for community settings in the future. Areas for improvement included (1) improving feedback/supervision and (2) improving communication among program staff, school staff, and parents. Eight BTs and three Directors/Supervisors (all sites) made suggestions to provide feedback for those in leadership positions, increase access to supervisors, and increase the consistency and amount of feedback and supervision opportunities. Overall, all BTs indicated that feedback and supervision “definitely could happen more.” Five BTs and three Directors/Supervisors (all sites) suggested including more staff training with a focus on communication skills to work with community members outside the program staff and establishing clearer guidelines for communication shared via email and in meetings (e.g., content and/or frequency). Lastly, one BT (community site) emphasized the importance of communicating more information about the overall purpose of the program “…more meetings where they are laying out what the purpose is or what purposes some of the tasks are for the children.”

Discussion

The case study demonstrated the use of sequential mixed methods (QUAN→QUAL) to highlight frequency of and experiences with implementation strategies utilized in one university- and two community-based settings. A quantitative survey of implementation strategy use (ISSS) provided information on the range of strategies reportedly used to implement ELI, along with perceived satisfaction with endorsed strategies within the preceding year. Qualitative interviews then provided context underlying implementation strategy use, expanding on how these strategies were used, what did or did not work well, and suggestions to improve future implementation strategy use in community settings.

Notably, findings illustrated that all three sites delivering the ELI program utilized multiple implementation strategies, rather than utilizing one discrete strategy. For instance, regardless of setting, sites most often endorsed three general categories of strategies: quality management, sustainment, and restructuring strategies. Further, many discrete strategies were reportedly utilized within each general category and varied by setting. While some studies have shown that there are advantages to using multi-faceted strategies versus one discrete strategy for more effective implementation, the evidence is mixed (Powell et al., 2019; Squires et al., 2014). However, these findings emphasize the need to consider context in determining the appropriate number of strategies, as well as expected barriers to the implementation process for a given setting. Other studies have suggested that implementation strategies may be reported as endorsed but not implemented in a systematic manner, particularly in community settings (Drahota et al., 2021). Future studies are encouraged to explore the fit of implementation strategies to agency characteristics as well as fidelity of implementation strategy use to broaden our understanding on how the effectiveness of strategies utilized in community sites relies on the number of strategies used simultaneously, fit, and/or fidelity of strategy use (Powell et al., 2017; Slaughter et al., 2015).

Interestingly, financial strategies were the least endorsed across settings, with significant mean differences found between settings. Although these strategies were reportedly used less frequently than other implementation strategies, the university site tended to report more use of financial strategies than community sites. This is worth noting given prior evidence on how financial strategies may effectively facilitate EBP implementation, particularly in behavioral health care systems providing disability services (Dopp et al., 2020).

Implementation strategies and role

Findings highlighted how role is related to the endorsement of implementation strategy use. In the case study, Directors/Supervisors consistently endorsed significantly more implementation strategies than BTs regardless of category. Role also demonstrated a significant negative correlation with the number of general implementation strategies, planning strategies, financial-based strategies, restructuring, quality, and sustainment strategies endorsed. Overall, these findings suggest a unidirectional relationship between role and implementation strategy endorsed use. Prior literature on staff roles consistently highlights the direct influence of supervisors on implementation climate (Bunger et al., 2019; Guerrero et al., 2020; Sterba et al., 2020). Given this, it is possible that Directors/Supervisors influenced implementation climate and BTs perspectives on implementation strategy use leading to distinct implementation strategy reporting. Further, significant relationships were found between the total number of general implementation strategies endorsed and perceived satisfaction with their utilization. That is, staff who endorsed a higher number of general implementation strategies also tended to endorse higher rates of satisfaction. These relationships are evidenced in the literature, demonstrating that staff characteristics—role, perception, and/or experiences with prior implementation practices—and implementation processes are related and can potentially impact implementation quality and processes (Bunger et al., 2019; Guerrero et al., 2020).

Consequently, it is important to understand the extent to which community stakeholders are actively involved or aware of implementation strategies utilized in their setting (Slaughter et al., 2015). For instance, qualitative findings indicated that some staff were unaware of strategies utilized. Modest endorsements of implementation strategies from BTs when compared to Directors/Supervisors (from the survey data) also reflect this uncertainty. These findings suggest that staff across levels may lack awareness of implementation strategies being used to integrate ELI into their organizations. These findings warrant discussion about whether implementation data should be collected from all stakeholders or the most appropriate stakeholder, particularly if supervisors were primary decision-makers. However, limiting our scope to the most appropriate stakeholder (e.g., those in higher managerial roles) may limit our understanding of team dynamics in implementation strategy use as well as perceived satisfaction with their use in practice. Moreover, the dynamics of organizational influence is driven by the social exchange of staff from various roles and responsibilities (Gottfredson & Aguinis, 2017), underscoring the value of capturing implementation strategy use among different staff involved with the EBP. Thus, increasing staff awareness of implementation strategies being utilized to facilitate EBP implementation and sustainment may be an important component to address in future implementation efforts for practice. Other tracking tools for implementation strategy utilization have also called for more inclusive stakeholder input to align preferences with improved outcomes, as well as engaging all stakeholder groups earlier on in the process (Walsh-Bailey et al., 2021).

Communication strategies

Qualitative findings also highlighted important communication strategies utilized throughout the implementation process. Findings indicated that communication around new initiatives within the organization, staff changes, and other announcements were disseminated to staff in formal and informal ways. Previous research illustrates the importance and value of timely and effective communication (e.g., providing all relevant staff with information regarding plans or impact of change) to foster favorable attitudes for EBPs (Drahota et al., 2021; Grau, 1994). Without the use of communication strategies, staff may be resistant to organizational changes, such as the implementation of new EBPs. However, given the importance of communication strategies in EBP implementation (Drahota et al., 2021; Lanham et al., 2009; Manojlovich et al., 2015), future efforts in community contexts should consider the utilization of more systematic and formal communication strategies, as this may facilitate staff buy-in and support.

Taken together, the methodological approach applied in this case study illustrates one method for tracking implementation strategy use and eliciting stakeholder perspectives regarding implementation strategies, within the context of both community and university based settings providing services to youth on the autism spectrum. These findings can inform next steps to improve tracking and reporting for EIBI implementation of autism services provided in community settings. For example, findings can be used to generate a list of strategies actively utilized in practice to assess variations across roles and sites and to identify areas for support. Site directors may also use the list of strategies to create a quality improvement checklist to determine whether the number of implementation strategies used may be effective or whether all sites are using the same quality of strategies. Additionally, findings highlight how strategies, as named currently, are often misunderstood, particularly for community agencies providing ASD treatment. Although we attempted to clarify strategies by providing standardized definitions, community providers were still uncertain in how implementation strategies were conceptualized. This prompts future research and practice to tailor implementation strategies that fit specific community settings. Incorporating community-based participatory principles or human centered design into implementation strategies to co-create with providers or other participants may be most beneficial to improve implementation of ASD treatments delivered through community settings.

Limitations

First, the study included a relatively small sample size, which may have generated inflated statistical findings. Future studies with larger sample sizes (above 30) are needed to confirm whether statistical results can be replicated with different parameters. Thus, findings may not be generalizable to other ASD-CBOs or broader populations. However, great effort was taken to normalize the data and limit analysis assumptions. Second, the study was exploratory in its approach and attempted to capture naturally occurring processes of reported implementation strategy use. Therefore, results cannot ascertain causality of any kind. Third, during interview procedures, community providers interpreted the standardized definitions for the ERIC compilation of implementation strategies differently than expected (e.g., many BTs discussed buy-in strategies within the Planning category but were uncertain whether it was appropriate for the discussion). Along with recall bias due to timing of the surveys, participants may have been limited in their ability to report on strategy use. Importantly, this study did not utilize all strategies in the updated glossary of the ERIC compilation or fully expand on details from Proctor's guidelines for reporting. While these findings may direct next steps to enhance definitions for clarity and relevance in the context of autism services (Cook et al., 2019; Perry et al., 2019; Proctor et al., 2013), future studies should utilize the updated compilation to more accurately explore implementation strategy use and satisfaction within this context, and utilize the guidelines to increase the potential for replication in both research and practice.

Contributions

This paper contributes to one of the five priorities to enhance the impact of implementation strategies by describing a practical approach to “improve tracking and reporting” of implementation strategy use within university- and community-based agencies serving young children on the autism spectrum. Tying implementation strategies to the ERIC compilation helped synthesize and communicate information about reported implementation strategy use with a shared language, a recommendation asserted to enhance the application of implementation efforts across settings (Moullin et al., 2020). The study's approach emphasizes the importance of understanding community context to develop successful strategies for EIBI implementation and scale-up (Brookman-Frazee & Stahmer, 2018; Nahmias et al., 2019). By involving mixed methods, findings revealed a feasible, useful, and quantifiable measure of reported strategy use and then contextualized these with qualitative details from ASD providers, imparting details on the what and how of implementation strategy use (Palinkas et al., 2011). Directors in agencies serving children on the autism spectrum can make use of this practice through ongoing discussions about strategies being utilized (e.g., monthly/bimonthly meetings to discuss successful use of strategies or ongoing process evaluations with the integration of community providers’ voices). As reflected in our approach, understanding community provider's preferences by eliciting their experiences and input directly can help develop strategies that may readily fit into implementation-as-usual processes, with the ultimate goal of facilitating effective implementation in community settings (Leeman et al., 2017; Powell et al., 2015).