Individual and organizational factors as predictors of early evidence-based practice adoption in Michigan high schools: Baseline data from an implementation trial

Abstract

Background

Adolescents increasingly access mental health services in schools. School mental health professionals (SPs; school counselors, social workers, etc.) can offer evidence-based mental health practices (EBPs) in schools, which may address access gaps and improve clinical outcomes. Although some studies have assessed factors associated with EBP adoption in schools, additional research focusing on SP- and school-level factors is warranted to support EBP implementation as SPs’ mental health delivery grows.

Methods

Baseline data were collected from SPs at Michigan high schools participating in a statewide trial to implement SP-delivered cognitive behavioral therapy (CBT) to students. Models examined factors associated with attitudes about EBPs, implementation climate, and implementation leadership, and their associations with CBT knowledge, training attendance, and pre-training CBT delivery.

Results

One hundred ninety-eight SPs at 107 schools (87%) completed a baseline survey. The mean Evidence-Based Practice Attitude Scale (EBPAS) total score was 2.9, and school-aggregated mean scores of the Implementation Climate Scale (ICS) and Implementation Leadership Scale (ILS) were 1.83 and 1.77, respectively, all on a scale ranging from 0 (low) to 4 (high). ICS and ILS scores were lower than typically reported in clinical settings, while EBPAS scores were higher. School characteristics were not significantly associated with EBPAS, ICS, or ILS scores, but scores did differ by SP role. Higher EBPAS scores were associated with more CBT knowledge (average marginal effect for 1 SD change [AME] = 0.15 points) and a higher probability of training completion (AME = 8 percentage points). Higher ICS scores were associated with a higher probability of pre-training CBT delivery (AME = 6 percentage points), and higher ILS scores were associated with higher probability of training completion (AME = 10 percentage points).

Conclusions

Our findings suggest that SPs’ attitudes toward EBPs and organizational support were positively associated with early signs of implementation success. As schools increasingly fill the adolescent mental healthcare access gap, efforts to strengthen both provider attitudes toward EBP and strategic organizational factors supporting EBP delivery will be key to encouraging EBP uptake in schools.

Plain Language Summary

Schools are an important setting in which adolescents receive mental healthcare. We need to better understand how to implement evidence-based practices (EBPs) in this setting to improve student mental health. This study examined the attitudes and perceptions of school professionals (SPs) as key contributors to the implementation of a particular EBP, the delivery of cognitive behavioral therapy (CBT) in schools. The study found that implementation climate and leadership scores in participating schools were lower than scores typically reported in clinical settings, while scores for SP attitudes about EBP adoption were higher than typical scores in clinical settings. Results further suggest that SPs with more positive attitudes toward EBPs are more knowledgeable of CBT and more likely to complete a 1-day CBT training. We also found that higher implementation climate scores were associated with SPs reporting pre-training CBT delivery (although this association was not statistically significant), and more implementation leadership was associated with SPs completing the CBT training. These findings suggest that SP attitudes toward EBPs and organizational support in schools are positively associated with early signs of implementation success. Early, low-intensity efforts to (1) improve SP attitudes about mental health EBPs, and (2) increase schools’ support for implementation may scaffold more intensive implementation efforts in schools down the road.

Keywords

schools mental health CBT ILS ICS EBPAS evidence-based practice

Introduction

As adolescent mental illness rates rise, schools are increasingly tasked with addressing gaps in access to effective mental healthcare (Kern et al., 2017; Stempel et al., 2019; Twenge et al., 2019). Only 40%–45% of youth with mental illness have accessed mental healthcare (Costello et al., 2014; SAMHSA, 2021); of those accessing mental health support, this was most commonly through school-based mental health services (Ali et al., 2019; Duong et al., 2020). However, the implementation of mental health evidence-based practices (EBPs) is limited in school settings (Langley et al., 2010; Mychailyszyn et al., 2011; O’Reilly et al., 2018). This study explored provider- and organizational-level conditions that may help to promote EBP implementation in schools.

School-Based Mental Health Services for Adolescents

Schools are well suited to deliver effective mental healthcare to students. Schools circumvent many barriers youth experience when seeking mental healthcare, including stigma, confidentiality concerns, cost/lack of insurance coverage, transportation challenges, and limited provider availability (Radez et al., 2021; Reardon et al., 2017). school-based mental health services are provided locally using currently employed school professionals (SPs; e.g., school counselors, school social workers, school psychologists, and school nurses); this is advantageous given the limited availability of providers for youth mental healthcare, especially in rural areas (Findling & Stepanova, 2018; Larson et al., 2016). Moreover, SPs have pre-established relationships with students, which can encourage service utilization for students seeking help (Herzig-Anderson et al., 2012; Masia Warner et al., 2016).

Studies, however, have pointed to various reasons limiting mental health EBP implementation in schools. Identified barriers include a lack of accessible and affordable training in mental health EBPs for SPs, along with SPs’ competing responsibilities and administrative demands (Bruns et al., 2016; Corteselli et al., 2020; Hoover & Bostic, 2021; Kern et al., 2017; Ringle et al., 2015). Studies have suggested various strategies to promote implementation, including garnering school administrator support (Forman & Barakat, 2011; Langley et al., 2010), providing SP support via training and consultation for EBP delivery (Beidas et al., 2012; Eiraldi et al., 2020; Koschmann et al., 2019; Nadeem et al., 2013), allocating protected time for SPs (Zhang et al., 2022), and adapting EBPs to fit the school environment (Lyon & Bruns, 2019; Mychailyszyn et al., 2011). For a more rigorous understanding, increased attention to implementation science can help highlight the factors that are most suited to encouraging SPs delivery of EBPs in schools (Kilbourne et al., 2018; Lyon & Bruns, 2019; Owens et al., 2014).

Inner-Context Factors to Facilitate Adoption of Evidence-Based Practices

Of the various conditions for EBP adoption studied in implementation science, one topic that requires further study in school settings are “inner-context” implementation factors (Locke, Lawson, et al., 2019; Lyon & Bruns, 2019), as described in various implementation frameworks, including the Exploration, Preparation, Implementation, Sustainment (EPIS) framework (Aarons et al., 2011). Inner-context factors of organizations implementing EBPs, including provider attitudes and organizational strategic context, have been shown to be influential for EBP implementation (Aarons et al., 2009; Moullin et al., 2019; Sijercic et al., 2020). While these factors have been studied frequently in clinical healthcare settings, more studies are needed to understand their mechanisms when scaled out to non-specialty settings like schools (Aarons, Sklar, et al., 2017; Wolk et al., 2022). Moreover, focusing on the inner context is particularly salient in schools as inner-context factors are relatively malleable and modifiable, may be less constrained than outer-context factors like district-level leadership and policies, and may have more direct impacts on teachers and school staff (Forman & Barakat, 2011; Melgarejo et al., 2020). Recent studies have increasingly examined these inner-context factors in schools, including (as discussed below) developing and validating measurements for schools based on those originally developed in clinical settings.

Provider Attitudes About Evidence-Based Practices

Provider attitudes about EBPs (e.g., willingness to adopt EBPs and openness to change) can be predictive of their decision to use EBPs and can be an important mechanism for behavioral change (Aarons, 2004; Lyon et al., 2019; Sanetti et al., 2014). Studies have shown that school counselors’ involvement with pre-implementation strategies that aim to improve their EBP attitudes and perceptions, grounded in the Theory of Planned Behavior (Ajzen, 1991) and the Health Action Process Approach (Schwarzer, 2008), could lead to increased engagement in the early stages of implementation process (Larson et al., 2021; Locke, Lawson, et al., 2019; Lyon et al., 2019).

For linkages between provider attitudes and EBP uptake, the evidence has been strong in healthcare settings (Aarons et al., 2009; Becker-Haimes et al., 2017; Beidas et al., 2014; Farahnak et al., 2020), but mixed in school settings (Locke, Lawson, et al., 2019; Odom et al., 2022; Suhrheinrich et al., 2020). In the context of special education teachers and staff serving elementary-aged students with autism spectrum disorder, for example, one study found that positive attitudes toward EBPs were significantly associated with the intensity of use for only one of three EBPs examined (Locke, Lawson, et al., 2019). Another study found its positive association with EBP training satisfaction, but no significant relationships with intervention fidelity or with daily use (Suhrheinrich et al., 2020). These studies often used the Evidence-Based Practice Attitude Scale (EBPAS) to measure provider attitudes, which was developed for mental health providers (Aarons, 2004; Aarons et al., 2012) and psychometrically validated with providers in other domains, including school-based behavioral health consultants (Cook et al., 2018).

In terms of factors associated with EBPAS, studies with mental healthcare providers have found that more positive attitudes are associated with provider characteristics (e.g., higher educational attainment, fewer years of experience, and training in social work compared to psychology), and organizational characteristics, including private (vs. public) organizations/programs and constructive (vs. defensive) culture (Aarons, 2004; Aarons et al., 2009, 2010; Aarons & Sawitzky, 2006). In school settings, less-experienced school counselors and those using cognitive/behavioral theoretical orientation showed more positive attitudes (Mullen et al., 2018).

Strategic Organizational Factors to Implement Evidence-Based Practices

Another purported important inner-context factor for increased EBP uptake is strategic organizational contexts, including implementation climate and implementation leadership (Aarons et al., 2014, 2015; Aarons, Ehrhart, et al., 2017; Ehrhart et al., 2014). Implementation climate captures the shared perception of EBP adoption within the organization (Ehrhart et al., 2014; Jacobs et al., 2014; Weiner et al., 2011), which is often measured with the Implementation Climate Scale (ICS). The ICS was originally developed and validated in community mental health contexts (Ehrhart et al., 2014); initial validation work in education sectors suggests overall applicability, but one of its constructs, related to financial incentives and rewards for staff using EBPs, may not translate to schools (Locke, Lee, et al., 2019; Lyon et al., 2018). More recently, the school version of ICS, validated with teachers implementing a universal prevention program, suggested seven dimensions of ICS (Thayer et al., 2022). Implementation leadership focuses on strategic behaviors of first-level leaders (e.g., direct supervisors) to support and facilitate EBP implementation within their organizations (Aarons et al., 2014; Gifford et al., 2017; Guerrero et al., 2020). It is often measured using the Implementation Leadership Scale (ILS), which was originally developed and validated in the clinical setting (Aarons et al., 2014), and showed its applicability in schools with minor changes (Lyon et al., 2018). More recently, the seven-factor, school version of ILS was validated with elementary school teachers (Lyon et al., 2022).

Little is known about predictors of better or worse implementation climate or leadership in schools: some studies suggest that school demographics (such as racial composition, percentage of special education students, and school size) have a negligible effect on schools’ ability to support EBP implementation (Lyon et al., 2022; Thayer et al., 2022), while others suggest potential differences in the inner context due to school location or size (Moore et al., 2021; Suhrheinrich et al., 2021). In terms of linkages between strategic organizational factors and EBP implementation, studies have found mixed support in clinical settings (Meza et al., 2021; Powell et al., 2021) as well as in schools. Studies found no significant effect of these organizational factors on teachers’ decision to use EBPs for autism spectrum disorder (Locke, Lawson, et al., 2019; Melgarejo et al., 2020), while higher ICS was associated with teachers’ perceived feasibility of implementing an universal EBP for social, emotional, and behavioral skill building (Corbin et al., 2022). Given the complexity of EBP implementation in schools, as well as the broad range of SPs supporting students with mental health concerns, more studies are needed to explore how SP and school characteristics are associated with inner-context factors, and if (and how) these factors may help stimulate EBP implementation in schools (Locke, Lawson, et al., 2019; Lyon et al., 2018; Wolk et al., 2022).

Research Questions

This paper examined inner-context factors associated with SPs’ early signs of EBP implementation in schools in the “Preparation” phase of EPIS. It used the data collected during a 3-month, pre-randomization “run-in” phase of a type III hybrid implementation-effectiveness trial testing implementation strategies to support implementation of school-based cognitive behavioral therapy (CBT) in Michigan. Specifically, this paper aimed to answer the following research questions:

How did SPs rate their attitudes about EBP adoption, as well as their schools’ implementation climate and implementation leadership?

What school- and SP-level characteristics were associated with higher versus lower provider attitudes about EBPs and/or implementation climate/leadership?

Was higher SP support for EBPs or school implementation climate/leadership associated with early signs of EBP implementation, such as higher CBT knowledge, increased completion of training, or more pre-training CBT delivery?

This study furthers understanding of inner-context factors for EBP implementation in schools to improve youth mental health access.

Methods

The Evidence-Based Practice and Implementers

CBT is one mental health EBP with well-established evidence showing positive effects on youth with anxiety and depression (Haugland et al., 2020; Higa-McMillan et al., 2016; Spirito et al., 2011), across racial groups (Ginsburg & Drake, 2002; Rosselló & Bernal, 1999) and through modular or components-based programs (Becker et al., 2012; Chiu et al., 2013). Studies have shown that CBT can be effectively delivered by SPs in school settings (Ginsburg et al., 2020; Koschmann et al., 2019; Masia Warner et al., 2016; Sanchez et al., 2018). Core components of CBT include psychoeducation, relaxation, mindfulness, cognitive restructuring, behavioral activation, and exposure (Chorpita & Daleiden, 2009; Chorpita et al., 2005).

This study partnered with the Transforming Research into Action to Improve the Lives of Students program (TRAILS; https://trailstowellness.org). TRAILS aims to increase equitable student access to effective mental healthcare. It does this by providing training, resources, and implementation support to school staff who then offer mental health EBPs in schools. In the present study, TRAILS provided: CBT manuals adapted for SP needs, CBT trainings, and all implementation support to schools.

Data Source, Participants, and Procedures

This paper used data collected during the 3-month Preparation phase of the Adaptive School-based Implementation of CBT (ASIC) trial (Kilbourne et al., 2018). Figure 1 shows the data collection timeline for this period. The trial was approved by the University of Michigan Institutional Review Board and full study details and primary aim results are published elsewhere (Kilbourne et al., 2018; Smith et al., 2022). Further details on study recruitment and participant eligibility criteria are provided in Appendix A. Once consented, SPs began receiving support for implementing CBT via an implementation strategy informed by the Replicating Effective Programs (REP) framework (Kegeles et al., 2000; Kilbourne et al., 2007). This support, detailed in other publications (Kilbourne et al., 2018; Smith et al., 2022), included access to TRAILS web-based CBT resources (e.g., CBT manuals, worksheets, and other student resources), on-demand technical support from a clinical psychologist, and a 1-day didactic CBT training.

Figure 1

Data Collection Timeline During Preparation Phase

SPs were asked to complete a SP baseline survey for a $10 incentive any time prior to training, which included questions about demographics, attitudes toward EBP adoption, implementation climate and leadership, and CBT knowledge. SPs were also asked to begin using the study data collection “dashboard” to report (or “practice reporting”) weekly CBT delivery (Smith et al., 2022). SPs were incentivized $3 per weekly report and could provide up to 10 reports prior to the first training if registered at the earliest date.

The current paper used data from SPs who completed the SP baseline survey. Of the 227 SPs at 115 schools who initially consented, N = 198 SPs (87%) from k = 107 schools completed the survey (Appendix Figure A.1). Twenty-six schools (24%) had three SPs complete the survey, 39 (36%) had two SPs and 42 (39%) had one SP. Table 1 summarizes the basic characteristics of SPs and their schools; the majority of SPs had no graduate training or prior professional development in CBT.

Table 1

Participant Characteristics

School level (k = 107)	% (k)
School size: >500 students (vs. ≤500)	66.4% (71)
School free/reduced lunch rate: ≥50% students eligible	35.5% (38)
School location: non-rural (vs. rural)	43.9% (47)
SP level (N = 198)	% (n)/Mean (SD; range)
Professional role
School counselor	60.1% (119)
School social worker	22.2% (44)
Other^a	17.7% (35)
White	91.9% (182)
Female	86.4% (171)
Years in the current position	8.1 years (SD = 7.8; 0–45)
CBT training in graduate school
No graduate training	67.2% (133)
1–2 lectures on CBT as part of a larger course	29.8% (59)
Quarter/semester course exclusively on CBT	2.0% (4)
Didactic course on CBT with associated clinical supervision of one or more cases	0.5% (1)
Did not disclose	0.5% (1)
Professional development on CBT in the past 5 years (choose all that apply)
None	53.0% (105)
Brief presentation or workshop	25.3% (50)
Self-directed (e.g., online course, reading, YouTube videos)	25.3% (50)
1- to 2-day training	9.1% (18)
Mentioned in other classes/trainings	4.0% (8)
Training with follow-up consultation or supervision of a case	4.6% (9)

Note. SD = standard deviation; SP = school professional; CBT = cognitive behavioral therapy.

Other reported roles include school psychologist (n = 7), behavioral intervention specialist (n = 4), special education teacher (n = 4), general education teacher (n = 4), school success worker (n = 4), administrator (n = 5), school nurse (n = 1), wellness therapist/coordinator (n = 1), and other (n = 5).

Measures

Individual-Level Factors

Attitudes About Evidence-Based Practices

The 36-item EBPAS (Rye et al., 2017) was used to measure SP attitudes about EBPs. It captures the original four attitude domains (Requirement, Appeal, Openness, and Divergence; Aarons, 2004) and eight additional domains (Limitations, Fit, Monitoring, Balance, Burden, Job security, Organizational support, and Feedback). All items were rated on a 5-point Likert scale ranging from 0 (not at all) to 4 (very great extent); the scale showed adequate to excellent internal consistency (Cronbach's α for subscales and total: .60–.91; Rye et al., 2017).

SP Characteristics

SPs provided information on their current professional position, years of tenure in current position, prior CBT training, as well as gender and race.

Organizational-Level factors

Implementation Climate

The 18-item ICS was used to measure implementation climate (Ehrhart et al., 2014). The original validated ICS includes six subscales—Focus, Educational Support, Recognition, Rewards, Selection for EBP, and Selection for Openness, and items are rated on a 5-point Likert scale ranging from 0 (not at all) to 4 (very great extent). In school settings, the exclusion of ICS-Rewards was recommended due to the rarity of school-level financial incentives and promotions for EBP use; the five remaining subscales showed good to excellent internal consistency (Cronbach's α: .88–.97; Lyon et al., 2018). Accordingly, we computed total ICS excluding ICS-Rewards, and SP scores were aggregated to school level to reflect that implementation climate is an organizational-level construct.

Implementation Leadership

The 12-item ILS was used for implementation leadership (Aarons et al., 2014). It includes four subscales: Proactive, Knowledgeable, Supportive, and Perseverant leadership; items are rated on the same 5-point Likert scale as the ICS. The ILS was also validated in school settings, with excellent internal consistency (Cronbach's α: .94–.97; Lyon et al., 2018). SP scores were again aggregated to school level for implementation leadership.

School Characteristics

School characteristics, including school size, location (non-rural vs. rural), and free/reduced lunch rate, were derived from a state data source (e.g., MI School Data; https://www.mischooldata.org/collected).

Outcomes

To examine how the inner-context factors work as determinants of EBP implementation, we chose three variables capturing early signs of successful EBP implementation collected during the Preparation phase. While these variables are undoubtedly proximal, and success in even all three is not necessarily a harbinger of successful downstream implementation, they could be early indicators of mechanisms likely to spur successful implementation: EBP knowledge (objective CBT knowledge), engagement in the implementation process (completion of TRAILS' 1-day training), and any early use of the practice, or the intention or initial decision to employ an EBP (Proctor et al., 2011; pre-training CBT delivery). They are also likely correlated with other commonly documented early implementation outcomes, like acceptability, appropriateness, and/or feasibility (Proctor et al., 2011; Wolk et al., 2022).

Objective CBT Knowledge

The baseline survey included a battery of CBT objective knowledge questions developed by clinical experts on the TRAILS team (Rodriguez-Quintana et al., 2021). One open-ended item asked SPs to identify components of the CBT model, and 22 multiple-choice questions were asked to identify appropriate CBT components or CBT-based treatments based on six vignettes about youth experiencing depression and/or anxiety. We assessed the number of correct answers out of 23.

Completion of Training

SPs were asked to complete a daylong didactic CBT training to be deemed eligible for inclusion in the trial. Training was offered by TRAILS at no cost at various locations across Michigan. SPs were coded as completing the training or not.

Pre-Training CBT Delivery

Weekly CBT reports were used to construct a binary measure indicating whether SPs reported any versus no CBT delivery during the Preparation (pre-training) phase.

Data Analysis

First, SPs’ ICS and ILS scores were averaged within a school to compute school-aggregated scores, after assessing SP within-group agreement using an interrater agreement index (Brown & Hauenstein, 2005). Univariate analyses examined the distribution of SP-reported EBPAS and school-aggregated ICS and ILS scores; the correlations between these measures were assessed using Pearson correlation coefficients. Second, ordinary least squares regressions with clustered robust standard errors were conducted to explore whether each of these inner-context factors (EBPAS, ICS, and ILS) was associated with school- and SP-level characteristics: school size (>500 vs. ≤500 students), school location (non-rural vs. rural), free/reduced lunch rate (≥50% vs. <50% students eligible), SP role (school social worker, school counselor, or other), race (White vs. non-White), sex (female vs. male), tenure in years, and prior CBT training (formal, informal, or none). Third, linear and logistic regression models were used to examine whether and how each of EBPAS and school-aggregated ICS and ILS was associated with SPs’ early signs of implementation: CBT knowledge (range: 0–23), training completion (yes vs. no), and pre-training CBT delivery (yes vs. no). These models used robust standard errors to account for clustering within schools and included school- and SP-level characteristics discussed above as control variables. Average marginal effects (AMEs) were computed to show the average change in the outcomes (i.e., the predicted values for the continuous outcome and predicted probability for the binary outcome) for a given change in the independent variable (e.g., EBPAS, ICS, and ILS). Stata 16 was used for all analyses.

Results

Distribution of Inner-Context Factors

SP-Reported Attitudes About EBPs

The average SP-level EBPAS total score was 2.90 (SD = 0.39), on a scale ranging from 0 (not favorable) to 4 (highly favorable attitudes about EBP adoption; Figure 2). Cronbach's α for EBPAS total and subscales ranged from .52 to .97, with two subscales below .70 (Balance: .52, Divergence: .66; see Appendix Table B.1 for details of subscales). EBPAS subscales for Limitations (reverse-coded), Fit, and Appeal had the highest scores, at 3.58, 3.45, and 3.31, respectively; the lowest subscale score was for Job Security (1.48).

Figure 2

Distribution of EBPAS, ICS, and ILS

School-Aggregated Implementation Climate and Implementation Leadership Scores

For ICS and ILS total scores, school-aggregated scores were computed by averaging across SPs. The within-group agreement (a_wg(J)) was >.90 for subscale and total scores (with 1–3 raters). For the schools with three SPs (k = 26 schools), a_wg(J) was >.65 for ICS and >.70 for ILS. Mean scores of school-aggregated ICS and ILS were 1.83 (SD = 0.60) and 1.77 (SD = 0.76), respectively, on a scale ranging from 0 (not supportive) to 4 (very supportive); internal consistency was excellent (Table 2). ICS and ILS scores were also highly correlated (r = .77), and showed weak positive correlations with EBPAS (Table 2). In terms of ICS subscales, Focus on EBP (2.41) and Selection for Openness (2.38) had highest scores. As expected, ICS-Rewards had the lowest score by a wide margin, at 0.23; this subscale was excluded for computing the ICS total. Within the ILS, the Supportive Leadership subscale had the highest score (2.36) and Proactive Leadership the lowest score (1.30; Appendix Table B.1).

Table 2

EBPAS, ICS, and ILS Total Score Bivariate Correlations and Summary Statistics

	1	2	3	n	M	SD	Cronbach's α
1. EBPAS total	—	—	—	194	2.90	0.39	.877
2. ICS total	.176*	—	—	195	1.83	0.60	.938
3. ILS total	.141	.769***	—	194	1.77	0.76	.964

Note. EBPAS = Evidence-based Practice Attitudes Scale; ICS = school-aggregated Implementation Climate Scale; ILS = school-aggregated Implementation Leadership Scale; M = Mean; SD = standard deviation.

***p < .001. *p < .05.

School- and SP-Level Characteristics and Inner-Context Factors

The second research question explored if school- and SP-level characteristics were associated with each of inner-context factors. Results are shown in Table 3. Little support was found for associations between school characteristics (size, free/reduced lunch rate, or location) and any SP-reported EBPAS or school-aggregated ICS or ILS scores.

Table 3

Associations Between School- and SP-Level Characteristics and Inner-Context Factors

	EBPAS	ICS	ILS
School-level factors
School size: >500 students (vs. ≤500)	.005 [−.13, .14]	.005 [−.30, .31]	.09 [−.28, .46]
School free/reduced lunch rate: ≥50% students eligible (vs. <50%)	.04 [−.07, .15]	−.06 [−.28, .15]	−.09 [−.36, .19]
School location: non-rural (vs. rural)	−.03 [−.17, .10]	−.05 [−.30, .20]	−.12 [−.47, .23]
SP-level factors
SP: professional role
School social worker vs. School counselor	.24*** [.12, .36]	.09 [−.14, .31]	.11 [−.18, .40]
Other (school psychologist, etc.) vs. School counselor	.15 [−.04, .34]	.38** [.10, .65]	.45** [.16, .73]
SP: White	.09 [−.16, .33]	.24 [−.09, .57]	.26 [−.21, .73]
SP: female	.14 [−.06, .33]	−.21* [−.42, −.003]	−.38** [−.62, −.13]
SP: years in the current position	−.01* [−.01, −.001]	−.0004 [−.01, .01]	−.01 [−.02, .003]
SP: prior CBT training
Only informal training vs. none	.001 [−.13, .13]	.03 [−.17, .23]	.10 [−.17, .36]
Formal training vs. none	.07 [−.05, .20]	.16 [−.03, .35]	.001 [−.25, .25]
(Constant)	2.65*** [2.38, 2.93]	1.69*** [1.21, 2.18]	1.84*** [1.22, 2.47]
Model summary
Number of SPs (n)	194	195	194
Number of schools (k)	107	106	105
R²/Pseudo R²	.10	.09	.11
Model test	F(10, 106) = 2.56**	F(10, 105) = 2.33*	F(10, 104) = 3.83***

Note. Coefficients [95% confidence interval] from the OLS regression with clustered robust standard errors. EBPAS = Evidence-Based Practice Attitude Scale; ICS = school-aggregated Implementation Climate Scale; ILS = school-aggregated Implementation Leadership Scale; CBT = cognitive behavioral therapy; SP = school professional.

***p < .001. **p < .01. *p < .05.

EBPAS total scores, however, varied significantly by SP role and tenure. Specifically, school social workers reported higher EBPAS scores compared to school counselors, by 0.24 points on average (95% CI = [0.12, 0.36]; p < .001), all else equal. Newer SPs had higher EBPAS scores: scores decreased by 0.01 per additional year of experience (95% CI = [−0.01, –0.001]; p = .034), all else equal.

School-aggregated ICS and ILS total scores were associated with gender and professional role. ICS scores were lower for female SPs than male by 0.21 points (95% CI = [−0.42, –0.003]; p = .047); 0.38 points higher among SPs identified as working in “other” positions (e.g., school psychologists) compared to school counselors (95% CI = [0.10, 0.65]; p = .008), all else equal. Similar relationships were found for ILS, with lower scores for female SPs than male by 0.38 points (95% CI = [−0.62, –0.13]; p = .003) and higher scores among SPs in “other” positions (0.45 points higher compared to school counselors; 95% CI = [0.16, 0.73]; p = .003), all else equal.

Inner-Context Factors and Early Signs of EBP Implementation

Finally, we examined whether any inner-context factors were associated with outcomes reflecting early indication of successful implementation: CBT knowledge, training completion, and pre-training CBT delivery. These results are summarized below for each outcome (see Appendix Tables C.1–C.3 for full model results). Figures 3–5 present the average marginal effects of each inner-context factor on each outcome of interest.

Figure 3

Average Marginal Effects of EBPAS on Early Implementation Outcomes.

Figure 4

Predicted Probability of Early Implementation Indicators Over the ICS Total Scores

Figure 5

Predicted Probability of Early Implementation Indicators Over the ILS Total Scores.

Outcomes

SPs correctly answered a mean of 12 (of 23; 52%) CBT knowledge questions (SD = 4.21; Mdn = 12). For analyses, we standardized this variable to the mean with unit variance to improve interpretability. Of N = 198 SPs, 85% (n = 168) completed TRAILS' 1-day CBT training. Thirty-three percent (n = 65) reported delivering at least one individual or group CBT session during the Preparation phase; remaining SPs either reported no CBT delivery (60%; n = 120) or did not report (7%; n = 13) during this phase.

Individual Attitudes About EBPs and Outcomes

Higher scores on EBPAS were associated with more correct CBT knowledge answers and higher probability of completing training; they were not associated with pre-training CBT delivery. On average, one SD (centered) increase in EBPAS scores (approximately 0.39 points) increased CBT knowledge scores by 0.15 SD (about 0.6 questions) [95% CI: 0.003, 0.29], and changed the probability of completing training by 8 percentage points [95% CI: .03, .14] (Figure 3).

Organizational Implementation Climate/Leadership and Outcomes

Higher ICS and ILS scores were associated with higher probability of reporting pre-training CBT delivery and training completion. On average, one SD increase in ICS (approximately 0.60 points) increased the probability of reporting pre-training CBT delivery by 6 percentage points [95% CI: −.002, .12] (Figure 4). One SD increase in ILS (approximately 0.76 points), on average, increased the probability of training completion by 10 percentage points [95% CI: .05, .15] (Figure 5).

Discussion

This observational study used pre-randomization data from an implementation trial to explore the inner-context factors identified by frameworks like EPIS and their associations with both school- and SP-level factors and early signs of implementation success for an SP-delivered CBT program in Michigan high schools. We explored: (1) the distribution of inner-context factors for EBP implementation (EBPAS, ICS, and ILS); (2) the relationship between school- and SP-level characteristics and these inner-context factors; and (3) the associations between inner-context factors and early signs of implementation success. SP EBPAS scores were relatively higher and school-aggregated ICS and ILS total scores were relatively lower. School characteristics were not significantly associated with EBPAS, ICS, or ILS scores; but scores did differ by SP role. In regards to outcome variables, EBPAS scores were positively associated with CBT knowledge and training completion, whereas school-aggregated ILS scores were positively associated with training completion. ICS scores were positively associated with pre-training CBT delivery, but not statistically significant.

Inner-Context Factors for EBP Implementation in Schools

SPs reported higher average scores on EBPAS than those found in traditional clinical settings (Appendix Figure B.1 presented the distribution relative to other clinical domains), consistent with prior findings (Cook et al., 2018; Locke, Lawson, et al., 2019). With the exception of EBPAS-Job Security, the EBPAS subscale scores were generally high, indicating that SPs generally view EBP adoption positively, see EBP use as useful for their students, and are willing to learn and use them, although EBP may not contribute to their job security.

It is perhaps not surprising that SPs who chose to enroll in our study reported positive attitudes about adopting EBPs. However, echoing prior findings in education sectors (Appendix Figure B.1) (Locke, Lawson, et al., 2019; Lyon et al., 2018), our SPs rated organizational support for EBP implementation, via the ICS and ILS, lower than in clinical settings. This may reflect challenges in integrating mental health EBPs into schools, as SPs navigate competing priorities (Hoover & Bostic, 2021). Alternatively, it may be because some dimensions of implementation climate and leadership work differently in schools than in clinical settings due to, for example, differences in organizational structures between these settings (Locke, Lee, et al., 2019). Distribution of the subscales provides additional insights: some aspects of organizational support for EBPs may be more prevalent than other constructs in schools (Lyon et al., 2018). Relatively higher scores on ILS-Supportive, compared to ILS-Proactive, may suggest that school administrators and principals provide general support for implementation efforts, but do not necessarily anticipate barriers and solve problems in the implementation process. Relatively lower scores on ICS-Selection for EBPs and ICS-Rewards may reflect that schools rarely have much discretion in selecting/recognizing those who use EBPs or that SPs often are not clearly informed about the hiring process (Locke, Lee, et al., 2019; Lyon et al., 2018).

School- and SP-Level Characteristics and Inner-Context Factors

None of our school characteristics, including size, geography, or free/reduced lunch rate, saw significant associations with inner-context factors (Table 3). They also did not show significant relationships with early signs of EBP implementation (Appendix C). This may be edifying for implementation scientists and practitioners as it highlights that, even in the face of challenging, harder-to-change structural factors, implementation efforts focused on inner context can still be impactful. Furthermore, implementation scientists and practitioners should be careful not to make assumptions about schools’ readiness for implementing EBPs based on their external characteristics. Future studies should further explore if school characteristics may moderate how inner-context factors impact implementation.

At the SP level, EBPAS scores did vary across SP roles and tenure. Consistent with prior studies (Aarons et al., 2010), SPs newer to their professional position showed higher EBPAS scores; school social workers reported higher EBPAS scores compared to school counselors and other roles. It is unknown whether and how professional disciplines may affect provider attitudes (Aarons et al., 2010), but our findings suggest that school social workers show more flexibility and willingness to get feedback and try new practices that fit with their students’ needs.

School-aggregated ICS and ILS scores did vary by SP gender (female SPs had lower scores, on average) and role (school counselors reported the lowest scores, on average), all else equal. Interestingly, we also found that school social workers (vs. counselors) had higher CBT knowledge, holding ICS scores constant; female SPs had higher CBT knowledge, holding ILS scores constant. Future studies should explore how SPs’ role or previous trainings and/or experiences may affect their perceptions about organizational implementation support and downstream implementation success.

Inner-Context Factors and Early Signs of EBP Implementation in Schools

The third research question asked if SP support for EBPs or school implementation climate/leadership was associated with early indicators of implementation success, including CBT knowledge, training completion, and reported CBT delivery during the Preparation phase. Our findings partially support that both individual attitudes toward EBPs and organizational support for EBPs are positively associated with these early indicators. Other studies have similarly found mixed results when examining associations between inner-context factors and implementation outcomes (Locke, Lawson, et al., 2019; Odom et al., 2022).

Our results show that only SP attitudes toward EBPs, and not organizational support, had a significant association with CBT knowledge. Relative to other outcomes, however, this may not be surprising—as gaining knowledge in CBT is something that SPs are likely to have more ability to explore given an interest even without organizational support, time, or resources. It may also reflect a reverse-directional pathway: SPs with more CBT knowledge exhibit more supportive attitudes toward EBPs. This finding also supports prior research in clinical settings showing no significant association between ILS/ICS and CBT knowledge (Powell et al., 2017).

With respect to training completion, all else equal, SPs with more positive attitudes toward EBPs were also more likely to complete training, as were SPs from schools with leadership that were more aligned with EBP implementation. SP attendance at training did impose some burden. Although Continuing Education Units were offered and trainings were held in five locations across the state (and one virtual option), it was a full-day commitment that required travel and presumably administrative approval.

We did not find strong evidence to support linkages between inner-context factors and pre-training CBT delivery. ICS/ILS scores were positively associated with pre-training CBT delivery, but average increases were not statistically significant. Nonetheless, these findings may suggest that SPs in schools that were viewed as more supportive of EBP implementation took initiative to use CBT materials earlier on. However, it may also be the case that SPs reported more positive organizational support if they were already delivering some form of CBT. Future analyses will leverage the experimental aspects of this study to further investigate the role that implementation climate and leadership may play in spurring EBP implementation in schools.

Implications for Research and Practice

These findings overall suggest that, at least among those who participated in our implementation trial, SPs generally have supportive attitudes about EBP adoption, but lack clear organizational support, by way of leadership and climate, for spending the necessary time and resources to learn and deliver these practices. Again, this may reflect a selection effect of study participation, as perhaps SPs with strong organizational support may not have felt it necessary to enroll in a trial to receive further implementation support. Nonetheless, these findings speak to the gap between SPs and schools in terms of their support and interest in EBPs, suggesting that there remains work to be done to address implementation climate and leadership barriers to EBP implementation in schools. They also provide further evidence that provider-level EBP support and organizational support are distinct “inner-context” features and the latter are not epiphenomenal, but also raise important questions for the field as to how these concepts (co-) develop and interrelate, and how their importance might differ depending on intervention or implementation effort, particularly in school settings. Potential mechanisms and relationships among these inner-context factors need to be considered in future studies—for example, implementation leadership may contribute to developing implementation climate, which in turn influences provider attitudes and their decision to use EBPs (Williams et al., 2020, 2022). Moreover, our findings suggest that the identification of various correlates may help to inform pathways for modification and successful downstream implementation, while we generally accept that implementation leadership and climate are concepts that are both predictive of successful implementation and modifiable.

Limitations

This study had several limitations. First, the data used in these analyses were largely cross-sectional, collected at baseline of a multi-year study; therefore, no causal relationships can be inferred. Future studies using longitudinal data are needed to understand how organizational- and individual-level factors may influence EBP uptake, directly or indirectly. Second, SPs in this study agreed to participate in a trial to implement CBT; as such, they may not reflect the broader population of SPs/schools with respect to attitudes about EBPs or their relationship with early indicators of implementation success. Third, the ICS/ILS scores were school-level average ratings from only the SPs enrolled in the study. In school buildings, teachers, staff, and administrators may have different perceptions about organizational support of EBPs, which need to be further studied to fully understand the role of implementation climate and leadership in schools. Future studies could also use more recent versions of EBPAS/ICS/ILS measures validated in schools which were not yet available at the time of data collection (Lyon et al., 2022; Merle, 2021; Thayer et al., 2022). Fourth, our selected outcomes related to early indicators of implementation success are supported by theory and prior literature in implementation science broadly, but limited research has examined the extent to which they would be indicative of successful SP implementation in school settings. They also reflect some dimensions of implementation (e.g., acceptability and feasibility) but not others (e.g., fidelity), which several scholars have noted are key to mental health delivery in schools (Locke, Lawson, et al., 2019; Lyon et al., 2018). Future analyses from this study will evaluate fidelity of delivery and its association with inner-context factors.

Conclusion

This paper sought to understand inner-context factors, including attitudes about EBPs, implementation climate, and implementation leadership, associated with SPs’ school mental health EBP delivery. Our results highlight that SPs rated their schools’ ILS and ICS lower than providers in clinical settings, but their own attitudes, measured via the EBPAS, were higher. These inner-context factors were partially connected to our outcomes, which were selected to reflect early indications of implementation success. These findings highlight a need for further research on and efforts to strengthen individual and organizational support for EBP adoption in school settings as important mediators for improved EBP implementation. At the same time, our analyses did not find school size, location, or free/reduced lunch rate to be associated with higher EBPAS, ILS, or ICS scores, which highlights the malleability of these inner-context factors. More work is needed to understand how existing systems and networks can strengthen these individual and organizational attitudes toward, and ultimately implementation of, SP mental health delivery in schools.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work is supported by the National Institute of Mental Health (R01MH114203). Transforming research into action to improve the lives of students program (TRAILS) is supported by funds from the Centers for Medicare and Medicaid Services through the Michigan Department of Health and Human Services, the Michigan Health Endowment Fund, the Ethel and James Flinn Foundation, and other Michigan-based foundations.

ORCID iDs

Amy Rusch

Annalise Lane

Carolyn Andrews

Morgan Perry

Previous Presentation

This work has been presented at the Academy Health 2019 Annual Research Meeting, Washington, DC, 6/2019, the American Public Health Association 2019 Annual Meeting & Expo, Philadelphia, PA, 11/2019, and the 12th Annual Conference on the Science of Dissemination and Implementation in Health, Washington, DC, 12/2019.

Appendix A

Appendix B

Appendix C

References

Aarons

G. A.

(2004). Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research, 6(2), 61–74. https://doi.org/10.1023/B:MHSR.0000024351.12294.65

Aarons

G. A.

Cafri

Lugo

Sawitzky

(2012). Expanding the domains of attitudes towards evidence-based practice: The Evidence Based Practice Attitude Scale-50. Administration and Policy in Mental Health and Mental Health Services Research, 39(5), 331–340. https://doi.org/10.1007/s10488-010-0302-3

Aarons

G. A.

Ehrhart

M. G.

Farahnak

L. R.

(2014). The implementation leadership scale (ILS): Development of a brief measure of unit level implementation leadership. Implementation Science, 9(1), Article 45. https://doi.org/10.1186/1748-5908-9-45

Aarons

G. A.

Ehrhart

M. G.

Farahnak

L. R.

Hurlburt

M. S.

(2015). Leadership and organizational change for implementation (LOCI): A randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implementation Science, 10(1), Article 11. https://doi.org/10.1186/s13012-014-0192-y

Aarons

G. A.

Ehrhart

M. G.

Moullin

J. C.

Torres

E. M.

Green

A. E.

(2017). Testing the leadership and organizational change for implementation (LOCI) intervention in substance abuse treatment: A cluster randomized trial study protocol. Implementation Science, 12(1), Article 29. https://doi.org/10.1186/s13012-017-0562-3

Aarons

G. A.

Glisson

Hoagwood

Kelleher

Landsverk

Cafri

(2010). Psychometric properties and U.S. national norms of the Evidence-Based Practice Attitude Scale (EBPAS). Psychological Assessment, 22(2), 356–365. https://doi.org/10.1037/a0019188

Aarons

G. A.

Hurlburt

Horwitz

S. M.

(2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health, 38(1), 4–23. https://doi.org/10.1007/s10488-010-0327-7

Aarons

G. A.

Sawitzky

A. C.

(2006). Organizational culture and climate and mental health provider attitudes toward evidence-based practice. Psychological Services, 3(1), 61–72. https://doi.org/10.1037/1541-1559.3.1.61

Aarons

G. A.

Sklar

Mustanski

Benbow

Brown

C. H.

(2017). “Scaling-out” evidence-based interventions to new populations or new health care delivery systems. Implementation Science, 12(1), Article 111. https://doi.org/10.1186/s13012-017-0640-6

10.

Aarons

G. A.

Sommerfeld

D. H.

Walrath-Greene

C. M.

(2009). Evidence-based practice implementation: The impact of public versus private sector organization type on organizational support, provider attitudes, and adoption of evidence-based practice. Implementation Science, 4(1), Article 83. https://doi.org/10.1186/1748-5908-4-83

11.

Ajzen

(1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179–211. https://doi.org/10.1016/0749-5978(91)90020-T

12.

Ali

M. M.

West

Teich

J. L.

Lynch

Mutter

Dubenitz

(2019). Utilization of mental health services in educational setting by adolescents in the United States. The Journal of School Health, 89(5), 393–401. https://doi.org/10.1111/josh.12753

13.

Becker

E. M.

Becker

K. D.

Ginsburg

G. S.

(2012). Modular cognitive behavioral therapy for youth with anxiety disorders: A closer Look at the use of specific modules and their relation to treatment process and response. School Mental Health, 4(4), 243–253. https://doi.org/10.1007/s12310-012-9080-2

14.

Becker-Haimes

E. M.

Okamura

Wolk

C. B.

Rubin

Evans

A. C.

Beidas

R. S.

(2017). Predictors of clinician use of exposure therapy in community mental health settings. Journal of Anxiety Disorders, 49(1), 88–94. https://doi.org/10.1016/j.janxdis.2017.04.002

15.

Beidas

R. S.

Edmunds

Ditty

Watkins

Walsh

Marcus

Kendall

(2014). Are inner context factors related to implementation outcomes in cognitive-behavioral therapy for youth anxiety? Administration and Policy in Mental Health, 41(6), 788–799. https://doi.org/10.1007/s10488-013-0529-x

16.

Beidas

R. S.

Mychailyszyn

M. P.

Edmunds

J. M.

Khanna

M. S.

Downey

M. M.

Kendall

P. C.

(2012). Training school mental health providers to deliver cognitive-behavioral therapy. School Mental Health, 4(4), 197–206. https://doi.org/10.1007/s12310-012-9074-0

17.

Brown

R. D.

Hauenstein

N. M.

(2005). Interrater agreement reconsidered: An alternative to the rwg indices. Organizational Research Methods, 8(2), 165–184. https://doi.org/10.1177/1094428105275376

18.

Bruns

E. J.

Duong

M. T.

Lyon

A. R.

Pullmann

M. D.

Cook

C. R.

Cheney

McCauley

(2016). Fostering SMART partnerships to develop an effective continuum of behavioral health services and supports in schools. The American Journal of Orthopsychiatry, 86(2), 156–170. https://doi.org/10.1037/ort0000083

19.

Chiu

A. W.

Langer

D. A.

McLeod

B. D.

Har

Drahota

Galla

B. M.

Jacobs

Ifekwunigwe

Wood

J. J.

(2013). Effectiveness of modular CBT for child anxiety in elementary schools. School Psychology Quarterly, 28(2), 141–153. https://doi.org/10.1037/spq0000017

20.

Chorpita

B. F.

Daleiden

E. L.

(2009). Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology, 77(3), 566–579. https://doi.org/10.1037/a0014565

21.

Chorpita

B. F.

Daleiden

E. L.

Weisz

J. R.

(2005). Modularity in the design and application of therapeutic interventions. Applied and Preventive Psychology, 11(3), 141–156. https://doi.org/10.1016/j.appsy.2005.05.002

22.

Cook

C. R.

Davis

Brown

E. C.

Locke

Ehrhart

M. G.

Aarons

G. A.

Larson

Lyon

A. R.

(2018). Confirmatory factor analysis of the Evidence-Based Practice Attitudes Scale with school-based behavioral health consultants. Implementation Science, 13(1), Article 116. https://doi.org/10.1186/s13012-018-0804-z

23.

Corbin

C. M.

Hugh

M. L.

Ehrhart

M. G.

Locke

Davis

Brown

E. C.

Cook

C. R.

Lyon

A. R.

(2022). Teacher perceptions of implementation climate related to feasibility of implementing schoolwide positive behavior supports and interventions. School Mental Health, 14(4), 1057–1069. https://doi.org/10.1007/s12310-022-09528-z

24.

Corteselli

K. A.

Hollinsaid

N. L.

Harmon

S. L.

Bonadio

F. T.

Westine

Weisz

J. R.

Price

M. A.

(2020). School counselor perspectives on implementing a modular treatment for youth. Evidence-Based Practice in Child and Adolescent Mental Health, 5(3), 271–287. https://doi.org/10.1080/23794925.2020.1765434

25.

Costello

E. J.

Sampson

N. A.

Kessler

R. C.

Merikangas

K. R.

(2014). Services for adolescents with psychiatric disorders: 12-month data from the national comorbidity survey–adolescent. Psychiatric Services, 65(3), 359–366. https://doi.org/10.1176/appi.ps.201100518

26.

Duong

M. T.

Bruns

E. J.

Lee

Cox

Coifman

Mayworm

Lyon

A. R.

(2020). Rates of mental health service utilization by children and adolescents in schools and other common service settings: A systematic review and meta-analysis. Administration and Policy in Mental Health and Mental Health Services Research, 48(3):420–439. https://doi.org/10.1007/s10488-020-01080-9

27.

Ehrhart

M. G.

Aarons

G. A.

Farahnak

L. R.

(2014). Assessing the organizational context for EBP implementation: The development and validity testing of the Implementation Climate Scale (ICS). Implementation Science, 9(1), Article 157. https://doi.org/10.1186/s13012-014-0157-1

28.

Eiraldi

Khanna

Jawad

A. F.

Power

T. J.

Cacia

Cabello

Schwartz

B. S.

Swift

Kanine

Orapallo

McCurdy

Mautone

J. A.

(2020). Implementation of targeted mental health interventions in urban schools: Preliminary findings for impact of training strategy on program fidelity. Evidence-Based Practice in Child and Adolescent Mental Health, 5(4), 437–451. https://doi.org/10.1080/23794925.2020.1784056

29.

Farahnak

L. R.

Ehrhart

M. G.

Torres

E. M.

Aarons

G. A.

(2020). The influence of transformational leadership and leader attitudes on subordinate attitudes and implementation success. Journal of Leadership & Organizational Studies, 27(1), 98–111. https://doi.org/10.1177/1548051818824529

30.

Findling

R. L.

Stepanova

(2018). The workforce shortage of child and adolescent psychiatrists: Is it time for a different approach? Journal of the American Academy of Child and Adolescent Psychiatry, 57(5), 300–301. https://doi.org/10.1016/j.jaac.2018.02.008

31.

Forman

S. G.

Barakat

N. M.

(2011). Cognitive-behavioral therapy in the schools: Bringing research to practice through effective implementation. Psychology in the Schools, 48(3), 283–296. https://doi.org/10.1002/pits.20547

32.

Gifford

Graham

I. D.

Ehrhart

M. G.

Davies

B. L.

Aarons

G. A.

(2017). Ottawa Model of Implementation Leadership and Implementation Leadership Scale: Mapping concepts for developing and evaluating theory-based leadership interventions. Journal of Healthcare Leadership, 9(1), 15–23. https://doi.org/10.2147/JHL.S125558

33.

Ginsburg

G. S.

Drake

K. L.

(2002). School-based treatment for anxious African-American adolescents: A controlled pilot study. Journal of the American Academy of Child & Adolescent Psychiatry, 41(7), 768–775. https://doi.org/10.1097/00004583-200207000-00007

34.

Ginsburg

G. S.

Pella

J. E.

Pikulski

P. J.

Tein

J.-Y.

Drake

K. L.

(2020). School-Based Treatment for Anxiety Research Study (STARS): A randomized controlled effectiveness trial. Journal of Abnormal Child Psychology, 48(3), 407–417. https://doi.org/10.1007/s10802-019-00596-5

35.

Guerrero

E. G.

Frimpong

Kong

Fenwick

Aarons

G. A.

(2020). Advancing theory on the multilevel role of leadership in the implementation of evidence-based health care practices. Health Care Management Review, 45(2), 151–161. https://doi.org/10.1097/HMR.0000000000000213

36.

Haugland

B. S. M.

Haaland

Å. T.

Baste

Bjaastad

J. F.

Hoffart

Rapee

R. M.

Raknes

Himle

J. A.

Husabø

Wergeland

G. J.

(2020). Effectiveness of brief and standard school-based cognitive-behavioral interventions for adolescents with anxiety: A randomized noninferiority study. Journal of the American Academy of Child & Adolescent Psychiatry, 59(4), 552–564.e2. https://doi.org/10.1016/j.jaac.2019.12.003

37.

Herzig-Anderson

Colognori

Fox

J. K.

Stewart

C. E.

Warner

C. M.

(2012). School-based anxiety treatments for children and adolescents. Child and Adolescent Psychiatric Clinics of North America, 21(3), 655–668. https://doi.org/10.1016/j.chc.2012.05.006

38.

Higa-McMillan

C. K.

Francis

S. E.

Rith-Najarian

Chorpita

B. F.

(2016). Evidence base update: 50 years of research on treatment for child and adolescent anxiety. Journal of Clinical Child & Adolescent Psychology, 45(2), 91–113. https://doi.org/10.1080/15374416.2015.1046177

39.

Hoover

Bostic

(2021). Schools as a vital component of the child and adolescent mental health system. Psychiatric Services, 72(1), 37–48. https://doi.org/10.1176/appi.ps.201900575

40.

Jacobs

S. R.

Weiner

B. J.

Bunger

A. C.

(2014). Context matters: Measuring implementation climate among individuals and groups. Implementation Science, 9(1), Article 46. https://doi.org/10.1186/1748-5908-9-46

41.

Kegeles

S. M.

Rebchook

G. M.

Hays

R. B.

Terry

M. A.

O’Donnell

Leonard

N. R.

Kelly

J. A.

Neumann

M. S.

(2000). From science to application: The development of an intervention package. AIDS Education and Prevention, 12(5 Suppl), 62–74.

42.

Kern

Mathur

S. R.

Albrecht

S. F.

Poland

Rozalski

Skiba

R. J.

(2017). The need for school-based mental health services and recommendations for implementation. School Mental Health, 9(3), 205–217. https://doi.org/10.1007/s12310-017-9216-5

43.

Kilbourne

A. M.

Neumann

M. S.

Pincus

H. A.

Bauer

M. S.

Stall

(2007). Implementing evidence-based interventions in health care: Application of the replicating effective programs framework. Implementation Science, 2(1), Article 42. https://doi.org/10.1186/1748-5908-2-42

44.

Kilbourne

A. M.

Smith

S. N.

Choi

S. Y.

Koschmann

Liebrecht

Rusch

Abelson

J. L.

Eisenberg

Himle

J. A.

Fitzgerald

Almirall

(2018). Adaptive school-based implementation of CBT (ASIC): Clustered-SMART for building an optimized adaptive implementation intervention to improve uptake of mental health interventions in schools. Implementation Science, 13(1), Article 119. https://doi.org/10.1186/s13012-018-0808-8

45.

Koschmann

Abelson

J. L.

Kilbourne

A. M.

Smith

S. N.

Fitzgerald

Pasternak

(2019). Implementing evidence-based mental health practices in schools: Feasibility of a coaching strategy. The Journal of Mental Health Training, Education and Practice, 14(4), 212–231. https://doi.org/10.1108/JMHTEP-05-2018-0028

46.

Langley

A. K.

Nadeem

Kataoka

S. H.

Stein

B. D.

Jaycox

L. H.

(2010). Evidence-based mental health programs in schools: Barriers and facilitators of successful implementation. School Mental Health, 2(3), 105–113. https://doi.org/10.1007/s12310-010-9038-1

47.

Larson

E. H.

Patterson

D. G.

Garberson

L. A.

Andrilla

C. H. A.

(2016). Supply and distribution of the behavioral health workforce in rural America [Data Brief #160]. WWAMI Rural Health Research Center, University of Washington.

48.

Locke

Kang-Yi

Frederick

Mandell

D. S.

(2020). Individual and organizational characteristics predicting intervention use for children with autism in schools. Autism, 24(5), 1152–1163. https://doi.org/10.1177/1362361319895923

49.

Locke

Lawson

G. M.

Beidas

R. S.

Aarons

G. A.

Xie

Lyon

A. R.

Stahmer

Seidman

Frederick

Spaulding

Dorsey

Mandell

D. S.

(2019). Individual and organizational factors that affect implementation of evidence-based practices for children with autism in public schools: A cross-sectional observational study. Implementation Science, 14(1), Article 29. https://doi.org/10.1186/s13012-019-0877-3

50.

Locke

Lee

Cook

C. R.

Frederick

Vázquez-Colón

Ehrhart

M. G.

Aarons

G. A.

Davis

Lyon

A. R.

(2019). Understanding the organizational implementation context of schools: A qualitative study of school district administrators, principals, and teachers. School Mental Health, 11(3), 379–399. https://doi.org/10.1007/s12310-018-9292-1

51.

Lyon

A. R.

Bruns

E. J.

(2019). From evidence to impact: Joining our best school mental health practices with our best implementation strategies. School Mental Health, 11(1), 106–114. https://doi.org/10.1007/s12310-018-09306-w

52.

Lyon

A. R.

Cook

C. R.

Brown

E. C.

Locke

Davis

Ehrhart

Aarons

G. A.

(2018). Assessing organizational implementation context in the education sector: Confirmatory factor analysis of measures of implementation leadership, climate, and citizenship. Implementation Science, 13(1), Article 5. https://doi.org/10.1186/s13012-017-0705-6

53.

Lyon

A. R.

Cook

C. R.

Duong

M. T.

Nicodimos

Pullmann

M. D.

Brewer

S. K.

Gaias

L. M.

Cox

(2019). The influence of a blended, theoretically-informed pre-implementation strategy on school-based clinician implementation of an evidence-based trauma intervention. Implementation Science, 14(1), Article 54. https://doi.org/10.1186/s13012-019-0905-3

54.

Lyon

A. R.

Corbin

C. M.

Brown

E. C.

Ehrhart

M. G.

Locke

Davis

Picozzi

Aarons

G. A.

Cook

C. R.

(2022). Leading the charge in the education sector: Development and validation of the School Implementation Leadership Scale (SILS). Implementation Science, 17(1), Article 48. https://doi.org/10.1186/s13012-022-01222-7

55.

Masia Warner

Colognori

Brice

Herzig

Mufson

Lynch

Reiss

P. T.

Petkova

Fox

Moceri

D. C.

Ryan

Klein

R. G.

(2016). Can school counselors deliver cognitive-behavioral treatment for social anxiety effectively? A randomized controlled trial. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 57(11), 1229–1238. https://doi.org/10.1111/jcpp.12550

56.

Melgarejo

Lind

Stadnick

N. A.

Helm

J. L.

Locke

(2020). Strengthening capacity for implementation of evidence-based practices for autism in schools: The roles of implementation climate, school leadership and fidelity. The American Psychologist, 75(8), 1105–1115. https://doi.org/10.1037/amp0000649

57.

Merle

J. L.

(2021). Teacher attitudes toward evidence-based practices: Confirmatory and predictive analyses of the school-adapted evidence-based practice attitude scale [Doctoral dissertation]. University of Minnesota. http://conservancy.umn.edu/handle/11299/224575

58.

Meza

R. D.

Triplett

N. S.

Woodard

G. S.

Martin

Khairuzzaman

A. N.

Jamora

Dorsey

(2021). The relationship between first-level leadership and inner-context and implementation outcomes in behavioral health: A scoping review. Implementation Science, 16(1), Article 69. https://doi.org/10.1186/s13012-021-01104-4

59.

Moore

S. A.

Landa

Azad

(2021). Organizational context in general and special education: An exploratory investigation to describe the perspective of school leaders. Global Implementation Research and Applications, 1(4), 233–245. https://doi.org/10.1007/s43477-021-00023-2

60.

Moullin

J. C.

Dickson

K. S.

Stadnick

N. A.

Rabin

Aarons

G. A.

(2019). Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implementation Science, 14(1), Article 1. https://doi.org/10.1186/s13012-018-0842-6

61.

Mullen

P. R.

Stevens

Chae

(2018). School Counselors’ attitudes toward evidence-based practices. Professional School Counseling, 22(1), 1–11. https://doi.org/10.1177/2156759X18823690

62.

Mychailyszyn

M. P.

Beidas

R. S.

Benjamin

C. L.

Edmunds

J. M.

Podell

J. L.

Cohen

J. S.

Kendall

P. C.

(2011). Assessing and treating child anxiety in schools: Assessing and treating anxiety in schools. Psychology in the Schools, 48(3), 223–232. https://doi.org/10.1002/pits.20548

63.

Nadeem

Gleacher

Beidas

R. S.

(2013). Consultation as an implementation strategy for evidence-based practices across multiple contexts: Unpacking the black box. Administration and Policy in Mental Health, 40(6), 439–450. https://doi.org/10.1007/s10488-013-0502-8

64.

Odom

S. L.

Sam

A. M.

Tomaszewski

(2022). Factors associated with implementation of a school-based comprehensive program for students with autism. Autism, 26(3), 703–715. https://doi.org/10.1177/13623613211070340

65.

O’Reilly

Svirydzenka

Adams

Dogra

(2018). Review of mental health promotion interventions in schools. Social Psychiatry and Psychiatric Epidemiology, 53(7), 647–662. https://doi.org/10.1007/s00127-018-1530-1

66.

Owens

J. S.

Lyon

A. R.

Brandt

N. E.

Warner

C. M.

Nadeem

Spiel

Wagner

(2014). Implementation science in school mental health: Key constructs in a developing research agenda. School Mental Health, 6(2), 99–111. https://doi.org/10.1007/s12310-013-9115-3

67.

Powell

B. J.

Mandell

D. S.

Hadley

T. R.

Rubin

R. M.

Evans

A. C.

Hurford

M. O.

Beidas

R. S.

(2017). Are general and strategic measures of organizational context and leadership associated with knowledge and attitudes toward evidence-based practices in public behavioral health settings? A cross-sectional observational study. Implementation Science, 12(1), Article 64. https://doi.org/10.1186/s13012-017-0593-9

68.

Powell

B. J.

Mettert

K. D.

Dorsey

C. N.

Weiner

B. J.

Stanick

C. F.

Lengnick-Hall

Ehrhart

M. G.

Aarons

G. A.

Barwick

M. A.

Damschroder

L. J.

Lewis

C. C.

(2021). Measures of organizational culture, organizational climate, and implementation climate in behavioral health: A systematic review. Implementation Research and Practice, 2(Jan–Dec 2021), 1–29. https://doi.org/10.1177/26334895211018862

69.

Proctor

Silmere

Raghavan

Hovmand

Aarons

Bunger

Griffey

Hensley

(2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7

70.

Radez

Reardon

Creswell

Lawrence

P. J.

Evdoka-Burton

Waite

(2021). Why do children and adolescents (not) seek and access professional help for their mental health problems? A systematic review of quantitative and qualitative studies. European Child & Adolescent Psychiatry, 30(2), 183–211. https://doi.org/10.1007/s00787-019-01469-4

71.

Reardon

Harvey

Baranowska

O’Brien

Smith

Creswell

(2017). What do parents perceive are the barriers and facilitators to accessing psychological treatment for mental health problems in children and adolescents? A systematic review of qualitative and quantitative studies. European Child & Adolescent Psychiatry, 26(6), 623–647. https://doi.org/10.1007/s00787-016-0930-6

72.

Ringle

V. A.

Read

K. L.

Edmunds

J. M.

Brodman

D. M.

Kendall

P. C.

Barg

Beidas

R. S.

(2015). Barriers to and facilitators in the implementation of cognitive-behavioral therapy for youth anxiety in the community. Psychiatric Services, 66(9), 938–945. https://doi.org/10.1176/appi.ps.201400134

73.

Rodriguez-Quintana

Choi

S. Y.

Bilek

Koschmann

Albrecht

Prisbe

Smith

(2021). The Cognitive Behavioral Therapy Competence Scale (CCS): Initial development and validation. Cognitive Behaviour Therapist, 14(1), Article e39. https://doi.org/10.1017/S1754470X21000362

74.

Rosselló

Bernal

(1999). The efficacy of cognitive-behavioral and interpersonal treatments for depression in Puerto Rican adolescents. Journal of Consulting and Clinical Psychology, 67(5), 734–745. https://doi.org/10.1037//0022-006x.67.5.734

75.

Rye

Torres

E. M.

Friborg

Skre

Aarons

G. A.

(2017). The Evidence-Based Practice Attitude Scale-36 (EBPAS-36): A brief and pragmatic measure of attitudes to evidence-based practice validated in US and Norwegian samples. Implementation Science, 12(1), Article 44. https://doi.org/10.1186/s13012-017-0573-0

76.

SAMHSA. (2021). Key substance use and mental health indicators in the United States: Results from the 2020 National Survey on Drug Use and Health (HHS Publication No. PEP21-07-01-003, NSDUH Series H-56). Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration (SAMHSA). https://www.samhsa.gov/data/sites/default/files/reports/rpt35325/NSDUHFFRPDFWHTMLFiles2020/2020NSDUHFFR1PDFW102121.pdf

77.

Sanchez

A. L.

Cornacchio

Poznanski

Golik

A. M.

Chou

Comer

J. S.

(2018). The effectiveness of school-based mental health services for elementary-aged children: A meta-analysis. Journal of the American Academy of Child and Adolescent Psychiatry, 57(3), 153–165. https://doi.org/10.1016/j.jaac.2017.11.022

78.

Sanetti

L. M. H.

Collier-Meek

M. A.

Long

A. C. J.

Kim

Kratochwill

T. R.

(2014). Using implementation planning to increase teachers’ adherence and quality to behavior support plans: Implementation planning. Psychology in the Schools, 51(8), 879–895. https://doi.org/10.1002/pits.21787

79.

Schwarzer

(2008). Modeling health behavior change: How to predict and modify the adoption and maintenance of health behaviors. Applied Psychology: An International Review, 57(1), 1–29. https://doi.org/10.1111/j.1464-0597.2007.00325.x

80.

Sijercic

Lane

J. E. M.

Gutner

C. A.

Monson

C. M.

Stirman

S. W.

(2020). The association between clinician and perceived organizational factors with early fidelity to cognitive processing therapy for posttraumatic stress disorder in a randomized controlled implementation trial. Administration and Policy in Mental Health and Mental Health Services Research, 47(1), 8–18. https://doi.org/10.1007/s10488-019-00966-7

81.

Smith

S. N.

Almirall

Choi

S. Y.

Koschmann

Rusch

Bilek

Lane

Abelson

J. L.

Eisenberg

Himle

J. A.

Fitzgerald

K. D.

Liebrecht

Kilbourne

A. M.

(2022). Primary aim results of a clustered SMART for developing a school-level, adaptive implementation strategy to support CBT delivery at high schools in Michigan. Implementation Science, 17(1), Article 42. https://doi.org/10.1186/s13012-022-01211-w

82.

Spirito

Esposito-Smythers

Wolff

Uhl

(2011). Cognitive-behavioral therapy for adolescent depression and suicidality. Child and Adolescent Psychiatric Clinics of North America, 20(2), 191–204. https://doi.org/10.1016/j.chc.2011.01.012

83.

Stempel

Cox-Martin

M. G.

O’Leary

Stein

Allison

M. A.

(2019). Students seeking mental health services at school–based health centers: Characteristics and utilization patterns. Journal of School Health, 89(10), 839–846. https://doi.org/10.1111/josh.12823

84.

Suhrheinrich

Melgarejo

Root

Aarons

G. A.

Brookman-Frazee

(2021). Implementation of school-based services for students with autism: Barriers and facilitators across urban and rural districts and phases of implementation. Autism, 25(8), 2291–2304. https://doi.org/10.1177/13623613211016729

85.

Suhrheinrich

Rieth

S. R.

Dickson

K. S.

Stahmer

A. C.

(2020). Exploring associations between inner-context factors and implementation outcomes. Exceptional Children, 86(2), 155–173. https://doi.org/10.1177/0014402919881354

86.

Thayer

A. J.

Cook

C. R.

Davis

Brown

E. C.

Locke

Ehrhart

M. G.

Aarons

G. A.

Picozzi

Lyon

A. R.

(2022). Construct validity of the school-implementation climate scale. Implementation Research and Practice, 3(1), Article 26334895221116065. https://doi.org/10.1177/26334895221116065

87.

Twenge

J. M.

Cooper

A. B.

Joiner

T. E.

Duffy

M. E.

Binau

S. G.

(2019). Age, period, and cohort trends in mood disorder indicators and suicide-related outcomes in a nationally representative dataset, 2005–2017. Journal of Abnormal Psychology, 128(3), 185–199. https://doi.org/10.1037/abn0000410

88.

Weiner

B. J.

Belden

C. M.

Bergmire

D. M.

Johnston

(2011). The meaning and measurement of implementation climate. Implementation Science, 6(1), Article 78. https://doi.org/10.1186/1748-5908-6-78

89.

Williams

N. J.

Hugh

M. L.

Cooney

D. J.

Worley

J. A.

Locke

(2022). Testing a theory of implementation leadership and climate across autism evidence-based interventions of varying complexity. Behavior Therapy, 53(5), 900–912. https://doi.org/10.1016/j.beth.2022.03.001

90.

Williams

N. J.

Wolk

C. B.

Becker-Haimes

E. M.

Beidas

R. S.

(2020). Testing a theory of strategic implementation leadership, implementation climate, and clinicians’ use of evidence-based practice: A 5-year panel analysis. Implementation Science, 15(1), Article 10. https://doi.org/10.1186/s13012-020-0970-7

91.

Wolk

C. B.

Arnold

K. T.

Proctor

E. K.

(2022). Implementing evidence-based practices in nonspecialty mental health settings. Families, Systems & Health, 40(2), 274–282. https://doi.org/10.1037/fsh0000506

92.

Zhang

Cook

C. R.

Lyon

A. R.

(2022). A simple matter of time? School-level analysis of the relationship between time allocation, treatment integrity, and student outcome. School Mental Health, 14(1), 73–87. https://doi.org/10.1007/s12310-021-09412-2