Sage Journals: Discover world-class research

Abstract

Background: A national shortage of mental health providers for youth exists in the United States. Implementation support for mental health services in schools, where students are most likely to access care, can help to fill these gaps. Coaching consists of in vivo modeling and support during service delivery and is effective in supporting the implementation of evidence-based practices (EBPs). This implementation report describes the recruitment and training of community providers to become coaches as a part of a modified train-the-trainer model of implementation support. Method: An EBP implementation program, Transforming Research into Action to Improve the Lives of Students (TRAILS), trained community providers in Michigan to increase knowledge of cognitive behavioral therapy (CBT) and position them as coaches in schools. The development of the coach network involved five stages prior to the initiation of a randomized controlled trial: (1) recruitment, (2) a one-day clinical training, (3) 12 weeks of individualized consultation, (4) evaluation, and (5) training in the coaching protocol. Results: A total of 347 individuals attended an initial training, and 187 were paired with a consultant. Eighty-six clinicians from 47 of Michigan's 83 counties successfully became coaches by completing all required elements of training. Coaches showed significant improvements in the use and knowledge of CBT across consultation. Conclusion: Statewide networks of trained mental health professionals can address gaps in mental healthcare for youth. This article demonstrates one strategy for strengthening and leveraging community expertise to support the implementation of EPBs in schools.

Plain Language Summary

Although youth face many barriers in accessing effective mental health care, schools are one setting where they can more easily receive treatment. Research shows that training and supports are needed for school mental health professionals to provide effective care. Coaching, which involves in vivo support for school mental health professionals, is one helpful strategy. We describe the process of recruiting and training community clinicians to become coaches. The Transforming Research into Action to Improve the Lives of Students (TRAILS) program successfully recruited and trained a network of 86 community clinicians to become coaches. Clinicians attended two day-long trainings and participated in 12 weeks of personalized consultation. Clients to whom these clinicians provided CBT showed significant symptom improvement. Clinicians also reported that they increased their knowledge and use of core CBT strategies, including psychoeducation, exposure, and behavioral activation. Community mental health professionals who are trained as coaches can address gaps in access to care for youth. We demonstrate one strategy for strengthening and leveraging clinicians’ knowledge of CBT to support the school-based implementation of CBT.

Keywords

Consultation coaching training school-based intervention cognitive behavioral therapy

Background

About one in four youth will experience a severe mental health disorder in their lifetime (Merikangas et al., 2010), and most will not receive any mental health services (Merikangas et al., 2011). Although evidence-based practices (EBPs) are effective in treating a variety of mental health concerns, a research-to-practice gap exists between the development of EBPs and their use in routine-care settings (Jensen & Foster, 2010; Weisz et al., 2017). Many barriers, including the lack of adequate training and resources for providers, limit EBP implementation (Meyer et al., 2020; Ringle et al., 2015). Furthermore, access to mental healthcare by youth varies across geographic contexts and sociodemographic groups, with those living in rural areas and minoritized youth experiencing lower access and greater barriers to care (Cook et al., 2016; Thomas et al., 2009).

Among youth who do access mental health care, many (35%–56%) do so only in schools (Ali et al., 2019; Green et al., 2013). Access to care and thus outcomes for youth could be substantially improved if school mental health professionals (SMHPs) were trained and equipped to deliver EBPs (Lyon et al., 2011; Warner et al., 2016). Coaching, an implementation strategy that provides in vivo training and support, is shown to improve EBP implementation and fidelity (Becker et al., 2013; Cappella et al., 2012). Furthermore, coaching is feasible and acceptable in school-based health clinics (Koschmann et al., 2019). This article describes the development of a statewide coach network, a modified train-the-trainer model, to support downstream EBP implementation in schools. Specifically, the coach network consisted of community clinicians trained in both EBPs and providing implementation support (coaching) to school-based mental health professionals. It was developed to increase access to evidence-based mental health treatment for youth via two pathways: through (1) direct care provision in community settings and (2) supporting the implementation of EBPs in schools.

Evidence-Based Training Approaches to Support EBP Implementation in Schools

Trainings for mental health providers in EBPs frequently take the form of didactic workshops that lack follow-up consultation, feedback, or supervision (Frank et al., 2020; Herschell et al., 2010). Such trainings alone are insufficient for sustained EBP implementation (Beidas & Kendall, 2010). Although these one-time trainings can increase clinicians’ knowledge and self-efficacy, they do not substantively impact clinician behaviors, EBP adherence, or clinical outcomes (Beidas et al., 2012; McHugh & Barlow, 2010; Scott et al., 2016). However, approaches that involve active learning and ongoing implementation supports, such as expert consultation and coaching, have been associated with higher EBP adherence and skill (Beidas et al., 2012; Powell et al., 2015). Frank et al. (2020) reviewed current training approaches and found support for the effectiveness of intensive trainings, specifically when combined with ongoing implementation support.

Coaching shares some elements with consultation—such as guided practice and expert feedback. The primary difference is that coaching is typically delivered in the natural environment during the live delivery of treatment sessions, whereas consultation typically occurs at a separate time and as a separate activity from service delivery (Koschmann et al., 2019). The advantages of coaching make it an implementation support particularly well-suited for school settings. School-based mental health providers carry large caseloads and have extremely limited time (McCarthy et al., 2010; Mullen et al., 2021), impeding their capacity to train in and practice new techniques. Coaching provides “‘active practice’ with real cases and their inherent complexities” and also limits the need for additional time and effort from the primary provider beyond the clinical service being delivered (Koschmann et al., 2019, p. 215). This approach allows school staff to actively learn and practice new skills in the context of service delivery, a crucial element to learning (Beidas & Kendall, 2010). Given these benefits, implementation support in schools has recently favored a greater focus on in vivo coaching (Schultz et al., 2015) and coaching has been classified as an implementation support that warrants further research (Novins et al., 2013).

The feasibility (Koschmann et al., 2019) and promise (Novins et al., 2013) of coaching to promote school professionals’ use of EBPs suggests that coaching may be one pipeline to address the vast unmet need for youth mental health services. Pragmatically, coaches must be geographically proximal to training sites, in order to be physically present where the services are being provided, and also must have expertise in the clinical intervention. Given the shortage of mental health professionals serving youth nationwide, and especially in rural areas (Thomas et al., 2009), one approach is to train local community clinicians in EBPs to serve as coaches in local schools, an adaptation to the train-the-trainer model (Yarber et al., 2015).

We describe the development of a statewide coach network of community clinicians trained to provide implementation support to school-based clinicians. Expanding beyond university-affiliated coaches was important for the program's capacity, feasibility, and sustainability, as well as covering a wider geographic range. The network was developed in anticipation of an NIMH-funded sequential multiple-assignment trial randomized at the school level —Adaptive School-Based Implementation of cognitive behavioral therapy (CBT) (ASIC; Kilbourne et al., 2018). This trial sought to evaluate the effectiveness of an adaptive implementation intervention for CBT for school-age youth delivered by school staff. Herein, we describe the development of the associated coach network by (1) describing the coach recruitment and training process and (2) reporting on key outcomes related to training (i.e., change in CBT competence), implementation (e.g., placement of coaches across counties in the state), and clinical indicators (i.e., depression and anxiety symptoms for practice clients).

Method

Transforming Research into Action to Improve the Lives of Students (TRAILS) is an implementation program designed to increase sustainable youth access to EBPs, specifically CBT, in school settings. TRAILS aims to increase EBP expertise and use among SMHPs by leveraging trained community coaches to augment initial training and provide implementation support. To increase coaching capacity for the ASIC trial, TRAILS identified, trained, and mobilized a network of coaches across the state of Michigan. The training of coaches was informed by best practice literature and included didactic trainings, personalized consultation, live observation, and standardized behavioral rehearsals—strategies demonstrated to improve implementation and clinical outcomes (Beidas et al., 2014; Frank et al., 2020). Thus, the TRAILS model is an upstream approach to train coaches who subsequently provide implementation support and feedback to SMHPs in the context of direct treatment delivery to students. To assess the outcomes of and barriers to coach recruitment and training, this article presents a retrospective review of administrative and self-report data.

Procedure

The development of a statewide coach network proceeded in five phases: (1) recruitment, (2) clinical training, (3) consultation, (4) evaluation, and (5) coach protocol training. All potential coaches, called “Coaches in Training” (CITs), were grouped into five sequential cohorts, roughly grouped by region of the state. The consultation process is classified as a non-regulated quality improvement project by the University of Michigan Medical School institutional review board; therefore, no written consent from either the CIT or their clients was requested. Each CIT followed their agency's processes for obtaining consent for participation in treatment from their clients.

Recruitment

TRAILS staff oversaw statewide recruitment of all potential coaches (CITs). Recruitment occurred from September 2016 through June 2018. Recruitment was limited to individuals in the state of Michigan who were interested and anticipated being available to provide local school-based coaching during the 2019-2020 school year. The preference was for masters- or doctoral-level licensed mental health professionals, although exceptions were made (e.g., an occupational therapist) in remote areas. In addition to direct contacts, advertisements were posted on social media sites (e.g., Facebook).

To identify CITs, TRAILS contacted state-level stakeholders, community mental health agencies, and private practitioners. Recruitment strategies included identifying both individual providers and agencies interested in having their staff trained to become coaches. To incentivize agency participation, the initial six-hour clinical training and associated continuing education units were offered for free to CIT attendees and up to ten additional clinicians from their respective agencies. CITs were also informed that, if selected as a coach, they would be eligible for a coaching stipend of $2,000 for each group for which they provided coaching.

Clinical Training

Following recruitment, all CITs attended one of several daylong trainings offered between January 2017 and June 2018. Trainings were provided in five regions across the state to facilitate access. Content included an overview of the TRAILS program, the rationale for coaching, and education and interactive practice in a set of five core components of CBT that have been identified as common elements of evidence-based-practice for youth with internalizing symptoms (Chorpita et al., 2005): (1) psychoeducation, (2) cognitive restructuring, (3) relaxation, (4) behavioral activation, and (5) exposure. Mindfulness strategies were integrated throughout the training.

Consultation

Following the clinical training, CITs were assigned a consultant. Consultants were 13 clinicians employed by TRAILS at the University of Michigan, who specialized in CBT. Three consultants held doctoral degrees, and the remaining 10 consultants held master's degrees. Consultants adhered to a formal consultation protocol and received ongoing mentorship and supervision from TRAILS lead clinical staff.

To participate in consultation, CITs were required to select one to two clinical cases with whom to practice CBT skills in the context of consultation. CITs were instructed that these practice clients should be 12 to 19 years old (to approximate the age range of students later served by SMHPs), who were experiencing symptoms of anxiety, depression, or both. Throughout consultation, CITs provided CBT to practice cases in their usual care setting, guided by a 10-session manual for treating combined anxiety and depression, or the seven-session manuals for treating anxiety and depression separately (current versions of these manuals can be viewed on TRAILStoWellness.org). CITs were expected to practice all five CBT components with their clinical cases.

Consultants provided virtual one-to-one consultation to CITs for approximately one hour per week for 12–15 weeks. During consultation meetings, consultants reviewed CBT skills and discussed the application of these skills to CITs practice cases. Consultants also administered five standardized behavioral rehearsals, one for each CBT component. Finally, consultants observed in vivo (via phone or video conference) one or two client sessions, during which the CIT was expected to demonstrate specific CBT techniques.

Evaluation

To determine suitability for coaching, CITs were evaluated on proficiency in CBT, professionalism, and communication skills. Measures used to assess clinical proficiency included a pre- and post-training measure of CBT competence (Rodriguez-Quintana et al., 2021); performance on five structured behavioral rehearsals; skills observed during live session observation; CIT self-assessments; and a post-consultation evaluation of CIT clinical skill completed by the consultant. Specifically, the behavioral rehearsals included a fidelity checklist of key components that should be presented by the CIT during the role play. Although no firm criteria were required to pass the behavioral rehearsals, consultants were instructed to have CITs redo rehearsals for which they received less than 80% fidelity scores. The performance of each role play was also assessed on a five-point Likert scale (poor to excellent). CIT professionalism and communication were evaluated using their consultant's clinical judgment. These measures were used as guideposts to evaluate clinical and CBT proficiency and professionalism and to determine whether the CIT would be likely to develop into a strong and effective coach. Ultimately, referral to be a coach was based on both objective and subjective measures of CBT skill, as well as CIT professionalism and communication. Finally, CIT practice clients completed the PHQ-8 and GAD-7 at both pre- and post-consultation to assess the symptoms of depression and anxiety. Table 1 describes the measures used for the evaluation of the CIT.

Table 1.

Measures used to assess Coach-in-Training (CIT) competency in CBT and mindfulness.

Measure	Source	Timepoint
Measure	Source	Pre-clinical training	Pre-consultation	Consultation sessions	Treatment sessions	Post-consultation
Behavioral rehearsals	Consultant			X
In-session observations	Consultant				X
Post-consultation evaluation	Consultant					X
Cognitive behavioral therapy competence scale (Rodriguez-Quintana et al., 2021)	CIT	X				X
Weekly self-assessment description	CIT				X
PHQ-8	Client		X			X
GAD-7	Client		X			X

Note. PHQ-8 = Patient Health Questionnaire-8; GAD-7 = Generalized Anxiety Disorder-7.

Coaching Protocol Training and Support

CITs who met the requirements of consultation and were recommended as coaches by their consultants were invited to attend a one-day training in the coaching protocol. Attendees at coach protocol trainings were again offered continuing education units, and trainings were delivered regionally. In addition to a brief review of CBT techniques, the Coach Protocol Training focused heavily on coaching tools, techniques, and expectations. In addition to receiving a comprehensive coach manual, CITs were taught strategies for supporting SMHP use of CBT, assessing clinical competence, providing effective feedback, and communicating with TRAILS and the ASIC study team.

Following the coach protocol training, CITs were considered qualified to serve as coaches. Throughout the active coaching phase of the ASIC clinical trial, TRAILS provided further support for coaches consisting of (1) an online TRAILS coach portal housing all coach tools and CBT materials, (2) monthly coach support calls, (3) monthly newsletters with clinical updates and study-related information, and (4) email and phone communication as needed.

Attrition

Not all CITs completed the training process, either because they chose to drop out or because they did not meet evaluation standards. Reasons for drop-out or attrition were reported either by the consultant or during CIT correspondence with the program but were not available for CITs who did not indicate a reason for dropout. These reasons were coded post hoc by the review of administrative data, using domains identified by the Consolidated Framework for Intervention Research (CFIR; Damschroder et al., 2009). CFIR includes five broad domains: (1) intervention characteristics, (2) outer setting, (3) inner setting, (4) individual characteristics of practitioners, and (5) implementation process.

Measures

CIT Measures

Demographics. Available characteristics include gender, race, ethnicity, professional degree, level of education, location, and type of agency. Demographic measures were collected only from coaches who were paired with a school and thus are not available for all trained coaches.

Geographic Coverage of State. Primary counties of the coaches (as identified by county of agency) were classified as urban, suburban, small-to-medium metropolitan, or rural using the National Center for Health Statistics (NCHS) classification scheme for counties. Counties were classified on a one-to-six scale, with one indicating large central metropolitan areas, two through four indicating small metro or suburban areas, and five and six indicating non-metropolitan (rural) areas (National Center for Health Statistics, 2017). Assessing the urban-to-rural classification of the coach's primary county provides one indication of the penetration of the program throughout the state.

CBT Competence. Competence in CBT was assessed via the CBT Competence Scale (CCS; Rodriguez-Quintana et al., 2021). This measure is a 33-item tool designed to evaluate provider competence in CBT and consists of four subscales: (1) non-behavioral skills, (2) behavioral skills, (3) CBT perceptions, and (4) CBT knowledge. This article analyzes data only from the first two scales—non-behavioral skills and behavioral skills—due to their relevance to training outcomes. The scales consist of items measuring self-reported expertise (0 = very low to 4 = very high) and frequency of use (0 = never to 4 = always) in clinical intervention. Thus, higher scores indicate greater expertise and use.

The non-behavioral skills scale is comprised of 10 items related to expertise and use of psychoeducation, relaxation, and cognitive restructuring. The behavioral skills scale is comprised of eight items related to the expertise and use of behavioral activation and exposure. Psychometric properties have been reported elsewhere (Rodriguez-Quintana et al., 2021) and consist of excellent internal consistency for the total scale, good convergent validity, and good-to-excellent construct validity. Internal consistency for this sample was high (Cronbach's α = 0.93 for both behavioral skills and non-behavioral skills scales; see Supplemental Tables 1 and 2). CITs completed the CCS both prior to the first clinical training and following the completion of consultation.

Confidence in Implementing Cognitive and Behavioral Skills. Following consultation, CITs also reported on whether they would feel confident in implementing the following cognitive and behavioral skills: psychoeducation, physical coping, cognitive restructuring, behavioral activation, gradual exposure, and mindfulness, with a goal of measuring clinicians’ self-report of their learning.

Clinician Attitudes. Following consultation, CITs were asked how satisfied they were with the consultation process, ranging from extremely dissatisfied (1) to extremely satisfied (5).

Impact on Service Delivery. Following consultation, CITs reported on the number of clients whose treatment was informed by skills and knowledge gained during training and consultation. CITs also reported (yes or no) on whether they would feel confident in implementing the specific components of CBT. (e.g., psychoeducation, cognitive coping, and behavioral activation).

Client Measures

Patient Health Questionnaire-8 (PHQ-8; Kroenke & Spitzer, 2002 ). The PHQ-8 is an eight-item measure of depressive symptoms that asks respondents to indicate the extent to which they have experienced any of a list of problems over the last two weeks, rated on a four-point scale, ranging from 0 to 3 (not at all to nearly every day). The PHQ-8 has good internal consistency and high specificity and sensitivity (Kroenke et al., 2001). Both a total score (with a possible range from 0 to 24) and an impairment score (sum of reported impairment in academic, social, and physical health domains; possible range: 0–9) were calculated. Scores of 10 or above indicate clinically significant depressive symptoms (Kroenke et al., 2009).

Generalized Anxiety Disorder-7 (GAD-7; Spitzer et al., 2006 ). The GAD-7 is a seven-item self-report scale that assesses symptoms of anxiety. The measure assesses the degree to which respondents have experienced any problems over the last two weeks. Respondents use a four-point scale from 0 to 3 (not at all to nearly every day). The GAD-7 has good reliability, criterion and construct validity, and internal consistency (Cronbach alpha = 0.92; Spitzer et al., 2006). Both a total score (possible range: 0–27) and an impairment score (sum of impairment in academic, social, and physical health domains; possible range: 0–9) were calculated. Scores of 10 or greater on the GAD-7 indicate clinically significant anxious symptoms (Spitzer et al., 2006).

Analytic Plan

Changes in CBT competence from pre-to-post-consultation by CITs were evaluated using dependent sample t-tests. For measures without pre- and post-data (i.e., the impact of consultation on service delivery; satisfaction with consultation), only post-consultation data are presented. Dependent sample t-tests were also used to assess change in client-reported anxious and depressive symptoms across treatment. All analyses were completed in the open-source statistical package R.

Results

Primary findings are developed from an administrative review of program data as well as surveys and measures completed by coaches. The term “CITs” is used when considering the full intent-to-train sample, whereas “coaches” refer to individuals who completed all required training and consultation. The primary population under consideration are coaches, those individuals who completed all required training and consultation steps. Sample size for each measure is included in Supplementary Table 3.

Training Attrition

CIT attrition could occur at five timepoints: (1) immediately following clinical training, (2) after being paired with a consultant, but before attending a consultation meeting, (3) during consultation, (4) with a negative recommendation following consultation, or (5) with positive recommendation to become a coach, but before attending the coach protocol training.

A total of 347 individuals attended the initial clinical training. Subsequently, 187 CITs were paired with a consultant after the training, and 85% (n = 159) entered consultation. In total, 66.8% (n = 125) of those paired with a consultant completed consultation, with 57.7% (n = 108) being recommended to attend the coach protocol training. Ultimately, 45% (n = 86) attended the coach protocol training to be considered TRAILS coaches. Figure 1 presents the flow of recruitment, training, and consultation with CITs as a part of the development of a statewide network of coaches.

Figure 1.

Flow of CITs through training and consultation.

Reasons for Attrition

Reasons for attrition were coded using the CFIR approach, and the selection of categories was data-driven to identify relevant barriers. CFIR has five broad domains. Intervention characteristics refers to features of the treatment itself. Outer settings refers to the economic, political, and social settings in which an organization exists, whereas inner setting refers to “features of structural, political, and cultural contexts through which the implementation process will proceed.” The practitioners domain refers to the individuals within the organization who may be interacting with the intervention. Finally, the implementation process refers to the process of change to promote individual and organizational uptake of the intervention. We defined reasons for attrition as related to (1) training or requirements related to identification of a clinical practice case (intervention characteristics domain), (2) resources (inner setting domain), (3) clinical setting or role within system (inner setting domain), and (4) individual or personal characteristics (individual characteristics of practitioners domain). Other attrition categories were related to an absence of information about cause for attrition: (5) CIT was uninterested or no longer available and (6) other or unknown reason.

Other or unknown was the most common reason (n = 37), but unavailability of or disinterest by the CIT was the second most common reason (n = 20). Among the attritors, 13 CITs discontinued consultation due to the training requirements, including not having access to a clinical case or discomfort with in-session observations or role play (n = 13). Less commonly-endorsed reasons involved resources such as time or money (n = 9) and inner system factors (n = 6), such as the lack of agency support to continue with consultation. When comparing CITs who successfully became coaches and those who attrited, there were no differences in the urban–rural designation of their primary county.

Demographics

Coach demographics are available in Table 2. Most of the sample identified as White or Caucasian (84%), female (87%), and non-Hispanic or Latinx (98%). Most coaches were master's level clinicians (99%) with an MSW (79%). Approximately one-third (37%) reported providing outpatient care, and approximately one-quarter (24%) reported providing in-home services. The remainder reported working in a school setting (22%) or a blend of settings (17%).

Table 2.

Demographics of coaches.

	N (% coaches)
Race^a
Black or African American	3 (5%)
White	48 (84%)
American Indian or Alaska Native	1 (2%)
Bi-racial or multi-racial	5 (8%)
Female^a	50 (87%)
Hispanic or Latinx^a	1 (2%)
Education level^b
Masters	71 (99%)
PhD	1 (1%)
License credential^a
Licensed marriage and family therapist or limited license marriage and family therapist	2 (3%)
Licensed master social worker or limited license master social worker	57 (79%)
Licensed professional counselor	12 (17%)
Limited license psychologist	2 (3%)
Primary practice setting^b
Outpatient	27 (37%)
School-based	17 (24%)
Home-based	16 (22%)
Other	12 (17%)

Demographic data (race, sex, and ethnicity) were available for 56 of the 86 TRAILS-trained coaches.

Degree and education data were available for 72 of the 86 coaches. One coach had both a LPC and LLMFT and thus is listed twice.

Geographic Distribution of Coaches

The 86 TRAILS-trained coaches covered 47 of 83 counties in the state of Michigan. As presented in Table 3, we were relatively more successful in training coaches located in urban counties (100%) and small metropolitan to suburban counties (66.6%) compared to rural counties (50.9%). Among the 86 trained coaches, 8 were in an urban county, 35 were in a small metropolitan or suburban county, and 40 were located in a rural county. Figure 2 indicates counties in which coaches were successfully recruited and trained.

Figure 2.

Michigan counties with a trained coach.

Table 3.

Comparing counties with and without coach coverage.

County geographic category	Covered counties N (%)^a	Uncovered counties N (%)^a	Entire state N (%)^b
Rural	29 (50.9%)	28 (49.1%)	57 (68.7%)
Small metro–suburban	16 (66.7%)	8 (33.3%)	24 (28.9%)
Urban	2 (100%)	0 (0%)	2 (2.4%)
Total Ns	47	36	83

% indicates row percentages.

% indicate column percentages.

Training Outcomes

Self-Report of Competence

Average self-reported competence and frequency of use of non-behavioral skills was 2.87 (0–4 scale) prior to consultation, and 3.34 following consultation, with significant improvement from pre- to post-consultation (t = 6.26, p < .001) and a medium effect size (Cohen's d = 0.45). Average self-reported competence and frequency of use of behavioral skills was 1.46 (0–4 scale) prior to consultation, with a significant increase to 3.01 following consultation (t = 18.05, p < .001; Cohen's d = 1.99).

Nearly all coaches reported they would feel confident implementing psychoeducation (98.7%), physical coping (97.4%), cognitive restructuring (97.4%), and behavioral activation (96.2%). A smaller, but still high proportion of coaches reported they would feel confident in using Mindfulness (93.6%) and Exposure (88.6%).

Acceptability of Consultation

The acceptability of the consultation experience was rated by coaches after finishing consultation. Among those who completed consultation and were recommended to be coaches, most (94%) reported being “extremely satisfied” with the TRAILS consultation process.

Impact on Service Delivery

Coaches reported providing CBT to a total of 165 practice clients between March 2017 and July 2018. Most coaches reported one (n = 35), two (n = 24) or three (n = 2) practice clients. The remainder (n = 6) reported having between four and 25 clients during consultation. Data were not available to determine if CITs saw clients individually or in a group setting.

Coaches also reported a median number of six additional clients whose care was informed by TRAILS training and consultation (range: 1–75), for a total of 450 youth. Eighty-seven percent of coaches reported that they “most of the time” or “always” use the CBT skills with clients outside of their TRAILS clinical case(s). Regarding future use, 99% reported that they “probably” or “definitely” intended to use CBT skills in the future.

Client Outcomes

Pre-post PHQ-8 and GAD-7 data were available for n = 69 youth treated by n = 62 TRAILS coaches during the consultation period whose pre–post data was 14 or more days apart. Pre-treatment only data were available for an additional 48 youth from n = 33 coaches, with no significant differences in the pre-treatment scores between the groups. Most coaches (n = 56) reported pre–post clinical data for one client, with a few having two (n = 6) or three clients (n = 1). A mean of 92.3 days (SD = 31.6; range: 14–186) passed between pre- and post-timepoints.

Pre-Treatment Anxiety and Depression Scores Indicated Clinical Significant Symptoms. The mean pre-treatment PHQ-8 score was 10.27 (SD = 5.60) and decreased to 5.85 post-treatment (SD = 4.21), a significant difference indicative of a large effect size (t = 7.46; p < .001, Cohen's d = 0.89). The mean pre-treatment GAD-7 score was 10.23 (SD = 5.56) and decreased to 6.37 (SD = 5.31) post-treatment, also a significant difference suggestive of a large effect size (t = 6.83; p < .001; Cohen's d = 0.82).

The impairment subscales of the PHQ-8 and GAD-7 showed similar improvements. Mean PHQ-8 impairment scale at pre-treatment was 3.24 (SD = 2.17) and decreased to 2.20 (SD = 1.88) at post-treatment, suggesting a medium effect size (t = 4.16; p < .001; Cohen's d = 0.50).

Mean GAD-7 impairment scale at pre-treatment was 3.06 (SD = 2.44) and significantly decreased to 1.93 at post-treatment (SD = 1.86), suggesting a medium effect size (t = 4.26; p < .001; Cohen's d = 0.52).

Discussion

The healthcare system in the United States cannot adequately meet the mental health needs of youth (Turner et al., 2019). As presented, one solution is to expand school-based mental health services. Training SMHPs in EBPs, however, requires tailored support. In vivo coaching delivered by community clinicians is a promising and feasible approach (Koschmann et al., 2019). We have shared the process used to develop a statewide network of CBT-trained clinician-coaches and presented implementation and clinical outcomes.

Reasons for Attrition

A detailed consideration of clinician dropout and attrition can support the identification of barriers to training of community clinicians in EBPs. Few studies describe reasons for training attrition (Gleacher et al., 2011 is an exception) or the proportion of clinicians who successfully reach competence standards (McHugh & Barlow, 2010). In this sample, approximately half of the CITs who were paired with a consultant ultimately completed the consultation process and were deemed to have the necessary clinical skills, knowledge, and availability to become competent and collaborative coaches. Gleacher and colleagues (2011) described the development of a similar network through a series of training workshops and consultations. In their analysis, 55%–57% of clinicians who participated in the initial training completed the program, which is somewhat comparable to our completion rate of 45.9%. Attrition rates during training have been noted previously as an area in need of systemic study (Beidas & Kendall, 2010). Thus, transparent reporting contributes to this literature and allows implementation organizations to anticipate barriers when training community clinicians.

In the current data, the predominant reason for attrition was unknown, which certainly limits interpretation of the data. Some known reasons for attrition in the present sample were inherent to expectations of the training process itself, such as the requirement for CITs to have a client with whom to practice the CBT skills. Although sustained access to a practice client can be a barrier to entry, if clinicians-in-training do not implement the new skills relatively quickly, the likelihood of doing so in the future decreases (Herschell et al., 2010; Lau et al., 2020), highlighting an implementation double-bind. These findings suggest value in clearly communicating the expectation, with advance notice, to prospective CITs that they will need to identify and be ready to practice skills with a client as an element of training.

Other factors associated with attrition were related to the CIT's inner setting, such as the lack of agency support, conflict of the CIT's role within an agency, or inadequate time to participate in weekly consultation. Time and setting barriers are well-researched barriers to training and implementation of EBPs (Stewart et al., 2012). Implementation efforts would benefit from addressing these barriers early in the training process (e.g., anticipating whether a clinician's role in the agency would be appropriate to serve as a coach). Future research should also examine the predictors of successful progression through consultation to increase efficiency and decrease training costs.

Training and Clinical Outcomes

CITs were recommended to become coaches if they were shown to demonstrate adequate knowledge and skillful delivery of CBT, as well as professionalism. The CCS showed increases in self-reported expertise and use of non-behavioral skills (e.g., relaxation and cognitive restructuring) and behavioral skills (exposure and behavioral activation), with greater change in behavioral skills than non-behavioral skills. This improvement in behavioral skills is notable, especially given that component analyses have suggested that exposure and behavioral activation may account for most of the variance in symptom improvement (Ritschel et al., 2011; Whiteside et al., 2015).

For practice clients, pre-treatment symptom measures were indicative of moderate depression and moderate anxiety and above the cut-off for clinically significant symptoms. By post-treatment, mean scores for both domains were within the mild range. Clinically significant change across the practice clients is one indicator of the skillful delivery of the 10-session treatment and thus gives some suggestion that the coaches were competent in the principles of CBT. Furthermore, the opportunity to observe symptom improvement in their clients likely increased coaches’ self-efficacy and commitment to the program, and ultimately to coaching. In addition to these benefits, assessing symptom change exposed clinicians to use of measurement-based care. Measurement-based care has been found to streamline the assessment process and increase the accuracy of clinician judgments and is considered an evidence-based intervention in its own right (Lewis et al., 2019; Scott & Lewis, 2015). In this way, direct experience by CITs with this measurement-based care offers an additional advantage to our training approach.

This report has many strengths, including presenting comprehensive results across training and clinical outcomes. Clinicians reported increased knowledge and use of CBT skills, and youth clients experienced clinically significant improvement in anxious and depressive symptoms. Consistent with the nature of real-world implementation, we transparently report on attrition rates and reasons for attrition, which is somewhat unusual in implementation science (Carlson et al., 2012). The report provides an exemplar of program implementation while navigating commonplace barriers to the training of community clinicians.

Limitations include incomplete data for some coaches and youth clients. Data were collected as pragmatic data collection, and thus, measures may have evolved or lacked a perfect response rate. Additionally, this report places a selective focus on those who completed all training components. Although the choice of this sample was intentional to highlight a proof-of-concept, training outcomes may be somewhat better than for those who did not successfully complete consultation, and further may not be representative of all community clinicians who seek training in EBP. Finally, although clinical skills were assessed directly, coaching skills were assessed by proxy, by considering the professionalism and clinical skill of CITs. Other programs (e.g., Triplett et al., 2020) have directly observed individuals providing trainings, but this strategy was not feasible in this statewide clinical trial.

Future Directions

This article raises topics for further investigation and solutions. Given that most CITs who began the process did not complete it in its entirety, a comparison of completers and non-completers would offer useful avenues for the improvement of the training process. Investigating similarities or differences in clinical skills, contextual factors, and attitudes toward EBPs among those who did and did not complete the consultation process may offer further insight into barriers and solutions for training community clinicians.

This article documents the development of a coach network across the state of Michigan; however, this development was not without challenges. First, and foremost, TRAILS was dependent on community partnerships for scaling. As a result, TRAILS often faced obstacles connecting with agencies in communities, either during the recruitment or the training process. Furthermore, many communities within the United States face a shortage of providers, especially in rural regions (Thomas et al., 2009). Recruiting and training CITs in these rural counties was somewhat more difficult than mid-geographic or urban counties. In the period since data collection, state funding has allowed the modification of the presented implementation model, including partnering directly with school districts to train their staff and personnel as well as virtual coaching. Emerging data suggests that virtual coaching is as effective and acceptable as in-person coaching (Warner et al., 2021), but an additional evaluation of these models is needed. Continuing to address both provider availability in rural counties, and parent and district needs is an ongoing area of development. Future research should investigate whether this offers a viable alternative to in-person coaching and can help overcome barriers to access in underserved regions.

Ultimately, addressing unmet youth mental health needs will rely on solutions involving multiple systems and at a scale grander than current capacity. School-based mental health services have been shown to narrow disparities among racial and ethnic groups in accessing mental healthcare (Cummings et al., 2010). The presented model can inform other programming attempting to scale into hard-to-reach or underserved areas. Training community providers to become coaches provides the opportunity to increase clinicians’ knowledge of CBT for use in clinical practice while also positioning them as trainers for school professionals is one avenue by which to ultimately increase youth access to services.

Supplemental Material

sj-docx-1-irp-10.1177_26334895221101215 - Supplemental material for Developing a statewide network of coaches to support youth access to evidence-based practices

Supplemental material, sj-docx-1-irp-10.1177_26334895221101215 for Developing a statewide network of coaches to support youth access to evidence-based practices by Allison E Meyer, Natalie Rodriguez-Quintana, Kristen Miner, Emily L Bilek, Jennifer Vichich, Shawna N Smith and Elizabeth Koschmann in Implementation Research and Practice

Footnotes

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article:Associate Editor Daniel Almirall, although not an author, had a major role in the design of the grant-funded project and is also a frequent collaborator. Dr Almirall was not involved in the editorial process. for this manuscript.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Institute of Mental Health (ASIC: R01 MH 114203) and by funds from the Centers for Medicare and Medicaid Services through the Michigan Department of Health and Human Services. The views expressed are those of the authors and do not necessarily represent the views of the state of Michigan or other public entity.

ORCID iD

Allison E Meyer

Supplemental material

Supplemental material for this article is available online.

References

Ali

M. M.

West

Teich

J. L.

Lynch

Mutter

Dubenitz

(2019). Utilization of mental health services in educational setting by adolescents in the United States. The Journal of School Health, 89(5), 393–401. https://doi.org/10/ghgbj8

Becker

Bradshaw

Domitrovich

Ialongo

(2013). Coaching teachers to improve implementation of the good behavior game. Administration and Policy in Mental Health, 40(6), 482–493. https://doi.org/10/f5ccjt

Beidas

R. S.

Cross

Dorsey

(2014). Show me, don’t tell me: Behavioral rehearsal as a training and analogue fidelity tool. Cognitive and Behavioral Practice, 21(1), 1–11. https://doi.org/10/gjj2nr

Beidas

R. S.

Edmunds

J. M.

Marcus

S. C.

Kendall

P. C.

(2012). Training and consultation to promote implementation of an empirically supported treatment: A randomized trial. Psychiatric Services, 63(7), 660–665. https://doi.org/10/ghp69d

Beidas

R. S.

Kendall

P. C.

(2010). Training therapists in evidence-based practice: A critical review of studies from a systems-contextual perspective. Clinical Psychology, 17(1), 1–30. https://doi.org/10/b2jw9f

Cappella

Hamre

B. K.

Kim

H. Y.

Henry

D. B.

Frazier

S. L.

Atkins

M. S.

Schoenwald

S. K.

(2012). Teacher consultation and coaching within mental health practice: Classroom and child effects in urban elementary schools. Journal of Consulting and Clinical Psychology, 80(4), 597–610. https://doi.org/10/f3742g

Carlson

J. S.

Mackrain

M. A.

van Egeren

L. A.

Brophy-Herb

Kirk

R. H.

Marciniak

Falvay

Zheng

Bender

S. L.

Tableman

(2012). Implementing a statewide early childhood mental health consultation approach to preventing childcare expulsion. Infant Mental Health Journal, 33(3), 265–273. https://doi.org/10/f3x544

Chorpita

B. F.

Daleiden

E. L.

Weisz

J. R.

(2005). Identifying and selecting the common elements of evidence based interventions: A distillation and matching model. Mental Health Services Research, 7(1), 5–20. https://doi.org/10/fkf4kc

Cook

B. L.

Trinh

N.-H.

Hou

S. S.-Y.

Progovac

A. M.

(2016). Trends in racial-ethnic disparities in access to mental health care, 2004–2012. Psychiatric Services, 68(1), 9–16. https://doi.org/10/ghfvg6

10.

Cummings

J. R.

Ponce

N. A.

Mays

V. M.

(2010). Comparing racial/ethnic differences in mental health service use among high-need subpopulations across clinical and school-based settings. Journal of Adolescent Health, 46(6), 603–606. https://doi.org/10/brk32d

11.

Damschroder

L. J.

Aron

D. C.

Keith

R. E.

Kirsh

S. R.

Alexander

J. A.

Lowery

J. C.

(2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(1), 50. https://doi.org/10/ck4bd2

12.

Frank

H. E.

Becker-Haimes

E. M.

Kendall

P. C.

(2020). Therapist training in evidence-based interventions for mental health: A systematic review of training approaches and outcomes. Clinical Psychology: Science and Practice, 27(3), e12330. https://doi.org/10/ghp69j

13.

Gleacher

A. A.

Nadeem

Moy

A. J.

Whited

A. L.

Albano

A. M.

Radigan

Wang

Chassman

Myrhol-Clarke

Eaton Hoagwood

(2011). Statewide CBT training for clinicians and supervisors treating youth: The New York state evidence based treatment dissemination center. Journal of Emotional and Behavioral Disorders, 19(3), 182–192. https://doi.org/10/ch5cg2

14.

Green

J. G.

McLaughlin

K. A.

Alegría

Costello

E. J.

Gruber

M. J.

Hoagwood

Leaf

P. J.

Olin

Sampson

N. A.

Kessler

R. C.

(2013). School mental health resources and adolescent mental health service use. Journal of the American Academy of Child & Adolescent Psychiatry, 52(5), 501–510. https://doi.org/10/f2xkbb

15.

Herschell

A. D.

Kolko

D. J.

Baumann

B. L.

Davis

A. C.

(2010). The role of therapist training in the implementation of psychosocial treatments: A review and critique with recommendations. Clinical Psychology Review, 30(4), 448–466. https://doi.org/10/fh8vxw

16.

Jensen

P. S.

Foster

(2010). Closing the research to practice gap in children's mental health: Structures, solutions, and strategies. Administration and Policy in Mental Health and Mental Health Services Research, 37(1), 111–119. https://doi.org/10/ctckwv

17.

Kilbourne

A. M.

Smith

S. N.

Choi

S. Y.

Koschmann

Liebrecht

Rusch

Abelson

J. L.

Eisenberg

Himle

J. A.

Fitzgerald

Almirall

(2018). Adaptive school-based implementation of CBT (ASIC): Clustered-SMART for building an optimized adaptive implementation intervention to improve uptake of mental health interventions in schools. Implementation Science, 13(1), 119. https://doi.org/10/gd7jt2

18.

Koschmann

Abelson

J. L.

Kilbourne

A. M.

Smith

S. N.

Fitzgerald

Pasternak

(2019). Implementing evidence-based mental health practices in schools: Feasibility of a coaching strategy. The Journal of Mental Health Training, Education and Practice, 14(4), 212–231. https://doi.org/10/ghfrxx

19.

Kroenke

Spitzer

R. L.

(2002). The PHQ-9: A new depression diagnostic and severity measure. Psychiatric Annals, 32(9), 509–515. https://doi.org/10/gf6zrm

20.

Kroenke

Spitzer

R. L.

Williams

J. B. W.

(2001). The PHQ-9. Journal of General Internal Medicine, 16(9), 606–613. https://doi.org/10/btcq9f

21.

Kroenke

Strine

T. W.

Spitzer

R. L.

Williams

J. B. W.

Berry

J. T.

Mokdad

A. H.

(2009). The PHQ-8 as a measure of current depression in the general population. Journal of Affective Disorders, 114(1–3), 163–173. https://doi.org/10/d4rhch

22.

Lau

A. S.

Lind

Crawley

Rodriguez

Smith

Brookman-Frazee

(2020). When do therapists stop using evidence-based practices? Findings from a mixed method study on system-driven implementation of multiple EBPs for children. Administration and Policy in Mental Health and Mental Health Services Research, 47(2), 323–337. https://doi.org/10/gktngs

23.

Lewis

C. C.

Boyd

Puspitasari

Navarro

Howard

Kassab

Hoffman

Scott

Lyon

Douglas

Simon

Kroenke

(2019). Implementing measurement-based care in behavioral health: A review. JAMA Psychiatry, 76(3), 324–335. https://doi.org/10/gf7skv

24.

Lyon

A. R.

Charlesworth-Attie

Stoep

A. V.

McCauley

(2011). Modular psychotherapy for youth with internalizing problems: Implementation with therapists in school-based health centers. School Psychology Review, 40(4), 569–581. https://doi.org/10/gkxtdq

25.

McCarthy

Van Horn Kerne

Calfa

N. A.

Lambert

R. G.

Guzmán

(2010). An exploration of school counselors’ demands and resources: Relationship to stress, biographic, and caseload characteristics. Professional School Counseling, 13(3), 146–158. https://doi.org/10.1177/2156759x1001300302.

26.

McHugh

R. K.

Barlow

D. H.

(2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65(2), 73–84. https://doi.org/10/dpdktm

27.

Merikangas

K. R.

Burstein

Swanson

S. A.

Avenevoli

Cui

Benjet

Georgiades

Swendsen

(2010). Lifetime prevalence of mental disorders in US adolescents: Results from the national comorbidity survey replication–adolescent supplement (NCS-A). Journal of the American Academy of Child & Adolescent Psychiatry, 49(10), 980–989. https://doi.org/10/bz9v5v

28.

Merikangas

K. R.

Burstein

Swendsen

Avenevoli

Case

Georgiades

Heaton

Swanson

Olfson

(2011). Service utilization for lifetime mental disorders in U.S. adolescents: Results of the national comorbidity survey–adolescent supplement (NCS-A). Journal of the American Academy of Child & Adolescent Psychiatry, 50(1), 32–45. https://doi.org/10/cgrdpj

29.

Meyer

A. E.

Reilly

E. E.

Daniel

K. E.

Hollon

S. D.

Jensen-Doss

Mennin

D. S.

Muroff

Schuler

T. A.

White

B. A.

Teachman

B. A.

(2020). Characterizing evidence-based practice and training resource barriers: A needs assessment. Training and Education in Professional Psychology, 14(3), 200–208. https://doi.org/10/ghfsf3

30.

Mullen

P. R.

Chae

Backer

Niles

(2021). School counselor burnout, job stress, and job satisfaction by student caseload. NASSP Bulletin, 105(1), 25–42. https://doi.org/10/gjv524

31.

National Center for Health Statistics. (2017). NCHS urban-rural classification scheme for counties. https://www.cdc.gov/nchs/data_access/urban_rural.htm#:∼:text=NCHS%20has%20developed%20a%20six,of%20nonmetropolitan%20%E2%80%9Cnoncore%E2%80%9D%20counties.

32.

Novins

D. K.

Green

A. E.

Legha

R. K.

Aarons

G. A.

(2013). Dissemination and implementation of evidence-based practices for child and adolescent mental health: A systematic review. Journal of the American Academy of Child & Adolescent Psychiatry, 52(10), 1009–1025.e18. https://doi.org/10/f2xkhs

33.

Powell

B. J.

Waltz

T. J.

Chinman

M. J.

Damschroder

L. J.

Smith

J. L.

Matthieu

M. M.

Proctor

E. K.

Kirchner

J. E.

(2015). A refined compilation of implementation strategies: results from the expert recommendations for implementing change (ERIC) project. Implementation Science, 10(1), 21. https://doi.org/10/f635v6

34.

Ringle

V. A.

Read

K. L.

Edmunds

J. M.

Brodman

D. M.

Kendall

P. C.

Barg

Beidas

R. S.

(2015). Barriers to and facilitators in the implementation of cognitive-behavioral therapy for youth anxiety in the community. Psychiatric Services, 66(9), 938–945. https://doi.org/10/f8bxvs

35.

Ritschel

L. A.

Ramirez

C. L.

Jones

Craighead

W. E.

(2011). Behavioral activation for depressed teens: A pilot study. Cognitive and Behavioral Practice, 18(2), 281–299. https://doi.org/10/bz6gbr

36.

Rodriguez-Quintana

Choi

S. Y.

Bilek

Koschmann

Albrecht

Prisbe

Smith

(2021). The Cognitive Behavioral Therapy Competence Scale (CCS): Initial development and validation. The Cognitive Behaviour Therapist, 14, E39. https://doi.org/10.1017/S1754470X21000

37.

Schultz

B. K.

Arora

Mautone

J. A.

(2015). Consultation and coaching to increase the uptake of evidence-based practices: Introduction to the special issue. School Mental Health, 7(1), 1–5. https://doi.org/10/gjv53b

38.

Scott

Klech

Lewis

C. C.

Simons

A. D.

(2016). What did they learn? Effects of a brief cognitive behavioral therapy workshop on community therapists’ knowledge. Community Mental Health Journal, 52(8), 998–1003. https://doi.org/10/f9c2p4

39.

Scott

Lewis

C. C.

(2015). Using measurement-based care to enhance any treatment. Cognitive and Behavioral Practice, 22(1), 49–59. https://doi.org/10/f6x2sx

40.

Spitzer

R. L.

Kroenke

Williams

J. B. W.

Löwe

(2006). A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine, 166(10), 1092–1097. https://doi.org/10/d7z8rz

41.

Stewart

R. E.

Chambless

D. L.

Baron

(2012). Theoretical and practical barriers to practitioners’ willingness to seek training in empirically supported treatments. Journal of Clinical Psychology, 68(1), 8–23. https://doi.org/10/ccmfrp

42.

Thomas

K. C.

Ellis

A. R.

Konrad

T. R.

Holzer

C. E.

Morrissey

J. P.

(2009). County-level estimates of mental health professional shortage in the United States. Psychiatric Services, 60(10), 1323–1328. https://doi.org/10/gjvcgm

43.

Triplett

N. S.

Sedlar

Berliner

Jungbluth

Boyd

Dorsey

(2020). Evaluating a train-the-trainer approach for increasing EBP training capacity in community mental health. The Journal of Behavioral Health Services & Research, 47(2), 189–200. https://doi.org/10/gjv52s

44.

Turner

Rhyan

Ehrlich

Stanik

(2019). Access to behavioral health care in Michigan. http://mihealthfund.org/wp-content/uploads/2019/07/BehavioralHealthAccessBrief_PrivateIns.pdf

45.

Warner

C. M.

Colognori

Brice

Herzig

Mufson

Lynch

Reiss

P. T.

Petkova

Fox

Moceri

D. C.

Ryan

Klein

R. G.

(2016). Can school counselors deliver cognitive-behavioral treatment for social anxiety effectively? A randomized controlled trial. Journal of Child Psychology and Psychiatry, 57(11), 1229–1238. https://doi.org/10/gh3fwx

46.

Warner

Miner

Albrecht

(2021, September 23). Virtual coaching to improve school mental health professional competence in CBT. 7th Social Work Research Symposium, Ann Arbor, MI.

47.

Weisz

J. R.

Kuppens

M. Y.

Eckshtain

Ugueto

A. M.

Vaughn-Coaxum

Jensen-Doss

Hawley

K. M.

Krumholz Marchette

L. S.

Chu

B. C.

Weersing

V. R.

Fordwood

S. R.

(2017). What five decades of research tells us about the effects of youth psychological therapy: A multilevel meta-analysis and implications for science and practice. The American Psychologist, 72(2), 79–117. https://doi.org/10/ggs57c

48.

Whiteside

S. P. H.

Ale

C. M.

Young

Dammann

J. E.

Tiede

M. S.

Biggs

B. K.

(2015). The feasibility of improving CBT for childhood anxiety disorders through a dismantling study. Behaviour Research and Therapy, 73, 83–89. https://doi.org/10/f7tf72

49.

Yarber

Brownson

C. A.

Jacob

R. R.

Baker

E. A.

Jones

Baumann

Deshpande

A. D.

Gillespie

K. N.

Scharff

D. P.

Brownson

R. C.

(2015). Evaluating a train-the-trainer approach for improving capacity for evidence-based decision making in public health. BMC Health Services Research, 15(1), 1–10. https://doi.org/10/f738bc

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB