Sage Journals: Discover world-class research

Abstract

Background

Sustained delivery of evidence-based treatments (EBTs) is essential to addressing the public health impacts of youth mental health problems, but is complicated by the limited and fragmented funding available to youth mental health service agencies. Supports are needed that can guide service agencies in accessing sustainable funding for EBTs. We conducted a pilot evaluation of the Fiscal Mapping Process, an Excel-based strategic planning tool that helps service agency leaders identify and coordinate financing strategies for their EBT programs.

Method

Pilot testing of the Fiscal Mapping Process was completed with 10 youth mental health service agencies over a 12-month period, using trauma-focused cognitive-behavioral therapy or parent–child interaction therapy programs. Service agency representatives received initial training and monthly coaching in using the tool. We used case study methods to synthesize all available data (surveys, focus groups, coaching notes, document review) and contrast agency experiences to identify key findings through explanation building.

Results

Key evaluation findings related to the process and outcomes of using the Fiscal Mapping Process, as well as contextual influences. Process evaluation findings helped clarify the primary use case for the tool and identified the importance—and challenges—of engaging external collaborators. Outcome evaluation findings documented the impacts of the Fiscal Mapping Process on agency-reported sustainment capacities (strategic planning, funding stability), which fully explained reported improvements in outcomes (extent and likelihood)—although these impacts were incremental. Findings on contextual factors documented the influence of environmental and organizational capacities on engagement with the tool and concerns about equitable impacts, but also the view that the process could usefully generalize to other EBTs.

Conclusions

Our pilot evaluation of the Fiscal Mapping Process was promising. In future work, we plan to integrate the tool into EBT implementation initiatives and test its impact on long-term sustainment outcomes across various EBTs, while increasing attention to equity considerations.

Plain Language Summary Title

Pilot-Testing a Tool for Planning the Sustainable Financing of Youth Mental Health Treatments that Work

Plain Language Summary

Youth mental health treatments that work must be consistently available to improve youth mental health in our communities, but funding for these treatments is often limited and hard to access. Youth mental health service agencies need tools that can help guide them in accessing sustainable funding for evidence-based treatments. We developed the Fiscal Mapping Process, an Excel-based strategic planning tool for planning sustainable financing of youth mental health treatment programs, and conducted a 1-year pilot-testing evaluation with 10 youth mental health service agencies. We used case study methods to compare and contrast agency experiences with using the tool, related to the process, outcomes, and contextual influences on using the Fiscal Mapping Process. Key findings included clarification of the ideal characteristics of contributors and treatment programs for using the tool; initial confirmation that the tool can improve agency-reported capacities for sustaining treatments that work and long-term sustainment outlooks, although these impacts were incremental; and documentation of the influence of environmental and organizational capacities on engagement with the tool, concerns about equitable impacts, and user views that the process could be applied to a wide range of treatment models. In summary, our pilot evaluation of the Fiscal Mapping Process showed that this tool is promising for supporting the financial sustainment of treatments that work in youth mental health services. In future research, we plan to incorporate the tool into real-world training initiatives with mental health service agencies, test its impact on long-term sustainment across a variety of treatment models, and incorporate attention to equity considerations.

Keywords

financing strategies evidence-based treatment youth mental health services sustainment strategic planning tailored implementation strategies

Introduction

U.S. mental health systems urgently need to address the societal impacts of youth mental health problems with extensive and sustained delivery of high-quality child and adolescent mental health treatments (Alves-Bradford et al., 2020; Shim, 2021). One in five youth experience mental health problems (Bitsko et al., 2022; Duong et al., 2021; Okwori, 2022), and mental health conditions are now one of the leading health problems among youth (Mei et al., 2020) with economic impacts of $250 billion each year (Doupnik et al., 2020; Torio et al., 2015). Numerous evidence-based treatments (EBTs) have been shown in rigorous research to improve youth mental health outcomes and be ready for widespread implementation (Masters et al., 2017; Weisz & Kazdin, 2017), that is, adoption and integration in usual care settings (Bauer & Kirchner, 2020). To improve youth mental health at a population level, youth mental health service agencies (hereafter, “service agencies”) must implement EBTs far more widely and consistently (Kazak et al., 2010; Williams & Beidas, 2019).

Despite their promise, the increased direct costs (e.g., training, materials) and indirect expenses (i.e., lost productivity or billable hours) of delivering EBTs are often a major barrier to their sustainment in usual care (Bond et al., 2014; Lang & Connell, 2017; Roundfield & Lang, 2017; Scudder et al., 2017). Sustainment refers to the continued use of an EBT after the initial implementation process is complete (Shelton et al., 2018; Urquhart et al., 2020), and is essential to public health benefits (Aarons et al., 2014). Financially, sustainment requires service agency staff to navigate the complex, multilevel, and dynamic influences to secure ongoing resources for treatment delivery (Chambers et al., 2013; Shelton et al., 2018; Stewart et al., 2016; Willging et al., 2015). The present study evaluated a novel strategic planning tool, the Fiscal Mapping Process, that we designed to support service agencies in navigating financial sustainment of EBTs.

Design of the Fiscal Mapping Process

The Fiscal Mapping Process (Dopp et al., 2023a, 2020) is a structured Microsoft Excel workbook that guides service agency representatives through completion of five steps, focused on strategic planning for sustained funding of EBTs. Figure 1 illustrates the five steps of the Fiscal Mapping Process: (1) identify resources needed, (2) specify funding objectives, (3) brainstorm financing strategies, (4) complete the Fiscal Map for EBT by linking financing strategies to objectives, and (5) monitor progress over time. We adapted these steps from an established implementation mapping process (Fernandez et al., 2019), an approach recommended for strategic, tailored selection of implementation strategies (Powell et al., 2017). Resource tabs within the workbook contain instructions and useful information (e.g., completed examples, step-specific resources).

Figure 1

Illustration of the Five Steps of the Fiscal Mapping Process

Details of the development process have been published along with the tool (Dopp et al., 2023b). In brief, we designed the Fiscal Mapping Process in partnership with 48 expert participants, representing 10 U.S. youth mental health service agencies and their partners from intermediary organizations (which provide training and guidance on the delivery of specific EBTs; Franks & Bory, 2015) and funding agencies (diverse public and private sources). We used surveys to achieve consensus on the Fiscal Mapping Process steps and incorporate a compendium of 23 potential financing strategies (Dopp et al., 2020; North et al., 2023), then refined the tool through a series of focus groups and 1 year of pilot testing by service agencies.

The Fiscal Mapping Process is grounded in the Public Health Sustainability Framework (Schell et al., 2013), which details core EBT sustainment capacity domains (i.e., structures and processes supporting sustainment). The two most relevant domains for the tool are funding stability and strategic planning, and it was designed to target increasing those capacities as its primary outcomes. Funding stability capacity is defined as having a consistent base of financial resources for an EBT program. The Fiscal Mapping Process helps service agencies achieve funding stability by securing resources for EBT implementation and sustainment activities through financing strategies (Dopp et al., 2020), such as increased fee-for-service reimbursement, grant funding, and contracts (Stroul, 2007; Stroul et al., 2009). Financing strategies are a subset of implementation strategies, which are any activities or methods that support the use of EBTs in usual care (Powell et al., 2015; Waltz et al., 2015).

Strategic planning is defined as the processes guiding a program's directions, goals, and strategies. It is the central sustainment capacity in Schell et al.'s (2013) framework because it coordinates all other capacities (including funding stability) into a sustainment-oriented plan. For example, a single funding source is rarely sufficient for sustainment of EBTs (North et al., 2023), so the Fiscal Mapping Process guides service agencies through strategically combining and coordinating various financing strategies to achieve funding stability (Jaramillo et al., 2019; Mundey et al., 2021; Willging et al., 2015). Success might require leveraging professional networks to identify viable financing strategies (partnerships capacity) or providing information about their EBT program to potential funders (communications capacity). For these reasons, we developed the Fiscal Mapping Process as a strategic planning tool for financial sustainment, rather than focusing it solely on funding issues (e.g., a budgeting-focused tool).

The Present Evaluation

We next sought to evaluate the preliminary impacts of the Fiscal Mapping Process on EBT sustainment outcomes during pilot testing. Specifically, service agencies piloted using the tool to support sustainment of two well-established youth mental health EBTs: parent–child interaction therapy (PCIT) for disruptive behavior problems in youth ages 2–7 and their caregivers (Kaminski & Claussen, 2017; Thomas et al., 2017), and trauma-focused cognitive-behavioral therapy (TF-CBT) for traumatic stress symptoms in youth ages 3–18 (Dorsey et al., 2017; Pollio et al., 2014). By including widely disseminated EBTs with different clinical foci, age groups, and formats, we sought to maximize generalizability of the Fiscal Mapping tool across a wide variety of financial sustainment challenges within the available resources for pilot testing.

We used comparative case study methods (Bartlett & Vavrus, 2017; Yin, 2018), synthesizing a variety of quantitative and qualitative data sources to understand service agencies’ efforts to sustain EBTs through the tool. Comparative case studies are useful for deriving higher-order understanding of a phenomenon by comparing and contrasting the experiences of multiple cases, permitting in-depth insights even with small samples and nonrandomized evaluations. Other applications of these methods in implementation research have evaluated EBT implementation and sustainment (Folker et al., 2018; Thompson et al., 2022), tools for supporting high-quality EBT delivery (Zeitlin et al., 2016), and system- and policy-level influences on the implementation process (Bullock & Lavis, 2019; Dickson et al., 2022).

This study was a pilot evaluation of the Fiscal Mapping Process that combined diverse sources of input in a comparative case study analysis. Our research questions were: (1) How was the Fiscal Mapping Process used by service agencies? (2) What impacts did the Fiscal Mapping Process have on EBT sustainment capacities and outcomes at service agencies? and (3) What factors influenced service agencies’ abilities to use the Fiscal Mapping Process? The questions cover the evaluation of process, outcomes, and contextual influences, respectively.

Method

We followed the Standards for Reporting Implementation Studies (StaRI; Pinnock et al., 2017) to describe this evaluation, and the Template for Intervention Description and Replication (TIDieR; Hoffman et al., 2014) to describe the Fiscal Mapping Process. Full study details are available in a protocol paper (Dopp et al., 2022); here, we only summarize methods for the comparative case studies.

Study Design

Our comparative case study design (Yin, 2018) synthesized mixed-method data sources (Palinkas et al., 2011). Specifically, we used an embedded, multiple-case design in which each youth mental health service agency represented a case (k = 10) of using the Fiscal Mapping Process for EBT sustainment planning. The mixed-method approach was qualitative and quantitative (qualitative-dominant) and examined convergence between data sources; we emphasized qualitative data more because the quantitative effects from small pilot studies are not reliable indicators of effects (Leon et al., 2011).

Our design was “embedded” (Yin, 2018) because each case included multiple units of analysis (i.e., perspectives from different roles) and “multiple-case” (i.e., comparative), grounded in a replication logic in which conclusions were generated by examining patterns of findings across cases. Expert guidance recommends including approximately 10 cases for subtle between-case comparisons (Yin, 2018), representing multiple perspectives in each case (Bartlett & Vavrus, 2017).

Pilot testing and data collection lasted from August 2021 to November 2022. This timeline coincided with intensive periods of the COVID-19 pandemic, but all research activities were conducted virtually which helped to minimize disruption.

Case and Participant Recruitment

Our study protocol (Dopp et al., 2022) provides an in-depth description of the recruitment process. Briefly, we recruited 10 service agencies across the United States, each represented by participants from three key roles in youth mental health services: service agency, intermediary, and funding agency. We sought a diverse range of service agencies, but to be eligible, the provision of mental health services needed to be one of the organization's primary functions. Using snowball sampling methods, intermediary representatives (expert EBT trainers) nominated service agencies with whom they had worked to implement PCIT or TF-CBT in the prior 5 years. We invited those agencies to apply to the project; as part of the application process, applicants identified service agency representatives (including clinical and/or financial management roles) and funding agency partners. To be eligible, service agencies need to have implemented the EBT of focus (PCIT or TF-CBT) with at least one clinician and enroll at least one affiliated service agency representative and funding agency representative.

Procedure

Participants provided informed consent electronically during project enrollment, and also at the beginning of each data collection activity (either electronically or verbally). All procedures were reviewed by the RAND Institutional Review Board and determined to not constitute human subjects research (Protocol #2020-N0607); nevertheless, we followed all ethical principles for the protection of human research participants to minimize any risk of harm. For example, all communications about data collection activities (e.g., focus group scheduling and facilitation) were conducted by a team member other than the Fiscal Mapping Process coach (described next), to avoid undue influence.

Table 1 provides a summary and timeline of all pilot testing and data collection activities relevant to the comparative case studies and three research questions. Participation in specific activities was optional, so each service agency varied in its engagement and data sources.

Table 1

Pilot-Testing and Data Collection Activities with Service Agencies for Evaluating the Fiscal Mapping Process

Activity	Data contributed to evaluation (with relevant RQs listed)	Number of agencies contributing data (of 10)	Participants beyond service agency reps	Notes
Initial training (August 2021–October 2021)	Agency application (RQ1)	10	Fiscal Mapping coach	Version 1 of Fiscal Mapping Process used for training
Initial training (August 2021–October 2021)	Training field notes (RQ1, 2, 3)	10		Version 1 of Fiscal Mapping Process used for training
Coaching sessions (up to 12 months)	Coaching log (RQ1)	10	Fiscal Mapping coach	Introduced Version 2 of Fiscal Mapping tool in March 2022
Coaching sessions (up to 12 months)	Coaching field notes (RQ1, 2, 3)	9	Fiscal Mapping coach	Introduced Version 2 of Fiscal Mapping tool in March 2022
Web survey 1^a (August 2021–November 2021)	ICS, AFSS (RQ2)	9	None	Compensation: $30 Contributed to design of Version 2
Web survey 1^a (August 2021–November 2021)				Compensation: $30 Contributed to design of Version 2
Focus group 1^b (December 2021–February 2022)	Semistructured protocol (RQ1, 2, 3)	7	None None None	Compensation: $50 Contributed to design of Version 2
Focus group 1^b (December 2021–February 2022)	PSAT, PRESS, and intention to sustain ratings (RQ2, 3)	6	None None None	Compensation: $50 Contributed to design of Version 2
Focus group 2 (June 2022–August 2022)			Intermediary and funding agency reps	Compensation: $60
	Semistructured protocol (RQ1, 2, 3)	6	Intermediary and funding agency reps	Compensation: $60
	PSAT, PRESS, and intention to sustain ratings (RQ2, 3)	5	None
Documents for review (RQ1, 2, 3)		6	None

Note. RQ = research question; AFSS = Agency Financial Status Scales (Maxwell et al., 2021; ICS = Implementation Climate Scale (Ehrhart et al., 2014); PSAT = Program Sustainability Assessment Tool (Center for Public Health System Science, 2012); PRESS = Provider REport of Sustainment Scale (Moullin et al., 2021).

All participants were also invited to complete a second web survey in months 6–9, but those data were for development rather than evaluation.

For Focus Group 1, separate groups were held with intermediary and funding agency representatives, but those data were not used in the evaluation. c For focus group 1, separate groups were held with intermediary and funding agency representatives, but those data were not used in the evaluation

Pilot-Testing Activities

The published Fiscal Mapping Process tool (Dopp et al., 2023b) includes a report detailing our development and pilot-testing process, which we summarize briefly here. Before initiating pilot testing, we created an initial prototype of the Fiscal Mapping Process (Version 1) which was subsequently refined throughout pilot testing based on participant feedback and coach experiences. Two Fiscal Mapping coaches (the principal investigator and project manager, both with training in mental health service delivery and EBT implementation) supported five service agencies each. Pilot testing began with an initial training for the service agency representatives, followed by up to 12 months of coaching. Service agency representatives sometimes engaged with funding agency or intermediary representatives between coaching sessions, but only service agency representatives participated in coaching sessions (see Table 1).

Fiscal Mapping Process Tool

We used Version 1 of the Fiscal Mapping Process for the training and initial coaching sessions. After 6 months of pilot testing, we provided an updated Version 2 that incorporated feedback from coaching, Web Survey 1, and Focus Group 1. Major changes included adding details to the tool steps; revising content on the resource tabs; and creating space for documenting action items. After introducing Version 2, we developed a Completion Checklist for self-monitoring completion of the tool and piloted it with service agencies; this checklist was incorporated into the published tool. The final, published Fiscal Mapping Process incorporated additional feedback from participants and quality assurance processes during publication, but these additions were relatively minor (e.g., updates to the color scheme, optional budgeting worksheet) and not relevant for this evaluation because that version of the tool was finalized after pilot testing.

Initial Training

Each service agency separately completed an initial training in the Fiscal Mapping Process upon project enrollment. This training was a 2.5- to 3-hour virtual session, led by the assigned Fiscal Mapping coach while the other coach provided technical support and took detailed field notes. One to three service agency representatives participated per training. The agenda included introductions; a practical, interactive overview of the Fiscal Mapping Process steps (including time for representatives to start completing them); and planning for coaching and data collection activities.

Coaching Sessions

Each service agency was offered monthly virtual coaching sessions up to 12 months posttraining; k = 9 agencies completed two or more sessions. Coaching sessions were brief and focused on providing support for applying the tool to the agency's EBT program (e.g., answering practical questions, troubleshooting barriers, offering support and encouragement). The coaches also gathered feedback on the Fiscal Mapping Process, introduced tool updates to the service agencies over time (e.g., Version 2, Completion Checklist), and encouraged plans for continued use of the tool after pilot testing. The coaches maintained a log of all contacts with assigned service agencies in a shared Microsoft Excel spreadsheet and took detailed field notes during each coaching session.

Data Collection

We collected qualitative and quantitative evaluation data from service agencies through surveys, focus groups, and document review (see Table 1), and also used coaching records (i.e., field notes, coaching log, agency project applications). Service agency representatives were involved in all data collection activities. Funding agency and intermediary representatives participated in the second round of focus groups together with service agency representatives, but otherwise mainly contributed to the development of the tool. Table 1 reports details of timing and compensation for each activity.

Web-Based Survey

We used SelectSurvey, a secure web-based platform, to administer a survey at the start of the project (a second survey was fielded later, but those data were used for development, not evaluation). Survey 1 respondents (n = 16, k = 9 agencies) rated their service agency's EBT implementation climate (Implementation Climate Scale [ICS]; Ehrhart et al., 2014) and financial status for EBT implementation (Agency Financial Status Scales [AFSS]; Maxwell et al., 2021).

Focus Groups

We held two rounds of focus groups with service agency representatives—at the mid-point and end of pilot testing—in which they discussed the impacts of the Fiscal Mapping Process on EBT program sustainment. Focus groups were organized separately for each service agency, and conducted as an interview if the agency only had one representative. In the second focus group, the service agency's funding and intermediary partners were also invited to participate. A note-taker recorded detailed notes and we audio-recorded the discussions for later verification. Following each focus group, service agency representatives completed a web-based questionnaire that included the Program Sustainability Assessment Tool (PSAT; Center for Public Health System Science, 2012; Calhoun et al., 2014), which measures Public Health Sustainability Framework domains (including strategic planning and funding stability, the capacities targeted by the Fiscal Mapping Process); (b) the three-item Provider REport of Sustainment Scale (PRESS; Moullin et al., 2021) to measure extent of EBT sustainment; and (c) a one-item rating of their likelihood of sustaining the EBT over the next year. All measures asked about the EBT of focus for pilot testing (PCIT or TF-CBT). Agency participation rates in Focus Groups 1 and 2 were 70% and 60%, respectively; in both cases, representatives from one agency did not complete the follow-up questionnaire.

Document Review

Throughout pilot testing, we asked service agencies to share any documents relevant to their use of the Fiscal Mapping Process, including copies of completed tools. k = 6 service agencies provided at least one document; completed tools were the most common (five of six), but we also received examples of financing strategies (e.g., grant proposals, service contracts) and EBT program evaluation or community outreach materials. Given the range of documents provided, we used document review as a secondary data source.

Analytic Strategy

Initial Data Processing

Before beginning the case study analysis, we analyzed each data source individually. We calculated overall scores and subscale scores for each quantitative scale (ICS, AFSS, PSAT, PRESS), and summarized scores at the agency level for each timepoint. For focus groups, we conducted preliminary rapid qualitative analyses (Hamilton & Finley, 2019; Taylor et al., 2018) of notes to highlight important themes; many of these themes related to the development of the Fiscal Mapping Process (e.g., suggestions for improvement), so we do not present them here. We still reviewed all notes again for each agency's case study analysis, but the initial rapid analysis helped guide our approach and interpretation.

Comparative Case Study

Our comparative case study analysis (Yin, 2018) involved synthesizing all available quantitative and qualitative data for each case (i.e., service agency) into a descriptive summary, identifying the contributions of specific sources to the conclusions. To structure our approach to case studies, we created a case summary template (see Supplemental File 1 for a blank template) that we completed for each case—that is, service agency—following the instructions in an accompanying development guide (see Supplemental File 2).

Analysis proceeded as follows: (a) the first and third authors (the Fiscal Mapping coaches) compiled and reviewed all data sources, then organized the evidence within Section A of the case study template for each case they coached; (b) the second or fourth author peer-reviewed each Section A, and the coach incorporated their feedback; (c) the second author completed the deidentified case summary narrative (Section B of the template) for each case, summarizing characteristics of the service agency, EBT program, and state context as well as documenting information relevant to the three research questions; (d) the first author reviewed each Section B and the second author incorporated their feedback; and (e) all coauthors reviewed and refined the final case summaries.

Once the individual case summaries were complete, we compared and contrasted experiences across the 10 cases through the replication logic of a multiple case study (Yin, 2018). Financial sustainment is a complex, dynamic process and service agencies had varied experiences with the Fiscal Mapping Process, so we did not encounter literal replication, in which the same result occurs across cases. Instead, we focused on theoretical replication, in which different findings occur across cases for understandable reasons. Given the evaluative purpose of our research questions, our primary analysis technique (Yin, 2018) was “explanation building,” in which explanatory propositions were derived from the data, then evaluated against other data sources (from the same case(s) and across cases) to carefully rule out plausible rival explanations, and refined as needed. We prioritized explanations that best represented all available evidence, lacked plausible rival explanations, and reflected the most significant aspects of the service agencies’ experiences with the Fiscal Mapping Process. We organized the findings by research question and identified illustrative quotes for each explanation presented.

Results

Table 2 summarizes baseline characteristics reported by the 10 service agency cases, including state, agency, and EBT program factors. Supplemental File 3 presents the final case summaries.

Table 2

Baseline Characteristics Reported by 10 Youth Mental Health Service Agencies that Piloted the Fiscal Mapping Process

	Youth Mental Health Service Agency
Variable	#1	#2	#3	#4	#5	#6	#7	#8	#9	#10
U.S. Census Division	South Atlantic	South Atlantic	South Atlantic	South Atlantic	W-N Central	W-S Central	W-S Central	W-S Central	W-N Central	Pacific
Type of agency	Nonprofit	CMHC	CAC	Nonprofit	CAC	For-profit	CMHC	CMHC	For-profit	CMHC
No. of sites	23	3	11	2	1	5	23	1	4	2
EBT model	TF-CBT	PCIT	TF-CBT	TF-CBT	TF-CBT	PCIT	PCIT	TF-CBT	PCIT	TF-CBT
No. of therapists delivering the EBT	11	5	18	8	2	2	7	10	3	4
No. of EBT clients in the past 12 months	24	20	880	250	12	27	12	1,179	10	3
% BIPOC	38%	60%	33%	90%	10%	88%	16%	48%	NR	100%
% in poverty	17%	70%	50%–75%	75%	75%	77%	92%	64%	80%	100%
% Rural	25%	10%	50%–75%	0%	25%	100%	67%	18%	70%	0%
Implementation Climate score^a (Ehrhart et al., 2014)	2.28 (n = 1)	2.74 (n = 3)	NR	3.53 (n = 3)	2.89 (n = 1)	2.56 (n = 1)	2.53 (n = 2)	3.67 (n = 2)	2.28 (n = 1)	1.97 (n = 2)
Agency Financial Status score^a (Maxwell et al., 2021)	2.40 (n = 1)	2.99 (n = 3)	NR	2.22 (n = 3)	2.28 (n = 1)	2.44 (n = 1)	3.07 (n = 2)	3.11 (n = 2)	2.19 (n = 1)	2.51 (n = 2)

Note. CAC = Children's Advocacy Center; CMHC = community mental health center; EBT = evidence-based treatment; PCIT = Parent–Child Interaction Therapy; TF-CBT = Trauma-Focused Cognitive-Behavioral Therapy; BIPOC = Black, Indigenous, and other people of color; NR = not reported.

Average score: 0 = not at all, 1 = slight, 2 = moderate, 3 = great, 4 = very great.

Research Question 1: How the Fiscal Mapping Process Was Used by Service Agencies

Table 3 summarizes the characteristics of pilot-testing participants (e.g., assigned coach, service agency roles represented) and activities (e.g., training, coaching) involved at each service agency. Agencies participated in coaching for an average of 9 months (M = ∼6 sessions, range = 0–10) and coaching sessions averaged ∼22 min each (range = 16–42).

Table 3

Summary of Fiscal Mapping Process Participants and Activities for the 10 Pilot-Testing Service Agencies

	Youth Mental Health Service Agency										Average or total
Variable	#1	#2	#3	#4	#5	#6	#7	#8	#9	#10	Average or total
Coach assigned	AD	MG	AD	MG	MG	AD	AD	MG	AD	MG	N/A
No. of service agency representatives who engaged in the process	3	3	2	3	1	3	3	2	1	3	2.4
Financial manager	Mixed	Mixed	Mixed	Mixed	N	N	Y	N	Y	N	20% Y 40% mixed
Clinical manager	Y	Y	Y	N	Mixed	Y	Y	Y	Y	Y	80% Y 10% mixed
Senior leader	N	Mixed	Y	Mixed	Y	Mixed	Mixed	Mixed	Y	Mixed	30% Y 60% mixed
Training complete	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	100% Y
Coaching complete	N	N	Y	N	N	N	Y	Y	Y	Y	50% Y
No. of coaching sessions	2	4	9	0	6	4	10	6	10	6	5.7
Months of coaching	3	10	12	0	7	6	12	11	12	12	8.5
Average coaching session length (min)	19.5	17.8	23.8	N/A	16.2	16.3	41.7	17.5	31.2	18.2	22.5

Note. AD and MG = coach initials (first and third authors, respectively). Y = yes, N = no, mixed = intermittent or temporary engagement, N/A = not applicable.

Table 4 reports four findings identified from the process evaluation. First, we found the best “use case” for the Fiscal Mapping Process was with (a) EBT programs that were integrated into the agency's core services and that (b) had financial needs or barriers as a high-priority issue for sustainment. This combination of greater level of commitment and complexity for a program, plus relevance of the tool to sustainment needs, made engagement in the Fiscal Mapping Process worthwhile. Notably, outpatient EBT programs were seen as harder to sustain than programs in residential services, due to the latter receiving more adequate reimbursement from payors.

Table 4

Comparative Case Study Findings for the Process Evaluation of Piloting the Fiscal Mapping Process (Research Question 1)

Key Finding	Case comparisons that contributed to finding	Case contrasts that contributed to finding	Illustrative quote(s)
The Fiscal Mapping Process was best-suited for EBT programs that were integrated into the agency's core services, but also had financial sustainment needs (most often in outpatient services).	Agencies #1, 3, 4, 5, 8, 9 had EBT programs at project enrollment. Agencies #2, 7 used the Fiscal Mapping Process to help formalize an EBT program. Outpatient EBT programs had more financial sustainment challenges—compared to, e.g., residential services—and thus prioritized for Fiscal Mapping (Agencies #1, 3, 7).	Agencies #6, 10 sustained EBTs on a clinician (rather than program) level, so the complexity of Fiscal Mapping was less worthwhile. Agencies #1, 4, 8 prioritized addressing nonfinancial sustainment barriers; less engaged than #2, 3, 7, 9. Agency #5 dropped out after EBT program was stopped due to turnover.	"This process has been beneficial as a company because it helped us see what it really requires to have a program that pays for itself and kind of functions as its own program. That's not the way we usually do things. We have cost centers, but they’re based on clinics not on programs." (Service Agency—Financial Manager)
The Fiscal Mapping Process required collaboration among team members with clinical manager, financial manager, and senior leadership roles (but note that the same team member may fulfill multiple roles, depending on the organization's structure).	Agencies #3, 7, 9 had consistent engagement from all three roles; team members found the Fiscal Mapping tool and coaching useful for expanding their understanding of EBT sustainment needs and reducing within-agency siloing of roles. Agency #8 had a strong EBT program already, with shared understanding across roles.	Agencies that lost (#1, 5) or always lacked (#4) clinical manager input did not engage with coaching or tool. Agencies without engagement from financial manager (#1, 10) or senior leadership (#2, 6) lacked the information and/or decision-making authority to make significant progress.	"We are conducting interviews for [clinical manager's] replacement, but it will take time to get the new person acclimated … to the [Fiscal Mapping] tool. Given that there's no [clinical manager] on our team, [financial manager] and I are really at a loss for how to use this tool at the moment." (Service Agency—Senior Leader)
Coaching was valuable for making full use of the Fiscal Mapping Process, but also difficult to prioritize.	Agencies #2, 3, 7, 9, 10 significantly engaged in coaching and reported it was helpful in supporting their use of the tool (created protected time, answered questions, external motivation and support). Attendance was inconsistent and all reported needing significant effort to prioritize coaching.	Agency #3 specifically noted finding coaching more useful than the tool itself. Agencies #1, 4, 5, 6 stopped coaching due to competing priorities; none used the tool outside of coaching. Agencies #6, 8 cited the level of effort required for coaching as a significant barrier.	"Coaching is important. It helps to be accountable, stay on track, [and] gives us a new perspective when we get stuck. If one-on-one [coaching] isn’t available [outside of this pilot study], maybe [offer] some kind of consultation call." (Service Agency—Clinical Manager)
The Fiscal Mapping Process can support collaboration with funding partners, but this was challenging and sensitive.	Agency #3 worked with state representatives to navigate Medicaid transformation. Agency #9 collaborated with the grant manager to develop a sustainment plan and was involved in a lobbying effort that increased Medicaid rates. Agency #8 had a strong collaboration with the funding agency already (resulting in minimal sustainment needs).	Agencies #3, 9 still found it difficult to identify state-level contacts to engage with. Agency #6 attempted to collaborate with funders but the results were discouraging. Agencies #2, 5, 7, 10 focused on internal collaboration around EBT sustainment during the year of coaching; communication with funders would go through colleagues not on Fiscal Mapping team. N/A for Agencies #1, 4.	"[Funders] and providers may be in agreement about an evidence-based practice, but not how to fund it. We may provide funding but maybe not in a way that [they] want it. [Fiscal Mapping] could show how [to improve funding for evidence-based practices]." (Funding Agency Representative)

Note. EBT = evidence-based treatment.

Second, successful use of the Fiscal Mapping Process required collaboration among team members with clinical manager, financial manager, and senior leadership roles (sometimes with one team member fulfilling multiple roles) to synthesize expertise. Lack of clinical manager input made it unlikely that an agency would engage with the tool or coaching at all (they were the “driving force”) whereas teams without financial manager and/or senior leader input found it challenging to make progress because they lacked key information or decision-making authority.

Third, coaching was valuable for making full use of the Fiscal Mapping Process, but also difficult to prioritize. Agencies with high engagement in coaching reported that their coach helped them create protected time, answered questions, and provided external motivation and support; one agency relied exclusively on coaching and did not use the Excel tool. However, attendance at coaching sessions was inconsistent and participants agreed that prioritizing it among their competing demands was challenging, regardless of their engagement level.

Fourth, the Fiscal Mapping Process can support collaboration with funding partners, but only a few service agencies did this. Many other Fiscal Mapping teams reported that communication with funders had to go through someone not on their team. The agencies that used the tool for this purpose still found it challenging to identify and connect with relevant contacts at funding agencies (who often changed or were unresponsive), and described these interactions as sensitive due to the power dynamic in which the funder held most or all of the decision-making authority.

Research Question 2: Impacts of the Fiscal Mapping Process on EBT Sustainment Capacities and Outcomes

Table 5 reports scores from the follow-up questionnaires administered after each focus group (PSAT, PRESS, likelihood of sustainment) including interim ratings, final ratings, and the change in score between ratings. Table 6 details three findings from the outcome evaluation.

Table 5

Interim and Final Sustainment Capacity and Outcome Ratings for Service Agencies Pilot Testing the Fiscal Mapping Process

	Youth Mental Health Service Agency
Rating	#1	#2	#3	#4	#5	#6	#7	#8	#9	#10
Interim Ratings (∼6 mo)	n = 0	n = 3	n = 0	n = 0	n = 1	n = 0	n = 1	n = 2	n = 1	n = 3
PSAT total score^a	NR	4.1	NR	NR	4.5	NR	1.4	5.6	4.6	2.7
Funding Stability	NR	4.2	NR	NR	4.8	NR	1.4	4.9	5.0	2.4
Strategic Planning	NR	3.8	NR	NR	4.6	NR	1.8	6.5	3.6	2.5
Environmental Support	NR	5.2	NR	NR	5.0	NR	1.8	7.0	5.4	4.4
Organizational Capacity	NR	4.6	NR	NR	5.0	NR	1.4	7.0	5.4	3.0
PRESS total score^b	NR	2.8	NR	NR	4.0	NR	1.0	4.0	2.0	1.3
Likelihood of sustainment^c	NR	2.7	NR	NR	3.0	NR	0.0	3.0	3.0	1.7
Final Ratings (∼12 mo)	n = 0	n = 1	n = 0	n = 0	n = 0	n = 0	n = 2	n = 2	n = 1	n = 2
PSAT total score^a	NR	4.7	NR	NR	NR	NR	4.1	6.8	5.0	2.4
Funding Stability	NR	3.8	NR	NR	NR	NR	5.1	5.6	5.2	2.5
Strategic Planning	NR	4.6	NR	NR	NR	NR	3.8	7.0	5.0	1.8
Environmental Support	NR	5.8	NR	NR	NR	NR	4.9	7.0	4.8	4.1
Organizational Capacity	NR	4.2	NR	NR	NR	NR	4.1	7.0	5.8	2.7
PRESS total score^b	NR	3.7	NR	NR	NR	NR	2.3	4.0	3.0	1.5
Likelihood of sustainment^c	NR	2.0	NR	NR	NR	NR	2.0	3.0	3.0	1.0
Change in Scores from Interim to Final Ratings
PSAT total score	N/A	+0.6	N/A	N/A	N/A	N/A	+2.7	+1.2	+0.4	−0.3
Funding Stability	N/A	−0.4	N/A	N/A	N/A	N/A	+3.7	+0.7	+0.2	+0.1
Strategic Planning	N/A	+0.8	N/A	N/A	N/A	N/A	+2.0	+0.5	+1.4	−0.7
Environmental Support	N/A	+0.6	N/A	N/A	N/A	N/A	+3.1	0	−0.6	−0.3
Organizational Capacity	N/A	−0.4	N/A	N/A	N/A	N/A	+2.7	0	+0.4	−0.3
PRESS total score	N/A	+0.9	N/A	N/A	N/A	N/A	+1.3	0	+1.0	+0.2
Likelihood of sustainment	N/A	−0.7	N/A	N/A	N/A	N/A	+2.0	0	0	−0.7

Note. PSAT = Program Sustainability Assessment Tool; PRESS = Provider REport of Sustainment Scale; NR = not reported; N/A = not applicable.

PSAT was scored as follows: 1 = to little or no extent, 7 = to a very great extent.

PRESS was scored as follows: 0 = not at all, 1 = slight extent, 2 = moderate extent, 3 = great extent, 4 = very great extent.

Likelihood of sustainment was scored as follows: −3 = very unlikely, −2 = likely, −1 = somewhat unlikely, 0 = neither likely or unlikely, 1 = somewhat likely, 2 = likely, 3 = very likely.

Table 6

Comparative Case Study Findings for the Outcome Evaluation of Piloting the Fiscal Mapping Process (Research Question 2)

Key Finding	Case comparisons that contributed to finding	Case contrasts that contributed to finding	Illustrative quote(s)
Service agencies that engaged with the Fiscal Mapping Process reported increased capacities to strategically plan for financial sustainment of EBT programs.	Agencies #2, 3, 7, 9 used the Fiscal Mapping Process to make strategic decisions that obtained financial resources for EBT sustainment; usually this funding supplemented inadequate insurance rates via, e.g., grants, contracts, or shifting internal funds.	Agencies #1, 5, 6, 8, 10 did not report improvements, but this was explained by contextual factors (see Table 7) that influenced tool use and engagement in coaching. All but Agency #6 (and #2 leadership) still viewed the Fiscal Mapping Process as useful for strategic planning.	"We’re an agency with so many different programs and funding streams … a lot of competing needs. [By] changing conversations internally to show that there are gaps and how we’re gonna fill them … this tool brings up more awareness." (Service Agency—Clinical Manager)
Increases in agency-reported strategic planning for financial sustainment consistently led to increases in their reported extent of and intentions for EBT sustainment.	Agencies #3, 7, 9 had the most consistent engagement and reported the greatest improvements to EBT sustainment over time. Only these agencies were working on scaling up the EBT to additional clinicians and/or sites (but note that Agency #8 was already fully scaled up).	Sustainment at Agencies #2, 5 was uncertain due to clinical manager and staff turnover; funding capacity was not the major barrier in either case. Agencies #1, 6, 8, 10 did not report changes in sustainment status, consistent with their lack of change on strategic planning or funding stability.	"I know how to make things fiscally work, but the tool shows me how the agency can evolve which allows us to be smarter about how to support evidence-based practices in the future." (Service Agency—Financial Manager)
Impacts of the Fiscal Mapping Process were incremental and best understood over multiyear timespans.	The agencies that made the most progress (#3, 7, 9) still made incremental progress and did not achieve all objectives in the project year.	Training and coaching laid groundwork for agencies with less certain sustainment status to potentially use the tool later, if conditions changed.	"This tool is gonna be really helpful soon … Managed Care has been passed in our state so it will happen within the year, that's a new ball game … I would pull out [the Fiscal Mapping Process] and discuss with our Managed Care entities to see what I could broker for getting things done." (Service Agency—Financial Manager)

Note. EBT = evidence-based treatment. Agency #4 disengaged from the project too early to be included in the Research Question 2 analysis.

First, we found that service agencies that engaged with the Fiscal Mapping Process also reported increased capacities to (a) use strategic decision-making (i.e., strategic planning) for (b) obtaining financial resources for EBT sustainment (i.e., funding stability). The resulting Fiscal Map usually combines service reimbursement from Medicaid or other government sources with supplemental funds from sources like grants, contracts, or shifting internal funds. (As an illustration, see Supplemental File 4 for two hypothetical completed examples of Fiscal Maps, which are also an embedded resource in the published tool; Dopp et al., 2023a.)

Second, when agencies reported increases in strategic planning for financial sustainment, there was a fully consistent association with increased EBT sustainment extent and/or likelihood; that is, in all cases where capacity ratings changed, so did sustainment ratings, and anytime sustainment did not change, we also did not find changes in the targeted capacities. Many agencies with lower engagement with the Fiscal Mapping Process reported maintaining a preproject level of sustainment, but only the agencies with increased capacities reported increased or expanded sustainment (e.g., all planned to scale up the EBT to additional clinicians and/or sites).

Third, the impacts of the Fiscal Mapping Process were incremental and may be best understood over long timespans. None of the agencies in the sample achieved all their funding objectives during the year-long pilot, suggesting that multiple years of follow-up are likely needed to fully understand the impact of the tool.

Research Question 3: Factors That Influenced Use of Fiscal Mapping

Table 5 also reports interim, final, and change scores for two additional PSAT capacities, environmental support and organizational capacity, that were especially relevant to Research Question 3. Furthermore, PSAT total scores are presented, which summarize all eight sustainment capacity domains captured by that measure.

Table 7 presents the five findings from our analysis of contextual influences. Before describing those findings, it is important to recognize that the COVID-19 pandemic was a major contextual influence shared by all service agencies in this project. Although none of our findings were pandemic-specific, the strain on communities and health systems certainly contributed to the dynamics of these factors and was noted by many participants (see Supplemental File 3).

Table 7

Comparative Case Study Findings for Contextual Influences on Piloting the Fiscal Mapping Process (Research Question 3)

Key Finding	Case comparisons that contributed to finding	Case contrasts that contributed to finding	Illustrative quote(s)
Youth-serving system capacities and strategies for funding EBTs influenced the impact of the Fiscal Mapping Process by determining the funding sources available to coordinate for EBT delivery and sustainment activities.	Agencies #1, 3, 5, 6, 10 had limited funding sources available for EBT resource needs, which generally did not cover actual costs (e.g., fee-for-service reimbursement, which actually decreased in multiple cases over the course of pilot testing), was difficult to renew (e.g., grants), or was outside the EBT program's control (e.g., external budget allocations)	Agencies #2, 7, 9 were able to identify a greater diversity of funding sources, but those still had limited feasibility or effectiveness for sustaining EBTs. Only Agency #8 reported no major concerns with system funding options. Agencies #3, 7, 8 identified the availability of statewide EBT training as benefiting sustainment, especially if a state agency covered the costs	"With Medicaid transformation most funders [in the state] are not doing an enhanced rate [for TF-CBT] anymore. You would think 'transformation' would be a positive thing, but really it's very disheartening. We know a lot of agencies that shut down or stopped taking Medicaid. [We are] in a good place mainly due to grant funding." (Service Agency—Clinical Manager)
Service agency leadership support for engaging in EBT-specific sustainment planning via the Fiscal Mapping Process influenced the impact of the tool by setting priorities for the organization and team members; this was especially influential when the EBT program was based in an organization with a broader mission beyond mental health.	Agencies #1, 2, 5, 6, 10 had low leadership support for the use of Fiscal Mapping; generally, cost-reduction measures (e.g., use of contractor clinicians) were preferred to identifying additional funding sources for EBT sustainment. Leaders for Agencies #5, 10 also had a broader nonmental health mission. (Leader priorities were also influenced by the service system supports)	Agencies #3, 7, 9 had support or engagement of their leadership for Fiscal Mapping, but had difficulty routinely making it a priority with key decision-makers; senior leaders have limited capacity for any one task. Agency #8 did not prioritize Fiscal Mapping due to lack of financial sustainment barriers.	"I think we do have good support from our organization, but our needs [as an evidence-based treatment program] are more minor in comparison to other things that must be done for the hospital." (Service Agency—Senior Leader)
Turnover of clinicians and managers impacted the impact of the Fiscal Mapping Process by threatening agency capacity to deliver EBTs, creating urgent needs that were prioritized over sustainment planning, and through turnover in key members of Fiscal Mapping teams.	Agencies #1, 2, 3, 5, 8, 9, 10 experienced challenges with turnover of EBT clinicians and managers, resulting in vacant positions and the need for managers and leadership to cover clinical responsibilities and maintain service delivery. Agency #7 had turnover of clinicians willing to be part of an EBT-specific program. A member of the Fiscal Mapping Team left Agencies #2, 4, and 5 during pilot testing.	EBT champions prioritized ongoing delivery and training for the EBTs, helping mitigate the impacts of turnover by ensuring that some current staff were trained in the EBT (Agencies #6, 9). Turnover was a bigger threat when champions were not present (#5) or left the agency (#2).	"The biggest challenge is lack of staffing. I am helping fill in for someone on maternity leave so I have more responsibilities, but in general, there is limited time to dedicate to [Fiscal Mapping]." (Service Agency—Senior Leader)
Financial sustainment planning with the Fiscal Mapping Process was even more challenging when the service agency prioritized populations experiencing mental health inequities, especially with low-income or BIPOC populations.	All cases prioritized low-income populations; tension between commitment to serving these populations and inadequate reimbursement. Agencies #3, 6 considered shifting to other funding sources but were concerned this would compromise their focus on underserved youth. BIPOC youth were prioritized by Agencies #2, 4, 6, 10, who had smaller EBT programs and less engagement from senior leaders and/or financial managers in Fiscal Mapping. All stopped coaching or had limited benefits.	Agencies #3, 6, 7, 9 prioritized rural populations. Access issues affected EBT delivery in rural communities, such as low caseloads (#3, 9) or challenges delivering EBTs via telehealth or home-based delivery (#3, 6, 7), but Fiscal Mapping tended to still be impactful for those cases. The exception was Agency #6, which also prioritized BIPOC communities and dropped out.	"We are working with private [insurance] contractors to increase the rate for PCIT. This can supplement [government payors with low rates] but we don’t have that many private cases. [Agency] really focuses on underserved populations." (Service Agency—Senior Leader/Clinical Manager)
Service agencies viewed the Fiscal Mapping Process as likely to generalize across a variety of behavioral health EBTs.	Agencies #7, 8, 9, 10 reported intentions to use the tool for sustaining other treatments beyond PCIT or TF-CBT.	N/A—no agencies used the Fiscal Mapping Process for a different EBT, so we cannot compare between cases.	"So even if the tool is only designed for certain [evidence-based treatments], I think it could be used for other things. It's a strategic planning tool that helps you plan the tactical day to day [of program sustainment]." (Service Agency—Clinical Manager)

Note. Agency #4 disengaged from the project too early to be included in the Research Question 3 analysis. BIPOC = Black, Indigenous, and other people of color; EBT = evidence-based treatment; PCIT = Parent–Child Interaction Therapy; TF-CBT = Trauma-Focused Cognitive-Behavioral Therapy.

Our first finding was that youth-serving system capacities and strategies for funding EBTs influenced the impact of the Fiscal Mapping Process (extra-organizational environmental support). The available funding determined the options that could be coordinated for EBT delivery and sustainment activities through the tool. Funding sources were often limited in feasibility (for service agencies to obtain) and effectiveness (in providing adequate funding to sustain EBTs), hence the need to coordinate multiple funding sources. Medicaid and other government reimbursement rates were viewed as inadequately low, and several agencies experienced rate decreases during the project, which made Fiscal Mapping feel less useful.

Second, service agency leadership support for engaging in EBT sustainment planning and decision-making influenced the impact of the Fiscal Mapping Process (internal environmental support). Leadership set priorities for the organization and team members, including the relative importance of Fiscal Mapping, and those priorities were influenced by external system supports as well. Consistently prioritizing strategic planning for one EBT was challenging even with supportive leaders, given the numerous administrative demands they experienced; when leaders were not supportive, they generally favored reducing costs instead. Leadership priorities were especially influential in two cases where the EBT program was based in an organization with a broader mission beyond mental health services.

Third, the turnover of clinicians and managers influenced the impact of the Fiscal Mapping Process by focusing attention on the organizational capacity to deliver the EBT. Reasons for turnover varied (e.g., leaving the agency due to low pay, COVID-related issues, vacancy created by promotion, clinician unwillingness to serve in an EBT-specific program) but consistently created a dynamic in which maintaining service delivery had to be prioritized over long-term strategic planning. Nearly every agency experienced turnover-related challenges, including turnover in key members of their Fiscal Mapping teams. However, the presence of individuals who championed (i.e., made a priority) delivery of the EBT mitigated the impact in some cases, as champions worked to ensure that some current staff were always trained in the EBT.

Fourth, Fiscal Mapping was especially challenging when the service agency prioritized populations experiencing mental health inequities. When serving low-income populations, service agencies experienced tension between their commitment to serving these populations and reimbursement sources for low-income populations (e.g., Medicaid) being inadequate to sustain EBTs. Agencies in our sample that prioritized serving Black, Indigenous, and other people of color (BIPOC) faced even more unique challenges and were least likely to benefit. These agencies tended to be smaller (i.e., fewer sites and/or EBT clients) and did not have consistent input from senior leaders and/or financial managers on their Fiscal Mapping teams; one agency also noted that training in culturally adapted EBT was expensive and difficult to access. In contrast, participating agencies that prioritized rural populations, despite still encountering access challenges, were relatively successful with the tool.

Finally, service agency representatives viewed the Fiscal Mapping Process as likely to generalize across a variety of behavioral health EBTs beyond PCIT and TF-CBT. Nearly half of the agencies reported intentions to use the tool for sustaining other treatments, although they typically described it as broadly applicable and did not detail plans for using the tool with specific treatment models.

Discussion

We used comparative case methods to complete a pilot evaluation of the Fiscal Mapping Process, a tool that helps youth mental service agencies coordinate financing strategies for sustaining EBTs. Using mixed-method data sources, we compared and contrasted the process, outcomes, and contextual influences experienced by 10 U.S. service agencies that pilot-tested the Fiscal Mapping Process to plan for the sustainment of TF-CBT or PCIT. Our comparative case study design enabled us to identify complex and nuanced explanations across diverse sources of evidence despite the small sample available for this pilot study, similar to the rich findings of prior comparative case studies from implementation research (Bullock & Lavis, 2019; Dickson et al., 2022; Folker et al., 2018; Thompson et al., 2022; Zeitlin et al., 2016). Moreover, our findings provide in-depth, real-world illustrations of theoretical concepts from public finance and nonprofit services research, showing how organizations pursue their goals—in this case, sustaining EBTs—by navigating complex resource networks to identify and secure diverse funding sources (see e.g., Hillman et al., 2009; Weber & Waeger, 2017).

Our process evaluation findings helped clarify the primary use case for the Fiscal Mapping Process was with service agencies that (a) had or were developing EBT programs; (b) focused on EBT programs with immediate financial sustainment challenges; and (c) engaged clinical manager, financial manager, and senior leader representatives. Other process findings highlighted the importance—and challenges—of service agencies engaging with external collaborations, such as coaches and funding partners, while using the tool. Our outcome evaluation findings documented impacts of Fiscal Mapping Process use on both sustainment capacities (strategic planning and funding stability) and outcomes (extent and likelihood) ratings for engaged service agencies. Furthermore, the relation between capacities and outcomes was fully consistent, providing robust initial evidence that the Fiscal Mapping Process supports EBT sustainment through the capacities it targeted.

Full understanding of implementation mechanisms must account for contextual factors, as demonstrated by the Research Question 3 findings. Multiple dimensions of environmental support (both external and internal) and organizational capacity (especially related to workforce turnover) influenced the impact of the Fiscal Mapping Process, largely explaining cases in which service agencies did not strongly engage with and obtain benefit from the tool. Given these contextual influences, it is important to treat financial sustainment of EBTs as a complex system that the Fiscal Mapping Process can help service agencies navigate, rather than seeking a singular mechanism by which the tool exerts its effects. For example, a recent study using the Public Health Sustainability Framework found that multiple combinations of sustainment capacities were adequate for sustainment, depending on the context (consistent with our case examples, e.g., improving strategic planning was less relevant for the service agency that already had high sustainment capacity in other domains; Cooper et al., 2022).

The youth population served was another important contextual influence, with notably limited benefits of the Fiscal Mapping Process for agencies that primarily serve low-income and especially BIPOC youth—a major concern for equitable outcomes. This builds on equity-focused implementation research, which consistently documents how efforts to implement and sustain EBTs often maintain or increase mental health disparities unless achieving health equity is centered as the primary goal (Baumann & Cabassa, 2020; Park et al., 2018; Shelton et al., 2023). More broadly, U.S. mental health systems are chronically underfunded (Dieleman et al., 2020), with serious workforce issues such as low pay, high education debt, and limited professional development (Conrad, 2023; Goldman et al., 2020) fueling turnover and poor service outcomes—especially in public systems that primarily serve low-income and BIPOC families (Last et al., 2022). We must incorporate an equity focus into implementation tools and strategies like the Fiscal Mapping Process, to make it more responsive and useful for navigating the major system-level issues that inequities in mental health service recipients and providers.

There are important limitations to the generalizability of our findings. We piloted the Fiscal Mapping Process with a small sample of service agencies that had the capacity and interest for a long-term research project. We were also limited to a single year and only able to observe incremental impacts, rather than larger-scale changes in, for example, EBT delivery volume. All pilot-testing service agencies were already sustaining an EBT, which may have limited how much benefit Fiscal Mapping could provide; for example, measures of agency implementation climate (ICS) and financial status (AFSS) were not as useful as we anticipated for our case study analysis, because they showed limited variability (see Table 2). Nevertheless, these limitations are all reasonable considering that pilot studies are meant to understand the feasibility and potential of novel approaches, rather than provide replicable estimates of intervention effects (Leon et al., 2011). Furthermore, we must continue adapting the Fiscal Mapping Process to disseminate it broadly. Many service agencies found coaching to be essential, but our coaching model was research-funded and not scalable. Furthermore, participants viewed the Fiscal Mapping Process as generalizable to other behavioral health treatments and service models, but we were not able to directly examine generalizability beyond PCIT and TF-CBT; it remains unclear how much customization of the step content and resources would be required for other EBTs, especially beyond youth mental health EBTs designed for individual, outpatient delivery (e.g., complex, high-intensity models like Multisystemic Therapy; Henggeler & Schaeffer, 2019).

To address these evaluation and generalizability limitations, we plan to support a more widespread scale-up of the Fiscal Mapping Process by embedding it in Learning Collaboratives and evaluating its effects. Learning Collaboratives are an implementation strategy in which an intermediary brings together multiple service agencies and roles (e.g., clinicians, senior leaders) to learn an EBT and develop plans to sustain it (Ebert et al., 2012). We envision intermediaries coaching service agencies in use of the Fiscal Mapping Process, and brokering conversations with senior leaders and funding agencies to improve funding stability (Lengnick-Hall et al., 2020). Learning Collaboratives could also support the use of Fiscal Mapping across implementation phases (Moullin et al., 2021), such as preimplementation decisions about which EBTs are financially viable. Regarding future evaluation, we plan to examine the effects of the Fiscal Mapping Process (vs. usual Learning Collaboratives without the Fiscal Mapping Process) on a range of implementation and service outcomes (e.g., number of EBT recipients; Proctor et al., 2011) across multiple years and implementation phases. It will also be important to further test strategic planning and funding stability capacities as mechanisms of the tool's effects (Lewis et al., 2018), and whether environmental and organizational capacities can also be targeted.

In sum, we pilot-tested and evaluated a Fiscal Mapping Process for promoting financial sustainment of EBTs in youth mental health service agencies. We found that the tool shows promise for sustainment planning, but needs refinements to be scalable within existing implementation supports (e.g., Learning Collaboratives)—especially system issues that drive mental health inequities. We will continue to pursue development and large-scale evaluation of the Fiscal Mapping Process to help direct resources where they are most needed to support effective practices and promote health.

Supplemental Material

sj-docx-1-irp-10.1177_26334895241249394 - Supplemental material for Pilot evaluation of the Fiscal Mapping Process for sustainable financing of evidence-based youth mental health treatments: A comparative case study analysis

Supplemental material, sj-docx-1-irp-10.1177_26334895241249394 for Pilot evaluation of the Fiscal Mapping Process for sustainable financing of evidence-based youth mental health treatments: A comparative case study analysis by Alex R. Dopp, Maddison N. North, Marylou Gilbert, Jeanne S. Ringel, Jane F. Silovsky, Mellicent Blythe, Dan Edwards, Susan Schmidt and Beverly Funderburk in Implementation Research and Practice

Supplemental Material

sj-docx-2-irp-10.1177_26334895241249394 - Supplemental material for Pilot evaluation of the Fiscal Mapping Process for sustainable financing of evidence-based youth mental health treatments: A comparative case study analysis

Supplemental material, sj-docx-2-irp-10.1177_26334895241249394 for Pilot evaluation of the Fiscal Mapping Process for sustainable financing of evidence-based youth mental health treatments: A comparative case study analysis by Alex R. Dopp, Maddison N. North, Marylou Gilbert, Jeanne S. Ringel, Jane F. Silovsky, Mellicent Blythe, Dan Edwards, Susan Schmidt and Beverly Funderburk in Implementation Research and Practice

Supplemental Material

sj-docx-3-irp-10.1177_26334895241249394 - Supplemental material for Pilot evaluation of the Fiscal Mapping Process for sustainable financing of evidence-based youth mental health treatments: A comparative case study analysis

Supplemental material, sj-docx-3-irp-10.1177_26334895241249394 for Pilot evaluation of the Fiscal Mapping Process for sustainable financing of evidence-based youth mental health treatments: A comparative case study analysis by Alex R. Dopp, Maddison N. North, Marylou Gilbert, Jeanne S. Ringel, Jane F. Silovsky, Mellicent Blythe, Dan Edwards, Susan Schmidt and Beverly Funderburk in Implementation Research and Practice

Supplemental Material

sj-xlsx-4-irp-10.1177_26334895241249394 - Supplemental material for Pilot evaluation of the Fiscal Mapping Process for sustainable financing of evidence-based youth mental health treatments: A comparative case study analysis

Supplemental material, sj-xlsx-4-irp-10.1177_26334895241249394 for Pilot evaluation of the Fiscal Mapping Process for sustainable financing of evidence-based youth mental health treatments: A comparative case study analysis by Alex R. Dopp, Maddison N. North, Marylou Gilbert, Jeanne S. Ringel, Jane F. Silovsky, Mellicent Blythe, Dan Edwards, Susan Schmidt and Beverly Funderburk in Implementation Research and Practice

Footnotes

Acknowledgments

First and foremost, we are grateful to the service agency, funding agency, and intermediary representatives who partnered with us to design and evaluate the Fiscal Mapping Process. Second, we thank Isabelle González and Grace Hindmarch for their assistance with administering focus groups. Third, we appreciate the contributions of expert consultants Dana Hagele, Douglas Luke, David Mandell, and Byron Powell to the design and execution of this project. Fourth, we thank Trista Campos, Olatunda Martin, and Natalie Richards for project administration, including the management of our project newsletter. Finally, we are grateful to Monique Martineau, Kristen Meadows, Emily Cantin, and Haley Okuley for their contributions to the design and development of the Fiscal Mapping Process.

Authors’ Contributions

AD designed and oversaw all aspects of the study, including all data collection and analysis, and wrote the first draft of this article. MN and MG also contributed to data collection and analysis; AD and MG wrote the first draft of each case summary, and MN refined and standardized all case summaries, after which all three authors completed the comparative case study analysis. All other authors contributed to conceptualization of the study, development of the data collection and analysis plans, and interpretation of findings, with an order of authorship determined by the extent of their contributions. All authors reviewed and provided feedback on this article.

Availability of Data and Materials

Quantitative data generated and analyzed during this study are available through the NIMH Data Archive: . Due to the small sample size and in-depth case descriptives provided, our qualitative data are not available through any data archives, but deidentified data are available from the corresponding author upon reasonable request with adequate protections for confidentiality in place.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethics Approval and Consent to Participate

All procedures were reviewed by the RAND Institutional Review Board and determined to not constitute human subjects research (Protocol #2020-N0607); nevertheless, we followed all ethical principles for the protection of human research participants to minimize any risk of harm during their participation, including the collection of informed consent at the start of the project and as part of each data collection activity.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was directly supported by an award from the U.S. National Institute of Mental Health (R21MH122889; principal investigator: Alex R. Dopp). The funder did not influence the content of this article; furthermore, the views expressed are those of the authors and do not necessarily represent the views of the National Institute of Mental Health.

ORCID iDs

Alex R. Dopp

Beverly Funderburk

Supplemental Material

Supplemental material for this article is available online.

References

Aarons

G. A.

Green

A. E.

Willging

C. E.

Ehrhart

M. G.

Roesch

S. C.

Hecht

D. B.

Chaffin

M. J.

(2014). Mixed-method study of a conceptual model of evidence-based intervention sustainment across multiple public-sector service settings. Implementation Science, 9, 183. https://doi.org/10.1186/s13012-014-0183-z

Alves-Bradford

J. M.

Trinh

N. H.

Bath

Coombs

Mangurian

(2020). Mental health equity in the twenty-first century: Setting the stage. Psychiatric Clinics, 43(3), 415–428. https://doi.org/10.1016/j.psc.2020.05.001

Bartlett

Vavrus

(2017). Comparative case studies: An innovative approach. Nordic Journal of Comparative and International Education (NJCIE), 1(1), 5–17. https://doi.org/10.7577/njcie.1929

Bauer

M. S.

Kirchner

(2020). Implementation science: What is it and why should I care? Psychiatry Research, 283, 112376. https://doi.org/10.1016/j.psychres.2019.04.025

Baumann

A. A.

Cabassa

L. J.

(2020). Reframing implementation science to address inequities in healthcare delivery. BMC Health Services Research, 20, 190. https://doi.org/10.1186/s12913-020-4975-3

Bitsko

R. H.

Claussen

A. H.

Lichstein

Black

L. I.

Jones

S. E.

Danielson

M. L.

Hoenig

J. M.

Jack

S. P. D.

Brody

D. J.

Gyawali

Maenner

M. J.

Warner

Holland

K. M.

Perou

Crosby

A. E.

Blumberg

S. J.

Avenevoli

Kaminski

J. W.

Ghandour

R. M.

(2022). Mental health surveillance among children – United States, 2013-2019. MMWR Supplements, 71(2), 1–42. https://doi.org/10.15585/mmwr.su7102a1

Bond

G. R.

Drake

R. E.

McHugo

G. J.

Peterson

A. E.

Jones

A. M.

Williams

(2014). Long-term sustainability of evidence-based practices in community mental health agencies. Administration and Policy in Mental Health and Mental Health Services Research, 41, 228–236. https://doi.org/10.1007/s10488-012-0461-5

Bullock

H. L.

Lavis

J. N.

(2019). Understanding the supports needed for policy implementation: A comparative analysis of the placement of intermediaries across three mental health systems. Health Research Policy and Systems, 17, 82. https://doi.org/10.1186/s12961-019-0479-1

Calhoun

Mainor

Moreland-Russell

Maier

R. C.

Brossart

Luke

D. A.

(2014). Using the program sustainability assessment tool to assess and plan for sustainability. Preventing Chronic Disease, 11, 130185. https://doi.org/10.5888/pcd11.130185

10.

Center for Public Health System Science. (2012). Program sustainability assessment tool . https://www.sustaintool.org/

11.

Chambers

D. A.

Glasgow

R. E.

Stange

K. C.

(2013). The dynamic sustainability framework: Addressing the paradox of sustainment amid ongoing change. Implementation Science, 8, 117. https://doi.org/10.1186/1748-5908-8-117

12.

Conrad

(2023, February 8). A call for a national strategy to strengthen the youth behavioral health workforce . The Commonwealth Fund. https://doi.org/10.26099/ayvv-7m44

13.

Cooper

B. R.

Hill

L. G.

Parker

Jenkins

G. J.

Shrestha

Funaiole

(2022). Using qualitative comparative analysis to uncover multiple pathways to program sustainment: Implications for community-based youth substance misuse prevention. Implementation Science Communications, 3, 55. https://doi.org/10.1186/s43058-022-00303-4

14.

Dickson

K. S.

Sklar

Chen

S. Z.

Kim

(2022). Characterization of multilevel influences of mental health care transitions: A comparative case study analysis. BMC Health Services Research, 22, 437. https://doi.org/10.1186/s12913-022-07748-2

15.

Dieleman

J. L.

Cao

Chapin

Chen

Liu

Horst

Kaldjian

Matyasz

Scott

K. W.

Bui

A. L.

Campbell

Duber

H. C.

Dunn

A. C.

Flaxman

A. D.

Fitzmaurice

Naghavi

Sadat

Shieh

, … & Murray

C. J. L.

(2020). US health care spending by payer and health condition, 1996–2016. JAMA, 323(9), 863–884. https://doi.org/10.1001/jama.2020.0734

16.

Dopp

A. R.

Gilbert

North

M. N.

Martineau

Ringel

J. S.

(2023a). The Fiscal Mapping Process: A strategic planning tool for sustainable financing of evidence-based treatment programs in youth behavioral health services. https://doi.org/10.7249/TLA2678-1

17.

Dopp

A. R.

Gilbert

North

M. N.

Martineau

Ringel

J. S.

. (2023b). The Fiscal Mapping Process: Development of a strategic planning tool for sustainable financing of evidence-based treatment programs in youth behavioral health services . The RAND Corporation.

18.

Dopp

A. R.

Gilbert

Silovsky

Ringel

J. S.

Schmidt

Funderburk

Jorgensen

Powell

B. J.

Luke

D. A.

Mandell

Edwards

Blythe

Hagele

(2022). Coordination of sustainable financing for evidence-based youth mental health treatments: Protocol for development and evaluation of the fiscal mapping process. Implementation Science Communications, 3(1), 1–15. https://doi.org/10.1186/s43058-021-00234-6

19.

Dopp

A. R.

Narcisse

M. R.

Mundey

Silovsky

J. F.

Smith

A. B.

Mandell

Funderburk

B. W.

Powell

B. J.

Schmidt

Edwards

Luke

Mendel

(2020). A scoping review of strategies for financing the implementation of evidence-based practices in behavioral health systems: State of the literature and future directions. Implementation Research and Practice, 1.

20.

Dorsey

McLaughlin

K. A.

Kerns

S. E. U.

Harrison

J. P.

Lambert

H. K.

Briggs

E. C.

Cox

J. R.

Amaya-Jackson

(2017). Evidence base update for psychosocial treatments for children and adolescents exposed to traumatic events. Journal of Clinical Child & Adolescent Psychology, 46(3), 303–330. https://doi.org/10.1080/15374416.2016.1220309

21.

Doupnik

S. K.

Rodean

Feinstein

Gay

J. C.

Simmons

Bettenhausen

J. L.

Markham

J. L.

Hall

Zima

B. T.

Berry

J. G.

(2020). Health care utilization and spending for children with mental health conditions in Medicaid. Academic Pediatrics, 20(5), 678–686. https://doi.org/10.1016/j.acap.2020.01.013

22.

Duong

M. T.

Bruns

E. J.

Lee

Cox

Coifman

Mayworm

Lyon

A. R.

(2021). Rates of mental health service utilization by children and adolescents in schools and other common service settings: A systematic review and meta-analysis. Administration and Policy in Mental Health and Mental Health Services Research, 48, 420–439. https://doi.org/10.1007/s10488-020-01080-9

23.

Ebert

Amaya-Jackson

Markiewicz

J. M.

Kisiel

Fairbank

J. A.

(2012). Use of the breakthrough series collaborative to support broad and sustained use of evidence-based trauma treatment for children in community practice settings. Administration and Policy in Mental Health and Mental Health Services Research, 39(3), 187–199. https://doi.org/10.1007/s10488-011-0347-y

24.

Ehrhart

M. G.

Aarons

G. A.

Farahnak

L. R.

(2014). Assessing the organizational context for EBP implementation: The development and validity testing of the Implementation Climate Scale (ICS). Implementation Science, 9, 157. https://doi.org/10.1186/s13012-014-0157-1

25.

Fernandez

M. E.

Ten Hoor

G. A.

Van Lieshout

Rodriguez

S. A.

Beidas

R. S.

Parcel

Ruiter

R. A. C.

Markham

C. M.

Kok

(2019). Implementation mapping: Using intervention mapping to develop implementation strategies. Frontiers in Public Health, 7, 158. https://doi.org/10.3389/fpubh.2019.00158

26.

Folker

A. P.

Mathiasen

Lauridsen

S. M.

Stenderup

Dozeman

Folker

M. P.

(2018). Implementing internet-delivered cognitive behavior therapy for common mental health disorders: A comparative case study of implementation challenges perceived by therapists and managers in five European internet services. Internet Interventions, 11, 60–70. https://doi.org/10.1016/j.invent.2018.02.001

27.

Franks

R. P.

Bory

C. T.

(2015). Who supports the successful implementation and sustainability of evidence-based practices? Defining and understanding the roles of intermediary and purveyor organizations. New Directions for Child and Adolescent Development, 2015(149), 41–56. https://doi.org/10.1002/cad.20112

28.

Goldman

M. L.

Druss

B. G.

Horvitz-Lennon

Norquist

G. S.

Kroeger Ptakowski

Brinkley

Greiner

Hayes

Hepburn

Jorgensen

Swatz

M. S.

Dixon

L. B.

(2020). Mental health policy in the era of COVID-19. Psychiatric Services, 71(11), 1158–1162. https://doi.org/10.1176/appi.ps.202000219

29.

Hamilton

A. B.

Finley

E. P.

(2019). Qualitative methods in implementation research: An introduction. Psychiatry Research, 280, 112516. https://doi.org/10.1016/j.psychres.2019.112516

30.

Henggeler

S. W.

Schaeffer

C. M.

(2019). Multisystemic therapy®: Clinical procedures, outcomes, and implementation research. In Fiese

B. H.

Celano

Deater-Deckard

Jouriles

E. N.

Whisman

M. A.

(Eds.), APA handbook of contemporary family psychology: Family therapy and training (pp. 205–220). American Psychological Association. https://doi.org/10.1037/0000101-013

31.

Hillman

A. J.

Withers

M. C.

Collins

B. J.

(2009). Resource dependence theory: A review. Journal of Management, 35(6), 1404–1427. https://doi.org/10.1177/0149206309343469

32.

Hoffmann

T. C.

Glasziou

P. P.

Boutron

Milne

Perera

Moher

Altman

D. G.

Barbour

Macdonald

Johnston

Kadoorie

S. E. L.

Dixon-Woods

McCulloch

Wyatt

J. C.

Chan

Michie

(2014). Better reporting of interventions: Template for intervention description and replication (TIDieR) checklist and guide. British Medical Journal, 348. g1687. https://doi.org/10.1136/bmj.g1687

33.

Jaramillo

E. T.

Willging

C. E.

Green

A. E.

Gunderson

L. M.

Fettes

D. L.

Aarons

G. A.

(2019). “Creative financing”: Funding evidence-based interventions in human service systems. The Journal of Behavioral Health Services & Research, 46, 366–383. https://doi.org/10.1007/s11414-018-9644-5

34.

Kaminski

J. W.

Claussen

A. H.

(2017). Evidence base update for psychosocial treatments for disruptive behaviors in children. Journal of Clinical Child & Adolescent Psychology, 46(4), 477–499. https://doi.org/10.1080/15374416.2017.1310044

35.

Kazak

A. E.

Hoagwood

Weisz

J. R.

Hood

Kratochwill

T. R.

Vargas

L. A.

Banez

G. A.

(2010). A meta-systems approach to evidence-based practice for children and adolescents. American Psychologist, 65(2), 85–97. https://doi.org/10.1037/a0017784

36.

Lang

J. M.

Connell

C. M.

(2017). Measuring costs to community-based agencies for implementation of an evidence-based practice. The Journal of Behavioral Health Services & Research, 44, 122–134. https://doi.org/10.1007/s11414-016-9541-8

37.

Last

B. S.

Schriger

S. H.

Becker-Haimes

E. M.

Fernandez-Marcote

Dallard

Jones

Beidas

R. S.

(2022). Economic precarity, financial strain, and job-related stress among Philadelphia’s public mental health clinicians. Psychiatric Services, 73(7), 774–786. https://doi.org/10.1176/appi.ps.202100276

38.

Lengnick-Hall

Willging

Hurlburt

Fenwick

Aarons

G. A.

(2020). Contracting as a bridging factor linking outer and inner contexts during EBP implementation and sustainment: A prospective study across multiple US public sector service systems. Implementation Science, 15, 43. https://doi.org/10.1186/s13012-020-00999-9

39.

Leon

A. C.

Davis

L. L.

Kraemer

H. C.

(2011). The role and interpretation of pilot studies in clinical research. Journal of Psychiatric Research, 45(5), 626–629. https://doi.org/10.1016/j.jpsychires.2010.10.008

40.

Lewis

C. C.

Klasnja

Powell

B. J.

Lyon

A. R.

Tuzzio

Jones

Walsh-Bailey

Weiner

(2018). From classification to causality: Advancing understanding of mechanisms of change in implementation science. Frontiers in Public Health, 6, 136. https://doi.org/10.3389/fpubh.2018.00136

41.

Masters

Anwar

Collins

Cookson

Capewell

(2017). Return on investment of public health interventions: A systematic review. Journal of Epidemiology and Community Health, 71(8), 827–834. https://doi.org/10.1136/jech-2016-208141

42.

Maxwell

C. A.

Ehrhart

M. G.

Williams

N. J.

Moore

T. M.

Kendall

P. C.

Beidas

R. S.

(2021). The organizational financial context of publicly-funded mental health clinics: Development and preliminary psychometric evaluation of the agency financial status scales. Administration and Policy in Mental Health and Mental Health Services Research, 48, 780–792. https://doi.org/10.1007/s10488-021-01128-4

43.

Mei

Fitzsimons

Allen

Alvarez-Jimenez

Amminger

G. P.

Browne

Cannon

Davis

Dooley

Hickie

I. B.

Iyer

Killackey

Malla

Maion

Mathias

Pennell

Purcell

Rickwood

Singh

S. P.

, … & McGorry

P. D.

(2020). Global research priorities for youth mental health. Early Intervention in Psychiatry, 14(1), 3–13. https://doi.org/10.1111/eip.12878

44.

Moullin

J. C.

Sklar

Ehrhart

M. G.

Green

Aarons

G. A.

(2021). Provider report of sustainment scale (PRESS): Development and validation of a brief measure of inner context sustainment. Implementation Science, 16, 86. https://doi.org/10.1186/s13012-021-01152-w

45.

Mundey

Slemaker

Dopp

A. R.

Beasley

L. O.

Silovsky

J. F.

(2021). Sustaining treatment for youth with problematic sexual behavior: Administrator and stakeholder perspectives following implementation. The Journal of Behavioral Health Services & Research, 48, 410–426. https://doi.org/10.1007/s11414-020-09726-0

46.

North

M. N.

Dopp

A. R.

Silovsky

J. F.

Gilbert

Ringel

J. S.

(2023). Perspectives on financing strategies for evidence-based treatment implementation in youth mental health systems. Journal of Mental Health Policy and Economics, 26(3), 115–130.

47.

Okwori

(2022). Prevalence and correlates of mental health disorders among children & adolescents in US. Children and Youth Services Review, 136, 106441. https://doi.org/10.1016/j.childyouth.2022.106441

48.

Palinkas

L. A.

Aarons

G. A.

Horwitz

Chamberlain

Hurlburt

Landsverk

(2011). Mixed method designs in implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 38, 44–53. https://doi.org/10.1007/s10488-010-0314-z

49.

Park

A. L.

Tsai

K. H.

Guan

Chorpita

B. F.

(2018). Unintended consequences of evidence-based treatment policy reform: Is implementation the goal or the strategy for higher quality care? Administration and Policy in Mental Health and Mental Health Services Research, 45, 649–660. https://doi.org/10.1007/s10488-018-0853-2

50.

Pinnock

Barwick

Carpenter

C. R.

Eldridge

Grandes

Griffiths

C. J.

Rycroft-Malone

Meissner

Murray

Patel

Sheikh

Taylor

S. J. C.

(2017). Standards for reporting implementation studies (StaRI) statement. BMJ, 356: i6795. https://doi.org/10.1136/bmj.i6795

51.

Pollio

McLean

Behl

L. E.

Deblinger

(2014). Trauma-focused cognitive behavioral therapy. In Reece

R. M.

Hanson

R. F.

Sargent

(Eds.), Treatment of child abuse: Common ground for mental health, medical, and legal practitioners (pp. 31–38). Johns Hopkins University Press.

52.

Powell

B. J.

Beidas

R. S.

Lewis

C. C.

Aarons

G. A.

McMillen

J. C.

Proctor

E. K.

Mandell

D. S.

(2017). Methods to improve the selection and tailoring of implementation strategies. The Journal of Behavioral Health Services & Research, 44, 177–194. https://doi.org/10.1007/s11414-015-9475-6

53.

Powell

B. J.

Waltz

T. J.

Chinman

M. J.

Damschroder

L. J.

Smith

J. L.

Matthieu

M. M.

Proctor

E. K.

Kirchner

J. E.

(2015). A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10, 21. https://doi.org/10.1186/s13012-015-0209-1

54.

Proctor

Silmere

Raghavan

Hovmand

Aarons

Bunger

Griffey

Hensley

(2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38, 65–76. https://doi.org/10.1007/s10488-010-0319-7

55.

Roundfield

K. D.

Lang

J. M.

(2017). Costs to community mental health agencies to sustain an evidence-based practice. Psychiatric Services, 68(9), 876–882. https://doi.org/10.1176/appi.ps.201600193

56.

Schell

S. F.

Luke

D. A.

Schooley

M. W.

Elliott

M. B.

Herbers

S. H.

Mueller

N. B.

Bunger

A. C.

(2013). Public health program capacity for sustainability: A new framework. Implementation Science, 8, 15. https://doi.org/10.1186/1748-5908-8-15

57.

Scudder

A. T.

Taber-Thomas

S. M.

Schaffner

Pemberton

J. R.

Hunter

Herschell

A. D.

(2017). A mixed-methods study of system-level sustainability of evidence-based practices in 12 large-scale implementation initiatives. Health Research Policy and Systems, 15, 102. https://doi.org/10.1186/s12961-017-0230-8

58.

Shelton

R. C.

Cooper

B. R.

Stirman

S. W.

(2018). The sustainability of evidence-based interventions and practices in public health and health care. Annual Review of Public Health, 39, 55–76. https://doi.org/10.1146/annurev-publhealth-040617-014731

59.

Shelton

R. C.

Hailemariam

Iwelunmore

(2023). Making the connection between health equity and sustainability. Frontiers in Public Health, 11, 1226175. https://doi.org/10.3389/fpubh.2023.1226175

60.

Shim

R. S.

(2021). Dismantling structural racism in psychiatry: A path to mental health equity. American Journal of Psychiatry, 178(7), 592–598. https://doi.org/10.1176/appi.ajp.2021.21060558

61.

Stewart

R. E.

Adams

D. R.

Mandell

D. S.

Hadley

T. R.

Evans

A. C.

Rubin

Erney

Neimark

Hurford

M. O.

Beidas

R. S.

(2016). The perfect storm: Collision of the business of mental health and the implementation of evidence-based practices. Psychiatric Services, 67(2), 159–161. https://doi.org/10.1176/appi.ps.201500392

62.

Stroul

B. A.

(2007). Issue brief 1: Effective strategies to finance a broad array of services and supports. Louis de la Parte Florida Mental Health Institute (FMHI), Research and Training Center for Children’s Mental Health, University of South Florida.

63.

Stroul

B. A.

Pires

S. A.

Armstrong

M. I.

McCarthy

Pizzigati

Wood

G. M.

Echo-Hawk

(2009). Effective financing strategies for systems of care: Examples from the field—A resource compendium for financing systems of care: Second edition (RTC study 3: Financing structures and strategies to support effective systems of care). Louis de la Parte Florida Mental Health Institute (FMHI), Research and Training Center for Children’s Mental Health, University of South Florida.

64.

Taylor

Henshall

Kenyon

Litchfield

Greenfield

(2018). Can rapid approaches to qualitative analysis deliver timely, valid findings to clinical leaders? A mixed methods study comparing rapid and thematic analysis. BMJ Open, 8(10), e019993. https://doi.org/10.1136/bmjopen-2017-019993

65.

Thomas

Abell

Webb

H. J.

Avdagic

Zimmer-Gembeck

M. J.

(2017). Parent–child interaction therapy: A meta-analysis. Pediatrics, 140(3), 1-15. https://doi.org/10.1542/peds.2017-0352

66.

Thompson

H. M.

Clement

A. M.

Ortiz

Preston

T. M.

Quantrell

A. L. W.

Enfield

King

A. J.

Klosinski

Reback

C. J.

Hamilton

Milburn

(2022). Community engagement to improve access to healthcare: A comparative case study to advance implementation science for transgender health equity. International Journal for Equity in Health, 21(1), 104. https://doi.org/10.1186/s12939-022-01702-8

67.

Torio

C. M.

Encinosa

Berdahl

McCormick

M. C.

Simpson

L. A.

(2015). Annual report on health care for children and youth in the United States: National estimates of cost, utilization and expenditures for children with mental health conditions. Academic Pediatrics, 15(1), 19–35. https://doi.org/10.1016/j.acap.2014.07.007

68.

Urquhart

Kendell

Cornelissen

Madden

L. L.

Powell

B. J.

Kissmann

Richmond

S. A.

Willis

Bender

J. L.

(2020). Defining sustainability in practice: Views from implementing real-world innovations in health care. BMC Health Services Research, 20, 87. https://doi.org/10.1186/s12913-020-4933-0

69.

Waltz

T. J.

Powell

B. J.

Matthieu

M. M.

Damschroder

L. J.

Chinman

M. J.

Smith

J. L.

Proctor

E. K.

Kirchner

J. E.

(2015). Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: Results from the Expert Recommendations for Implementing Change (ERIC) study. Implementation Science, 10, 109. https://doi.org/10.1186/s13012-015-0295-0

70.

Weber

Waeger

(2017). Organizations as polities: An open systems perspective. Academy of Management Annals, 11(2), 886–918. https://doi.org/10.5465/annals.2015.0152

71.

Weisz

J. R.

Kazdin

A. E.

(Eds.). (2017). Evidence-based psychotherapies for children and adolescents. Guilford Press.

72.

Willging

C. E.

Green

A. E.

Gunderson

Chaffin

Aarons

G. A.

(2015). From a “perfect storm” to “smooth sailing” policymaker perspectives on implementation and sustainment of an evidence-based practice in two states. Child Maltreatment, 20(1), 24–36. https://doi.org/10.1177/1077559514547384

73.

Williams

N. J.

Beidas

R. S.

(2019). Annual research review: The state of implementation science in child psychology and psychiatry: A review and suggestions to advance the field. Journal of Child Psychology and Psychiatry, 60(4), 430–450. https://doi.org/10.1111/jcpp.12960

74.

Yin

R. K.

(2018). Case study research: Design and methods (6th ed.). Sage Publications.

75.

Zeitlin

Altschul

Samuels

(2016). Assessing the utility of a toolkit for modifying evidence-based practice to increase cultural competence: A comparative case study. Human Service Organizations: Management, Leadership & Governance, 40(4), 369–381. https://doi.org/10.1080/23303131.2016.1153551

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB

0.04 MB

0.27 MB

0.08 MB