Implementing a complex psychosocial intervention for unstably housed Veterans: A realist-informed evaluation case study

Abstract

Background

Only 7% of individuals with co-occurring mental health and substance use disorder (COD) receive services for both conditions. We implemented and evaluated maintaining independence and sobriety through systems integration, outreach and networking-Veteran's edition (MISSION-Vet), an evidence-based manualized psychosocial intervention for Veterans with CODs. This paper identifies the generative mechanisms that explain “how, for whom, and under what conditions” MISSION-Vet adoption, implementation, and fidelity work when applied in a complex adaptive system with facilitation support.

Methods

Within two VA healthcare systems (Sites A and B), a hybrid III trial tested facilitation to implement MISSION-Vet. We conducted a two-site case study based on 42 semi-structured consolidated framework for implementation research (CFIR) guided interviews with site leadership, implementers (social workers, peer specialists), and team members (facilitators, site leads). Interviews were coded and CFIR constructs used to generate “Context + Mechanism = Outcome” configurations to understand the conditions of MISSION-Vet adoption, implementation, and fidelity.

Results

Site A was low, and Site B was high in adoption, implementation, and fidelity. Adoption hesitancy/eagerness (outcome) resulted from the interaction of “external policy” (context) dampening/encouraging a “tension for change” (mechanism). Implementation intensity (outcome) was based on how “peer pressure” or practice culture (context) activated staff “self-efficacy” (mechanism) to engage with MISSION-Vet and appraise its “relative advantage” over current practices (mechanism). Fidelity relied on how “staffing structure and availability” (context) activated/muted “facilitation” (mechanism) to result in strategy and intervention adaptation (outcome).

Conclusions

We delineated how specific contexts activated certain mechanisms to drive the different stages of implementation of a multi-faceted COD treatment intervention.

Trial registration

ClinicalTrials.gov, NCT02942979.

Plain language abstract

Implementation is inherently dynamic and influenced by interdependent factors operating at the individual, organizational, and system levels. This is especially true for complex interventions addressing co-occurring mental health and substance use disorders because such interventions involve multiple treatment modalities delivered simultaneously, in busy practice settings, with challenging populations. This paper pairs consolidated framework for implementation research (CFIR) constructs with a realist evaluation approached to generate configurations important to the adoption, implementation, and adaptation stages of a highly complex intervention addressing the behavioural health and housing needs of a vulnerable population. Each configuration describes how contextual factors trigger mechanisms to generate implementation outcomes and answers “what works for whom, in what circumstances and in what respects, and how?” These findings further our understanding of possible mechanisms of change and push us to be more precise about identifying causal relationships among constructs that contribute to the success of implementing complex interventions. This work also moves us to think theoretically and methodologically in a more dynamic fashion, thereby leading to more responsive implementation practice.

Keywords

Facilitation psychosocial intervention Veterans realist-informed

Over 90% of individuals with co-occurring mental health and substance use disorders (COD) do not receive treatment for both conditions (Substance Abuse & Mental Health Services Administration, 2020). Recognizing and responding to the substantial morbidity and mortality of CODs, psychosocial interventions are increasingly multi-faceted, responsive to social determinants of health, and expanding beyond traditional care settings (Peterson, 2013; Ziedonis, 2004). Therefore, psychosocial interventions for persons with CODs are often complex, delivered in lower-resourced settings, and difficult to implement with fidelity.

Veterans experiencing COD are a vulnerable population requiring tailored care. Veterans with the added burden of homeless are at even greater risk: COD affects over 80% of the 340,000 homeless US Veterans (Ding et al., 2018; Solari et al., 2016). The US Department of Veterans Affairs (VA) has a national goal to end Veteran homelessness by adhering to the Housing First model which prioritizes establishing housing followed by as-needed unmandated mental health and substance use disorder services (O'Toole et al., 2013). As such, the collaborative Department of Housing and Urban Development-VA Supportive Housing (HUD-VASH) program provides HUD housing vouchers and VA case management to help address housing and behavioral health needs (Kertesz et al., 2017).

The Maintaining Independence and Sobriety through Systems Integration, Outreach and Networking (MISSION) is a manualized COD intervention for unstably housed populations delivered by a case manager and a consumer provider team (“peer specialist”; Smelson et al., 2011). Clinical trials have found that MISSION improves care engagement, mental health and substance use outcomes, and reduces homelessness (Smelson et al., 2005; Smelson et al., 2012; Smelson et al., 2013; Smelson et al., 2015; Smelson et al., 2018). MISSION-Vet is an adaptation for Veterans receiving care in the VA. The first translation of MISSION-Vet into a HUD-VASH real-world setting found that, while implementation support in the form of manualized capacity building (i.e., “Getting to Outcomes”) increased uptake compared to usual implementation, overall implementation was relatively low and besides treatment engagement, no other meaningful clinical differences were detected as compared to usual care (Chinman et al., 2017). Accordingly, the goal of enhancing uptake and consistent delivery of COD programs through the efficient application of strategies in response to local context is a key implementation science area of study.

Recent work highlights how differences in adoption, implementation, adaptation, and sustainment of interventions relate not only to features of the intervention itself, but also to the effective and efficient use of implementation strategies that are appropriate to the context (Lewis et al., 2018; Williams & Beidas, 2019). However, there is a dearth of information on the necessary pre-conditions and drivers of effective implementation and scale-up of complex COD interventions.

The present study builds on previous MISSION-Vet implementation results and the extant literature to test a flexible implementation strategy—facilitation—to increase intervention uptake and use (Lewis et al., 2018; Neal & Neal, 2019). As implementation is a socially mediated process, facilitation is a multi-faceted strategy ideal for working in a context that recognizes the need for improvement and is supportive and tactical about interpersonal relationships (Curran et al., 2005; Harvey & Kitson, 2015; Kilbourne et al., 2017; Kirchner et al., 2012; Powell et al., 2015; Ritchie et al., 2017). There is a growing understanding of the importance of contextual influences and interdependencies when translating interventions from research to practice (McCullough et al., 2015).

The unpredictable and non-linear nature of COD treatment and the implementation support needed to ensure COD effectiveness calls for a theoretical lens such as critical realism which not only answers “what” happens, but also “why” and “for whom and under what conditions” (Rycroft-Malone et al., 2012). Realist-informed evaluations, especially for behavioral health interventions, may be critical to uncovering causal mechanisms as to why certain outcomes do or do not occur within specific contexts (Duncan et al., 2018). The realist ‘Context + Mechanism = Outcome’ configuration serves as a heuristic to help generate causative explanations about which contextual factors influence mechanisms of change to achieve an implementation outcome (Greenhalgh et al., 2015; Pawson et al., 1997).

The Consolidated Framework for Implementation Research (CFIR) is a multi-level implementation determinant framework that helps structure the consistent study of contextual and intervention factors associated with effective implementation (Damschroder et al., 2009; Kislov et al., 2019). Bringing together the power of theory through a mechanism-based analytic lens, such as the realist-informed approach, and a comprehensive well-operationalized implementation framework, CFIR, provides an approach that allows the for mid-range theorizing and pragmatic study of dynamic relationships in a complex system. Together, the integrated application of CFIR/realist-informed evaluation furthers the building of a cumulative implementation science (Papoutsi et al., 2016).

The purpose of this paper is to identify the generative mechanisms that explain “how, for whom, and under what conditions” MISSION-Vet adoption, implementation, and fidelity work when applied in a complex adaptive system with facilitation support (Dalkin et al., 2015). Our findings extend a growing movement in implementation science to examine and gain a richer understanding of how contexts activate or mute mechanisms and what results from that interaction (Hawe et al., 2004; McCullough et al., 2015). This work can help drive understanding of how to support complex COD intervention implementation in different settings.

Methods

Design

This work was conducted as part of a hybrid type III modified cluster randomized trial comparing MISSION-Vet usual implementation to MISSION-Vet with facilitation across two VA healthcare systems. Details of the project's design and facilitation package were previously published (Simmons et al., 2017). This project conforms to the Consolidated Criteria for Reporting Qualitative Research (Supplemental Appendix 1). Informed consent was obtained from all participants at the time of the interview. The project was reviewed by the Institutional Review Board at the VA Bedford Healthcare System and determined to be quality improvement.

Conceptual framework

Our combined CFIR/realist-informed approach allows for the explication of relationships between implementation determinants through the integrated use of a static determinant framework and a mechanism-based analytic lens. This is a case-based, realist-informed qualitative evaluation considering Realist and Meta-narrative Evidence Syntheses: Evolving Standards (RAMSES II; Wong et al., 2016). A realist-informed perspective is appropriate for this evaluation as it strives to answer, “what works (or does not), for whom, and under what conditions.” Realist evaluation helps clarify how certain contextual conditions trigger, or do not trigger, specific mechanisms to generate an outcome (Greenhalgh et al., 2009; Pawson et al., 1997). This interrelationship is expressed as a “configuration”: Context + Mechanism = Outcome. Context is the pre-existing micro to macro social, political, economic, etc., conditions in which implementation occurs. Mechanisms are the cognitive, emotional, motivational, psychological, psychosocial responses, actions, behaviors, or other changes activated by the Context. The combination of mechanisms in a context produce Outcomes which are results at individual, interpersonal, organizational, and structural levels, categorized as favorable/unfavorable, expected/unexpected, or proximal/distal.

The CFIR taxonomy guided our labeling of configuration components. The CFIR contains 39 well-operationalized constructs across five domains, including (1) intervention characteristics (e.g., evidence, relative advantage, complexity, adaptability), (2) outer setting (e.g., external policies, patient needs), (3) inner setting (e.g., organizational structure, culture, climate, relative priority), (4) individual characteristics (e.g., knowledge, beliefs, self-efficacy), and (5) implementation process (e.g., engaging, champions, executing). The CFIR is agnostic to setting, intervention, implementation phase, and study design thus making it a well-suited framework for studying context and mechanisms.

Setting and participants

This project took place in two VA healthcare systems: (1) Site A is a large facility and serves over 83,000 unique Veterans annually, and (2) Site B is a mid-size suburban facility serving over 18,000 unique Veterans annually. There were three levels of purposefully recruited participants: (1) leadership (i.e., healthcare system leaders, chiefs, middle managers), (2) frontline staff (i.e., case managers, Veteran peer specialists), and (3) project team members (i.e., MISSION-Vet developer, site leads, external facilitators).

MISSION-Vet intervention

MISSION-Vet, the evidence-based practice being implemented, includes three core components: (1) critical time intervention (CTI), a time-limited form of assertive community treatment that addresses situational barriers to care during care transitions, (2) dual recovery therapy (DRT), a psychosocial intervention comprised of 13 case manager-led sessions addressing mental health and substance use issues simultaneously, and (3) Peer Support, comprised of 11 peer-led sessions offering skill building and role modeling of successful recovery.

Implementation facilitation strategy

Usual MISSION-Vet implementation consists of a 1 time 120-min didactic training and sharing of materials, including the social worker and peer manuals, and participant workbooks. Implementation facilitation involves one external facilitator per setting conducting stakeholder engagement, site assessment, preparation/planning, ongoing process monitoring, education, program adaptation, marketing, problem identification, and problem solving. Site A facilitation was conducted exclusively virtually, and Site B facilitation was delivered in-person and virtually.

Data collection

We used a multi-method data collection approach. An anthropologist (MM) conducted semi-structured qualitative interviews with staff, leadership, and project team members between March 2016 and October 2019. Interviewees were asked questions from the CFIR Interview Guide (CFIR Research Team, 2020) tailored to MISSION-Vet implementation and as they related to CFIR's five domains (Damschroder et al., 2009). Adoption and implementation outcomes were based on MISSION-Vet sessions tracked at site and individual (staff) levels. The adaption/fidelity outcome was informed by data from facilitation tracking logs documenting time, frequency, and activities, and team meeting notes describing project deliberations and decisions.

Analysis

First, interviews were recorded and transcribed using a rapid deductive approach and tri-level analysis (Sobo et al., 2003) using the a priori CFIR domains/constructs. We also drew on inductive approaches to identify emerging patterns and themes in the data (De Munck & Sobo, 1998; Saldana, 2015). Each participant's tri-level sheet, facilitation tracking logs, and team meeting notes were then imported into NVivo 12 Plus for further analysis.

Next, we drew on realist evaluation principles and retroductive inference to iteratively examine and identify interrelationships among CFIR constructs to generate ‘Context + Mechanism = Outcome’ configurations (Eastwood et al., 2014; Fletcher, 2017). Our implementation outcomes were the degree of MISSION-Vet adoption, implementation, and fidelity which we classified as either “higher” or “lower” for the participating sites.

Results

Participant characteristics

Across the two sites, 42 interviews were conducted with 25 social workers (four social workers were interviewed twice), eight peers, and four leadership staff. This accounted for 37% of those trained in MISSION-Vet at Site A and 47% of those trained at Site B. Participants were mostly female (63%), between 26 and 72 years old, and had worked at the VA between one and 25 years. Five interviews were conducted with MISSION-Vet project team members, including the MISSION-Vet developer, project leads at Site A and B, and two external facilitators.

MISSION-Vet implementation

There were 15 MISSION-Vet training sessions (11 in Site A vs. 4 in Site B), each attended by 1 to 22 staff at a time, for a total of 108 staff. At Site A, there were 70 facilitation events with each event averaging 24-min in duration; at Site B, 40 events averaged 34-min. A total of 21 staff (13 in Site A vs. 8 in Site B) entered 424 DRT sessions (273 in Site A vs. 151 in Site B), and 89 peer-led sessions (32 in Site A vs. 57 in Site B). Site A was considered low in adoption (unfavorable), low in implementation (unfavorable), and low in fidelity (unfavorable). Site B was high in adoption (favorable), high in implementation (favorable), and high in fidelity (favorable).

Configurations

We identified three realist-informed ‘Context + Mechanism = Outcome’ configurations reflecting MISSION-Vet adoption, implementation, and fidelity. Each configuration highlights how CFIR constructs representing context and mechanism interact in ways that produce an unfavorable/favorable MISSION-Vet outcome. Below we describe each configuration and include illustrative quotes.

Configuration 1: Adoption hesitancy/eagerness (outcome) resulted from the interaction of “external policy” (context) dampening/encouraging the “tension for change” (mechanism) among case managers and peers.

Configuration 2: The degree of lower/higher implementation (outcome) was based on how “peer pressure” (context) activated/dampened staff “self-efficacy” (mechanism) to engage with MISSION-Vet, and positive/negative appraisal of “relative advantage” over current practices (mechanism). For Configuration 2, two interacting mechanisms worked in tandem.

Configuration 3: Fidelity inconsistent/consistent (outcome) strategy and intervention implementation relied on how “staffing structure and availability” (context) activated/muted “facilitation” (mechanism).

Configuration 1: Adoption

The first configuration reflected an “external policy” (context) triggers or mutes a “tension for change” (mechanism) to prompt adoption hesitancy, neutrality, or receptiveness (outcome). The context-mechanism relationship was mediated by a pre-existing “patient need” context but that need was not the activating context for the “tension for change” and eventual MISSION-Vet adoption.

At both sites, staff reported positive feedback about MISSION-Vet's purpose to engage homeless Veterans with CODs because it closely aligned with existing “national VA policy” to end homelessness (context). Although the VA is an integrated healthcare system, Sites A and B experienced the outer setting policy context distinctly. Because of their unique responses to the policy context, Site A muted a favorable “tension for change” and Site B activated a favorable “tension for change” resulting in lower and higher MISSION-Vet adoption, respectively.

The hyper-focus on Site A's homelessness programming reflected its status as the “epicenter of homelessness.” This context of “high patient need” mediated and helped explain the inverse relationship between the “policy” and “tension for change.” Site A staff consistently expressed they were “awash in initiatives” and operating in a work culture driven by goals to improve housing status, rather than focus limited efforts on MISSION-Vet's COD treatment components, including case management, peer support, vocational/educational support, and trauma-informed care. At Site A, a social worker lamented, “I don’t think it [MISSION-Vet] is a great fit for us…we need to overwhelmingly house people that we barely have time to do anything else.” This sentiment was congruent with a more global sense of feeling overextended and change fatigued (“things come in and go and then things go back to the way it was”) leading to a low expectation for change. Introduction of MISSION-Vet highlighted site challenges in addressing COD in a wholistic manner rather than activating a tension for change which would trigger receptiveness for its adoption.

Despite concerted external facilitation to reach multiple levels of Site A staff, high turnover across leadership positions precluded successful engagement: “And then with no warning they switched all the leadership. So, we start all over to see if they still agree with all the planning that we’ve done … that makes people burnt out.” Only after multiple attempts to form a coordinated plan, Site A succeeded at gaining leadership support, yet that failed to translate downstream to mid- and lower-management levels to enact the intervention. Site A's external facilitator recalled, “I was kind of waved off of engaging with … leaders of the caseworker or the peer specialist…it was like pulling teeth to find these people who are willing to give MISSION a try, and they [leadership and staff] didn’t really necessarily want me, and they were really busy.”

Conversely, Site B staff were well-positioned to adopt a new program because they experienced the national homelessness policy (context) as bolstering (or neutral rather than in opposition to) a “tension for change” (mechanism). While Site B's leadership positions were similarly tenuous, Site B's “tension for change” was positioned more at the staff rather than leadership level which allowed for both internal and external tension for change to be generated. These “fewer channels” or layers of communication for negotiating leadership and staff MISSION-Vet implementation decisions were paramount to Site B's success.

“Patient need” acted as a contextual mediator at both sites. Prior to MISSION-Vet introduction, the homelessness policy had intensified homeless programs throughout the VA. At Site B, a new HUD-VASH project-based housing facility had been recently built was served as the primary setting for MISSION-Vet implementation. In turn, there was an expectation to actively engage the recently housed Veterans as mandated by the HUD-VASH program. Unlike Site A, Site B's high caseload created a tension for change in practice. In turn, MISSION-Vet was appealing to staff “from a productivity standpoint” because it was possible to reach a greater number of Veterans more efficiently in a group setting compared to the typical one-on-one case management arrangement.

Configuration 2: Implementation

The second configuration reflected that implementation intensity (outcome) was based on how “peer pressure” (context) influenced staff “self-efficacy” (mechanism) and appraisal of MISSION-Vet's “relative advantage” over current practices (mechanism). Site A had lower implementation due to “peer pressure” (context) that neither permitted activation of staff self-efficacy nor encouraged positive appraisal of MISSION-Vet. Site B's higher implementation was spurred by an enabling context of positive “peer pressure” which allowed for a willingness to consider positively MISSION-Vet's “relative advantage.”

Overall, staff at both sites stated that MISSION-Vet resembled work they were already doing and thus initially struggled to pinpoint how it could improve outcomes over current practices. However, this concern was more pronounced at Site A and one social worker explained a state of decision paralysis: “We have so many choices. It is like when you go to the toilet paper aisle. You just need something quick and there are so many choices so what happens is you become familiar with what you usually refer to … rather than figuring out another plan.” Site A's “peer pressure” was focused on sustaining current practices rather than embracing new efforts. Although a Site A social worker said they were “always looking to try something different,” they were apprehensive about how MISSION-Vet could be embedded into their practice given the time and resource constraints they were operating under. There was a backdrop of caution and a lack of trust among staff, therefore many did not take initiative given the implicit lack of encouragement or support from peers and leadership. Nevertheless, there was a desire among some staff for a closer connection to one another and to resolve deficiencies: “We all have a responsibility to understand how we can work together … we can get super creative together, but we have to know what each other doing, right?”

“Peer pressure” was a central and potent positive context at Site B that was able to stimulate a greater sense of “self-efficacy” and acceptance of MISSION-Vet. There was strong support from the Chief of Social Work who oversaw the HUD-VASH program and was regarded as “trying to be kind of cutting edge.” Social workers paralleled leadership in their desire to maintain a “modern kind of approach” which stimulated the “relative advantage” mechanism. The “peer pressure” experience was unique to HUD-VASH staff because they, as one social worker described “…support each other because they’re kind of off doing their thing outside of the hospital grounds often times.” For this reason, Site B staff were willing to take implementation risks due to a heightened “self-efficacy”: “We kind of just ripped off the Band-Aid and were like, okay, we’re just going to start this group even though we don’t really know what we’re doing.”

Site B staff saw opportunities on how MISSION-Vet could either substitute or accompany current practices and be a means to “think outside the box … [as] sometimes we get stuck with the same kind of routine.” Both social workers and peers at Site B readily characterized MISSION-Vet as a “new way of presenting similar information” and a “good segue to other treatment.” Another example of how “peer pressure” activated increased perceptions of “relative advantage” and “self-efficacy” was that once Site B staff were more comfortable with the intervention, they undertook an informal train-the-trainer process to help one another learn and implement MISSION-Vet. No such cross-team sharing of MISSION-Vet occurred at Site A.

Configuration 3: Fidelity and adaptation

The third configuration indicated “structure and staffing” (context) triggered the “external facilitation” (mechanism) to be activated and this resulted in variable levels of MISSION-Vet fidelity (outcome). When peers for the intervention and champions for implementation were not available or did not have defined roles then there were more MISSION-Vet adaptations.

At the outset, Site A did not have the suitable staffing levels for implementing MISSION-Vet as intended (i.e., case manager paired with peer specialist for the intervention and a designated champion or internal facilitator for the facilitation strategy). Therefore, at Site A, MISSION-Vet was adapted both in delivery and content, and the facilitation approach was modified. With no local champion at Site A to organize staff and help with logistics, providing external facilitation remotely to a targeted group in a consistent manner was challenging. The Site A external facilitator held check-in calls and regularly sent check-in emails; however, calls were irregularly attended, and emails often went unanswered. In the instances when facilitation was provided the support was largely appreciated, yet staff recommended the facilitator shift from a passive “here to help” stance to a more directive approach. A social work intern from Site A recognized this need: “…a contact person for questions regarding specifically MISSION-Vet would be helpful.” The Site A external facilitator reported having “watered down facilitation and who was internal and who was external in order to … adapt to [Site A].”

Conversely, Site B was able to secure a staff member to serve in a champion role to oversee and support local MISSION-Vet implementation. Overall, Site B was open to guided support from the external facilitator and reported that it was advantageous to be co-located with the external facilitator because “being present matters.” Site B staff readily participated in scheduled check-ins and requested ad hoc meetings to resolve implementation questions because otherwise, as one social worker commented, “you don’t have that actual time carved out to connect about how the group's going.” Moreover, on-site facilitation allowed the external facilitator to establish a closer connection with staff assumes more internal facilitator tasks and characteristics.

MISSION-Vet is intended to be delivered by a case manager and peer specialist team, but Site A's HUD-VASH program was staffed solely by social workers with no regular access to peers which created a challenging implementation context that could not accommodate the MISSION-Vet model. The external facilitator noted this structural gap and there were attempts to “borrow” peers from other programs, “but it didn’t work well for having that peer accountable.” Considering MISSION-Vet's core components, implementing without peers was a significant and fidelity-inconsistent adaptation. This was a pronounced deficit given many staff expressed that MISSION-Vet's uniqueness was rooted in its peer component: “in my clinical fantasies it seems like a peer doing the engagement would probably be the most powerful for folks.”

Unlike Site A where peers were not available, Site B's peers were integrated into the HUD-VASH program and had established relationships with social workers. In turn, it was possible to extend peer tasks to readily incorporate MISSION-Vet. Moreover, all Site B peers had experience running groups and several had experience with manualized interventions. Given the context and mechanism interaction at Site B, there was high fidelity to the MISSION-Vet model in delivery.

MISSION-Vet intervention content was modified at both sites. External facilitators recognized the tension between supporting sites in implementing a “lighter” version of MISSION-Vet by altering core components: “You have a choice point. You either do MISSION-Vet as close as you can to the model as possible or we’re going to kind of roll it out in a way that's really much more inferior.” A social work supervisor at Site A described concerns about losing clinical autonomy by following a manualized protocol: “The only thing that I have about it, we tend to go on things and follow it to a ‘T’ that we can sometimes not use or follow our own clinical judgement.” Conversely, there was a perceived benefit that “giving people that level of choice makes them feel much more empowered about the model.”

Staff at both sites sought assistance implementing MISSION-Vet with fidelity: “I’m just hopeful that you guys [facilitators] keep us honest and keep us directed in regard to trying to implement it in the best way we can.” Some admitted they were “still experimenting” and “improvisation” was expected for group-level interventions with such a highly vulnerable population. When opportunities to modify MISSION-Vet were presented, the proposals were typically vetted with the MISSION-Vet developer. Generally, peers were more “lenient” and “liberal” than social workers with content adaptations. This was not surprising to the MISSION-Vet developer given “the field of peer support lacks any curriculum on how to standardize practices.”

Configuration summary

The three configurations generated also interact with each other. The interrelationship among the three configurations created a “ripple effect” such that the outcomes of the first configuration transformed the context of the next and so forth (Jagosh et al., 2015). Identifying such interrelationships accentuates the importance of temporality in implementation. Hence, context, in a tautological sense, operates in a feedback loop with other contextual elements. While intuitive, such overlapping contexts may be less relevant for interventions not as complex as MISSION-Vet. Bringing together CFIR and realist-informed evaluation shifts theory to a grounded approach in which empirical data iteratively informs and advances knowledge about the interactions among context, mechanism, and outcome. Figure 1 illustrates the interrelationships of not only the three context-mechanism pairs, but also the configurations themselves.

Figure 1

Illustration of the interrelationships of the three context-mechanism pairs and the configurations.

Discussion

In this realist-informed evaluation of a complex intervention for unstably housed Veterans with a co-occurring disorder, we generated context-mechanism-outcome configurations that illustrated the ‘how, for whom, and under what conditions’ MISSION-Vet adoption, implementation, and fidelity/adaptation occurred. Our combined CFIR/realist-informed approach served as a mid-range theorizing vehicle to reflect upon and investigate implementation determinants and the dynamic interdependencies between them to uncover generative mechanisms and guide subsequent empirical enquiry. We found that contexts related to policy, peer pressure, and staffing structures enabled mechanisms for activation of tension for change, positive appraisal of intervention relative advantage, and effective facilitator support, respectively. Our findings suggest that MISSION-Vet and other multi-faceted interventions may require substantial support across implementation stages to help trigger key behavior changes. These three configurations can help guide implementation support by identifying and prioritizing elements of the context amenable to intervention and providing a model for theoretically informed implementation research. This study advances implementation science by demonstrating how theory (realist-informed approaches) and empirical data (guided by CFIR) can elucidate the often underexplored and underarticulated generative mechanisms.

Adoption

Near universal endorsement of COD integrated treatments, such as provided through MISSION-Vet, exists though treatments range in scope and effectiveness (Chow et al., 2013). Our study affirms a general preference for integrated approaches, however, the modest adoption of MISSION-Vet in both sites signals that there may be issues with this integrated approach. For example, MISSION-Vet may not have been compatible with the tension for change on a local level activated by a national movement to reduce homelessness. Perhaps a wider adoption of prevention-centered homelessness policies across sites could help orient providers to incorporate approaches like MISSION-Vet for addressing overlapping risk factors in the population with COD that threaten housing stability (Culhane et al., 2011).

According to CFIR, our identified “tension for change” mechanism is one of six constructs (compatibility, relative priority, organizational incentives and rewards, goals and feedback, learning climate) within “implementation climate” which form the perceptions of expectations, supports, and rewards for implementation. Given the recognized importance of implementation climate, studies and instruments for measuring it have proliferated (Powell et al., 2021). In this study, we established a configuration of policy-activated tension for change and subsequent adoption or non-adoption of MISSION-Vet. This finding is supported by Williams et al.'s (2018) discussion of the relationship between implementation climate and psychotherapy techniques that is moderated by molar organizational climate (i.e., “clinicians’ shared perceptions of how the work environment influences their personal well-being”). This is particularly important in highly complex settings, where, for example, leadership churn may lead to redundancy in facilitator activities and inaction by staff (Aarons et al., 2009). As such, the importance of investigating contextual interactions and conditionality, rather than over-focusing on reductive cause-and-effect connections cannot be overstated.

Implementation

Realist-informed evaluations such as ours may advance how implementation determinants are conceived and analyzed. In this study, staff perceived conflicting internal and external pressures which left unresolved, led to an erosion of self-efficacy to implement a new practice or reconciling of pressures and a greater capacity for implementation.

At a structural level, the expertise required to implement complex psychosocial interventions may prove too challenging for many settings given only 9% of mental disorder programs and 18% of substance use disorder programs have the capacity to serve individuals with CODs (McGovern et al., 2014). Accordingly, self-efficacy is compromised because complex psychosocial interventions require the expertise and collaboration between multiple complex systems such which are not readily available. This challenge may prompt de-implementation or substitution of existing practices given resource-limited contexts. Decisions to retain or remove intervention parts may be related to the appraisal of an intervention's characteristics, and specifically, its relative advantage (Rogers, 2010). In comparison to other intervention characteristics, such as complexity, adaptability, or trialability, the relative advantage is highly context-reliant and therefore cannot be anticipated in the intervention development stage. Still, there is a growing movement to introduce user-centered design approaches to help ensure interventions are developed in response to real-world user needs and therefore are perceived higher relative advantage (Lyon & Bruns, 2019).

Fidelity and adaptation

The decision to adapt is a dynamic process based on many factors, thus identifying a triggering mechanism or mechanisms is a challenging implementation science core aim (Stirman et al., 2019). In the current study, adaptation occurred at the intervention and at the implementation strategy levels. MISSION-Vet's flexibility in delivery and, to some extent, its content were axes of adaptation and seen as both strengths and weaknesses. Greater adaptability of an intervention may encourage a practitioner's sense of ownership and willingness to implement and simultaneously risk intervention fidelity and thus negatively impact effectiveness (Movsisyan et al., 2019). Multi-component interventions like MISSION-Vet are particularly susceptible to adaptation and modification; however, caution is warranted about disassembling interventions into their component parts, which are evidence-based practices on their own, but have synergistic benefits when bundled. For instance, a potentially unfavorable adaptation of a solo social worker rather than a social worker/peer team remains to be explored. Recognizing the various adaptations undertaken to promote MISSION-Vet use, this study highlights a need for increased attention to elucidating fidelity-consistent adaptations and modifications, with an emphasis on refining low burden fidelity measures (Walton et al., 2017).

There was likewise adaptation at the implementation strategy level in our study. This finding is consistent with the nature of implementation as a social process and emblematic of facilitation's flexibility as an implementation strategy. Although facilitation is a common remedy to coexisting high need and low ability to absorb effective programming, it was largely not able to overcome the limiting context at Site A (Harvey & Kitson, 2015). This may be accounted for by the finding that complex settings face a constellation of challenges such as bureaucratic structures, a silo mentality, low improvement culture, limited capacity to innovate, and other competing demands (Allen et al., 2017). Accordingly, for Site A, the larger and more complex setting with a high-need population, the dual tasks of learning a new intervention and restructuring teams to accommodate the intervention hampered adoption and implementation, and increased adaptation. In contrast, the lower-complexity and higher-uptake Site B could tolerate a new integrated intervention and process change without substantial disruption. Efforts by Waltz et al. (2019) to systematize how CFIR constructs link to implementation strategies is an ongoing effort. Thus, it is possible that facilitation (or other strategies) may produce highly variable results across settings and contexts, while other strategies may be consistently applied and experienced across settings.

Strengths and limitations

This study was based in two VA healthcare systems; this limitation was mitigated by the fact that we had a large sample of staff participants at various levels of MISSION-Vet implementation, as well as perspectives of MISSION-Vet developers and facilitators. The transferability of our findings should be evaluated in light of the uniqueness of VA settings, and specifically the unique population of Veterans with CODs in care. Although we deviated from the RAMSES II, a strength of this study is its application of a realist-informed approach combined with a commonly used implementation framework. Determination of the context-mechanism-sustainment configuration was beyond the scope of the project.

Conclusions

Psychosocial interventions generally, and COD interventions specifically, are notoriously difficult to implement and implementation with fidelity is a particular concern (Brunette et al., 2008; Powell et al., 2014). Our realist-informed methods, including the use of CFIR, applied the context-mechanism-outcome heuristic to derive possible causal relations and determine why some contexts are more conducive than others to the adoption of complex interventions. We delineated constructs and identified construct interactions that influenced the adoption, implementation, and adaptation of a multi-faceted psychosocial intervention for unstably housed Veterans with CODs. This work moves us to consider construct interactions in a more theoretically and methodologically dynamic fashion to produce a more responsive implementation practice. Future studies could build on our findings by investigating facilitation's promising role as an adaptive implementation strategy. As there are likely other salient configurations that may lead to similar implementation outcomes, future work with a greater number of sites would be valuable to test and refine the identified configurations.

Supplemental Material

sj-docx-1-irp-10.1177_26334895211049483 - Supplemental material for Implementing a complex psychosocial intervention for unstably housed Veterans: A realist-informed evaluation case study

Supplemental material, sj-docx-1-irp-10.1177_26334895211049483 for Implementing a complex psychosocial intervention for unstably housed Veterans: A realist-informed evaluation case study by Vera Yakovchenko, Megan B. McCullough, Jeffrey L. Smith, Sonya Gabrielian, Thomas Byrne, Kathryn E. Bruzios, Ella Koosis and David A. Smelson in Implementation Research and Practice

Footnotes

Declaration of conflicting interests

The authors have no conflicts of interest to disclose.

Funding

All the authors are funded by a grant from the Health Services Research and Development Quality Enhancement Research Initiative, “Bridging the Care Continuum” (QUE 15-284).

ORCID iD

Vera Yakovchenko

Supplemental material

Supplemental material for this article is available online.

References

Aarons

G. A.

Sommerfeld

D. H.

Hecht

D. B.

Silovsky

J. F.

Chaffin

M. J.

(2009). The impact of evidence-based practice implementation and fidelity monitoring on staff turnover: Evidence for a protective effect. Journal of Consulting and Clinical Psychology, 77(2), 270–280. https://doi.org/10.1037/a0013223

Allen

J. D.

Towne Jr

S. D.

Maxwell

A. E.

DiMartino

Leyva

Bowen

D. J.

Linnan

Weiner

B. J.

(2017). Measures of organizational characteristics associated with adoption and/or implementation of innovations: A systematic review. BMC Health Services Research, 17(1), 591. https://doi.org/10.1186/s12913-017-2459-x

Brunette

M. F.

Asher

Whitley

Lutz

W. J.

Wieder

B. L.

Jones

A. M.

McHugo

G. J.

(2008). Implementation of integrated dual disorders treatment: A qualitative analysis of facilitators and barriers. Psychiatric Services, 59(9), 989–995. https://doi.org/10.1176/ps.2008.59.9.989

CFIR Research Team (2020). Consolidated framework for implementation research. https://cfirguide.org/.

Chinman

McCarthy

Hannah

Byrne

T. H.

Smelson

D. A.

(2017). Using Getting To Outcomes to facilitate the use of an evidence-based practice in VA homeless programs: a cluster-randomized trial of an implementation support strategy. Implement Sci, 12(1), 34. https://doi.org/10.1186/s13012-017-0565-0

Chow

C. M.

Wieman

Cichocki

Qvicklund

Hiersteiner

(2013). Mission impossible: Treating serious mental illness and substance use co-occurring disorder with integrated treatment: A meta-analysis. Mental Health and Substance Use, 6(2), 150–168.

Culhane

D. P.

Metraux

Byrne

(2011). A prevention-centered approach to homelessness assistance: A paradigm shift?. Housing Policy Debate, 21(2), 295–315. https://doi.org/10.1080/10511482.2010.536246

Curran

G. M.

Thrush

C. R.

Smith

J. L.

Owen

R. R.

Ritchie

Chadwick

(2005). Implementing research findings into practice using clinical opinion leaders: Barriers and lessons learned. The Joint Commission Journal on Quality and Patient Safety, 31(12), 700–707. https://doi.org/10.1016/s1553-7250(05)31091-9

Dalkin

S. M.

Greenhalgh

Jones

Cunningham

Lhussier

(2015). What's in a mechanism? Development of a key concept in realist evaluation. Implementation Science, 10(1), 49. https://doi.org/10.1186/s13012-015-0237-x

10.

Damschroder

L. J.

Aron

D. C.

Keith

R. E.

Kirsh

S. R.

Alexander

J. A.

Lowery

J. C.

(2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementaition Science, 4, 50. https://doi.org/10.1186/1748-5908-4-50

11.

De Munck

V. C.

Sobo

E. J.

(1998). Using methods in the field: A practical introduction and casebook. Rowman Altamira.

12.

Ding

Slate

Yang

(2018). History of co-occurring disorders and current mental health status among homeless veterans. BMC Public Health, 18(1), 751. https://doi.org/10.1186/s12889-018-5700-6

13.

Duncan

Weich

Fenton

S. J.

Twigg

Moon

Madan

Singh

S. P.

Crepaz-Keay

Parsons

Bhui

(2018). A realist approach to the evaluation of complex mental health interventions. The British Journal of Psychiatry: The Journal of Mental Science, 213(2), 451–453. https://doi.org/10.1192/bjp.2018.96

14.

Eastwood

J. G.

Jalaludin

B. B.

Kemp

L. A.

(2014). Realist explanatory theory building method for social epidemiology: A protocol for a mixed method multilevel study of neighbourhood context and postnatal depression. SpringerPlus, 3, 12. https://doi.org/10.1186/2193-1801-3-12

15.

Fletcher

A. J.

(2017). Applying critical realism in qualitative research: Methodology meets method. International Journal of Social Research Methodology, 20(2), 181–194. https://doi.org/10.1080/13645579.2016.1144401

16.

Greenhalgh

Humphrey

Hughes

Macfarlane

Butler

Pawson

(2009). How do you modernize a health service? A realist evaluation of whole-scale transformation in London. Milbank Quarterly, 87(2), 391–416. https://doi.org/10.1111/j.1468-0009.2009.00562.x

17.

Greenhalgh

Wong

Jagosh

Greenhalgh

Manzano

Westhorp

Pawson

(2015). Protocol—the RAMESES II study: Developing guidance and reporting standards for realist evaluation. BMJ open 5(8), e008567. http://dx.doi.org/10.1136/bmjopen-2015-008567

18.

Harvey

Kitson

(2015). PARIHS revisited: From heuristic to integrated framework for the successful implementation of knowledge into practice. Implementation Science, 11(1), 33. https://doi.org/10.1186/s13012-016-0398-2

19.

Hawe

Shiell

Riley

(2004). Complex interventions: How “out of control” can a randomised controlled trial be? BMJ (Clinical Research Ed.), 328(7455), 1561–1563. https://doi.org/10.1136/bmj.328.7455.1561

20.

Jagosh

Bush

P. L.

Salsberg

Macaulay

A. C.

Greenhalgh

Wong

Cargo

Green

L. W.

Herbert

C. P.

Pluye

(2015). A realist evaluation of community-based participatory research: Partnership synergy, trust building and related ripple effects. BMC Public Health, 15(1), 725. https://doi.org/10.1186/s12889-015-1949-1

21.

Kertesz

S. G.

Austin

E. L.

Holmes

S. K.

DeRussy

A. J.

Van Deusen Lukas

Pollio

D. E.

(2017). Housing first on a large scale: Fidelity strengths and challenges in the VA's HUD-VASH program. Psychological Services, 14(2), 118–128. https://doi.org/10.1037/ser0000123

22.

Kilbourne

A. M.

Elwy

A. R.

Sales

A. E.

Atkins

(2017). Accelerating research impact in a learning health care system: VA's quality enhancement research initiative in the choice act era. Medical Care, 55(7 Suppl 1), S4–S12. https://doi.org/10.1097/MLR.0000000000000683

23.

Kirchner

J. E.

Parker

L. E.

Bonner

L. M.

Fickel

J. J.

Yano

E. M.

Ritchie

M. J.

(2012). Roles of managers, frontline staff and local champions, in implementing quality improvement: Stakeholders’ perspectives. Journal of Evaluation in Clinical Practice, 18(1), 63–69. https://doi.org/10.1111/j.1365-2753.2010.01518.x

24.

Kislov

Pope

Martin

G. P.

Wilson

P. M.

(2019). Harnessing the power of theorising in implementation science. Implementation Science, 14, 103. https://doi.org/10.1186/s13012-019-0957-4

25.

Lewis

C. C.

Klasnja

Powell

B. J.

Lyon

A. R.

Tuzzio

Jones

Walsh-Bailey

Weiner

(2018). From classification to causality: Advancing understanding of mechanisms of change in implementation science. Frontiers in Public Health, 6, 136. https://doi.org/10.3389/fpubh.2018.00136

26.

Lyon

A. R.

Bruns

E. J.

(2019). User-centered redesign of evidence-based psychosocial interventions to enhance implementation—hospitable soil or better seeds? JAMA Psychiatry, 76(1), 3–4. https://doi.org/10.1001/jamapsy chiatry.2018.3060

27.

McCullough

M. B.

Chou

A. F.

Solomon

J. L.

Petrakis

B. A.

Kim

Park

A. M

Benedict

A. J.

Hamilton

A. B.

Rose

A. J.

(2015). The interplay of contextual elements in implementation: an ethnographic case study. BMC Health Serv Res, 15(1), 62. https://doi.org/10.1186/s12913-015-0713-7

28.

McGovern

M. P.

Lambert-Harris

Gotham

H. J.

Claus

R. E.

Xie

(2014). Dual diagnosis capability in mental health and addiction treatment services: An assessment of programs across multiple state systems. Administration and Policy in Mental Health and Mental Health Services Research, 41(2), 205–214. https://doi.org/10.1007/s10488-012-0449-1

29.

Movsisyan

Arnold

Evans

Hallingberg

Moore

O’Cathain

Pfadenhauer

L. M.

Segrott

Rehfuess

(2019). Adapting evidence-informed complex population health interventions for new contexts: A systematic review of guidance. Implementation Science, 14(1), 105. https://doi.org/10.1186/s13012-019-0956-5

30.

Neal

J. W.

Neal

Z. P.

(2019). Implementation capital: Merging frameworks of implementation outcomes and social capital to support the use of evidence-based practices. Implementation Science, 14(1), 16. https://doi.org/10.1186/s13012-019-0860-z

31.

O'Toole

Pape

Kane

(2013). Ending homelessness – then what? American Journal of Public Health, 103(Suppl 2), S185–S187. https://doi.org/10.2105/AJPH.2013.301730

32.

Papoutsi

Boaden

Foy

Grimshaw

Rycroft-Malone

(2016). Challenges for implementation science. In R. Raine, R. Fitzpatrick, H. Barratt, G. Bevan, N. Black, R. Boaden, P. Bower, M. Campbell, J-L. Denis, K. Devers, M. Dixon-Woods, L. Fallowfield, J. Forder, R. Foy, N. Freemantle, N. J. Fulop, E. Gibbons, C. Gillies, L. Goulding, R. Grieve, J. Grimshaw, E. Howarth, R. J. Lilford, R. McDonald, G. Moore,. Moore, R. Newhouse, A. O’Cathain, Z. Or, C. Papoutsi, S. Prady, J. Rycroft-Malone, J. Sekhon, S. Turner, S. I. Watson, & M. Zwarenstein. Challenges, solutions and future directions in the evaluation of service innovations in health care and public health. Health Services and Delivery Research, 4(16), 121–132. https://doi.org/10.3310/hsdr04160

33.

Pawson

Tilley

(1997). Realistic evaluation. Sage.

34.

Peterson

A. L.

(2013). Integrating mental health and addictions services to improve client outcomes. Issues in Mental Health Nursing, 34(10), 752–756. https://doi.org/10.3109/01612840.2013.809830

35.

Powell

B. J.

Mettert

K. D.

Dorsey

C. N.

Weiner

B. J.

Stanick

C. F.

Lengnick-Hall

Ehrhart

M. G.

Aarons

G. A.

Barwick

M. A.

Damschroder

L. J.

Lewis

C. C.

(2021). Measures of organizational culture, organizational climate, and implementation climate in behavioral health: A systematic review. Implementation Research and Practice. 2, 1–29. https://doi.org/10.1177/26334895211018862

36.

Powell

B. J.

Proctor

E. K.

Glass

J. E.

(2014). A systematic review of strategies for implementing empirically supported mental health interventions. Research on Social Work Practice, 24(2), 192–212. https://doi.org/10.1177/1049731513505778

37.

Powell

B. J.

Waltz

T. J.

Chinman

M. J.

Damschroder

L. J.

Smith

J. L.

Matthieu

M. M.

Proctor

E. K.

Kirchner

J. E.

(2015). A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10(1), 21. https://doi.org/10.1186/s13012-015-0209-1

38.

Ritchie

M. J.

Dollar

K. M.

Miller

C. J.

Oliver

K. A.

Smith

J. L.

Lindsay

J. A.

Kirchner

J. E.

(2017). Using implementation facilitation to improve care in the veterans health administration (version 2). Veterans Health Administration, Quality Enhancement Research Initiative (QUERI) for Team-Based Behavioral Health.

39.

Rogers

E. M.

(2010). Diffusion of innovations (4th ed.). Simon and Schuster.

40.

Rycroft-Malone

McCormack

Hutchinson

A. M.

DeCorby

Bucknall

T. K.

Kent

Schultz

Snelgrove-Clarke

Stetler

C. B.

Titler

Wallin

Wilson

(2012). Realist synthesis: Illustrating the method for implementation research. Implementation Science, 7(1), 33. https://doi.org/10.1186/1748-5908-7-33

41.

Saldana

(2015). The coding manual for qualitative researchers (3rd ed.). Sage.

42.

Simmons

M. M.

Gabrielian

Byrne

McCullough

M. B.

Smith

J. L.

Taylor

T. J.

O'Toole

T. P.

Kane

Yakovchenko

McInnes

D. K.

Smelson

D. A.

, (2017). A Hybrid III stepped wedge cluster randomized trial testing an implementation strategy to facilitate the use of an evidence-based practice in VA Homeless Primary Care Treatment Programs. Implementation Science, 12(1), 46. https://doi.org/10.1186/s13012-017-0563-2

43.

Smelson

Losonczy

Castles-Fonseca

Sussner

Rodrigues

Kaune

Ziedonis

(2005). Preliminary outcomes from a community linkage intervention for individuals with co-occurring substance abuse and serious mental illness. Journal of Dual Diagnosis, 1(3), 47–59. https://doi.org/10.1300/J374v01n03_05

44.

Smelson

D. A.

Sawh

Kane

Kuhn

Ziedonis

D. M.

(2011). MISSION-VET Treatment Manual. University of Massachusetts Medical School.

45.

Smelson

Kalman

Losonczy

M. F.

Kline

Sambamoorthi

St Hill

Castles-Fonseca

Ziedonis

(2012). A brief treatment engagement intervention for individuals with co-occurring mental illness and substance use disorders: Results of a randomized clinical trial. Community Mental Health Journal, 48(2), 127–132. https://doi.org/10.1007/s10597-010-9346-9

46.

Smelson

Kline

Kuhn

Rodrigues

O'Connor

Fisher

Sawh

Kane

(2013). A wraparound treatment engagement intervention for homeless veterans with co-occurring disorders. Psychological Services, 10(2), 161–167. https://doi.org/10.1037/a0030948

47.

Smelson

D. A.

Pinals

D. A.

Sawh

Fulwiler

Singer

Guevremont

Fisher

Steadman

H. J.

Hartwell

(2015). An alternative to incarceration: Co-occurring disorders treatment intervention for justice-involved veterans. World Medical & Health Policy, 7(4), 329–348. https://doi.org/10.1002/wmh3.168

48.

Smelson

Farquhar

Fisher

Pressman

Pinals

D. A.

Samek,

Duffy

M-K.

Sawh

(2018). Integrating a co-occurring disorders intervention in drug courts: An open pilot trial. Community Mental Health Journal, 55(2), 222–231. https://doi.org/10.1007/s10597-018-0255-7

49.

Sobo

E. J.

Simmes

D. R.

Landsverk

J. A.

Kurtin

P. S.

(2003). Rapid assessment with qualitative telephone interviews: Lessons from an evaluation of California's healthy families program & Medi-Cal for children. American Journal of Evaluation, 24(3), 399–408. https://doi.org/10.1016/S1098-2140(03)00053-5

50.

Solari

C. D.

Mooris

Shivji

de Souza

(2016). The 2015 annual homeless assessment report (AHAR) to congress .

51.

Stevenson

Ball

Haverhals

L. M.

Aron

D. C.

Lowery

(2018). Evaluation of a national telemedicine initiative in the Veterans Health Administration: Factors associated with successful implementation. Journal of Telemedicine and Telecare, 24(3), 168–178. https://doi.org/10.1177/1357633 ( 16677676

52.

Stirman

S. W.

Baumann

A. A.

Miller

C. J.

(2019). The FRAME: An expanded framework for reporting adaptations and modifications to evidence-based interventions. Implementation Science, 14(1), 1–10. https://doi.org/10.1186/s13012-019-0898-y

53.

Substance Abuse and Mental Health Services Administration (2020). Substance abuse treatment for persons with co-occurring disorders [HHS Publication No. (SMA) 13-3992] Treatment Improvement Protocol (TIP) Series, No. 42.

54.

Walton

Spector

Tombor

Michie

(2017). Measures of fidelity of delivery of, and engagement with, complex, face-to-face health behaviour change interventions: A systematic review of measure quality. British Journal of Health Psychology, 22(4), 872–903. https://doi.org/10.1111/bjhp.12260

55.

Waltz

T. J.

Powell

B. J.

Fernández

M. E.

Abadie

Damschroder

L. J.

(2019). Choosing implementation strategies to address contextual barriers: Diversity in recommendations and future directions. Implementation Science, 14(1), 1–5. https://doi.org/10.1186/s13012-019-0892-4

56.

Williams

N. J.

Beidas

R. S.

(2019). Annual research review: The state of implementation science in child psychology and psychiatry: A review and suggestions to advance the field. Journal of Child Psychology and Psychiatry, 60(4), 430–450. https://doi.org/10.1111/jcpp.12960

57.

Williams

N. J.

Ehrhart

M. G.

Aarons

G. A.

Marcus

S. C.

Beidas

R. S.

(2018). Linking molar organizational climate and strategic implementation climate to clinicians’ use of evidence-based psychotherapy techniques: Cross-sectional and lagged analyses from a 2-year observational study. Implementation Science, 13, 85. https://doi.org/10.1186/s13012-018-0781-2

58.

Wong

Westhorp

Manzano

Greenhalgh

Jagosh

Greenhalgh

(2016). RAMESES II reporting standards for realist evaluations. BMC Medicine, 14(1), 96. https://doi.org/10.1186/s12916-016-0643-1

59.

Ziedonis

D. M.

(2004). Integrated treatment of co-occurring mental illness and addiction: Clinical intervention, program, and system perspectives. CNS Spectrums, 9(12), 892–904, 925. https://doi.org/10.1017/s1092852900009718

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB