Abstract
Background:
Adaptations to evidence-based practices (EBPs) are common but can impact implementation and patient outcomes. In our prior research, providers in routine care made a fidelity-inconsistent adaptation to an EBP that improved health outcomes in people with serious mental illness (SMI). The purpose of this study was to characterize the process and reasons for the adaptation using a framework for reporting adaptations and modifications to EBPs, with a focus on equity.
Methods:
This study used qualitative data collected during a national implementation of the InSHAPE EBP addressing obesity in persons with SMI. We reviewed transcripts from five behavioral health organizations that made a successful fidelity-inconsistent adaptation to a core component of InSHAPE that was associated with cardiovascular risk reduction. We coded the data using the Framework for Reporting Adaptations and Modifications-Expanded (FRAME) with an emphasis on exploring whether the adaptation addressed inequities in using the EBP related to social determinants of health.
Results:
Across the five agencies, the fidelity-inconsistent adaptation was characterized as unplanned and reactive in response to challenges InSHAPE teams experienced delivering the intervention in community fitness facilities as intended. In all cases, the goal of the adaptation was to improve intervention access, feasibility, and fit. Social and economic disadvantage were noted obstacles to accessing fitness facilities or gyms among participants with SMI, which led agencies to adapt the program by offering sessions at the mental health center.
Conclusion:
Findings from this study show the advantages of applying a health equity lens to evaluate how obstacles such as poverty and discrimination influence EBP adaptations. Recommendations can also assist researchers and community partners in making proactive decisions about allowable adaptations to EBPs.
Plain Language Summary
Adaptations to evidence-based practices (EBPs) are common but can impact implementation and patient outcomes. Understanding why adaptations are made to EBPs by organizations and providers during implementation can help inform implementation strategies designed to guide adaptations that improve outcomes. We found that social and economic factors were driving inequities in access to a core intervention component of an EBP, which led agencies to adapt an EBP in a way that model developers considered to be inconsistent with fidelity but improved patient outcomes. These findings contribute to the growing literature on equitable implementation and adaptation by highlighting the advantages of considering when and how fidelity-inconsistent adaptations to an EBP may be in the service of reducing inequities in access to and use of EBPs for health disparity groups.
Background
Evidence-based practices (EBPs) are often adapted from their original designs during implementation, particularly when there are mismatches between the setting of the original EBP and new implementation contexts (Escoffery et al., 2018; Wiltsey Stirman et al., 2017). EBPs are also adapted to address inequities in the delivery of health care (Baumann & Cabassa, 2020)—for example, when social determinants of health such as poverty and discrimination create barriers to accessing and using EBPs in their original form (Cabassa & Baumann, 2013; Gandelman & Dolcini, 2012). Balancing adaptation and fidelity is critical in effective EBP implementation. Striving for strict adherence to fidelity protocols of an EBP may be counterproductive when adapting to context (e.g., allowing intervention activities and materials to be adapted to local cultural styles, literacy levels, or delivery system characteristics) may be more appropriate. However, the effectiveness of an EBP may be threatened when stakeholders make adaptations that reduce or preclude participant benefits.
People with serious mental illness (SMI) who disproportionately experience medical comorbidity and mortality compared with the overall population are among the least recognized health disparity group (Bartels & Dimilia, 2017). Health disparities are closely linked with economic, social, or environmental disadvantage, including poverty, unequal access to health care, lack of education, stigma, and racism (Braveman, 2014). As EBPs targeting people with SMI and other health disparity groups are translated from the research contexts in which they were originally tested to routine health care settings, providers may make adaptations in response to social determinants of health and individual social needs (Braveman et al., 2011). Such adaptations made during real-world implementation to improve the fit of an EBP for health disparity groups affected by social determinants of health may or may not alter the core elements or components of an EBP.
Wiltsey Stirman and colleagues developed and refined a system for classifying the types of modifications and adaptations made to EBPs implemented in routine care as fidelity-consistent or fidelity-inconsistent (Wiltsey Stirman et al., 2015, 2017, 2019). Fidelity-consistent modifications to an EBP are defined as those that preserve core elements of an intervention that are needed to be effective. In contrast, fidelity-inconsistent modifications are those that alter the EBP in a manner that fails to preserve its core elements. Within this conceptualization, fidelity-inconsistent adaptations have the potential to reduce an EBP’s effectiveness. However, little empirical research has examined the relationship between fidelity-consistent and fidelity-inconsistent adaptations and implementation, and patient-level outcomes.
In a prior implementation study, we found that an adaptation to an evidence-based lifestyle intervention (InSHAPE) for adults with SMI rated as fidelity-inconsistent by model developers was associated with reduced cardiovascular risk (Aschbrenner et al., 2020). InSHAPE consists of weekly individual “health mentoring” sessions with a fitness trainer that includes supported exercise and personalized dietary guidance (Bartels et al., 2013, 2015). The intervention developers specified in the InSHAPE manual and fidelity scale that a core component of InSHAPE involves health mentor sessions taking place at a fitness facility or gym in the community that is open to the general public to promote social integration and thereby support the health and well-being of people with SMI (Van Citters et al., 2010). During a recent national implementation study of InSHAPE in behavioral health organizations across the United States (Aschbrenner et al., 2019), our research team found that a subset of sites adapted InSHAPE by changing the venue of the health mentor sessions from a community-based fitness facility to the mental health agency. This adaptation was rated as fidelity-inconsistent by the model developers who considered integrated fitness or gym facility participation a core component of InSHAPE. However, this fidelity-inconsistent adaptation was positively associated with cardiovascular risk reduction among participants (Aschbrenner et al., 2020).
The purpose of this study was to characterize the process and reasons for the fidelity-inconsistent adaptation to InSHAPE that resulted in improved health outcomes for participants with SMI with a focus on the role of social determinants of health in the adaptation. We used the Framework for Reporting Adaptations and Modifications-Expanded (FRAME) (Wiltsey Stirman et al., 2019) to evaluate when and how modifications occurred, whether they were planned (i.e., intentionally occurring in advance of the implementation by design) or unplanned (i.e., occurring during or after the implementation without advance planning), their relationship to fidelity, and the reasons and goals for the modifications. In applying the FRAME, we evaluated whether social determinants of health such as poverty and discrimination created obstacles to access or use of InSHAPE and thus were reasons for the adaptation.
Methods
Our sample consisted of InSHAPE teams (health mentors and their supervisors) at behavioral health organizations participating in a Hybrid Type 3 implementation study. The InSHAPE implementation study was a cluster randomized controlled trial (RCT) evaluating the effectiveness of a Virtual Learning Collaborative (VLC) compared with Technical Assistance (TA) in the implementation of InSHAPE. Details about the InSHAPE implementation RCT and methods can be found elsewhere (Aschbrenner et al., 2019). Briefly, 55 mental health provider organizations were randomized to the implementation interventions and 39 sites were assessed at the 24-month follow-up period. In a prior study of adaptations to InSHAPE, we invited all 39 sites at 24-month follow-up to participate in an additional 1-hr semi-structured interview by phone to identify adaptations they had made to InSHAPE since the initial training and implementation (Aschbrenner et al., 2020). Of the 39 sites, 37 agreed to participate. The interviews were audio-recorded and transcribed for data analysis purposes. Participants were asked to provide verbal consent to participate in the telephone interviews after reviewing an information sheet detailing the study procedures. Dartmouth College Committee for the Protection of Human Subjects approved the protocol for this study.
InSHAPE lifestyle intervention
InSHAPE is an evidence-based lifestyle intervention developed for adults with SMI consisting of weekly individual meetings with a certified fitness trainer (i.e., health mentor) who provides individually tailored instruction on both exercise and healthy eating to reduce cardiovascular risk through weight loss and improved cardiorespiratory fitness. Evidence supporting InSHAPE as an EBP includes two RCTs (Bartels et al., 2013, 2015) and a statewide implementation study (Bartels et al., 2018) conducted through an academic–community partnership. The community partners who developed the manualized InSHAPE model specified its core components to include (1) weekly individual meetings with a health mentor who provides instruction and motivational support for an individually tailored program of exercise and healthy eating; (2) securing free or low-cost gym or fitness facility memberships for participants and holding health mentor sessions at a gym or fitness facility that is integrated with the general public; and (3) group motivational celebrations.
In a prior study, we identified adaptations to InSHAPE made by providers during implementation. InSHAPE model developers and training team rated the adaptations as fidelity-consistent or fidelity-inconsistent, and our research team evaluated the impact of adaptations overall and by type on implementation and patient-level outcomes (Aschbrenner et al., 2020). Five of the 37 sites we interviewed at 24-month follow-up made a fidelity-inconsistent adaptation to InSHAPE that involved changing the venue of health fitness mentor sessions from community fitness facilities (a core component of InSHAPE) to the mental health agency. This fidelity-inconsistent adaptation was significantly and positively associated with cardiovascular risk reduction among participants. This study uses transcripts from qualitative data collected at these five sites to characterize the process and reasons for the successful fidelity-consistent adaptation using an equity lens to evaluate the role of social determinants of health in the adaptation. Characteristics of the five sites are described using an Organizational Characteristics Survey administered during the implementation study (Aschbrenner et al., 2019).
Characterizing adaptations
We used the FRAME (Wiltsey Stirman et al., 2019) to document the process and reasons for the successful fidelity-inconsistent adaptation to InSHAPE. FRAME provides a guide for researchers to evaluate and report eight aspects of adaptations to EBPs: (1) when and how in the implementation process the modification was made, (2) whether the modification was planned/proactive or unplanned/reactive, (3) who determined that the modification should be made, (4) what is modified, (5) at what level of delivery the modification is made, (6) type or nature of context or content-level modifications, (7) the extent to which the modification is fidelity-consistent, and (8) the reasons for the modification, including (a) the intent or goal of the modification (e.g., improve fit, adapt to a different culture, reduce costs) and (b) contextual factors that influenced the decision, both directly and indirectly. In evaluating the reasons for adaptations guided by the FRAME domains, we focused on assessing whether social determinants of health affected participants’ access to and use of integrated fitness or gym facilities and thus played a role in the fidelity-inconsistent adaptation. Specifically, we considered whether social determinants, including poverty and discrimination based on having a mental illness, interacted with the sociopolitical environment, organization, provider, and/or recipient to create obstacles to access and use of InSHAPE in community-based fitness and gym facilities.
Qualitative data analysis
Two authors (K.A.A. and N.M.M.), who are implementation scientists with experience in qualitative research, reviewed the interview transcripts from the five sites that made the fidelity-inconsistent adaptation to InSHAPE. During the coding process, K.A.A. and N.M.M. independently read and coded the transcripts using a template outlining the domains from the FRAME as a guide to apply constructs that represented the process and reasons for the adaptation, and included a consideration of social determinants of health as reasons for the adaptation. This approach to qualitative coding was informed by template coding, a qualitative method used in implementation research (Hamilton & Finley, 2019). The pair met after coding each transcript to review the codes applied to the data. Any discrepancies in coding were discussed until consensus was reached on reporting the reasons for and processes of the adaptation, including examples of social determinants of health informing the fidelity-inconsistent adaptation.
Results
Transcripts of interviews with five behavioral health organizations were reviewed. Organizations were in suburban (n = 1), urban (n = 1), rural (n = 1), and mixed rural/urban (n = 2) locations. The volume of SMI clients served annually ranged from 256 to 3,472 across the sites (M = 1,812, SD = 1,162). The majority of clients with SMI paid for services with Medicaid (M = 64.8%, SD = 28.3%).
The results of the FRAME analysis of the transcripts across five sites are presented in Table 1. There were consistencies across the agencies in the reasons for and processes of adapting InSHAPE. For example, the fidelity-inconsistent adaptation of changing the venue of health mentor sessions from a community-based fitness facility or gym (a core component of InSHAPE) to the mental health agency was described by InSHAPE teams as unplanned and in response to challenges in delivering health mentor sessions as intended. In all cases, the goal of the adaptation was to improve feasibility of the health mentor sessions and fit of the program for participants.
Application of the FRAME to describe the process and reasons for the fidelity-inconsistent adaptation to InSHAPE.
Lack of reliable transportation, cost, and mental illness stigma were barriers to accessing and using health mentor sessions held at fitness facilities or gyms, leading agencies to adapt the program by offering sessions locally at the mental health agency.
Transportation
Transportation was described as a barrier in several ways. One InSHAPE team member discussed transportation as both not readily available in the community and something that her clients were hesitant to use: For all of our [name of county] folks who wanted to go to the [name of fitness facility] they would have to take a two hour bus ride and then walk half a mile from the bus stop to the [name of fitness facility] . . . A lot of our clients live either in like supervised apartments or like group home settings and have a big hesitancy to riding public transportation on their own. (Site 1, Health Mentor)
Similarly, another site discussed transportation as an issue for the health mentor, who was transporting clients to gyms for exercise sessions: I think some of the biggest obstacles is that we’re a very large county and there was a lot of driving involved, and cause she [health mentor] was transporting people back and forth to the gym and, of course, there it was only one on one [health mentor sessions], but she could do that, I think her difficulty was spending all that time driving, there were other programs, there were multiple programs that were pulling participants from and often times they were strapped or they couldn’t have staff helping transport, which I know would have helped her [health mentor] out. (Site 2, Supervisor)
Finally, one site stated that the cost of transportation to the mental health care facility was reimbursed by insurance but travel to the gym was not. Exercising at the mental health care facility improved attendance because it allowed participants to travel to the facility by taxi, a reliable, reimbursable mode of transportation: I think it’s a convenience factor. A lot of them [participants] already go there [mental health agency] or get transportation provided to that location. So, in terms of people not showing up or missing appointments, it’s helpful in that respect . . . I have a really good attendance rate, there, and I think it is because transportation is kind of an issue with [name of city] being so big. I had a lady tell me it would take her an hour and a half, one way, to be able to meet me one day. But because insurance provides cab service [to the mental health agency], she can get a cab there, which is fairly reliable. And I’ve had almost no no-shows. That’s really good. (Site 5, Health Mentor)
Cost of fitness facility or gym membership
InSHAPE teams discussed the cost of a gym membership as a barrier to exercising in the community. This barrier was compounded by resistance from clients, who felt that a gym membership was not aligned with their preferences, and resistance from the fitness facilities that actively discouraged clients from obtaining memberships: I’ve had people just not be interested in the [name of fitness facility] or a gym, just because like it’s a gym atmosphere, and then from money reason, and then just cause they don’t wanna go there, like it wasn’t something that they found necessary. (Site 1, Health Mentor) We had a big challenge when it came to our community [name of fitness facility]. They were very discouraging to our members about membership. They wouldn’t even work with us to provide a great environment for them to be in or a financial—you know, helping there. (Site 4, Health Mentor)
Mental illness stigma
InSHAPE teams described resistance from gym employees who were worried about clients using the gym unsupervised or displaying disruptive behavior: They [name of fitness facility] pretty much said they didn’t want unstable people in their gym. Because if they had a membership, they were gonna be able to go all the time. And they only wanted the client to be able to come with me, the health mentor. And that was kind of like exclusion and that was kind of defeating the purpose, because we wanted to get into the community . . . I met with a wellness worker at the [name of fitness facility], and [name of fitness facility] and they all said they all had concerns with our clients and their behavior . . . What if they act out when there were in there by themselves? (Site 2, Health Mentor)
Discussion
Adaptations to EBPs occur frequently, yet stakeholder decision-making processes behind adaptations generally remain undocumented. This makes it difficult to understand how and why interventions fail or succeed in different contexts and can preclude efforts to scale-up or sustain programs (Evans et al., 2019). The purpose of this study was to document and characterize the process and reasons for a fidelity-inconsistent adaptation that improved health outcomes in a health disparity group using the FRAME with an emphasis on assessing the role of social determinants of health in the adaptation process. We found that the reasons why behavioral health organizations implementing InSHAPE made a fidelity-inconsistent adaptation to a core component of the model (i.e., integrated exercise in the community at a gym or fitness facility open to the general public) were in response to social determinants of health that created barriers to access and use of the EBP. In all cases, the stated goal of the adaptation was to improve intervention access, feasibility, and fit of the program with participants. The results demonstrate the need to consider whether fidelity-inconsistent adaptations to EBPs may be in the service of reducing inequities in access and use of care for health disparity groups.
This study contributes to efforts to advance health equity within implementation science by highlighting how an existing implementation science framework (FRAME) can be used to examine the role of social determinants of health in adaptations to EBPs for health disparity populations. FRAME focuses on four domains to guide the evaluation of the reasons for adaptations: (1) sociopolitical, (2) organizational/setting, (3) provider, and (4) recipient. In applying the FRAME, we focused on whether and how social determinants of health interacted with each of these domains to explore why a fidelity-inconsistent adaptation was made by agencies during the course of an implementation study where fidelity to the intervention’s core components was taught, emphasized, reinforced, and assessed over a 24-month study period.
Findings from this study illustrate what could be gained in implementation science by emphasizing the impact of social determinants of health on access to and use of EBPs by health disparity groups and allowing for flexible adaption to intervention forms to create more equal access to these interventions. EBP fidelity is often a primary outcome of implementation evaluations (Proctor et al., 2011); however, fidelity to core intervention components may not be possible or desirable when social determinants of health create obstacles to delivering an intervention model as originally designed. EBP adaptations, even those that are fidelity-inconsistent, may serve to promote equitable implementation particularly when health disparity groups experience obstacles to participation in an EBP associated with economic, social, or environmental disadvantage.
Evaluating adaptations from an equity lens has the advantage of highlighting core components of fidelity as specified by model developers that may not be feasible or desirable to implement in health disparity populations (Baumann & Cabassa, 2020). Core functions are an intervention’s basic purposes, while its forms are the strategies used to meet each function (Perez Jolles et al., 2019). As discussed in our prior research (Aschbrenner et al., 2020), the fidelity-inconsistent adaptation of changing the venue of the health mentor sessions from an integrated gym or fitness facility located in the community and open to the public does not change the primary function of InSHAPE to address obesity in individuals with SMI by promoting weight loss and improvements in cardiorespiratory fitness through supported exercise and dietary guidance. However, the InSHAPE model developers rated changes to this model component as inconsistent with fidelity. By exploring how social determinants of health can create obstacles to access and use of an EBP in a health disparity population, EBP developers may be confronted with the need to reconceptualize core intervention components and identify the EBP’s core functions and forms to promote adaptation that enables participation by health disparity groups while preserving the effectiveness of an intervention (Kirk et al., 2020).
An area ripe for future research is the study of how social determinants of health influence implementation, adaptation, and sustainability when EBPs are implemented for health disparity groups, particularly in low-resourced health care settings, and what to do about them. Implementation outcomes (e.g., program fidelity), service outcomes (e.g., program participation), and participant outcomes (e.g., health outcomes) (Proctor et al., 2011) can be impacted by social and economic disadvantages experienced by the patient population served. Leaders in the field of implementation science have called for improving and using equity-relevant metrics, including measures of disadvantage that impact the context for implementation (Brownson et al, 2021) and thus could influence EBP adaptations. As implementation scientists build consensus around equity-relevant metrics and design and tailor implementation strategies to promote health equity, it will be critical to get input from stakeholders on measures of health equity and approaches to addressing equity (including allowable adaptations) that are pragmatic and useful in routine clinical practice.
Study limitations
Limitations of this study are the small sample size, retrospective nature of the adaptation interviews, missing patient-level data on cardiovascular risk reduction from one site in the qualitative analysis, and secondary use of qualitative interview transcripts to investigate the research question. In the previous adaptation study, we asked InSHAPE teams to recall any adaptations they made to the intervention during the course of the 2-year implementation study. It is possible that InSHAPE teams were not able to recall and thus report specific and important details about the processes and reasons for the fidelity-inconsistent adaptation characterized in this study. During the interviews, we did not explicitly ask the InSHAPE teams to describe any modifications they made to the intervention to address inequities in delivery of intervention components. Doing so might have revealed additional adaptations made in the service of promoting equitable implementation of the intervention.
Conclusion
Adapting the core components of an EBP may be an appropriate, desirable, and necessary strategy during implementation with health disparity populations, particularly when culture and context do not match the EBP (Cabassa & Baumann, 2013; Escoffery et al., 2018; Tu et al., 2014). Despite recent calls for an equity approach to implementation where EBPs may need to be adapted for health disparity populations (Baumann & Cabassa, 2020; Chinman et al., 2017; Eslava-Schmalbach et al., 2019), few, if any, empirical studies have applied an equity lens to evaluate the reasons why organizations and providers make adaptations to an EBP during implementation. Findings from the current study highlight the advantages of considering when and how fidelity-inconsistent adaptations to an EBP may be in the service of reducing inequities in access to and use of EBPs for health disparity groups. In evaluating the reasons for EBP adaptations, developers may be confronted with the need to reconceptualize core intervention components and identify an EBP’s core functions and forms to promote adaptation that enables participation by disadvantaged groups most affected by social determinants of health while preserving the effectiveness of an intervention.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by R01MH102325 awarded to S.J.B. N.M.M. was funded by T32CA057711. The research team received support from The Implementation Science Center for Cancer Control Equity (ISCCCE) P50P50 CA244433 to conduct this work. The funding sources had no role in the study design, execution, analyses, interpretation of the data, or decision to submit results.