Abstract
Background
To help improve the implementation of evidence-based substance use disorder (SUD) treatment in practice settings, the United States funds a support system called the Addiction Technology Transfer Center (ATTC) network. Prior implementation research in HIV care found the team-focused Implementation and Sustainment Facilitation (ISF) strategy as an effective addition to the ATTC's staff-focused training, feedback, and consultation (TFC) strategy. Using the ISF + TFC strategy as the control, this type-3 hybrid trial tested the effectiveness of adding a staff-focused incentivization (INC) strategy (ISF + TFC + INC vs. ISF + TFC). Staff-focused incentivization was selected because prior implementation research found it to be highly effective and cost-effective for improving SUD treatment implementation.
Methods
Twenty-six HIV service organizations (HSOs), their staff participants (N = 87), and their client participants (N = 341) were cluster-randomized to either the ISF + TFC control condition or ISF + TFC + INC experimental condition. The INC strategy rewarded/reinforced motivational interviewing brief intervention (MIBI) implementation (US$10 per MIBI delivered) and MIBI implementation at or above a pre-defined level of quality (US$10 per demonstration). In addition to these outcomes, past 4-week changes/reductions in client participant's days of primary substance use and anxiety symptoms were examined.
Results
The addition of the INC strategy had a large and significant (p < .05) effect on the number of MIBIs implemented (d = 1.30) and reduction in anxiety (d = −1.54). There was no significant impact on days of substance use.
Conclusions
The addition a staff-focused INC strategy improved implementation of an evidence-based brief intervention for adults with comorbid HIV and SUD, and also reduced anxiety. To help improve the integration of evidence-based SUD services in HSOs across the United States, use of the ISF + TFC + INC strategy by the ATTC network and/or the AIDS Education and Training Center (AETC) network is recommended.
Plain Language Title
Staff incentives improved implementation of a motivational interviewing brief intervention for substance use disorders within HIV service organizations
Plain Language Summary
To improve implementation of effective substance use disorder interventions in practice settings, the United States funds a support system called the Addiction Technology Transfer Center (ATTC) network. Prior research found the Implementation and Sustainment Facilitation (ISF) strategy was effective for improving the ATTC's training, feedback, and consultation (TFC) strategy. Building on that research, this research tested also adding a staff incentivization (INC) strategy, which is a strategy that was effective in a different setting. Twenty-six HIV service organizations, their staff (N = 87), and their clients (N = 341) were selected at random to receive either the ISF + TFC strategy or the ISF + TFC + INC strategy. The INC strategy rewarded motivational interviewing brief intervention (MIBI) implementation ($10 per MIBI delivered) and MIBI implementation at or above a pre-defined level of quality ($10 per demonstration). We examined these two outcomes, as well as examined client participant's days of primary substance use and anxiety symptoms. We found the addition of the INC strategy improved the number of MIBIs implemented and reduced anxiety but did not reduce days of substance use. To help improve the use of evidence-based substance use disorder services in HIV practice settings, use of the ISF + TFC + INC strategy is recommended.
Keywords
Introduction
To help improve the implementation of evidence-based substance use disorders (SUDs) interventions in practice settings, the Substance Abuse and Mental Health Services Administration's Center for Substance Abuse Treatment funds the Addiction Technology Transfer Center (ATTC) network—the support system within the Interactive Systems Framework for Dissemination and Implementation (Wandersman et al., 2008). SUDs have a substantial, adverse impact on antiretroviral treatment, quality of life, morbidity, and mortality among people with HIV (Doshi et al., 2021; Garner et al., 2022a; Lesko et al., 2023), as well as HIV transmission risk behaviors(Jenkins et al., 2023). In 2014, consistent with calls for support systems to use an “evidence-based system for innovation support” (Wandersman et al., 2012), the National Institute on Drug Abuse (NIDA) funded a type 2 hybrid trial to test (a) the incremental effectiveness of the team-focused Implementation and Sustainment Facilitation (ISF) strategy as an adjunct to the ATTC's staff-focused training, feedback, and consultation (TFC) strategy, and (b) the effectiveness of a motivational interviewing brief intervention (MIBI) for comorbid HIV and SUD. Relative to the TFC strategy, the ISF + TFC strategy was found to significantly improve both implementation effectiveness (i.e., the consistency and quality of MIBI implementation) and intervention effectiveness (i.e., the effectiveness of the MIBI for reducing client's days of primary substance use at follow-up) (Garner et al., 2020). This article presents the results of a type-3 hybrid trial to test the impacts of further adding a staff-focused incentivization (INC) strategy (ISF + TFC vs. ISF + TFC + INC).
Rationale for Testing Incentivization
The rationale for testing incentivization was threefold. First, incentivization is a strategy included as part of the Expert Recommendations for Implementing Change (ERIC) compilations (Powell et al., 2012, 2015), though it is named “alter incentive/allowance structures” and defined as “work to incentivize the adoption and implementation of the clinical innovation” (p. 8). Second, incentivization directly addresses the reward dimension of implementation climate (i.e., the extent to which implementation is expected, supported, and rewarded), which is a key mechanism of change according to the theory of implementation effectiveness (Helfrich et al., 2007; Klein et al., 2001; Klein & Sorra, 1996). Although implementation may be rewarded via financial or non-financial rewards/incentives, financial incentives were used as part of this incentivization study. More specifically, HIV service organization (HSO) staff participants had the opportunity to earn $10 per MIBI implemented with HSO client participants, as well as $10 per MIBI implemented at or above a benchmark level of MIBI quality. These incentivization amounts, which were determined based on discussion and consensus by the study team, were believed to represent a good balance between willingness-to-pay from the perspective of the AETC support system and sufficient-to-motivate from the perspective of the HSO staff. Third, prior implementation research found incentivization to be an effective and cost-effective strategy for improving implementation of an evidence-based SUD intervention (Garner et al., 2012, 2018). More specifically, although incentivization resulted in a 5% increase in cost, it resulted in a 116% increase implementation outcome and 325% increase in client outcome. An advantage of using incentivization as an implementation strategy is relatively simple to use and only increases costs when the desired behavior is demonstrated. In contrast, strategies like facilitation are relatively complex (i.e., training and supervision of the facilitators is required) and increases costs whether or not the facilitation leads to the desired behaviors.
Rationale for Trial Design
The current study used a parallel cluster-randomized design, staff clustered within HIV service organization (HSO; the unit of randomization) to minimize the likelihood of contamination across the project's two implementation conditions. A type-3 hybrid trial was used given the importance of examining implementation outcomes and client outcomes (Foy et al., 2015), including anxiety symptoms that are associated with SUDs among people with HIV (Brandt et al., 2017; Mannes et al., 2019) and have potential for improvement following SUD intervention (Bahorik et al., 2017; Wolitzky-Taylor, 2023).
Aims and Hypotheses
As illustrated in Figure 1, the aims of this type-3 hybrid trial included testing the incremental effectiveness of a staff-focused incentivization strategy for improving both implementation outcomes and client outcomes. We hypothesized that relative to MIBI staff trained and supported using the ISF + TFC strategy (control condition), MIBI staff trained, supported, and rewarded using the ISF + TFC + INC strategy (experimental condition) would achieve significantly greater: (1) MIBI implementation consistency, (2) MIBI implementation quality, (3) MIBI implementation effectiveness (i.e., the consistency and quality of implementation), and client participants in the experimental condition would achieve significantly greater, (4) reductions in days of primary substance use, and (5) reductions in anxiety symptom severity. We combined MIBI implementation consistency and MIBI implementation quality together to be consistent with the theory of implementation effectiveness, which operationalizes implementation effectiveness as the consistency and quality of implementation (Helfrich et al., 2007; Klein et al., 2001; Klein & Sorra, 1996). Although the MIBI does not specifically address anxiety, reductions in anxiety symptom severity was included as a secondary outcome to be responsive with the funding mechanism (RFA-MH-20-520: Implementation Research in HRSA Ryan White Sites: Screening and Treatment for Mental and Substance Use Disorders to Further the National Ending the HIV Epidemic Goals) and given prior research has found improvements in anxiety following SUD intervention (Bahorik et al., 2017; Wolitzky-Taylor, 2023).
Conceptual Overview of Project.
Methods
Trial Design
The design was a 26-site parallel cluster-randomized type-3 hybrid trial (Curran et al., 2012, 2022). Guided by the EPIS framework (Aarons et al., 2011) HSOs (and their staff) were recruited and randomized to one of two conditions (ISF + TFC or ISF + TFC + INC) during the exploration phase. During the hybrid preparation-implementation phase the ISF + TFC strategy was available to HSOs (and their staff) in both conditions. The INC strategy was available to MIBI staff cluster-randomized to the TFC + ISF + INC condition from onset of the preparation-implementation phase. A MIBI staff transitioned to implementation by implementing a MIBI with a client participant. Thus, HSO's were in a hybrid preparation-implementation phase until all MIBI staff had implemented the intervention with at least one client participant. The preparation-implementation phase was 12 months for the project's first cohort and 6 months for the project's second cohort. This unplanned difference was due to the project being stopped early when the principal investigator changed institutions. Approval and oversight of all research activities was provided by the Institutional Review Board of Advarra.
Context
The project's participating HSOs (N = 26) were located in 13 states and the District of Columbia within the United States. HSOs are an appropriate context given the high prevalence of SUDs among people with HIV (Garner et al., 2022a; Hartzler et al., 2017).
Participants
Staff Participants
For an HSO to be eligible for the project, it had to receive funding from the Health Resources & Services Administrations’ Ryan White program, have two or more staff members (e.g., case manager, behavioral specialist) willing to be trained to implement a MIBI for SUD, and have at least one leadership staff (e.g., director, manager, supervisor) willing to help ensure the MIBI staff were provided sufficient time for participation. The participating HSOs identified individual staff (typically case management staff) to be invited to participate. Case management staff were considered ideal because of their regular interactions with clients and greater ability to incorporate this into their existing work with clients. Staff were required to be at least 18 years of age. There were no any exclusion criteria for HSO or staff participation. After staff provided informed consent to participate in the project, which was obtained electronically, they completed an online survey. Each participant received a US$25 e-gift card upon completion. The survey assessed background characteristics (e.g., age, biological sex, ethnicity, race, highest level of education), implementation climate using the six-item measure developed and described by Jacobs et al. (2014), motivational interviewing knowledge and experience using a two-item index measure developed for this study and where staff self-rated their current baseline knowledge (0 = no knowledge at all, 1 = knowledgeable to a small extent, 2 = knowledgeable to a moderate extent, 3 = knowledgeable to a great extent) and experience (0 = none, 1 = beginner, 2 = intermediate, 3 = advanced, 4 = expert) using MI, setting-intervention fit using the six-item measure developed and described by Garner et al. (2022b), and organizational readiness for change using the 12-item measure developed and described by Shea et al. (2014).
Client Participants
To be eligible, clients had to be at least 18 years of age and HIV positive. There were no exclusion criteria for client participation. Client participants were recommended to receive a MIBI when they reported unhealthy alcohol use or endorsed two or more of the 11 Diagnostic and Statistical Manual of Mental Disorders (DSM-V) criteria (American Psychiatric Association, 2013) for at least one substance during the past 12 months. Client participants received a US$10 e-gift card upon completion of the online SUD screener. MIBI-recommended client participants were asked to complete additional online assessments. These included a pre-MIBI assessment completed within the 24-h period prior to receiving the MIBI and a follow-up assessment completed 4 weeks after receiving the MIBI. The MIBI was provided at no cost. Participants received e-gift cards upon completion of each assessment (US$20 for pre-MIBI assessment; US$10 for 4-week follow-up assessment). For the 4-week follow-up assessment, participants received a US$10 bonus for completion within 24 h of the online link being emailed or texted to them or a US$5 bonus for completion within 25–48 h of the online link being emailed or texted to them.
Strategies
As part of the SAT2HIV Project, the team-focused ISF Strategy was an effective adjunct strategy to the staff-focused TFC Strategy (ISF + TFC) (Garner et al., 2020). As such, ISF + TFC was the control strategy and ISF + TFC + INC as the experimental strategy in the SAT2HIV-II Project. Regarding temporality, ISF was the strategy HSOs and their MIBI staff received first. This was strategic and designed to help optimize MIBI staff's early understanding and motivation for the TFC Strategy's components. Importantly, for HSOs randomized to ISF + TFC + INC, the ISF strategy was intended in part to accelerate MIBI staff's understanding and motivation for achieving the incentivization strategy targets: MIBI consistency and MIBI quality. Each strategy is described below and Table 1. Recently developed, the Strategies Timeline, Activities, and Resources (STAR) Table (Garner et al., 2025) helps standardize reporting of strategies in accordance with recommendations by Proctor et al. (2013). All strategies were provided by the research team and therefore were external to the HSOs.
The Strategies Timeline, Activities, and Resources (STAR) Table.
Note. ISF = implementation and sustainment facilitation; HSO = HIV service organization; MI = motivational interviewing; MIBI = motivational interviewing brief intervention.
Implementation and Sustainment Facilitation
The ISF is a team-focused strategy that seeks to optimize MIBI consistency and MIBI quality via elevating implementation climate (i.e., the extent to which MIBI staff perceive these dimensions of implementation effectiveness to be expected, supported, and/or rewarded) (Garner et al., 2020; Garner, Zehner et al., 2017). It is team-focused in that it requires engagement of staff to deliver the intervention as well as leaders to help address implementation barriers (e.g., competing priorities, insufficient time). To help optimize staff understanding of the implementation initiative and motivation for the training provided as part of the preparation phase, it is recommended that the first ISF strategy meeting be completed prior to the initiation of the staff-focused training strategy. Consistent with other research that has used motivational interviewing as an implementation strategy (Hettema et al., 2014; Kauth et al., 2010), ISF uses key principles of motivational interviewing (i.e., engaging, focusing, evoking, planning) as a guide.
In this project, each HSO engaged in a virtual monthly ISF strategy meeting that included three or more of the four motivational interviewing principles. These meetings lasted 30–60 min. An ISF workbook programmed in Microsoft Excel helped standardize the ISF strategy, document HSO staff attendance, document ISF fidelity (i.e., at least three of the four motivational interviewing principles were employed), and document any meeting notes and/or action items. The ISF strategy was provided to each HSO by one of four master-level facilitators, two of which had provided ISF as part of the SAT2HIV Project. Weekly ISF team meetings supervised by a PhD-level ISF strategy facilitator were conducted for quality assurance purposes.
Training, Feedback, and Consultation
The TFC strategy, similar to the ATTC strategy in the SAT2HIV Project, is a staff-focused strategy composed of three discrete strategies. As part of the current project, MIBI staff were provided access to the project's MIBI manual, 4-h online introduction to motivational interviewing training course, and 12-h virtual training in the MIBI protocol. Although conducted by the same trainer as the SAT2HIV Project, the 12-h virtual training was a condensed version of the 2-day in-person training used in the prior trial. This approach was necessitated due to COVID-19 safety protocols. The training content provided as part of the project's in-person training was condensed into three 4-h virtual training sessions and was conducted over a four-week period. A one-week break between the second and third training sessions was provided to enable MIBI staff with time to practice the MIBI before the third training session. Free continuing education credits were provided for completion of both the online training and virtual training. Fidelity to the training strategy was ensured by using a single trainer from the Motivational Interviewing Network of Trainers.
Feedback on the quality of MIBI implementation was provided via a machine-learning-based feedback system for motivational interviewing (Imel et al., 2019). This machine-learning approach was selected due to being more efficient than using humans to rate MIBI quality. Using this system costs approximately $40 per staff and there is not a limit on the number of sessions rated per month, whereas it typically costs about $50 per session to have a human spend their time listening to the session and provide ratings of quality. In addition to the cost savings, use of this machine-learning approach enabled HSO staff to receive immediate and standardized high-fidelity feedback, as opposed to having to wait days or even weeks when ratings and feedback is provided by a human. For each MIBI session at least 10 min in length, a MIBI quality score was recorded. The range for each MIBI quality score was 0‒12, with higher scores indicative of higher quality. The MIBI quality score was provided to MIBI staff as part of a feedback report, which as detailed by Imel et al. (2019) is accessed by MIBI staff within the machine learning system and includes feedback on multiple motivational interviewing aspects (e.g., spirit, empathy, reflection-to-question ratio, percent open questions, percent complex reflections). MIBI staff were required to demonstrate at least average MIBI proficiency (a score of 4) via a practice role play session, which was based on the average MIBI proficiency provided as part of the MIBI effectiveness trial (Garner et al., 2020), hence sufficient for achieving intervention effectiveness. MIBI role play sessions were conducted with another MIBI staff at their HSO who played the role of the client using a standardized client scenario provided by the project.
In addition to training and feedback, MIBI staff were provided the opportunity to participate in group MIBI consultation meetings. Conducted separately for each condition, these meetings were provided monthly, were 30–60 min in length, and were conducted by the same trainer who conducted the virtual training sessions (i.e., a trainer from the Motivational Interviewing Network of Trainers). In addition to providing didactic booster training regarding motivational interviewing and our project's MIBI protocol, these group consultations provided MIBI staff with the opportunity to ask specific questions about how to improve the quality of MIBI implementation with client participants.
Incentivization
Incentivization (INC) was a staff-focused strategy that used financial incentives to explicitly target the reward dimension of implementation climate. Fidelity regarding the INC strategy was ensured by having the research team send MIBI staff earned incentives. For each MIBI session implemented, the MIBI staff received a US$10 incentive. Additionally, for each MIBI session at least 10 min in length and that received a 6 + quality rating via a machine-learning-based feedback system for motivational interviewing (Imel et al., 2019), the MIBI staff received an additional US$10 incentive. This machine-learning-based system enabled MIBI staff to receive immediate standardized high-fidelity feedback on session quality, including whether their overall score met or exceeded the project's pre-defined quality target. The project's target was a score of 6, which was the 80th percentile score obtained by MIBI staff as part of the MIBI effectiveness trial described above.
For the project's first cohort, incentivization of attendance at the group consultation meetings and ISF meetings was introduced after month 6, which was an unplanned adaptation of the INC strategy and intended to help improve attendance by HSO staff. Approved by the project's IRB, MIBI staff were asked to complete a new informed consent form to document their consent to receive a US$30 incentive per monthly group consultation meeting attended and a US$30 incentive per ISF meeting attended. Also approved by the IRB, leadership staff were asked to complete a new informed consent form that documented their consent to receive a US$30 incentive per ISF meeting attended. These additional incentives were only for the HSOs and their participating staff that were randomized to the ISF + TFC + INC condition. This revised incentivization protocol was offered from the start for the project's second cohort. During the first week of each month, research staff calculated the incentive amount earned by MIBI staff during the prior calendar month, sent a notification email to the staff, and had an e-gift card for the total incentive amount earned sent to the MIBI staff. There was no a limit on the amount MIBI staff were able to earn.
Intervention
The project's single-session MIBI protocol has been shown to be effective in multiple settings (Garner et al., 2020; Martino et al., 2018). It was designed to help motivate an individual with comorbid HIV and SUD to change their primary substance use by: (1) examining their reasons for change, (2) receiving feedback about some common negative interactions between substance use and HIV-related physical and mental health issues, (3) increasing the importance or confidence to reduce or stop their primary substance use, and (4) making a plan for change (Garner, Gotham et al., 2017).
Outcomes
Implementation Outcomes
According to the theory of implementation effectiveness (Helfrich et al., 2007; Klein et al., 2001; Klein & Sorra, 1996), implementation effectiveness is a multidimensional construct representing the consistency and quality of implementation by targeted users. Consistency is similar to measures like reach (Glasgow et al., 2019) and penetration (E. Proctor et al., 2011) and quality is similar to measures like fidelity (E. Proctor et al., 2011) and integrity (Dane & Schneider, 1998). MIBI consistency was operationalized as the total number of MIBIs a trained MIBI staff implemented during the project, with no minimum session length. MIBI quality was operationalized as the total quality score a trained MIBI staff demonstrated during the project. For a MIBI quality score to be generated a MIBI session had to be at least 10 min in length. Each MIBI at least 10 min in length received a quality score between 0 and 12, with higher scores indicative of higher quality. Following the standardization of each respective measure, these two measures were summed and standardized to create a measure of implementation effectiveness.
Client Outcomes
Consistent with its type-3 hybrid trial design, this project also examined the incremental impact of incentivization on client outcomes 4-week post-MIBI. The two client outcomes of interest were change in days of primary substance use and change in anxiety symptom severity, both of which were based on client participant's self-report. They were asked “out of the past 28 days (4 weeks), about how many days did you use [insert their primary substance].” The Generalized Anxiety Disorder 7-item (GAD-7) (Spitzer et al., 2006), with scores ranging from 0 to 21 (lowest to highest severity), was used to assess anxiety during the past two weeks. These client outcomes were aggregated for each MIBI staff. Thus, for each trained MIBI staff, we computed two client outcome impact measures. One for their impact on days of primary substance use, and one for their impact on anxiety. A change score was computed for each possible client participant (i.e., those who completed the 4-week follow-up assessment) by subtracting their respective pre-MIBI assessment measure from their respective MIBI follow-up assessment measure. Then, for each respective change score measure, we aggregated to the level of the corresponding MIBI staff. Negative values are preferable as they represent greater reductions in the outcome. If a MIBI staff member was trained but implemented the MIBI with zero client participants, their client impact scores were zero (i.e., no impact, no return on investment).
Targeted Sample Size
The targeted sample size was 30 HSOs and 90 MIBI staff. Assuming an intraclass correlation of .06 (the maximum ICC observed in the SAT2HIV project), power analysis indicated sufficient power to detect medium-sized effects.
Randomization Sequence Generation
For each project cohort, randomization of HSOs (the clusters) was completed by the project's principal investigator, project coordinator, and statistician. The principal investigator and project coordinator independently matched pairs of HSOs based on information collected as part of an organizational background form (e.g., number of people with HIV served per year, number of staff employed, number of staff who have been trained in any type of motivational interviewing). They then met to compare rankings and reach consensus on a final list of match pairs. The final list of matched pairs was given to the statistician who randomized each matched pair to one of the two implementation conditions.
Masking
Client participants were masked to implementation condition, as was the MIBI training staff
Statistical Methods
Statistical analyses were conducted using an intention-to-implement approach. All MIBI staff who received some component of the TFC strategy or ISF strategy were included in the analyses. All analyses were conducted using SAS (2023). An initial analytic step was to examine the partitioning of variance for each outcome measure. Mixed effects regression analyses were used for the impact of incentivization on implementation outcomes and general linear regression analyses were used for the impact of incentivization on client outcomes. Doubly robust estimation (i.e., combined propensity score weighting with outcome regression) was used to increase the models robustness to misspecification (Bang & Robins, 2005; Funk et al., 2011; Robins et al., 1994; Vermeulen & Vansteelandt, 2015). Factors used to create the staff propensity weight were age, gender, ethnicity, race, education, and experience, both with the organization and in their current position with the organization. All analyses controlled for project cohort. We examined the extent to which there were any significant interactions between condition assignment and the staff factors included in the model. If no significant interaction was found, we focused on the main effect of the implementation condition. Statistical significance was set at p < .05 and interpretation of effect sizes was guided by Cohen (1988): small effect size (d = 0.2), medium effect size (d = 0.5), and large (d ≥ 0.8).
Results
Participant Flow and Recruitment
Two cohorts of HSOs were recruited and randomized as part of the project. The first cohort, which cluster-randomized 12 HSOs, 44 MIBI staff, and 198 client participants, occurred from January 2021 through December 2021. The second cohort, which cluster-randomized 14 HSOs, 43 MIBI staff, and 143 client participants, occurred from November 2021 to April 2022. Figure 2 details the flow of the project's 26 HSO (87% of targeted sample), 87 MIBI staff (97% of targeted sample), and 341 client participants. The targeted sample size of 30 HSOs and 90 MIBI staff was not achieved because the project's principal investigator changed organizations and the trial was forced to stop early. A target number of client participants was not pre-specified because number of MIBIs implemented was a primary outcome of interest.
Participant Flow.
Half of the 26 HSOs were randomized to ISF + TFC and the other half were randomized to ISF + TFC + INC. In the ISF + TFC condition, one HSO withdrew from the project after randomization and before any of the project's preparation phase strategies (ISF meetings, MIBI training) could be provided. The number of MIBI staff per condition was nearly the same, with 44 in ISF + TFC and 43 in ISF + TFC + INC. The number of client participants in the ISF + TFC + INC condition (n = 224) was 91% more than in the TFC + ISF condition (n = 117). The positive screening rate was 14 percentage points higher in ISF + TFC + INC (80%) than in TFC + ISF (66%). There was little difference between conditions in terms of how many client participants withdrew from the study prior to completing the pre-MIBI assessment, with this rate being low for both conditions (10% for ISF + TFC and 8% for ISF + TFC + INC).
In addition to the SUD screening assessment, client participants were asked to complete a pre-MIBI assessment within the 24-h period prior to receiving the MIBI. If the pre-MIBI assessment and MIBI were not completed within 30 days of the SUD screen, the MIBI participant was documented as lost. The percentage of client participants lost between completion of the SUD screener and the pre-MIBI assessment was 18 percentage points lower in ISF + TFC + INC (30%; 50 of 165) than in ISF + TFC (48%; 33 of 69). Rates of completion for the 4-week post-MIBI assessment were nearly identical between conditions: 66% (23 of 35) for the ISF + TFC condition and 67% (70 of 104) for the ISF + TFC + INC condition. There was one baseline difference between client participants who completed the follow-up and those who did not, with a greater proportion of participants identifying as female completing the follow-up survey compared to those who did not (p = .04).
Participant Characteristics
Table 2 presents the baseline characteristics for MIBI staff participants for the overall sample (N = 87) and each condition (43 for ISF + TFC; 44 for ISF + TFC + INC). Most MIBI staff were female (63%), non-Hispanic (77%), non-White (63%), and did not have a graduate degree (72%). On average, MIBI staff were 40.0 (SD = 12.5) years of age, had 51.4 (SD = 67.6) months of experience with their HSO, and 37.3 months (SD = 53.1) of experience in their current position. As also noted in Table 2, there was one statistically significant difference between conditions, with a greater percentage of MIBI staff in the TFC + ISF condition having a graduate degree or higher (50%) compared to the TFC + ISF + INC condition (27.9%). Table 3 presents the characteristics of client participants who received the project's MIBI and completed the follow-up interview (N = 93).
Staff Demographic Characteristics.
Note. TFC = training, feedback, and consultation; ISF = implementation and sustainment facilitation; MIBI = motivational interviewing-based brief intervention; INC = incentivization.
* p < .05.
Client Baseline Characteristics.
Note. TFC = training, feedback, and consultation; ISF = implementation and sustainment facilitation; MIBI = motivational interviewing-based brief intervention; INC = incentivization; GAD = generalized anxiety disorder.
Strategy Dosage
Table 4 presents information on strategy dosage. In ISF + TFC, the average dose was 12.86 h (SD = 6.10) of training, 0.68 (SD = 1.61) times receiving feedback, 1.68 times (SD = 1.78) receiving consultation, and 3.52 (SD = 2.00) hours of ISF. For ISF + TFC + INC, the average dose was 13.12 h (SD = 5.74) of training, 2.21 (SD = 1.61) times receiving feedback, 2.72 times (SD = 1.78) receiving consultation, and 3.90 (SD = 2.00) hours of ISF. There were no statistically significant differences between conditions.
Strategy Dosage for Staff in the Two Study Conditions for Each Project Cohort.
Note. HSO = HIV service organization; TFC = training, feedback, and consultation; ISF = implementation & sustainment facilitation; INC = incentivization; N/A = not applicable.
Outcomes
Analyses focused on testing the incremental impact of incentivization (ISF + TFC vs. ISF + TFC + INC) on implementation outcomes (Table 5) and client outcomes (Table 6). None of the staff background characteristics were found to significantly moderate the condition assignment to outcome relationship. This was true for all five outcomes. Incentivization had a large effect on all three implementation outcomes but was only statistically significant (p < .05) for MIBI consistency, β = 1.30, 95% confidence interval (CI) = [0.002, 2.6], p = .0497). Beyond being statistically significant this difference is practically and clinically significant, with more than three times as many MIBIs implemented in the ISF + TFC + INC condition (115 MIBIs) than in the ISF + TFC condition (36 MIBIs). For MIBI quality and MIBI implementation effectiveness, the results were β = 1.24, 95% CI = [−0.18, 2.7], (p = .08) and β = 1.28, 95% CI = [−0.08, 2.6], (p = .06), respectively.
Results of Analyses of the Impact of the INC Strategy on Implementation Outcomes.
Note. CI = confidence interval; MIBI = motivational interviewing-based brief intervention; ISF = implementation & sustainment facilitation, TFC = training, feedback, and consultation; INC = incentivization.
*p < .10, **p < .05, ***p < .01.
Results of Analyses of the Impact of the INC Strategy on Client Outcomes.
Note. CI = confidence interval; PSU = past substance use; GAD = generalized anxiety disorder; ISF = implementation and sustainment facilitation; TFC = training, feedback, and consultation; INC = incentivization.
*p < .10, **p < .05, ***p < .01.
In terms of client outcomes, overall reduction in days of primary substance was greater for MIBI clients assigned to ISF + TFC + INC, β = −0.34, 95% CI = [−1.63, 0.95], but was not statistically significant (p = .59). Incentivization had a significant impact on reduction in client anxiety symptoms, β = −1.54, 95% CI = [−2.96, −0.11], p = .04). However, there was a significantly greater overall reduction in level of client anxiety for those whose MIBI staff were female (β = −1.66, 95% CI = [−3.12, −0.21], p = .03) and/or Hispanic (β = −2.63, 95% CI = [−4.47, −0.79], p = .007).
Discussion
Consistent with the intent of type-3 hybrid studies, the secondary aim of our study was to examine the extent to which there were impacts on client outcomes. This is important given improved implementation may not necessarily lead to improved client outcomes. Regarding the impact of the INC strategy on client outcomes, MIBI staff in ISF + TFC + INC had slightly greater overall changes/reductions in days of primary substance use by clients compared to ISF + TFC, but this was not statistically significant as a result of not achieving our targeted sample size and therefore being underpowered to detect a statistically significant difference between conditions. We did, however, find that client participants in the ISF + TFC + INC condition had significantly greater changes/reductions in anxiety symptom severity compared to those in the ISF + TFC condition. This finding is encouraging regarding the potential benefit of SUD intervention on anxiety outcomes. However, it must also be acknowledged that the more significant predictor of this outcome was if the MIBI staff person was female or Hispanic. There is research that has previously suggested a “female effect” (Bhati, 2014) in better mental health outcomes, but we are not aware of research that has found similar effect for interventionists who identify as Hispanic. Future research exploring factors associated with improvement in anxiety may therefore be warranted.
Limitations
The current research was not without limitations. First, while both implementation conditions were affected, conducting the research during COVID-19 was not ideal and may have limited the extent to which the MIBI was able to be implemented. Second, the trial's sample size and associated statistical power was lower than targeted, which as noted above in the methods section was due to the trial being stopped earlier than planned. Third, client outcomes were limited to two measures based on self-report. Fourth, only about two-thirds of follow-up assessments were completed, which was lower than the 70% rate targeted as a minimum follow-up rate, and therefore does adversely impact generalizability of client-level findings.
Conclusion
The consistency and quality of implementation (i.e., implementation effectiveness) is believed to be the greatest when it is expected, supported, and rewarded (Helfrich et al., 2007; Klein et al., 2001; Klein & Sorra, 1996). The current study findings support the effectiveness of a staff-focused incentivization strategy for improving implementation of an evidence-based brief intervention for adults with comorbid HIV and SUD. Thus, to help improve the integration of evidence-based SUD services in HSOs across the United States, use of the ISF + TFC + INC strategy by the ATTC network and/or the AETC network is recommended. Additionally, future research examining the effectiveness of staff-focused incentivization for improving the implementation of other clinical innovations that are both evidence-based and underutilized in real-world practice settings is warranted. This study also underscores the importance of not assuming improved implementation will lead to improved client outcomes. We therefore recommend more type-3 hybrid studies that examine implementation outcomes as the primary outcome and client outcomes as the secondary outcomes. Lastly, we note that although it is possible to provide incentivization to staff and/or organizations for achieving pre-defined improvements in client outcomes, we suggest incentivization is an implementation strategy most appropriately used as a staff-focused strategy and for improving staff-level implementation outcomes they have some level of control over, at least more control than over client outcomes.
Footnotes
Abbreviations
Acknowledgments
Between submitting the original manuscript and revising this manuscript, Dr. Bryan Garner, the principal investigator of the SAT2HIV-II study, passed away. Dr. Garner was a dedicated implementation scientist who was passionate about helping HIV service organizations and SUD treatment organizations to implement evidence-based practices to improve care. He forged mentoring relationships with early career investigators in implementation science and was loved by the addiction health services research community. He is and will continue to be deeply missed. The content is solely the responsibility of the authors and does not necessarily represent the official views of the government. The authors acknowledge that this project would not have been possible without both the participation and collaboration of the project's 26 organizations and their staff, as well as the participation of the organization's client participants. Additionally, this project would not have been possible without the assistance of Alyssa Toro and Liz Ball.
Ethical Considerations and Consent to Participate
The current study was conducted under the auspices of the Advarra Review Board.
Authors’ Contributions
BG, ST, MB, KS, DS, CR, TD, JM, SP, RR, MZ, MR, and JF contributed to conception and design of the work. ST, MB, KS, DS, CR, TD, JM, SP, RR, MZ, MR, and JF contributed to acquisition, analysis or interpretation of data. BG, ST, MB, KS, DS, CR, TD, JM, SP, RR, MZ, MR, and JF contributed to drafted or substantively revised the work.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Institute on Drug Abuse (R01DA052294; PI Garner).
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data Availability Statement
Upon reasonable request, which should be made to the corresponding author, study data, resources, or other materials may be made available.