Abstract
Background
Substance use disorders (SUDs) result in individual and societal burden. However, most individuals with SUD receive no treatment. Implementing SUD interventions in primary care could address this population's treatment needs. In the USA, reSET® and reSET-O® were the first prescription digital therapeutics (PDTs) for SUDs and opioid use disorder (OUD), respectively. The Digital Treatments for Substance Use Disorder (DIGITS) study tested the effectiveness of practice facilitation and health coaching strategies to support reSET and reSET-O implementation into primary care. A formative evaluation was conducted to monitor implementation, inform adaptations, and learn what promotes PDT sustainment.
Method
The Dynamic Sustainability Framework and the Framework for Reporting Adaptations and Modifications to Evidence-based Implementation Strategies guided the evaluation. Using rapid qualitative methods, we collected and analyzed observational fieldnotes, key informant interviews, and document sources (e.g., meeting minutes) for synthesis and dissemination to clinical partners and the study team via formative reports. We analyzed the reports to generate evaluation results.
Results
Twenty-four primary care clinics participated. Evaluation data included 98 observational fieldnotes, 16 interviews, and 253 document sources. We produced nine formative reports. The study encountered barriers and facilitators in each DSF domain (ecological system, practice setting, and intervention). In the ecological system, the PDT vendor enabled the study, but the COVID-19 pandemic, laws, regulations, and contracting delayed implementation. In the practice setting, staff shortages and low clinic capacity were implementation challenges, while electronic health record capabilities were both barriers and facilitators. At the intervention level, non-routine workflows, clinician burden, and low patient engagement were barriers despite clinicians’ efforts.
Conclusions
Digital therapeutics are promising SUD and OUD treatments, but integration into primary care requires conducive laws and regulations, organizational capacity, and patient and clinician engagement. Formative evaluation identified important lessons for future PDT implementation.
Plain Language Summary
Most people with substance use disorders (SUDs) receive no treatment. In the USA, two smartphone app-based digital treatments for SUD and opioid use disorders (OUDs) became available by prescription. The Digital Treatments for Substance Use Disorder (DIGITS) study, a randomized implementation trial, tested how best to integrate these digital treatments for SUD and OUD into primary care. Throughout the study, we conducted a formative evaluation to observe progress, recommend implementation improvements, and understand how digital treatments could be offered to patients after the study ended. For this evaluation, we collected qualitative data through observing study meetings and interviewing primary care leaders, clinicians, and clinical and study staff. The data were regularly summarized and reported to our clinical partners and the study team. We used two implementation frameworks to interpret the data: the Dynamic Sustainability Framework and the Framework for Reporting Adaptations and Modifications to Evidence-based Implementation Strategies. The evaluation revealed factors that helped and hindered implementation. Obstacles from outside the health care system included the COVID-19 pandemic, laws, regulations, and delays in setting up contractual agreements, whereas a partnership with the digital treatment vendor proved helpful. Clinics had difficulties with clinician burden, staff shortages, lack of time and available appointments, and performing new and unfamiliar tasks. The electronic health record system both supported and impended implementation. Last, few patients accepted the offered digital treatment, or used it persistently. Formative evaluation results suggest that digital treatments for SUD and OUD in primary care are promising, however their successful use requires supportive laws and regulations, health system resources, and increased patient and clinician engagement.
Keywords
Background
Substance use disorders (SUDs) result in individual and societal burden, but only 36% of people with past-year opioid use disorder (OUD) receive treatment and as few as 22% receive first-line medication (e.g., buprenorphine) (Degenhardt & Hall, 2012; Grant et al., 2015; Grant et al., 2016; Jones et al., 2023; SAMHSA, 2023). Treatment utilization for other SUDs including cannabis, stimulant, and alcohol use disorders is even lower (Frost et al., 2024; Grant et al., 2015; Grant et al., 2016; Wu et al., 2017). The National Academy of Medicine and field experts call for preventing and treating SUDs in primary care, which most individuals with SUD prefer (Barry et al., 2016).
In 2017, the U.S. FDA authorized two prescription digital therapeutics (PDTs) for SUDs. Pear Therapeutics reSET® for SUD and reSET-O® for OUD are smartphone-based versions of a computerized cognitive-behavioral SUD treatment with demonstrated effectiveness in specialty care (Bickel et al., 2008; Campbell et al., 2014; Christensen et al., 2014; Maricich et al., 2022; Marsch et al., 2014). They have potential in primary care because they target substance use reductions among people with an active SUD. Further, reSET-O supplements buprenorphine treatment for OUD, which can be prescribed in primary care (Barry et al., 2016).
Healthcare organization uptake of digital SUD interventions remains low (Miller-Rosales et al., 2023). Implementation challenges include persuading care teams to offer digital treatments and patients to initiate them, and ensuring patients receive adequate doses through regular use. The Digital Treatments for Substance Use Disorder (DIGITS) study investigated how to optimally integrate digital therapeutics into primary care clinics. We conducted a rigorous formative evaluation of implementation (Elwy et al., 2020; Stetler et al., 2006) throughout DIGITS, using empirical data from stakeholders to monitor and improve implementation; adapt it to maximize fit to the ecosystem, practice setting, and intervention; and learn what promoted sustainment.
Our formative evaluation can guide pragmatic, multimethod evaluations for timely, trustworthy improvement feedback to researchers and clinical partners throughout implementation. We describe evaluation goals, data sources, analytic methods, evaluation activities, and implementation barriers and facilitators. Methods for sharing evaluation findings with clinical partners and perceived impact on implementation illustrate how centering evaluations on “real-world” concerns informs decision-making in dynamic settings.
Method
Setting
Kaiser Permanente Washington (KPWA) integrates healthcare and coverage for approximately 700,000 members. The more than 30 primary care clinics in Washington State use an integrated mental health (IMH) model, with conditions such as depression and unhealthy substance use treated by primary care clinicians including licensed independent clinical social workers (LICSWs). (“Clinician” indicates qualified clinical practitioners who provide patients’ principal care.) (CMS, 2023; Tumulty, 1970).
Research Team and Reflexivity
The lead evaluator was a doctoral-level sociologist and qualitative methodologist with federally funded implementation science training (Palazzo). The bachelor-level evaluator (Dorsey) had experience in qualitative research and contributed to the implementation science literature. Both performed data collection and qualitative analysis and were full-time employees at KPWA Health Research Institute affiliated with KPWA. The study team included clinician scientists (social work, general internal medicine), implementation practitioners (public health), a health economist, project manager, and biostatisticians. Co-investigator clinical partners were leaders in KPWA Mental Health and Wellness (addiction psychiatry, social work). A Steering Committee comprised leaders in addiction, implementation science, and digital therapeutics.
DIGITS Study and Intervention
DIGITS tested the effectiveness of strategies for implementing reSET and reSET-O in primary care to achieve optimal reach, fidelity, and cost effectiveness. Researchers and KPWA leaders partnered to define clinically meaningful objectives and conduct the study.
The digital therapeutics were regulated by the FDA. FDA prescription labeling criteria dictated which patients could be prescribed the therapeutics (Watson et al., 2023). reSET-O was indicated for patients with OUD who were prescribed buprenorphine and reSET for patients with other SUDs (e.g., cannabis) (Maricich et al., 2021, 2022). Patients with a primary OUD not treated with buprenorphine or using alcohol with no other substance were not eligible.
PDTs were treatment adjuncts for patients under clinician care; patients could not download PDTs or use them without clinician assistance. KPWA clinicians trained in reSET and reSET-O, primarily LICSWs, assessed patient eligibility and offered the PDTs in primary care. If patients accepted, clinicians used the vendor's platform to create a new account, sending the patient an access code. Clinicians used the electronic health record (EHR) to prescribe the app by entering a standing order that a physician would co-sign.
reSET and reSET-O were chosen because their effectiveness data and commercial maturity promised sustainability if the health system institutionalized them. The vendor's help desk provided patient and clinician technical support, and a population-management dashboard allowed clinicians to track patients’ app initiation and use. The patient-facing app incorporated three evidence-based approaches—contingency management (Lussier et al., 2006), community reinforcement (Lussier et al., 2006; Marsch et al., 2014) (a cognitive-behavioral therapy, delivered in text- and video-based lessons) (Magill & Ray, 2009) and fluency training (Binder, 1996)—to support substance use abstinence or reduction. reSET and reSET-O prescriptions lasted approximately 12 weeks.
Implementation
Three implementation strategies promoted PDT adoption: (1) standard strategy, (2) practice facilitation, and (3) health coaching (Glass et al., 2023; Proctor et al., 2013).
Standard was a package of supports previously employed by KPWA to implement non-prescription digital therapeutics, including live or video-based clinician training from clinical leaders and the PDT vendor, and an implementation toolkit with clinician- and patient-facing materials, such as a clinician job aid about offering the PDTs, patient pamphlets, EHR tools and documentation templates, quality-monitoring performance reports, and marketing materials (e.g., huddle cards). A trained external practice facilitator assisted clinicians, provided audit and feedback, supported plan-do-study-act cycles, and engaged clinic leaders (Ritchie et al., 2017). Health coaching was led by a centralized medical assistant who contacted patients to support engagement, PDT use, and communication with healthcare teams.
DIGITS Phases
The four phases (Figure 1) were (1) quality improvement (QI) pilot (3 months), (2) trial pilot (3 months), (3) active implementation (up to 1 year per clinic), and (4) sustainment (initially planned for up to 1 year per clinic). In the QI pilot, we worked with two clinicians across two clinics to adapt, apply, and monitor use of the standard strategy. In the trial pilot, we observed application of practice facilitation and health coaching alone or in combination in the same 2 clinics (Mogk et al., 2024). In active implementation, 21 participating primary care clinics were cluster-randomized to (1) standard implementation alone, (2) standard + practice facilitation, (3) standard + health coaching, or (4) standard + facilitation and coaching. Sustainment assessed continued intervention reach and viability after reducing study-funded implementation supports.
DIGITS Study Phases and Corresponding Key Activities.
All KPWA primary care clinics not participating in our two-clinic pilot study were initially eligible for the trial. Due to COVID pandemic staffing vacancies, before trial start, we required clinics to have at least one clinician trained to offer the apps.
Formative Evaluation
Qualitative methods used a post-positivist approach (Guba & Lincoln, 1994) to assess implementation strategies, feasibility, real-time implementation, contextual factors and participant responses, and optimize adaptations. Goals included: (1) monitoring and documenting PDT implementation; (2) understanding barriers and facilitators to intervention fidelity, adoption, appropriateness, acceptability, and sustainment; and (3) identifying modifications and adaptations to strategies and/or the intervention. All pilot and trial clinics and study phases were evaluated. Clinical partner needs drove evaluation activities. Mainly evaluators, overseen by Glass, collected and analyzed data.
Dissemination of Findings
Every 1–3 months, evaluators presented to and discussed with clinical partners and the project team as a data-collection point and member-checking opportunity (Birt et al., 2016). Evaluators generated formative reports using resulting data and feedback. Approximately twice annually, the Steering Committee heard selected findings and gave feedback. Reports contained (1) standard sections with key implementation findings (e.g., successes, barriers, recommendations); and (2) special sections on emerging issues (e.g., adaptation outcomes), non-routine events (e.g., supplementary clinician training), or unexpected organizational-level change impact (e.g., departmental restructuring). Flexibility kept the evaluation relevant to a complex health system study.
Guiding Frameworks
The Dynamic Sustainability Framework (DSF) and the Framework for Reporting Adaptations and Modifications to Evidence-based Implementation Strategies (FRAME-IS) guided formative evaluation (Chambers et al., 2013; Miller et al., 2021; Wiltsey Stirman et al., 2019).
DSF comprises three domains—ecological system (practice setting environment), practice setting (intervention context), and intervention (PDTs, associated implementation strategies, clinical interventions)—that change over time.
Successful, sustainable implementation “maximizes fit” of these domains. The DSF is appropriate for PDTs, which are novel for healthcare. Adaptive strategies, including external facilitation were chosen based on the DSF. Before starting, evaluators mapped implementation components to DSF domains and constructs (Table 1). This allowed characterizing overall barriers and facilitators as arising in ecological system, practice setting, or intervention and helped identify constructs and domains targeted by adaptations.
DIGITS Implementation Constructs Mapped to the Dynamic Sustainability Framework.
Note. CRA = community reinforcement approach; CBT = cognitive behavioral therapy; IMH = integrated mental health; LICSW = licensed independent clinical social worker; PDT = prescription digital therapeutic; KPWA = Kaiser Permanente Washington; KP = Kaiser Permanente; EHR = electronic health record; FDA = U.S. Food and Drug Administration; HIPAA = Health Insurance Portability and Accountability Act; IT = information technology.
The FRAME-IS systematically documents modifications and adaptations (respectively, unplanned and planned changes) of evidence-based implementation strategies to improve implementation effectiveness. FRAME-IS captures and organizes content, rationale, decision-making level, and goals of tailoring implementation to fit conditions.
Data Sources
Observational Fieldnotes
Fieldnotes were collected at implementation meetings (internal practice facilitation and health coaching meetings, clinical staff check-ins, clinical partner meetings, clinical education calls), and meetings with the research team, KPWA partners, and primary care clinics (Table 2). Evaluators used a structured fieldnotes template informed by FRAME-IS and recommendations (Proctor et al., 2013) to specify and report implementation strategies by actor, action, action target, temporality, dose, outcomes affected, and justification. Unstructured observations and insights were captured as free text. Evaluators documented strategies, barriers, facilitators, modifications and adaptations, and clinician stories, including about patients. They collected data from virtual practice facilitation meetings with local clinic teams. Throughout the trial, evaluators attended practice facilitation meetings on a rotating basis (five clinics observed at any point), attending at least two per clinic. Meeting participants were aware of evaluators’ roles and verbally consented to activities.
Data Sources Used in the DIGITS Formative Evaluation.
Note. FRAME-IS = Framework for Reporting Adaptations and Modifications to Evidence-based Implementation Strategies; DSF = Dynamic Sustainability Framework; LICSW = licensed independent clinical social worker; QI = quality improvement; IMH = integrated mental health; EHR = electronic health record; SUDs = substance use disorders; KPWA = Kaiser Permanente Washington; PC = primary care; MAs = medical assistants.
Evaluators did not directly observe clinics that implemented standard implementation alone or standard + health coaching but not practice facilitation. To minimize clinic burden under COVID, evaluators attended clinic meetings only if scheduled as part of the trial and led by the practice facilitator (with or without the health coach). Our evaluation plan did not include observing patient encounters with clinicians or the health coach when reSET or reSET-O might be discussed. Data on clinics with standard implementation alone were obtained from trainings, departmental meetings, and from clinical partners. Data on clinics with standard + health coaching were extracted from health coaching implementation meetings.
Key Informant Interviews
Interviews with KPWA employees occurred between trial pilot end and two months into active implementation to explore successes and challenges of app implementation and understand how to adapt implementation strategies for the future. Participants were purposively sampled, including clinical leaders across the healthcare system and clinicians within the two pilot clinics. Interviews were by JM and TEM, overseen by JEG, informed by the DSF and FRAME-IS and asked about interviewees’ understanding of PDT implementation, what worked or was challenging. Full data collected and interview guides are published (Mogk et al., 2023) (Additional files in the supplemental material).
Document Sources
Evaluators reviewed implementation materials (e.g., training videos), implementation and other meeting minutes, emails, and online messages, using the observational fieldnotes template to extract data.
Analyses and Reports
Data sources were analyzed for actionable findings for clinical partners and the research team (Figure 2) and summarized in monthly formative reports using qualitative content analysis (Hsieh & Shannon, 2005). Approximately six months into active implementation, evaluators began reviewing reports, developing a coding template for barriers, facilitators, modifications, adaptations, and implementation strategies, and DSF domains and constructs. Evaluators used the template and Atlas.ti 9 (ATLAS.ti, 2023) to code reports, reviewing 10% of coding for quality assurance, and scrutinizing coded text for emerging patterns presented in the final assessment.
DIGITS Formative Evaluation Analytic Procedures by Data Source.
Rigor and Trustworthiness
Evaluators triangulated data sources and researcher and care delivery system partners’ perspectives, routinely discussing and cross-checking data and findings. During regular evaluation reports, evaluators elicited and incorporated feedback about results validity and accuracy from researchers, delivery partners, practice facilitators, and the health coach (member checking) (Birt et al., 2016), adhering to the Standards for Reporting Qualitative Research (Additional files in the supplemental material) (O’Brien et al., 2014).
Ethics Approval
The KPWA Institutional Review Board reviewed all study activities and granted a waiver of consent and HIPAA authorization to collect individual-level automated electronic medical record data necessary to randomize clinics and to collect information for the analysis. Primary data collection for the formative evaluation was exempt from IRB review.
Results
Implementation Experience
Of 24 participating primary care clinics, two started the QI pilot, each with one trained LICSW. Both entered the trial pilot, but one ended early when the LICSW left. The active implementation phase randomized 21 sites. Across all phases, 38 clinicians (mostly LICSWs) were trained to offer the PDTs. From QI pilot launch to study end, 89 prescriptions were created (68 for reSET), with 46 activated by patients (41 for reSET, Table 3). In April 2023, shortly after all clinics entered the sustainment phase, the vendor filed for bankruptcy, so the sustainment phase ended early.
reSET and reSET-O activity Activity Across Piloting and the DIGITS Implementation Trial (March 2, 2021, through March 31, 2023). a
Note. LICSW = licensed independent clinical social worker; PA = physician assistant; MD = medical doctor = RN = registered nurse; PDT = prescription digital therapeutic.
Date of the first and last event related to reSET and reSET-O prescription and activation.
Of the clinicians trained to prescribe the PDTs, one PA and one MD were buprenorphine prescribers.
22 clinics were randomized in 21 randomization units; two clinics were paired because of their geographical proximity and sharing of the LICSW.
Formative Evaluation
Between February 2021 (QI pilot) and March 2023 (early sustainment), we collected observational fieldnotes and meeting minutes. Data were from research team, implementation, and practice facilitation meetings, and events including trainings. Evaluators also analyzed training videos and presentations, and reviewed job aids, huddle cards, and secure messaging scripts. Evaluators directly observed and documented implementation at 13 clinics, indirectly observing remaining clinics via meetings with clinical staff and leaders (Table 2).
Key informant interviews were with eight care team members (LICSWs, physicians, and medical assistants), and eight others (implementation team members, clinical leaders). Interviewees described pros and cons of using reSET and reSET-O and other digital therapeutics in clinical practice, implementation barriers, and mitigation strategies (Table 4).
Key Informant Interview Themes and Representative Quotes Relevant to the DIGITS Study.
Note. MA = medical assistant; LICSW = licensed independent clinical social worker; PC = primary care.aRefers to suboxone, buprenorphine, and naloxone.
Evaluators generated nine formative reports across all phases.
Barriers and facilitators
Implementation barriers and facilitators pertained to all DSF domains. They limited, or to a lesser extent supported, the implementation strategies’ effectiveness, impacted intervention reach and other outcomes. Barriers in ecological system included COVID-19 pandemic disruptions and population needs; in practice setting, technical problems and adjustments, security and compliance protections, and clinical staffing and communication gaps; in intervention domain, increasing clinician burden, patient eligibility requirements, and challenging intervention features. Implementation facilitators were, in ecological system, PDT vendor cooperation; in practice setting, the EHR system, familiarity with digital tools, and frontline managers’ engagement; in intervention, clinician and leadership champions, responsive research teams, and beneficial intervention aspects (Table 5).
Barriers and Facilitators to Implementation by Dynamic Sustainability Framework (DSF) Domain and Data Source.
Note. IMH = integrated mental health; LICSW = licensed independent clinical social worker; PDT = prescription digital therapeutic; KPWA = Kaiser Permanente Washington; KP = Kaiser Permanente; EHR = electronic health record; SUD = substance use disorder; AUD = alcohol use disorder; CBT = cognitive behavioral therapy.
Refers to suboxone, buprenorphine, and naloxone.
Ecological System
Barriers
COVID-19 pandemic disruptions
The pandemic impact delayed the study launch. Pandemic mitigation resulted in clinic closures, staffing losses, and shifts in resource allocations (e.g., to business continuity). These factors reduced the health system's capacity to absorb new digital therapeutics. Implementation was also delayed by state and federal laws that health system administrators interpreted as potentially limiting the small patient financial incentives embedded in reSET and reSET-O as contingency management, a common SUD behavioral motivator. Contracting processes between the health system and the PDT vendor also delayed implementation: “We planned on doing this last year, and the contracting piece sort of dragged on and on” (Table 5, 1.a.1, table has full quotes and attributions). Health system leaders anecdotally reported that delays reduced implementation momentum.
Population Characteristics and Needs
According to LICSWs, despite SUDs’ general high prevalence in primary care, few referred patients had SUD, especially in smaller clinics. Some patients needed more specialized or intensive care, limiting LICSWs’ interactions with patients with SUD. Some eligible patients were deemed by clinicians to have competing challenges that precluded prescribing the PDT, including unstable housing, mental health crises, or serious medical conditions: [The LICSW said] “there was a patient from [medical center] …he is homeless, needed a crisis plan. He is not doing well mentally or physically” (Table 5, 2.c.2).
Facilitators
PDT Vendor's Cooperation
Regular communication and collaboration with the PDT vendor addressed issues as they arose (e.g., with technical support for patient account creation), aiding the practice facilitators and health coach. The vendor allowed the research team to tailor standard PDT training to accommodate clinicians’ time limitations: “Lots of aspects of the product have been customized to support the KPWA team” (Table 5, 3.b). PDT vendor-delivery system collaborations overcame compliance and regulatory concerns.
Practice Setting
Barriers
Information technology capabilities and clinicians’ role
Information technology capabilities (largely EHR-related) and LICSWs’ role emerged as intersecting barriers. LICSWs were tasked with offering the PDTs, but were not accustomed to creating prescriptions, including in the EHR, which is outside their scope of practice per legal statutes. The research team and mental health leadership facilitated regional changes to institute a standing order policy allowing LICSWs to immediately provision PDTs based on standard eligibility criteria, later obtaining a physician prescriber's approval and signature. The EHR allowed standing order implementation but required that clinicians execute new steps with multiple clicks to complete prescriptions. The novelty of this procedure was emphasized in LICSW training: “an authorized provider needs to co-sign entering an order for reSET. This is a new piece” (Table 5, 4.b.1).
Another EHR-related barrier was lack of integration with vendor information systems so clinicians could set up patient accounts and monitor app use within the EHR versus the vendor's web-based dashboard. Health system information technology leaders would not support two-way communication between the EHR and the vendor systems without experience and a long-term commitment to the products. Thus, during active implementation, workflows were not fully optimized to clinician needs and product capabilities, affecting intervention reach: “EHR integration is critical—we really need to shorten time between offering and the patient being able to use the app” (Table 5, 4. c.1).
Security and compliance protections
Documenting and communicating security and compliance protections about using software-based products to transmit and store protected health information outside the health system's firewall required significant effort. The need to protect patient privacy and the health system's brand and adhere to regulations about use and disclosure of patient data led to rigorous, lengthy, technical risk evaluations. “How do you balance the care provided to patients with the risk?” (Table 5, 5.c.). Risk assessment involved the regional and national health system, research team, and PDT vendor before business leader approval.
Clinical staffing and information gaps
Clinician losses and staffing gaps impeded reach. Several clinics lacking LICSWs who could be trained in the PDTs were excluded from implementation, and nine trained clinicians left during the study: “[two clinics] had social workers that had left their positions and so the social workers that were at [other clinics] ended up having to do a lot of coverage” (Table 5, 6.a.1).
New staffing vacancies coincided with reports of high clinician burden and undermined adoption as LICSWs had difficulties integrating new duties into their schedules. Some perceived their limited capacity was not considered when launching the PDTs: “There isn’t a whole lot of consent requested of the team at the clinic, like “hey, we’re thinking of starting this—do you have capacity for it?” It's more like “hey, we’re starting this—here's the meeting for the orientation.” (Table 4, 4.c.1, table has full quotes and attributions). Primary care physicians, physicians’ assistants, and registered nurses could refer patients to LICSWs to discuss the PDTs. However, many were not aware of the PDTs. In one example, at a staff meeting about the apps, a physician stated, “I have no idea what this is about” (Table 5, 6.b.1).
Others were unaccustomed to referring patients with SUD and OUD to LICSWs for treatment. These barriers affected PDT adoption, while reach was further hampered by few physicians prescribing buprenorphine for OUD, an eligibility requirement for reSET-O. KPWA organizational restructuring during the study changed LICSW staff oversight, enhancing change fatigue and affecting PDT prioritization by LICSWs.
Facilitators
EHR System
The existing EHR supported building dedicated tools to automate clinical processes including ordering, documentation, patient instructions, and population management: “Whether it's working with clinical guides or working with smart phrases but having that information available at a site where I would typically go, that's very helpful” (Table 4, 3.c.1).
Long-standing relationships between research informaticists and delivery system information technology staff were largely responsible for these achievements. Equally critical was working with the PDT vendor and delivery system on compliance and regulatory issues.
Familiarity with Digital Tools
DIGITS was not the only digital initiative at KPWA, with pandemic-motivated shifts to telehealth. Non-prescription depression and anxiety wellness apps were already available at KPWA. Clinicians reported that simultaneously obtaining access to multiple new digital programs, each with its own purpose and requirements, was sometimes confusing to clinicians and patients (Table 4, 2.c.1). However, it aided PDT implementation by providing process blueprints and familiarity with digital care delivery: “We use apps, frankly, for a lot of other diagnoses here as well—anxiety/depression, which oftentimes go along with substance use disorders…it's a very nice complement” (Table 5, 8.a).
Frontline Managers’ Engagement
IMH department restructuring that affected LICSWs’ perceived work environment introduced a new administrative layer, but also indirectly supported implementation. LICSW managers expressed interest in DIGITS and their staff's involvement, opening new lines of communication with the research team including a one-page study overview, reSET test accounts, and targeted email updates.
Intervention Domain
Barriers
PDTs’ demands on clinicians
Specific requirements for the PDTs, including the detailed patient-eligibility assessment and required prescription, were adoption obstacles. Clinicians often lacked time during appointments to determine eligibility and readiness for the PDT, availability for recommended monthly follow-ups, or time to monitor app use (Table 5, 10.a.1).
Prescribing the PDT and tracking module completion using the vendor's dashboard fell outside routine workflows, in part because of lack of integration into the EHR. These features decreased intervention feasibility.
Patient eligibility requirements
PDT FDA authorization did not include patients with only alcohol use disorder and required patients with OUD to have active buprenorphine prescriptions. Clinician training provided the rationale, but clinicians reported frustration and confusion about these narrow eligibility criteria, limiting the apps’ utility in everyday practice (Table 5, 11.b.2).
Challenging intervention features
Clinicians reported low, inconsistent patient engagement with SUD treatment generally and the PDTs, specifically. LICSWs and the health coach frequently could not contact patients who were or could be offered the PDTs. As with many apps, patients often accepted a prescription without activating it or completing learning modules.
Clinicians reported that some patients had technical difficulty installing the app, absorbing the instructional content, or fitting module completion into their schedules. Thus, some clinicians stopped offering the PDTs (Table 5, 12.c.2).
Facilitators
Clinician and Leadership Champions
Some LICSWs committed to supporting PDT implementation, for example by spreading information about the apps in huddles and meetings and primary care clinicians and clinic leaders championed the intervention. A strong evidence base bolstered clinicians’ belief about the PDT's benefits and potential to help some patients, motivating their endorsement (Table 5, 13.a.1). Support was not uniform, however. Competing priorities prevented consistent engagement from clinicians and leaders.
Responsive Research Team
Practice facilitators, the health coach, and other research team members answered challenges to intervention delivery with flexibility and responsiveness to local needs, for example, temporarily pausing practice facilitation in understaffed clinics, adapting the facilitation protocol and shortening clinic-based meetings to lessen clinician burden and improve implementation acceptability. A programmer generated reports to help clinicians identify eligible patients in practice facilitation clinics, and the health coach shared successful patient engagement strategies for clinicians (Table 5, 14.b.1). The research team presented at a continuing medical education event and offered PDT training to all clinicians to reduce reliance on LICSWs (manuscript describing adaptations with quantitative implementation results in preparation).
Beneficial Intervention Aspects
According to LICSWs and the health coach, patients who activated accounts and regularly engaged with reSET enjoyed it and were satisfied with the experience. They liked the content, embedded incentives, and lessons such as on building life skills to apply beyond substance use. Some clinicians liked that the PDTs offered “one more tool in the toolbox” that was non-stigmatizing and quickly accessible by patients (Table 5, 15. b.1).
Successes
Successes were mostly with clinicians and patients who had positive experiences with DIGITS and the PDTs. Reflecting on the experience of participation, one LICSW reported increased SUD awareness and confidence in discussing substance use with patients. Practice facilitation clinic clinicians expressed widespread appreciation for practice facilitators, their tools, and resources (e.g., reports, illustrative materials, PDT “elevator speech” for colleagues). Supporting materials formed a toolkit for collaboration and coordination of implementation activities (Additional files in the supplemental material): “I felt like the rollout of reSET was far superior to other initiatives in primary care” (Table 5, 14.a).
Fidelity in strategy delivery by the health coach within an adapted, expanded role was widely praised for helping patients connect with IMH care and engage with PDTs. These efforts identified effective approaches for sustained patient engagement, including stressing the low time commitment for instructional modules, reminding patients about built-in app incentives, and helping them master difficult or intimidating content.
Discussion
DIGITS tested strategies to optimally implement digital therapeutics for SUD and OUD in primary care. Our formative evaluation was critical to improving intervention fit with local contexts and identifying lessons from the implementation experience. It addresses the lack of knowledge on practical considerations to implementing evidence-based digital therapeutics for SUD and OUD that require prescription and clinician oversight. Our multimethod evaluation revealed real-time barriers and prompted rapid adaptations in partnership with the delivery system. We identified project and intervention facilitators to assist others in implementing PDTs. Conceptually grounding the evaluation in the DSF enabled identification of levels where implementation determinants emerged—ecological system, practice setting, intervention—and necessary adjustments.
COVID-19 and its aftermath in the ecological system created opportunities and benefits to digital care by improving access, but disrupted health systems. The impact reverberated across levels, leading to low capacity and staffing challenges in the practice-setting level and widespread clinician burden. These conditions were unfavorable to introducing a first-ever substance use PDT into routine care, despite the need for more and better treatment options. The impact of staffing challenges on implementation of evidence-based SUD interventions aligns with pre-COVID studies (Hagedorn et al., 2014; Rubinsky et al., 2018; Woltmann & Whitley, 2007). In the ecological system, common needs among people with SUD, such as mental health care and housing (Austin et al., 2021; Jones & McCance-Katz, 2019) were competing demands to offering PDTs for clinicians. Legal constraints and compliance issues at the ecological and practice levels delayed implementation. These are important factors for health systems to consider when planning for digital therapeutic adoption. At the intervention level, cumbersome prescribing and PDT use undermined clinicians’ and patients’ willingness to engage. PDT eligibility criteria precluded use for some with common treatment needs, contributing to poor uptake.
Skillful practice facilitation and health coaching augmented standard implementation strategies, assisting clinicians with workflow concerns. However, many multilevel barriers were outside strategy scope such as low clinic capacity and staffing losses, or workflow hurdles from the lack of EHR-PDT communication. Health coaching to assist patients and minimize clinician burden was hampered by inability to reach patients. The intervention and implementation strategies could not mitigate health or psychosocial conditions beyond SUDs.
A DIGITS goal was gaining insights for future PDT implementation. Our formative evaluation generated lessons and recommendations to improve implementing digital therapeutics in clinical care. First, PDTs versus non-prescription health and wellness apps promise unique benefits but with unique legal and regulatory challenges. This increases administrative tasks for implementors, healthcare administrators, and organizational leaders. The PDT prescription process clashed with clinicians’ scope of practice, leading to implementation delays as the research team and clinical leaders spurred institution of new health system policies. Research and implementation teams should know of these issues before selecting PDTs to implement, and thoroughly investigate organizational rules and federal and state laws pertaining to novel intervention features (e.g., monetary incentives). Maintaining connections and good working relationships with clinical leaders and organizational compliance and security officials was critically important to addressing obstacles.
Another lesson is that EHR limitations compound workflow issues. Limited vendor-EHR integration made it burdensome for clinicians to perform reSET and reSET-O implementation tasks, especially when outside their usual role, as supported by prior research (Quanbeck et al., 2018). Health system's information technologists refused to integrate the PDT into the EHR without evidence that the intervention would be sustained, creating a “chicken-and-egg” workflow limitation that reduced sustainment likelihood. We iteratively co-designed new workflows with the implementers, considering their needs and preferences, and identifying barriers promptly. Pilot clinicians acknowledged that EHR integration would be beneficial but did not see lack of integration as a major limitation, although our evaluation found it a significant implementation impediment. We recommend involving a broader group of clinicians in piloting, especially those less receptive to technology, to identify constraints before implementation. Benefits of co-design with patients, clinicians, and end-users are documented (Slattery et al., 2020).
Marketing digital therapeutics interventions to clinicians and leaders is essential. We prepared dissemination materials and invited LICSWs to share patient success stories with peers (Parchman et al., 2022) and clinical partners held a medical education session about reSET and reSET-O. This worked in some cases (e.g., with clinicians using our huddle cards). However, a more comprehensive, coordinated communication strategy led by health system leadership may be necessary with novel interventions with multilevel adoption barriers (Aarons et al., 2014).
Relationship building was critical to overcoming implementation obstacles and maintaining partnerships. We forged relationships with informational technologists and delivery system and clinical leaders before DIGITS, maintaining and expanding them to foster frontline managers’ engagement. During unavoidable organizational change, involvement of clinical implementation partners from study design to interpreting results will support long-standing and new connections. These efforts are an important investment to ensuring progress and results under difficult circumstances. Ongoing engagement, maintaining relationships, and fostering trust are crucial for implementation (Horton et al., 2018; Rose & Schlichter, 2013).
Sustainment of PDTs ended when the vendor declared bankruptcy and discontinued the apps. Nonetheless, digital interventions continue their promise to address health behaviors and improve patient access and treatment for SUDs and other diseases (Abernethy et al., 2022), although widespread adoption and reach may require contingency planning for sustainability and continuity as digital healthcare grows and evolves. Lessons from DIGITS provide implementation guidance for SUD digital therapeutics and emerging best practices for integrating new interventions into healthcare.
Strengths of our formative evaluation included rigorous, flexible, data collection and analysis to adjust to project and delivery system needs. Multiple data sources and regular member checking validated findings. Limitations included absence of direct patient contact. Patient PDT experiences were from clinicians and the health coach. We did not collect standardized measures of implementation constructs from clinicians (Clinton-McHarg et al., 2016; Fernandez et al., 2018). The COVID-19 pandemic limited our observations; therefore, we did not directly observe clinics randomized to standard implementation alone and had less information on clinics with health coaching without practice facilitation. We learned about experiences at these clinics through interactions with clinicians during training and department meetings. Vendor bankruptcy precluded identifying facilitators and barriers to sustainment.
Conclusion
The DIGITS formative evaluation offers a unique, close-up look into implementing PDTs for SUD and OUD in primary care. Embedding the evaluation within clinics provided rich context for the challenges of making digital therapeutics a routine care component. It surfaced implementation strengths and lessons to guide researchers and healthcare partners in realizing benefits of digital therapeutics for escalating public health problems.
Supplemental Material
sj-docx-1-irp-10.1177_26334895241301670 - Supplemental material for Formative evaluation of the implementation of digital therapeutics for opioids and other substance use disorders in primary care (DIGITS trial)
Supplemental material, sj-docx-1-irp-10.1177_26334895241301670 for Formative evaluation of the implementation of digital therapeutics for opioids and other substance use disorders in primary care (DIGITS trial) by Lorella Palazzo, Caitlin N. Dorsey, Jess Mogk, Tara Beatty, Deborah King, Kelsey Stefanik-Guizlo, Dustin Key, Tessa E. Matson, Mary Shea, Ryan M. Caldeiro, Angela Garza McWethy, Edwin S. Wong, Abisola E. Idu and Joseph E. Glass in Implementation Research and Practice
Footnotes
Availability of Data and Materials
Contributions
Palazzo designed and planned the formative evaluation and engaged in all evaluation activities. Dorsey collected and analyzed formative evaluation data and drafted reports. Glass conceptualized and led the study, obtained funding, and approved evaluation activities. All members of the DIGITS study team (Glass,Palazzo, Dorsey, Mogk, Beatty, King, Stefanik-Guizlo, Key, Matson, Shea, Wong, Idu) and clinical partners (Caldeiro, Garza McWethy) enabled data collection and interpretation. Palazzo and Dorsey completed the first manuscript draft. All authors reviewed and approved the final manuscript.
Consent for Publication
Pear Therapeutics was offered the opportunity to review a copy of this manuscript before publication solely to check for disclosures of proprietary content.
Declaration of Conflicting Interests
reSET® and reSET-O® are digital therapeutics for substance use disorder marketed by Pear Therapeutics (US), Inc. that are discussed in this manuscript. During the three-month quality improvement pilot study discussed in this manuscript, Pear Therapeutics (US), Inc. provided digital therapeutic prescriptions at no cost to Kaiser Permanente Washington. Pear Therapeutics (US), Inc. has not provided funding to the authors.
Ethical Approval and Consent to Participate
The Kaiser Permanente Washington Human Subjects Review Office (FWA00002344) determined that the formative evaluation activities are exempt from Institutional Review Board review according to federal regulations, per Category 4 and the clinician interviews are exempt per Category 2. This exempt research was conducted in accordance with the principles of the Belmont Report. All subjects participated voluntarily. Verbal informed consent was obtained from subjects who completed interviews. Staff who attended meetings used for formative evaluation were notified that notes and observations would be used for research purposes. Interviewees received a small compensation. Meeting attendees participated during work hours and did not receive additional compensation.
Funding
Research reported in this publication was supported by the National Institute on Drug Abuse of the National Institutes of Health under Award Number R01DA047954 and the National Institute on Alcohol Abuse and Alcoholism of the National Institutes of Health under Award Number K01AA023859. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
