Predicting non-initiation of care and dropout in a blended care CBT intervention: Impact of early digital engagement,sociodemographic,and clinical factors

Abstract

Objective

This study examines predictors of non-initiation of care and dropout in a blended care CBT intervention, with a focus on early digital engagement and sociodemographic and clinical factors.

Methods

This retrospective cohort analysis included 3566 US-based individuals who presented with clinical levels of anxiety and depression and enrolled in a blended-care CBT (BC-CBT) program. The treatment program consisted of face-to-face therapy sessions via videoconference and provider-assigned digital activities that were personalized to the client's presentation. Multinomial logistic regression and Cox proportional hazard survival analysis were used to identify predictors of an increased likelihood of non-initiation of therapy and dropout.

Results

Individuals were more likely to cancel and/or no-show to their first therapy session if they were female, did not disclose their ethnicity, reported poor financial status, did not have a college degree, endorsed more presenting issues during the onboarding triage assessment, reported taking antidepressants, and had a longer wait time to their first appointment. Of those who started care, clients were significantly more likely to drop out if they did not complete the digital activities assigned by their provider early in treatment, were female, reported more severe depressive symptoms at baseline, reported taking antidepressants, and did not disclose their ethnicity.

Conclusions

Various sociodemographic and clinical predictors emerged for both non-initiation of care and for dropout, suggesting that clients with these characteristics may benefit from additional attention and support (especially those with poor early digital engagement). Future research areas include targeted mitigation efforts to improve initiation rates and curb dropout.

Keywords

Blended care CBT digital dropout engagement

Introduction

Cognitive-behavioral therapy (CBT) is an efficacious treatment for anxiety and depressive disorders, with robust empirical support from numerous randomized controlled trials.^1,2 However, there are multiple barriers to accessing CBT that prevent individuals from receiving quality, evidence-based care. Specifically, issues related to cost, geographic constraints, limited availability of providers, transportation issues, and stigma are just a few of the many challenges that confront treatment-seeking individuals.^3–5 Internet-based CBT (iCBT) and telehealth solutions have emerged as promising methods to address some of these gaps, allowing for broader dissemination of quality interventions in a cost-effective manner.^6–8

Telehealth solutions often entail delivering CBT through a video-based platform, allowing sessions to occur face-to-face in real time with a therapist. Video-based teletherapy has demonstrated comparable clinical outcomes to in-person treatment, especially when targeting anxiety and depression.⁹ Similarly, iCBT also allows remote access to evidence-based care, though the treatment materials can be delivered asynchronously through various types of media (e.g., written content, videos, gamified/interactive content, audio).¹⁰ Studies examining the efficacy of iCBT programs have demonstrated significant decreases in depression and anxiety,^11,12 but the effects tend to be stronger when there is support from a therapist.¹³ Specifically, smaller treatment effects and increased attrition are observed in unguided versus therapist guided iCBT programs,¹⁴ and are magnified in real-world settings.¹⁵ Despite these technological advances in reducing barriers to accessing care, iCBT programs notably struggle with keeping users engaged, leading to limited uptake and dropout rates of up to 80%.¹⁶

To address these challenges, blended care models of CBT (BC-CBT) combine the advantages of teletherapy with a therapist and the support of digital therapy materials.¹⁷ Holding face-to-face sessions with a therapist (via videoconference or in person) confers the benefits of problem-solving emergent concerns, enhancing the precision of care, and increasing social accountability. Digital therapy materials in BC-CBT typically involve digital content (e.g., videos, guides, practices) that teach core CBT concepts and skills, allowing for asynchronous access and reinforcement of key therapeutic content. However, limited research exists on dropout in BC-CBT.

Dropout from therapy (more broadly) can be due to numerous reasons, including dissatisfaction with the treatment model, weak therapeutic alliance, low motivation, circumstantial barriers (e.g., transportation challenges, life events), or even earlier-than-expected improvement.^18–20 Internet-based interventions tend to demonstrate particularly high dropout rates,¹⁸ perhaps in part because they are inflexible, limited in scope, and lack the personalization, guidance, and accountability a therapist provides. Consequently, intervention engagement is largely up to the client's discretion, increasing the likelihood that clients will discontinue and drop out. Indeed, the “law of attrition” stipulates that dropout is a normal and common occurrence in internet-based interventions, and highlights the importance of investigations focused on dropout and accounting for it in evaluations of treatment efficacy.²¹

Various sociodemographic and clinical factors, such as male gender, lower educational levels, and higher baseline depression have been linked to higher dropout rates in CBT.^18,22–24 The timing of care can also be impactful, as some studies have suggested that the greatest risk of dropout occurs before the first session, while those who have been in therapy longer have decreased risks of dropping out.^18,25 However, findings have been mixed due to the heterogeneity of these interventions and study designs, underscoring the need for more nuanced investigations into these sociodemographic and clinical variables.^{23,24,26–28} Additionally, clients who drop out are a special subpopulation, as a sizable amount of individuals fail to even initiate care; delays of up to 14 years can occur between symptom onset and initiation of mental health care.^29,30 This stark delay in initiating care serves as a significant barrier to timely treatment and symptom amelioration.^31,32 Consequently, there remains a need to further examine this key subsample of individuals who would benefit from care and are considering it, but ultimately do not initiate care.

In order for clients to achieve treatment gains, engagement in therapy is paramount. Clients who exhibit high engagement tend to demonstrate greater buy-in to the treatment model, better session attendance, and higher between-session homework completion.³³ With digital interventions, personalization of the digital content is essential and has repeatedly surfaced as a key predictor of increased engagement in therapy.^34,35 Personal support, as provided by a coach or therapist, is also linked to greater engagement with internet-based interventions.³⁴ This added level of guidance affords greater accountability and further personalization of internet-based interventions, lending further support for a blended care model.³⁶ Despite these advantages, there has been limited systematic research on treatment dropout or initiation of care in BC-CBT specifically. Some initial studies have reported lower dropout rates within blended care models compared to treatment as usual,^37,38 and early qualitative evidence suggests that therapist guidance is beneficial in the completion of iCBT content.³⁹

To address this gap in research, this study seeks to examine potential predictors of the initiation of care and dropout from a BC-CBT program for depression and anxiety. Relevant factors may include early engagement with digital therapeutic content, as well as client sociodemographic and clinical variables (e.g., symptom severity).⁴⁰ Identifying the pertinent predictive factors can help facilitate efforts to encourage initiation of care and mitigate dropout, which is of particular importance in digital interventions.

Method

Study design and procedures

This pragmatic, retrospective study examines data collected as part of routine quality assurance procedures for the BC-CBT program at Lyra Health. Lyra Health partners with Lyra Clinical Associates to provide therapy services for clients in the BC-CBT program. Clients were individuals (and their dependents) who were eligible to receive these behavioral health services as part of their employer sponsored benefits. Interested clients completed an onboarding triage process to determine eligibility and fit for the program. Eligible clients were then matched with providers in the BC-CBT program and scheduled for treatment. Providers met with clients for face-to-face therapy sessions via videoconference and assigned digital activities (video lessons, digital exercises) between each session based on the client's presenting issues. Clients were assigned weekly symptom severity assessments (anxiety, depression) to complete for the duration of care. All therapy sessions, digital activities, and assessments were completed through a secure, Health Insurance Portability and Accountability Act (HIPAA)-compliant, online platform that is proprietary to Lyra Health. This retrospective analysis of routinely collected, deidentified data was determined to be exempt research by the Western-Copernicus Group (WCG) Institutional Review Board.

Participants and data inclusion

Participants included 3566 individuals who booked the first appointment between July 1st, 2020 and June 7th, 2021. To be eligible for this analysis, participants must have had scores above the clinical threshold for the Generalized Anxiety Disorder-7 (GAD-7; score ≥8) or the Patient Health Questionnaire-9 (PHQ-9; score ≥10) on a valid baseline assessment. Specifically, for individuals who initiated care, the baseline assessments were considered invalid if they were collected more than two weeks before the first therapy session or after the second therapy session with the provider.

Participants must have been at least 18 years of age and willing to engage in teletherapy sessions over video. Participants were excluded from the BC-CBT program for specific clinical reasons (i.e., if reporting active suicidality, self-harm, or homicidality, current diagnosis of severe alcohol/substance use disorder(s), unstable bipolar disorder, or psychiatric disorder with psychotic features not stabilized by medications). Participants were also excluded if the first appointment was canceled due to encountering any technical issues during the appointment. Additionally, participants were excluded if they did not provide gender information. The participant flow chart is displayed in Figure 1.

Figure 1.

Participant flow chart.

Blended care therapy program

The BC-CBT program included face-to-face teletherapy sessions with a provider and digital activities assigned between sessions. Therapy sessions were generally conducted weekly, with a gradual titration to biweekly as clinically indicated. Cognitive behavioral and transdiagnostic approaches were used in therapy, including the Unified Protocol,^41,42 Acceptance and Commitment Therapy,⁴³ and Dialectical Behavior Therapy.⁴⁴ Providers assigned digital activities to be done in between sessions, personalizing the content based on the client's presenting issues. Engagement with and completion of digital activities were tracked through the online platform. Providers and clients could also securely message each other through the platform (e.g., to problem-solve emergent issues, for the provider to send reminders or feedback, etc.). More detailed information regarding the BC-CBT program can be found elsewhere.⁴⁵

Digital activities

Digital activities included digital video lessons and/or digital exercises. Digital video lessons were created with a storytelling approach, which has been shown to help facilitate relatability and normalization of mental health challenges.⁴⁶ Digital video lessons follow a character through their own therapy journey, introducing and reinforcing core CBT concepts and skills. Example clinical topics include mindful awareness, thinking traps, assertive communication, and exposure therapy. Each video lesson is approximately 10 min in length, and concludes with a brief quiz to assess for understanding. Digital exercises are online analogues of traditional worksheets or handouts that facilitate the practice of CBT-based skills. These exercises include awareness-building activities (e.g., logging thoughts, feelings, and behaviors), as well as practice-oriented activities (e.g., cognitive restructuring, behavioral activation log).

Measures

Onboarding triage assessment

Individuals who are interested in seeking mental health care complete an onboarding triage assessment to determine eligibility and fit for the BC-CBT program. This process includes self-assessments of primary issues (e.g., anxiety, stress, depression/sadness, family or relationship issues), symptoms (e.g., irritability, obsessive thoughts, difficulty sleeping, etc.), treatment history (psychological and psychiatric), and sociodemographic variables (e.g., race/ethnicity).

Initial intake assessment

Eligible clients complete an initial intake assessment at baseline to report their gender, financial status, education/employment history, and relevant clinical information (e.g., symptom presentation, psychiatric history, social functioning).

Symptom severity assessments

The Generalized Anxiety Disorder-7 (GAD-7) is a 7-item self-report measure that assesses the presence and severity of anxiety.⁴⁷ A cutoff score of ≥8 is indicative of a likely diagnosis of an anxiety disorder.⁴⁸ The Patient Health Questionnaire-9 (PHQ-9) is a 9-item self-report measure that assesses the presence and severity of depression. A cutoff score of ≥10 is indicative of a likely diagnosis of major depression.⁴⁹

The GAD-7 and PHQ-9 are rated on Likert scales from 0 to 3, with higher scores indicating more severe symptomatology. Both measures have been extensively established and validated, demonstrating strong psychometric properties as evidenced by high reliability and validity.⁵⁰

Digital engagement

Early digital engagement was defined as client completion, prior to the second therapy session, of digital activities assigned by the provider after the first therapy session. Providers could assign digital video lesson(s), digital exercise(s), both, or none. Depending on the provider's assignment, clients could complete at least one digital video lesson, at least one digital exercise, both, or none. The categories for early digital engagement included all possible permutations of assignment and/or completion of digital video lessons and/or digital exercises (Appendix A). Assignment and completion of these digital activities were objectively logged in real-time through the online platform.

Initiation of care

Initiation of care for the BC-CBT program (initiate) was defined as clients who scheduled an initial session with a provider and attended the session. Clients who rescheduled and attended the rescheduled initial session were also included in this initiate group.

Conversely, clients who cancel and no-show were operationalized as those who scheduled an initial session with a provider and either canceled or no-showed the initial appointment. If these clients attempted to reschedule the appointment afterwards, but ultimately never attended the session, they were still included in the respective cancel and no-show categories.

Dropout

Dropout from the BC-CBT program was defined as clients who initiated care by attending at least one therapy session, but then dropped out of therapy prematurely. Providers coded clients as a “dropout” if they did not respond to attempts to reschedule appointments, declined further sessions, and prematurely ended treatment. Providers marked clients as “completed” if they graduated from care.

Data analyses

The present study used a two-stage analytic approach to evaluate predictors of failing to initiate care and dropout after care had been initiated. First, multinomial logistic regression was used to identify factors associated with non-initiation of care by differentiating among individuals who cancel, those who no-show, and those who initiate care (i.e., attend at least 1 therapy session). Multinomial regression analyses were conducted using initiate as the reference category, and again with cancel as the reference category. The former produces coefficients describing the change in the odds of cancel and no-show versus initiate, whereas the coefficients in the latter represent the change in the odds of initiate and no-show versus cancel. Among participants who initiate care, a Cox proportional hazard (PH) survival analysis was applied to identify factors associated with dropout (ending treatment prematurely, as coded by providers). The Cox model describes how the probability of experiencing the event (i.e., dropout) across the exposure period (i.e., treatment weeks 0–36) changes as a function of predictor variables. Although the logistic regression analysis describes predictors of client non-initiation (which is prior to any substantial interaction with providers), clients were under the care of a specific provider during the exposure period covered by the survival analysis; a random effect for provider was included in the survival model to account for this dependency.

The onboarding assessment allowed participants to omit or opt out of providing some background and demographic information, which resulted in a small amount of missing data for education (n_miss = 156, 4.37%), employment (n_miss = 31, 0.87%), and financial status (n_miss = 99, 2.78%). Rather than omitting participants with missing data from the primary analytic models, multiple imputation was used to account for missing responses. More specifically, a multiple imputation model was specified using SAS PROC MI to generate 30 imputed datasets using the fully conditional specification with 20 burn-in iterations. To further improve the quality of the imputation, all predictor variables used in the multinomial regression and Cox survival analyses were included in the imputation model. The multinomial regression models were estimated using SAS PROC LOGISTIC, the Cox survival analysis was conducted in SAS PROC PHREG, and the corresponding estimates from the imputed results were pooled using SAS PROC MIANALYZE.

Results

Table 1 includes participants’ baseline characteristics for the entire sample and each of the engagement categories (completed, dropout, cancel, and no-show), as well as the length of treatment (if applicable).

Table 1.

Baseline characteristics and length of treatment.

	Overall	Completed	Dropout	Cancel	No-Show
n	3566	2355	887	267	57
Age, mean (SD)	32.91 (8.78)	32.91 (8.76)	32.85 (8.42)	33.28 (10.00)	31.88 (9.08)
Gender, n (%)
Female	2316 (64.95)	1493 (63.40)	602 (67.87)	190 (71.16)	31 (54.39)
Male	1250 (35.05)	862 (36.60)	285 (32.13)	77 (28.84)	26 (45.61)
Race/Ethnicity, n (%)
Minority	1766 (49.52)	1153 (48.96)	470 (52.99)	116 (43.45)	27 (47.37)
White	1605 (45.01)	1107 (47.01)	360 (40.59)	115 (43.07)	23 (40.35)
Not disclosed	195 (5.47)	95 (4.03)	57 (6.43)	36 (13.48)	7 (12.28)
Highest education, n (%)
College graduate or above	2755 (77.26)	1897 (80.55)	651 (73.39)	180 (67.42)	27 (47.37)
No college degree	655 (18.37)	392 (16.65)	184 (20.74)	56 (20.97)	23 (40.35)
Missing	156 (4.37)	66 (2.80)	52 (5.86)	31 (11.61)	7 (12.28)
Employed, n (%)
Employed	3268 (91.64)	2183 (92.70)	816 (92.00)	227 (85.02)	42 (73.68)
Unemployed	267 (7.49)	171 (7.26)	67 (7.55)	19 (7.12)	10 (17.54)
Missing	31 (0.87)	1 (0.04)	4 (0.45)	21 (7.87)	5 (8.77)
Financial status, n (%)
Good	2199 (61.67)	1514 (64.29)	519 (58.51)	147 (55.06)	19 (33.33)
Fair	1111 (31.16)	721 (30.62)	285 (32.13)	84 (31.46)	21 (36.84)
Poor	157 (4.40)	88 (3.74)	46 (5.19)	12 (4.49)	11 (19.30)
Missing	99 (2.78)	32 (1.36)	37 (4.17)	24 (8.99)	6 (10.53)
Primary need, n (%)
Anxiety	1394 (39.09)	948 (40.25)	333 (37.54)	100 (37.45)	13 (22.81)
Depression / sadness	743 (20.84)	469 (19.92)	204 (23.00)	57 (21.35)	13 (22.81)
Stress	580 (16.26)	404 (17.15)	122 (13.75)	45 (16.85)	9 (15.79)
Family or relationship issues	523 (14.67)	338 (14.35)	135 (15.22)	42 (15.73)	8 (14.04)
Other	326 (9.14)	196 (8.32)	93 (10.48)	23 (8.61)	14 (24.56)
Baseline GAD-7, mean (SD)	11.74 (4.09)	11.68 (4.01)	11.82 (4.33)	11.85 (3.93)	12.60 (4.45)
Baseline PHQ-9, mean (SD)	10.72 (4.99)	10.40 (4.88)	11.33 (5.03)	11.03 (5.29)	13.09 (5.53)
Therapy sessions completed, mean (SD)	5.59 (3.68)	7.30 (3.04)	3.10 (2.12)	N/A	N/A
Duration of care (weeks), mean (SD)	7.94 (5.80)	9.81 (5.39)	2.95 (3.43)	N/A	N/A

Predicting initiation of care

Coefficients for the multinomial logistic regression analyses are provided in Table 2. The initial model featured the initiate care outcome status as the reference category, with cancel and no-show as the coded contrast categories. The following contrast describes the change in odds of cancel versus initiate. Sociodemographic variables. A significant coefficient emerged for gender (b_cancel = −0.29 [-0.57, −0.01], b/S.E. = −2.00, p = .045, OR = 0.75), suggesting that participants identifying as male had 25% lower odds of canceling. Individuals who did not disclose their race/ethnicity were approximately 3.5 times more likely to cancel (b_cancel = 1.24 [0.82, 1.66], b/S.E. = 5.78, p < .01, OR = 3.46) than individuals identifying as minority. Clinical variables. Each additional self-reported symptom on the onboarding triage assessment was associated with a 4% increase in the odds of canceling (b_cancel = 0.04 [0.01, 0.07], b/S.E. = 2.62, p < .01, OR = 1.04), and each additional day between the time of making the appointment and the scheduled date of first session was associated with a 14% increase in the odds of canceling (b_cancel = 0.13 [0.09, 0.17], b/S.E. = 6.32, p < .01, OR = 1.14).

Table 2.

Multinomial logistic regression coefficients for predicting initiation of care.

Parameter	Cancel vs. Initiate				No-show vs. Initiate				No-show vs. Cancel
Parameter	Est [95% CI]	b/S.E.	P value	Odds Ratio	Est [95% CI]	b/S.E.	P value	Odds Ratio	Est [95% CI]	b/S.E.	P value	Odds Ratio
Intercept	−3.54 [-4.35, -2.73]	-8.54	<.01**	0.03	-4.78 [-6.36, -3.19]	-5.90	<.01**	0.01	-1.24 [-2.99, 0.52]	-1.38	.17	0.29
Sociodemographic variables
Age	0.01 [-0.01, 0.02]	0.85	.40	1.01	-0.02 [-0.05, 0.01]	-1.09	.27	0.98	-0.02 [-0.06, 0.01]	-1.38	.17	0.98
Gender [Ref = Female]
Male	-0.29 [-0.57, -0.01]	-2.00	.045*	0.75	0.45 [-0.10, 1.01]	1.60	.11	1.58	0.74 [0.13, 1.36]	2.37	.02*	2.10
Race / Ethnicity [Ref = Minority]
White	0.06 [-0.22, 0.34]	0.41	.68	1.06	-0.10 [-0.71, 0.50]	-0.34	.74	0.90	-0.16 [-0.82, 0.50]	-0.48	.63	0.85
Not disclosed	1.24 [0.82, 1.66]	5.78	<.01**	3.46	0.97 [0.08, 1.86]	2.15	.03*	2.65	-0.27 [-1.22, 0.68]	-0.55	.58	0.76
Occupation [Ref = Unemployed]
Employed	0.01 [-0.50, 0.52]	0.03	.97	1.01	-0.71 [-1.48, 0.07]	-1.79	.07†	0.49	-0.72 [-1.61, 0.18]	-1.56	.12	0.49
Financial status [Ref = Good]
Fair	0.06 [-0.24, 0.35]	0.38	.71	1.06	0.53 [-0.12, 1.18]	1.59	.11	1.70	0.47 [-0.24, 1.18]	1.30	.20	1.60
Poor	-0.01 [-0.67, 0.64]	-0.04	.97	0.99	1.42 [0.55, 2.29]	3.21	<.01**	4.16	1.44 [0.38, 2.49]	2.67	<.01**	4.21
Education [Ref = No college degree]
College graduate or above	-0.24 [-0.59, 0.10]	-1.38	.17	0.78	-0.68 [-1.32, -0.04]	-2.07	.04*	0.51	-0.43 [-1.15, 0.28]	-1.19	.23	0.65
Clinical variables
Baseline GAD-7	-0.01 [-0.04, 0.03]	-0.32	.75	0.99	0.03 [-0.04, 0.10]	0.87	.38	1.03	0.04 [-0.04, 0.11]	0.94	.35	1.04
Baseline PHQ-9	0.00 [-0.03, 0.03]	0.11	.91	1.00	0.06 [-0.01, 0.12]	1.80	.07†	1.06	0.06 [-0.01, 0.12]	1.60	.11	1.06
SI indicated	0.09 [-0.37, 0.55]	0.39	.70	1.09	-0.32 [-1.28, 0.65]	-0.65	.52	0.73	-0.41 [-1.46, 0.64]	-0.76	.45	0.66
Primary need [Ref = Anxiety]
Depression / sadness	-0.00 [-0.38, 0.38]	0.00	1.00	1.00	0.07 [-0.79, 0.93]	0.16	.87	1.07	0.07 [-0.85, 0.99]	0.15	.88	1.07
Stress	0.06 [-0.32, 0.44]	0.31	.76	1.06	0.52 [-0.36, 1.40]	1.16	.25	1.68	0.46 [-0.48, 1.40]	0.95	.34	1.58
Family or relationship issues	0.17 [-0.22, 0.56]	0.84	.40	1.18	0.43 [-0.49, 1.35]	0.91	.36	1.53	0.26 [-0.73, 1.25]	0.52	.61	1.30
Other needs	0.13 [-0.38, 0.65]	0.51	.61	1.14	0.89 [-0.01, 1.78]	1.93	.05†	2.42	0.75 [-0.26, 1.76]	1.46	.14	2.12
Number of self-assessed symptoms	0.04 [0.01, 0.07]	2.62	<.01**	1.04	-0.05 [-0.11, 0.02]	-1.39	.16	0.95	-0.09 [-0.16, -0.01]	-2.37	.02*	0.92
Currently taking antidepressant or similar medication	0.19 [-0.16, 0.55]	1.08	.28	1.22	1.20 [0.50, 1.90]	3.35	<.01**	3.32	1.01 [0.24, 1.77]	2.58	.01*	2.73
Currently seeing a therapist	0.32 [-0.11, 0.75]	1.47	.14	1.38	0.15 [-0.79, 1.10]	0.32	.75	1.17	-0.17 [-1.17, 0.84]	-0.33	.74	0.84
Time to appointment (days)	0.13 [0.09, 0.17]	6.32	<.01**	1.14	0.02 [-0.08, 0.12]	0.38	.70	1.02	-0.11 [-0.21, -0.00]	-2.05	.04*	0.90

Note. N = 3566. ** p < .01, * p < .05, † p < .10. Significant findings are noted in bold.

The next contrast describes the change in odds of no-show versus initiate. Sociodemographic variables. Individuals who chose not to disclose their race/ethnicity were more than 2.6 times more likely to no-show (b_no−show = 0.97 [0.08, 1.86], b/S.E. = 2.15, p = .03, OR = 2.65). Those reporting poor financial status were greater than 4 times more likely to no-show (b_no−show = 1.42 [0.55, 2.29], b/S.E. = 3.21, p < .01, OR = 4.16), relative to those who described their financial status as “good.” In addition, individuals who reported earning a college degree or higher had 49% lower odds of no-showing (b_no−show = −0.68 [-1.32, −0.04], b/S.E. = −2.07, p = .04, OR = 0.51). Clinical variables. Individuals reporting current use of antidepressant or similar medication were more than 3.3 times more likely to not show up for their first session (b_no−show = 1.20 [0.50, 1.90], b/S.E. = 3.35, p < .01, OR = 3.32).

The final model featured cancel as the reference category. Sociodemographic variables. Clients who identified as male were greater than 2 times more likely to no-show (b_no−show = 0.74 [0.13, 1.36], b/S.E. = 2.37, p = .02, OR = 2.10). Those describing their financial status as “poor” had more than 4 times higher odds of no-showing (b_no−show = 1.44 [0.38, 2.49], b/S.E. = 2.67, p < .01, OR = 4.21), compared to those who described their financial status as “good.” Clinical variables. Each additional self-reported symptom was associated with a 8% reduction in the odds of no-showing (b_no−show = −0.09 [-0.16, −0.01], b/S.E. = −2.37, p = .02, OR = 0.92) and each additional day between scheduling the appointment and the first session was associated with a 10% decrease in odds (b_no−show = −0.11 [-0.21, −0.00], b/S.E. = −2.05, p = .04, OR = 0.90). Finally, clients reporting use of antidepressant or similar medication were nearly 3 times more likely to no-show (b_no−show = 1.01 [0.24, 1.77], b/S.E. = 2.58, p = .01, OR = 2.73). Coefficients describing the initiate versus cancel contrast are redundant with coefficients presented earlier.

In summary, relative to those who initiate care, individuals identifying as male were less likely to cancel and more likely to no-show, compared to those identifying as female. In addition, individuals who did not report race/ethnicity were substantially more likely to cancel and no-show, relative to those who reported their race/ethnicity. Those endorsing a greater number of symptoms during the onboarding triage assessment had higher odds of canceling and no-showing. Relative to those describing their financial status as good, individuals who reported poor financial status were more likely to no-show. Additionally, individuals who report use of an antidepressant or similar medication were also more likely to no-show. In contrast, individuals reporting a higher level of education were less likely to no-show. Finally, a longer latency between making an appointment and the date of the first session was associated with higher odds of canceling.

Predicting dropout from care

Unstandardized coefficient and hazard ratio (HR) estimates for the Cox PH survival analysis predicting dropout from care (per therapist coding) are provided in Table 3. The dummy-coded vectors identify different patterns of assignment and completion of BC-CBT digital activities; they indicate the change in risk of dropout for clients who did not complete at least one digital exercise and at least one digital lesson during the first week of treatment. Clients who were assigned both exercises and lessons, but completed only an exercise(s) (b = 0.96 [0.67, 1.24], b/S.E. = 6.60, p < .01, HR = 2.61) or completed only a lesson(s) (b = 0.89 [0.61, 1.17], b/S.E. = 6.27, p < .01, HR = 2.43) had more than 2 times higher risk of dropout, and those who completed neither an exercise nor a lesson had nearly 10 times higher risk (b = 2.30 [2.11, 2.49], b/S.E. = 23.74, p < .01, HR = 9.94). Clients who were not assigned any exercises or lessons (and therefore completed none) exhibited nearly 2 times higher risk of dropout (b = 0.64 [0.07, 1.21], b/S.E. = 2.20, p = .03, HR = 1.90). Those who were assigned a lesson but failed to complete it and were not assigned any exercises had more than 4.6 times higher risk of dropout (b = 1.54 [1.12, 1.97], b/S.E. = 7.08, p < .01, HR = 4.68). It is noted that there were small sample sizes for clients who were not assigned any digital lessons, but were assigned a digital exercise; clients who completed at least 1 exercise (n = 12) or did not complete any exercises (n = 7) were included in this category.

Table 3.

Survival analysis predicting time to dropout.

Parameter	Est [95% CI]	b/S.E.	P value	Hazard Ratio
Early engagement [Ref = Completed both]
Assigned both but only completed exercises	0.96 [0.67, 1.24]	6.60	<.01**	2.61
Assigned both but only completed lessons	0.89 [0.61, 1.17]	6.27	<.01**	2.43
Assigned both but not completed either	2.30 [2.11, 2.49]	23.74	<.01**	9.94
Completed lessons but not assigned any exercise	0.07 [-0.23, 0.37]	0.48	.63	1.08
Not completed any lessons and not assigned any exercises	1.54 [1.12, 1.97]	7.08	<.01**	4.68
Not assigned either	0.64 [0.07, 1.21]	2.20	.03*	1.90
Not assigned any lessons and either completed at least one exercise or not completed any exercise	0.84 [-0.00, 1.68]	1.96	.05†	2.31
Baseline GAD-7	-0.00 [-0.02, 0.02]	-0.09	.93	1.00
Baseline PHQ-9	0.02 [0.00, 0.03]	2.17	.03*	1.02
SI indicated	-0.07 [-0.32, 0.18]	-0.53	.59	0.93
Age	-0.00 [-0.01, 0.00]	-0.82	.41	1.00
Gender [Ref = Female]
Male	-0.21 [-0.36, -0.06]	-2.75	<.01**	0.81
Race / Ethnicity [Ref = Minority]
White	-0.11 [-0.26, 0.04]	-1.42	.16	0.90
Not disclosed	0.43 [0.14, 0.72]	2.91	<.01**	1.54
Occupation [Ref = Unemployed]
Employed	0.03 [-0.24, 0.29]	0.19	.85	1.03
Financial status [Ref = Good]
Fair	0.05 [-0.10, 0.21]	0.67	.50	1.06
Poor	0.21 [-0.12, 0.55]	1.23	.22	1.23
Education [Ref = No college degree]
College graduate or above	-0.10 [-0.28, 0.09]	-1.04	.30	0.91
Primary need [Ref = Anxiety]
Depression / sadness	0.01 [-0.19, 0.21]	0.10	.92	1.01
Stress	-0.17 [-0.38, 0.05]	-1.50	.13	0.85
Family or relationship issues	-0.01 [-0.23, 0.20]	-0.10	.92	0.99
Other needs	0.17 [-0.10, 0.43]	1.21	.23	1.18
Number of self-assessed symptoms	0.01 [-0.01, 0.02]	0.85	.39	1.01
Currently taking antidepressant or similar medication	0.43 [0.24, 0.62]	4.40	<.01**	1.54
Currently seeing a therapist	0.21 [-0.03, 0.46]	1.74	.08†	1.24
Time to appointment (days)	-0.01 [-0.04, 0.01]	-0.93	.35	0.99

Note. N = 3242. ** p < .01, * p < .05, † p < .10.

Clients reporting higher PHQ-9 scores at baseline (b = 0.02 [0.00, 0.03], b/S.E. = 2.17, p = .03, HR = 1.02), those who chose not to disclose their race/ethnicity (b = 0.43 [0.14, 0.72], b/S.E. = 2.91, p < .01, HR = 1.54), and those who reported taking an antidepressant or similar medication (b = 0.43 [0.24, 0.62], b/S.E. = 4.40, p < .01, HR = 1.54) were at higher risk for dropout. Conversely, clients who identified as male were less likely to drop out (b = −0.21 [-0.36, −0.06], b/S.E. = −2.75, p < .01, HR = 0.81).

In summary, relative to clients who were assigned and completed both a digital lesson and a digital exercise prior to the second session, those who were assigned both but completed neither had the greatest risk of dropping out. Not being assigned any digital lessons and/or exercises was also associated with a higher risk of dropping out. Several baseline characteristics of clients were associated with increased risk of dropout, including higher PHQ-9 scores, not disclosing race or ethnicity, and reporting use of an antidepressant (or similar medication), whereas identifying as male was associated with a lower risk of dropout.

Discussion

The goal of the study was to understand the predictors of non-initiation and dropout from BC-CBT for symptoms of depression and anxiety. Many investigations focus on dropout from care,^20,22,26,51 but often overlook factors impacting the initiation of care. By using this two-step approach to investigate both the non-initiation of care and dropout, this study was able to differentiate among individuals who may not have had the chance to drop out (due to early withdrawal) and/or withdrew for non-care-related reasons (e.g., technology barriers, scheduling issues).⁵² With the significant delay observed between symptom onset and initiation of mental health care, examining this subsample can help inform targeted intervention approaches to mitigate prolonged functional impairment and suffering.^29,30

Shorter wait times were associated with an increased likelihood of initiating care, highlighting the importance of connecting to care in a timely manner. This can be achieved, for instance, by showing the availability of providers after clients search for care and highlighting those with more immediate openings. Increased barriers may emerge as more time passes between scheduling the appointment and the actual time of the appointment, such as new scheduling conflicts, waning motivation, or deciding that their difficulties are self-manageable. Given that therapy is an engaging process that involves vulnerability, delaying the initial appointment or hesitating to disclose ethnicity may reflect an individual's level of motivation and readiness to engage in care more broadly. Of note, individuals belonging to a minority group (Black, Hispanic, Asian/Pacific Islander) may be less likely to disclose their ethnicity in healthcare surveys.⁵³ In this study, clients who did not disclose their ethnicity were significantly more likely to disengage from care than clients who did disclose belonging to a minority ethnic group. Consequently, there may be a unique confluence of belonging to an ethnic minority group and general hesitance of engaging in mental health care that is associated with increased disengagement rates. Additional attention and retention efforts should thus be provided to these individuals to curb further mental health care disparities.

Furthermore, endorsement of a greater number of issues and symptoms during the onboarding process was associated with greater risk of non-initiation of care and dropout, which is consistent with previously reported increases in dropout rates when comorbidities and greater clinical complexity were present.⁵⁴ This also highlights the utility of having a quantitative screening measure administered at intake (e.g., checklist of symptoms), like the current BC-CBT program incorporates, to help identify those at risk for not initiating care or dropping out. Similarly, individuals who reported being on antidepressants and endorsed higher baseline depression severity were less likely to initiate care and more likely to drop out. Individuals with depression who have had prior success with medications could be more likely to believe in the medical model, and in turn less likely to pursue psychotherapy.⁵⁵ Furthermore, individuals with more severe depressive symptoms are more likely to disengage due to the confluence of disorder-related functional impairment and how depressive symptoms inherently manifest (e.g., high withdrawal, pessimism, fatigue, amotivation).⁵⁶

The gender-related findings for the present study were interesting, given that the extant literature has generally shown greater dropout rates among male versus female clients in traditional in-person psychotherapy. Males in the present sample demonstrated relatively lower rates of dropout when compared to females; it is possible that this is related to modality (e.g., teletherapy, blended care), industry, geography, or other factors that would benefit from further research. Of note, predictors of dropout in iCBT have been more mixed for gender effects.^57,58 Studies examining attitudes about and use of online therapy have also noted different patterns of gender effects when compared to traditional in-person therapy, suggesting that engagement may be impacted by treatment modality.^59,60

Early digital engagement was a particularly potent and important factor when evaluating dropout from a blended care program. Given that clients were up to ten times more likely to drop out from care if they did not complete any digital activities that were assigned to them (compared to those who completed all activities), this suggests that early digital engagement is a robust signal for potential client dropout within a blended care model. Consequently, although therapists may have periodic inclinations to hold off on assignments between sessions (e.g., uncertainty about client motivation, concern about follow-through), assigning digital activities could still be beneficial; a client's early digital engagement can help signal whether there is a lack of buy-in for the model or therapy more broadly, and thus create an opportunity for targeted intervention. As with therapy in general, it is imperative that expectations regarding the treatment model are set early on to obtain strong buy-in and commitment. The advantages of blended care can be leveraged to facilitate this process, as providing the rationale for and personalizing the digital activities have been shown to increase engagement.^34,35 Indeed, early client follow-through with digital activities appears to be a protective factor against dropout. It importantly also facilitates in-between session homework assignment and completion, which are core pillars of CBT.

Given that blended care treatment harnesses the advantages of integrating technology into care, there are several clinical implications for these findings. Based on these predictor variables, an algorithm can be created to see who is at highest risk of not initiating care or dropping out. Identifying these individuals early on in the process increases the likelihood that a provider can intervene and problem-solve emergent concerns. Early identification also provides opportunities for targeted interventions at a critical time point, such as the beginning of care. For instance, if a client is flagged as high risk for dropout, it may be indicated for the provider to dedicate additional efforts towards increasing buy-in using motivational interviewing techniques.^61,62 The integration of technology in blended care also presents unique opportunities to address these issues. For example, the platform can be programmed to send supportive messages to sustain engagement for clients at higher risk for dropout. Additionally, providers could send additional, more personalized messages in between sessions to prompt clients who may be hesitant to complete assignments. Alternatively, specific digital activities could be created to directly address worries about starting care or how the blended care model can optimally benefit the client.

These results should be considered within the context of several limitations and suggestions for future directions. First, a circumscribed set of sociodemographic and clinical variables were analyzed as potential predictors in this study. Other potentially relevant factors, such as therapeutic alliance, previous treatment history, and psychiatric comorbidities,^20,22,63 should be investigated in future research. Second, no formal comparison groups were used to determine whether these predictors for initiating care and dropout were specific to blended care programs. Although this was the first study to quantitatively examine potential predictors in BC-CBT, future studies should seek to disentangle the unique impact of the blended care component through randomized controlled trials. Third, this study objectively evaluated predictors associated with initiating care and dropout (the “what”), but does not explore the reasoning behind them (the “why”). Future studies could benefit from a more qualitative exploration into how and why these variables may confer more risk, providing more nuanced insights to better inform targeted interventions. Indeed, further investigations are still needed to determine the best content and type of intervention to encourage initiation of care and curb dropout. Fourth, dropout was coded by the providers. While providers are well positioned to determine whether or not a client has ended treatment prematurely, this coding method tends to result in higher reported dropout rates when compared to other methods of coding dropout (e.g., client stopped attending sessions, failure to complete treatment protocol).^64,65 This binary coding system (“dropout” vs. “complete”) also does not fully capture the client's care journey. For instance, clients may end treatment because they believe their symptoms have improved enough, even if they did not meet all treatment goals or finish a specific treatment protocal.⁶⁶ Indeed, previously reported clinical improvement rates for this BC-CBT program (73% achieving reliable improvement, 83% recovered)⁴⁵ suggest that there is likely a meaningful proportion of therapist-coded dropouts that reached clinical success prior to withdrawal from care. Future studies should explore the impact of more nuanced coding methods for dropout and examine the diverse subsets of clients who drop out of care for various reasons. Furthermore, early engagement with the assigned digital activities was associated with a significantly lower risk of dropout. It is possible that this engagement was indicative of broader baseline activation at the outset of treatment, so clients may have been generally more willing to engage in therapy-related activities. Future research using dismantling methodology would be useful to help illuminate the specificity of different treatment components in predicting dropout. Additionally, future investigations into the type of digital activities that have the highest completion rates and/or lead to continued engagement in treatment would be valuable for maximizing retention. Lastly, our findings indicate that individuals with lower levels of education and financial status are associated with greater risks of not initiating mental health care. While digital mental health interventions are designed to increase accessibility, further research is needed to ensure that marginalized and underserved individuals are considered during the development of these interventions to address disparities.⁶⁷ Current solutions include review of written material to be simplified and readable across a broader education level, flexible appointment times, and access to asynchronous therapy content, but additional work remains.

Conclusions

There has been a rapid proliferation of alternative methods of delivering mental health care to increase accessibility, such as delivering therapy via a combination of self-guided digital tools and video-conferencing through a blended care model. This shift in modality also calls for concerted research efforts to optimize emerging approaches. This study is one of the first systematic investigations of predictors of (non)engagement with a BC-CBT program, and the results suggest areas of optimization, including minimizing wait times before starting care, promoting earlier digital engagement, and offering individuals with more severe and complex symptom presentations additional support. Continuing research on improving engagement with technology-based interventions will be instrumental for their advancement and, consequently, for bettering the mental health of individuals.

Footnotes

Acknowledgements

The authors would like to extend their gratitude for the blended care therapists who are dedicated to improving the lives of our clients. The authors would also like to thank Jocelynn Owusu, PhD, MPH for reviewing the manuscript.

Declaration of Conflicting Interests

MW, SC, and AL are employed by Lyra Health, receive income from Lyra Health, and have been granted equity in Lyra Health. AV and CC are employed by Lyra Health and Lyra Clinical Associates, receive income from Lyra Health and Lyra Clinical Associates, and have been granted equity in Lyra Health. RW is a paid consultant for Lyra Health. YL has no conflicts of interest to declare.

Ethical approval

This retrospective analysis of routinely collected, deidentified data was determined to be exempt research by the Western-Copernicus Group (WCG) Institutional Review Board.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Guarantor

Contributorship

MSW, SC, REW, YL, and AL conceived the study. MSW was responsible for obtaining ethical approval for the study. MSW, SC, REW, and YL wrote the first draft of the manuscript. SC and REW were responsible for data cleaning and analysis. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

ORCID iD

Monica S. Wu

Appendix

Appendix A. Distribution of categories of early digital content engagement

References

Butler

Chapman

Forman

, et al. The empirical status of cognitive-behavioral therapy: a review of meta-analyses. Clin Psychol Rev 2006; 26: 17–31.

Hofmann

Asnaani

Vonk

IJJ

, et al. The efficacy of cognitive behavioral therapy: a review of meta-analyses. Cogn Ther Res 2012; 36: 427–440.

Bishop

Seirup

Pincus

, et al. Population of US practicing psychiatrists declined, 2003–13, which may help explain poor access to mental health care. Health Aff Proj Hope 2016; 35: 1271–1277.

Patel

Chowdhary

Rahman

, et al. Improving access to psychological treatments: lessons from developing countries. Behav Res Ther 2011; 49: 523–528.

Payne

Myhr

. Increasing access to cognitive-behavioural therapy (CBT) for the treatment of mental illness in Canada: a research framework and call for action. Healthc Policy 2010; 5: e173–e185.

Bennett

Ruggero

Sever

, et al. Ehealth to redress psychotherapy access barriers both new and old: a review of reviews and meta-analyses. J Psychother Integr 2020; 30: 188–207.

Kannarkat

Smith

McLeod-Bryant

. Mobilization of telepsychiatry in response to COVID-19—moving toward 21st century access to care. Adm Policy Ment Health 2020; 47: 489–491.

Weightman

. Digital psychotherapy as an effective and timely treatment option for depression and anxiety disorders: implications for rural and remote practice. J Int Med Res 2020; 48: 0300060520928686.

Fernandez

Woldgabreal

Day

, et al. Live psychotherapy by video versus in-person: A meta-analysis of efficacy and its relationship to types and targets of treatment. Clin Psychol Psychother Epub ahead of print 7 April 2021. DOI: 10.1002/cpp.2594

10.

Andersson

Carlbring

Ljótsson

, et al. Guided internet-based CBT for common mental disorders. J Contemp Psychother 2013; 43: 223–233.

11.

Păsărelu

Andersson

Bergman Nordgren

, et al. Internet-delivered transdiagnostic and tailored cognitive behavioral therapy for anxiety and depression: a systematic review and meta-analysis of randomized controlled trials. Cogn Behav Ther 2017; 46: 1–28.

12.

Andrews

Basu

Cuijpers

, et al. Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: an updated meta-analysis. J Anxiety Disord 2018; 55: 70–78.

13.

Spek

Cuijpers

Nyklíček

, et al. Internet-based cognitive behaviour therapy for symptoms of depression and anxiety: a meta-analysis. Psychol Med 2007; 37: 319–328.

14.

Baumeister

Reichler

Munzinger

, et al. The impact of guidance on internet-based mental health interventions — A systematic review. Internet Interv 2014; 1: 205–215.

15.

Baumel

Edan

Kane

. Is there a trial bias impacting user engagement with unguided e-mental health interventions? A systematic comparison of published reports and real-world usage of the same programs. Transl Behav Med 2019; 9: 1020–1033.

16.

Saad

Bruno

Camara

, et al. Self-directed technology-based therapeutic methods for adult patients receiving mental health services: systematic review. JMIR Ment Health 2021; 8: e27404.

17.

Wentzel

van der Vaart

Bohlmeijer

, et al. Mixing online and face-to-face therapy: how to benefit from blended care in mental health care. JMIR Ment Health 2016; 3: e9.

18.

Fernandez

Salem

Swift

, et al. Meta-analysis of dropout from cognitive behavioral therapy: magnitude, timing, and moderators. J Consult Clin Psychol 2015; 83: 1108–1122.

19.

Bados

Balaguer

Saldaña

. The efficacy of cognitive-behavioral therapy and the problem of drop-out. J Clin Psychol 2007; 63: 585–592.

20.

Sharf

Primavera

Diener

. Dropout and therapeutic alliance: a meta-analysis of adult individual psychotherapy. Psychother Theory Res Pract Train 2010; 47: 637–645.

21.

Eysenbach

. The law of attrition. J Med Internet Res 2005; 7: e11.

22.

Karyotaki

Kleiboer

Smit

, et al. Predictors of treatment dropout in self-guided web-based interventions for depression: an ‘individual patient data’ meta-analysis. Psychol Med 2015; 45: 2717–2726.

23.

Christensen

Griffiths

Farrer

. Adherence in internet interventions for anxiety and depression: systematic review. J Med Internet Res 2009; 11: e1194.

24.

Issakidis

Andrews

. Pretreatment attrition and dropout in an outpatient clinic for anxiety disorders. Acta Psychiatr Scand 2004; 109: 426–433.

25.

Bentley

Cohen

Kim

, et al. The nature, timing, and symptom trajectories of dropout from transdiagnostic and single-diagnosis cognitive-behavioral therapy for anxiety disorders. Behav Ther 2021; 52: 1364–1376.

26.

Melville

Casey

Kavanagh

. Dropout from internet-based treatment for psychological disorders. Br J Clin Psychol 2010; 49: 455–471.

27.

Beatty

Binnion

. A systematic review of predictors of, and reasons for, adherence to online psychological interventions. Int J Behav Med 2016; 23: 776–794.

28.

Linardon

Fuller-Tyszkiewicz

. Attrition and adherence in smartphone-delivered interventions for mental health problems: a systematic and meta-analytic review. J Consult Clin Psychol 2020; 88: 1–13.

29.

Kessler

Olfson

Berglund

. Patterns and predictors of treatment contact after first onset of psychiatric disorders. Am J Psychiatry 1998; 155: 62–69.

30.

Christiana

Gilman

Guardino

, et al. Duration between onset and time of obtaining initial treatment among people with anxiety and mood disorders: an international survey of members of mental health patient advocate groups. Psychol Med 2000; 30: 693–703.

31.

Mojtabai

Olfson

Sampson

, et al. Barriers to mental health treatment: results from the national comorbidity survey replication. Psychol Med 2011; 41: 1751–1761.

32.

Andrade

Alonso

Mneimneh

, et al. Barriers to mental health treatment: results from the WHO world mental health surveys. Psychol Med 2014; 44: 1303–1317.

33.

Glenn

Golinelli

Rose

, et al. Who gets the most out of cognitive behavioral therapy for anxiety disorders? The role of treatment dose and patient engagement. J Consult Clin Psychol 2013; 81: 639–649.

34.

Saleem

Kühne

Santis

KKD

, et al. Understanding engagement strategies in digital interventions for mental health promotion: scoping review. JMIR Ment Health 2021; 8: e30000.

35.

Brouwer

Oenema

Crutzen

, et al. An exploration of factors related to dissemination of and exposure to internet-delivered behavior change interventions aimed at adults: A delphi study approach. J Med Internet Res 2008; 10: e956.

36.

Cavanagh

. Turn on, tune in and (don’t) drop out: engagement, adherence, attrition, and alliance with internet-based interventions. In: Oxford Guide to low intensity CBT interventions. New York, NY, US: Oxford University Press, 2010, pp.227–233.

37.

Campbell

ANC

Nunes

Matthews

, et al. Internet-delivered treatment for substance abuse: a multisite randomized controlled trial. Am J Psychiatry 2014; 171: 683–690.

38.

Christensen

Landes

Jackson

, et al. Adding an internet-delivered treatment to an efficacious treatment package for opioid dependence. J Consult Clin Psychol 2014; 82: 964–972.

39.

Wilhelmsen

Lillevoll

Risør

, et al. Motivation to persist with internet-based cognitive behavioural treatment using blended care: a qualitative study. BMC Psychiatry 2013; 13: 296.

40.

Puls

H-C

Schmidt

Herpertz

, et al. Adherence as a predictor of dropout in internet-based guided self-help for adults with binge-eating disorder and overweight or obesity. Int J Eat Disord 2020; 53: 555–563.

41.

Barlow

Farchione

Bullis

, et al. The unified protocol for transdiagnostic treatment of emotional disorders compared with diagnosis-specific protocols for anxiety disorders. JAMA Psychiatry 2017; 74: 875–884.

42.

Barlow

Farchione

Sauer-Zavala

, et al. Unified protocol for transdiagnostic treatment of emotional disorders: Therapist guide. 2nd ed. New York, NY: Oxford University Press, 2017.

43.

Hayes

Strosahl

Wilson

. Acceptance and commitment therapy, second edition: The process and practice of mindful change. New York, NY: Guilford Press, 2011.

44.

Linehan

. DBT Skills training manual, second edition. Second edition, Available separately: DBT skills training handouts and worksheets. Second edition. New York, NY: The Guilford Press, 2014.

45.

Lungu

Jun

Azarmanesh

, et al. Blended care-cognitive behavioral therapy for depression and anxiety in real-world settings: pragmatic retrospective study. J Med Internet Res 2020; 22: e18723.

46.

Williams

O’Moore

Mason

, et al. The effectiveness of internet cognitive behaviour therapy (iCBT) for social anxiety disorder across two routine practice pathways. Internet Interv 2014; 1: 225–229.

47.

Spitzer

Kroenke

Williams

JBW

, et al. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006; 166: 1092–1097.

48.

Plummer

Manea

Trepel

, et al. Screening for anxiety disorders with the GAD-7 and GAD-2: a systematic review and diagnostic metaanalysis. Gen Hosp Psychiatry 2016; 39: 24–31.

49.

Kroenke

Spitzer

Williams

. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001; 16: 606–613.

50.

Kroenke

Spitzer

Williams

, et al. The patient health questionnaire somatic, anxiety, and depressive symptom scales: a systematic review. Gen Hosp Psychiatry 2010; 32: 345–359.

51.

Gersh

Hallford

Rice

, et al. Systematic review and meta-analysis of dropout rates in individual psychotherapy for generalized anxiety disorder. J Anxiety Disord 2017; 52: 25–33.

52.

Mohr

Hart

Howard

, et al. Barriers to psychotherapy among depressed and nondepressed primary care patients. Ann Behav Med 2006; 32: 254–258.

53.

Dembosky

Haviland

Haas

, et al. Indirect estimation of race/ethnicity for survey respondents who do not report race/ethnicity. Med Care 2019; 57: e28–e33.

54.

Olfson

Mojtabai

Sampson

, et al. Dropout from outpatient mental health care in the United States. Psychiatr Serv 2009; 60: 898–907.

55.

Beshai

Watson

Meadows

TJS

, et al. Perceptions of cognitive-behavioral therapy and antidepressant medication for depression after brief psychoeducation: examining shifts in attitudes. Behav Ther 2019; 50: 851–863.

56.

Zivin

Pfeiffer

McCammon

, et al.

‘No-shows’: who fails to follow up with initial behavioral health treatment?

Am J Manag Care 2009; 15: 105–112.

57.

Schmidt

Forand

Strunk

Predictors of dropout in internet-based cognitive behavioral therapy for depression. Cogn Ther Res 2019; 43: 620–630.

58.

Edmonds

Hadjistavropoulos

Schneider

, et al. Who benefits most from therapist-assisted internet-delivered cognitive behaviour therapy in clinical practice? Predictors of symptom change and dropout. J Anxiety Disord 2018; 54: 24–32.

59.

March

Day

Ritchie

, et al. Attitudes toward e-Mental health services in a community sample of adults: Online survey. J Med Internet Res 2018; 20: e59.

60.

Tsan

Day

. Personality and gender as predictors of online counseling use. J Technol Hum Serv 2007; 25: 39–55.

61.

Arkowitz

Westra

. Integrating motivational interviewing and cognitive behavioral therapy in the treatment of depression and anxiety. J Cogn Psychother 2004; 18: 337–350.

62.

Zuckoff

Swartz

Grote

. Motivational interviewing as a prelude to psychotherapy of depression. In: Motivational interviewing in the treatment of psychological problems. New York, NY, US: The Guilford Press, 2008, pp.109–144.

63.

Sales

. Understanding prior dropout in psychotherapy. Int J Psychol Psychol Ther 2003; 3: 81–90.

64.

Wierzbicki

Pekarik

. A meta-analysis of psychotherapy dropout. Prof Psychol Res Pract 1993; 24: 190–195.

65.

Swift

Greenberg

. Premature discontinuation in adult psychotherapy: a meta-analysis. J Consult Clin Psychol 2012; 80: 547–559.

66.

Hynan

. Client reasons and experiences in treatment that influence termination of psychotherapy. J Clin Psychol 1990; 46: 891–895.

67.

Schueller

Hunter

Figueroa

, et al. Use of digital mental health for marginalized and underserved populations. Curr Treat Options Psychiatry 2019; 6: 243–255.