Utilization,Patient Characteristics,and Longitudinal Improvements among Patients from a Provincially Funded Transdiagnostic Internet-delivered Cognitive Behavioural Therapy Program: Observational Study of Trends over 6 Years

Abstract

French

Objective:

There is strong evidence supporting internet-delivered cognitive behaviour therapy (iCBT) and consequently growing demand for iCBT in Canada. Transdiagnostic iCBT that addresses both depression and anxiety is particularly promising as it represents an efficient method of delivering iCBT in routine care. The Online Therapy Unit, funded by the Saskatchewan government, has been offering transdiagnostic iCBT for depression and anxiety since 2013. In this article, to broadly inform implementation efforts, we examined trends in utilization, patient characteristics, and longitudinal improvements for patients receiving transdiagnostic iCBT over 6 years.

Methods:

Patients who completed telephone screening between November 2013 and December 2019 were included in this observational study. Patients provided demographics and mental health history at screening and completed measures at pre-treatment, post-treatment and at 3- to 4-month follow-up. Treatment engagement and satisfaction were assessed.

Results:

A total of 5,321 telephone screenings were completed and 4,283 of patients were accepted for treatment over the 6-year period (80.5% acceptance). The most common reason for referral to another service was high suicide risk/severe symptoms (47.1%). Examination of trends showed growing use of transdiagnostic iCBT over time (37% increase per year). There was remarkable stability in patient characteristics across years. Most patients were concurrently using medication (57.3%) with 11.9% reporting using iCBT while on a waiting list for face-to-face treatment highlighting the importance of integrating iCBT with other services. Consistent across years, large improvements in depression and anxiety symptoms were found and maintained at 3- to 4-month follow-up. There was strong patient engagement with iCBT and positive ratings of treatment experiences.

Conclusions:

As there is growing interest in iCBT in Canada, this large observational study provides valuable information for those implementing iCBT in terms of likely user characteristics, patterns of use, and improvements. This information has potential to assist with resource allocation and planning in Canada and elsewhere.

Keywords

depression anxiety internet-delivered cognitive behaviour therapy transdiagnostic

Introduction

Depression and anxiety contribute significantly to the global disease burden,¹ but many Canadians do not receive treatment for their concerns.² Individuals face both structural (e.g., location) and attitudinal (e.g., stigma) barriers to treatment.³ Public health crises, like COVID-19, highlight the need for digital mental health treatments.⁴ Consequently, the Canadian government announced “an investment of $240.5 million to develop, expand, and launch virtual care and mental health tools to support Canadians.”⁵

Within the digital mental health field, there is enormous evidence supporting both diagnostic-specific and transdiagnostic (concurrent treatment of depression and anxiety) therapist-assisted internet-delivered cognitive behaviour therapy (iCBT).^6

–11 Although heterogeneity among iCBT programs exists, in routine practice, it is common for programs to begin with screening and for treatment materials to be presented over several months using online weekly lessons. In routine practice, it is also common for trained providers (e.g., therapist, psychologist) to offer support via weekly asynchronous emails and or telephone calls¹⁰ and for programs to include persuasive design features such as automated emails.¹²

While iCBT research emerged over 20 years ago,¹³ within Canada, there has been limited access to publicly funded therapist-assisted iCBT programs coordinated with healthcare services.¹⁴ One question that emerges is what can be expected longer-term when therapist-assisted iCBT is routinely and freely available.

One example of a well-researched publicly funded iCBT program in Canada is the Online Therapy Unit in Saskatchewan. The unit screens patients for, and delivers, an 8-week transdiagnostic iCBT program called the Wellbeing Course that was originally, developed, validated, and then implemented by MindSpot, an Australian digital mental health service.¹¹ The transdiagnostic program creates efficiency in implementing iCBT (e.g., reduces time to train therapists in and to allocate patients to diagnostic-specific programs) and has been integrated into Saskatchewan mental health care by encouraging physicians and clinicians to direct patients to this service when this meets patients’ needs or when patients are on waiting lists. Furthermore, community mental health clinics have designated clinicians who deliver iCBT via the unit platform.

The Online Therapy Unit also has a research arm. Past research on the Wellbeing Course has shown outcomes are strong whether delivered by therapists who specialise in iCBT or by therapists who primarily deliver face-to-face treatment.^15,16 Furthermore, outcomes are comparable with varying support (i.e., once-weekly support compares to optional support,^17,18 twice-weekly support,¹⁹ and one-business-day support²⁰). Still, other trials have studied therapist behaviours in iCBT^21
–23 and outcomes of iCBT for alcohol misuse,²⁴ for various chronic health conditions,^25

–30 and for specific populations like postpartum depression/anxiety³¹ or university students.³²

With the growing imperative to offer iCBT in Canada, the purpose of this article is to examine patient characteristics as well as longitudinal improvements among patients screened for and offered transdiagnostic iCBT via the Online Therapy Unit over ∼6 years; there is no other such report of iCBT in routine care in Canada and to our knowledge only one other similar report from Australia,³³ which deserves replication. In this 7-year Australian study, utilization increased in the first 3 years and subsequently stabilized, with a higher demand for assessment services than iCBT. iCBT effects were stable across years. Over time, there was an increase in young adult users, Aboriginal or Torres Strait Islander, and individuals with anxiety.

As has been noted, “evaluation of trends over time in how mental health services are used can help guide decisions about resource allocation”³⁴ and program planning.³³ Specifically, monitoring trends helps identify user changes and gaps in care and informs new programming or care models (e.g., growth in younger users could suggest offering texting rather than telephone support). Examining trends also adds to evidence regarding iCBT in routine care as there are few reports of iCBT outcomes sustained over longer periods after initial implementation efforts conclude.³⁵ Research questions explored in this study were as follows: What growth will be observed in iCBT over time? Will users of iCBT change? Will there be stability in longitudinal symptom improvements over 6 years? Drawing on findings from Australia,³³ it was hypothesized there would be growth in iCBT over time and effects would be consistently large over the 6-year period. Hypotheses regarding changes in users of iCBT were exploratory.

Methods

Design and Ethics

Patients who requested iCBT between November 2013 and December 2019 are included in this uncontrolled observational cohort analysis. Ethics approval was obtained from the University of Regina Research Ethics Board. The article combines data across and between registered trials that examined the Wellbeing Course outcomes across settings (ISRCTN42729166), with motivational interviewing as a pre-treatment (NCT03684434) or with varying levels of support (ISRCTN14230906; NCT03304392; NCT03957330).

Participants

Participants learn about iCBT through various sources (which have remained consistent) including medical professionals (55.0%; n = 2,928), mental health professionals (16.0%; n = 851), word of mouth (13.0%; n = 689), online sources (6.9%; n = 367), printed materials (2.2%; n = 115), media (2.5%, n = 132), insurance providers (0.3%, n = 14), or non-specified means (4.0%; n = 214). Consistent across years, to be eligible for iCBT, patients have to identify as (1) ≥18 years of age, (2) Saskatchewan residents, (3) having >5 Patient Health Questionnaire 9-item (PHQ-9)³⁶ and/or >5 Generalized Anxiety Disorder Questionnaire 7-item (GAD-7)³⁷, (4) having Internet access and computer comfort, and (5) a healthcare provider emergency contact. Individuals meeting any of the following are excluded: (1) high risk of suicide, past year suicide attempt, or mental health hospitalization; (2) unmanaged psychosis, mania, and or drug/alcohol problems; (3) > twice-monthly face-to-face therapy; (4) not in Saskatchewan for 8-week treatment; (5) seeking help for different problem; or (6) not interested in iCBT. As individuals with subthreshold symptoms benefit from iCBT, diagnosis is not required for iCBT.³⁸

Online and Telephone Screening

Prospective participants complete an online screening questionnaire capturing demographics and questionnaires listed below. Thereafter, participants complete a telephone interview further assessing inclusion/exclusion criteria. Individuals ineligible for iCBT are referred to alternative resources. Eligible participants provide verbal treatment consent over the telephone followed by digital consent on the platform.

Intervention

All patients received the transdiagnostic iCBT Wellbeing Course (see Titov et al. for details¹¹), consisting of 5 lessons targeting anxiety and depression symptoms, released gradually over 8 weeks. Lessons consist of educational content, patient stories, and homework. Core strategies include monitoring physical, behavioural, and cognitive symptoms; thought challenging; controlled breathing; behavioural activation; graduated exposure; and relapse prevention. Additional resources (i.e., assertiveness, beliefs, communication, managing emergencies, mental skills, panic, post-traumatic stress, sleep, and structured problem solving/worry time) are also openly accessible throughout treatment.¹¹

Therapists

The Online Therapy Unit trained all therapists (n = 132) who have a primary affiliation with the unit (n = 46) or a government-funded Saskatchewan mental health clinic (n = 86 therapists from 11 clinics). Therapists have backgrounds in social work (n = 80), psychology (n = 36), addictions (n = 9), or nursing (n = 7) and are either registered professionals (n = 96) or supervised graduate students (n = 36). On average, therapists included in this dataset treated 31 patients (SD = 89.23; range = 1 to 626). In March 2018, to improve coordination of iCBT, the Unit increased the numbers of therapists working half-to full-time and reduced the numbers of therapists with small caseloads.²²

Therapist Support

Therapists contact clients on a designated day once-weekly for 8 weeks, although in some trials the amount of contact has varied (e.g., optional, twice-weekly).^17

–20 Most contact includes asynchronous secure emails, with therapists spending 15 to 20 minutes composing messages after reviewing patient iCBT progress, symptom measures, and messages. Therapists also phone patients when there is a large symptom increase, messages suggest suicide risk, patients have not logged in for a week, and or a phone call would more easily address patient concerns. Across trials, therapists follow a treatment manual; regular audit and feedback is used to ensure compliance with procedures.²⁴

Measures

All measures are administered pre-treatment, at 8 weeks, and 3 to 4 months post-treatment. There has been some minor variability in measures administered resulting in variable sample size for some measures.

Primary Follow-up Measures

PHQ-9

The PHQ-9 is a 9-item questionnaire assessing depression over the past 2 weeks (range 0 to 27).³⁶ Scoring ≥10 suggests a likely depression diagnosis.³⁸

GAD-7

The GAD-7 is a 7-item questionnaire measuring anxiety over the past 2 weeks (range 0 to 21).³⁷ Scoring ≥8 suggests a likely generalized anxiety disorder diagnosis.³³

Secondary Follow-up Measures

Panic Disorder Severity Scale Self-report (PDSS-SR)

The PDSS-SR is 7-item panic disorder questionnaire (ranges 0 to 28).³⁹ Scoring ≥8 suggests a likely panic disorder diagnosis.⁴⁰

Social Interaction Anxiety Scale and Social Phobia Scale (SIAS-6/SPS-6)

The SIAS-6/SPS-6 are 6-item social anxiety measures (ranges 0 to 48).⁴¹ SIAS-6 ≥7 and SPS-6 ≥ 2 suggest a likely social anxiety disorder diagnosis.⁴¹

Kessler Distress Scale (K10)

The K10 includes10 items of psychological distress (ranges 10 to 50).⁴² Score > 21 suggests significant distress.³³

Sheehan Disability Scale (SDS)

The SDS includes 3 items measuring disruption to work/school, social life, and family life⁴³ creating a 0 (unimpaired) to 30 (highly impaired) score.

Working Alliance Inventory–Short Revised (WAI-SR)

The 12-item WAI-SR, administered at 8 weeks, resulted in a total working alliance score (range 15 to 60), and task, goal, and bond sub-scores (range 5 to 20).⁴⁴

Healthcare use

From October 2017 to September 2018, at pre-treatment and 4-month follow-up, patients completed 3-month healthcare use questions (i.e., physician, psychiatrist, mental health provider, ER/hospital admission and other medical/health provider).

Treatment Satisfaction

At 8 weeks post-treatment, patients answered yes/no items about whether they would recommend the treatment to a friend, the course was worth their time, and they experienced negative treatment effects. Patients also rated satisfaction (1 to 5) with the course, materials, frequency and quality of therapist contact, confidence to manage symptoms, and motivation to seek future treatment.

Engagement

iCBT engagement was assessed by examining number of patient messages, therapist messages, phone calls, and lessons accessed.

Statistical Analyses

Data were analysed using SPSS version 26.0 with chi-square analyses of linear-by-linear associations (categorical variables) and analysis of variance (continuous variables) to assess yearly trends in patient characteristics, satisfaction, and engagement. As this study used data collected during and between trials, secondary analyses examined whether these variables differed by trials (see Supplementary Material).

Generalized estimating equation (GEE) models were run in R and examined symptoms from assessment to post-treatment to follow-up after controlling for enrolment year and therapist location.⁴⁵ Secondary analyses also controlled for trials. Consistent with intention-to-treat analyses, missing data were imputed before analysis.

Clinically significant change was examined by year (and secondarily by trials) through percentage symptom changes from baseline,⁴⁶ and within-group Hedges g effect sizes using GEE estimated marginal means. Separate analyses were conducted for the overall sample and for patients in the clinical range at pre-treatment (i.e., PHQ-9 > 10, GAD-7 > 8, PDSS > 8, SIAS >7 and SPS > 2, K10 > 21,³³ and SDS > 11)^. The SDS value was based on 1st quartile cut-off as no cut-off was identified. Clinical deterioration was defined as PHQ-9 increasing ≥ 6 and GAD-7 increasing ≥ 5³³; clinical recovery was defined as PHQ-9 decreasing ≥ 6 and GAD-7 decreasing ≥ 5, resulting in a patient moving from the clinical to the non-clinical range.³³ A Bonferroni correction was used for each set of analyses as a control for number of tests performed.

Results

Patient Flow

Figure 1 shows the number and proportion of patients who were screened (n = 5,321), started (n = 4,283) and completed all treatment materials (n = 2,831).

Figure 1.

Patient flow from screening to follow-up. Trial 1 (November 1, 2013, to July 20, 2015): Cohort comparison of specialized and nonspecialized therapists. Trial 2 (February 1, 2016, to July 25, 2016): Randomized controlled trial comparing optional support versus once-weekly support. Trial 3 (December 2, 2016, to June 8, 2017): Preference trial comparing patient preference for optional versus once-weekly support. Trial 4 (October 1, 2017, to September 14, 2018): Randomized trial comparing one-business-day support versus once-weekly support. Trial 5 (September 2018, to March 1, 2019): Randomized trial comparing once-weekly support with or without pre-treatment lesson on motivational interviewing. Trial 6 (May 2, 2019, to November 5, 2019): Randomized factorial trial comparing once-weekly versus twice-weekly support and homework reflection record versus no reflection record.

Background

Table 1 provides demographics for patients who started iCBT by year. There were no differences in patient characteristics over time, except for the proportion who indicated having children (range: 51.0% to 63.8%). Patient characteristics also did not differ by trial period (See Supplementary Table 1).

Table 1.

Demographic Characteristics by Year and Overall for Patients Who Started the Wellbeing Course.

	2013	2014	2015	2016	2017	2018	2019	Total	Significance
Patients started course (n)	28	275	407	571	715	812	1,195	4,003
Age
Mean age (SD)	38.14 (11.99)	38.97 (12.42)	38.94 (13.22)	37.36 (12.50)	37.25 (12.90)	37.56 (13.12)	36.58 (12.76)	37.42 (12.89)	F ₍ _6,4000 ₎ = 2.51, P = 0.02
[Age range]	[22 to 73]	[19 to 73]	[18 to 81]	[18 to 81]	[18 to 86]	[18 to 84]	[18 to 88]	[18 to 88]	F ₍ _6,4000 ₎ = 2.51, P = 0.02
Gender
Men	10.7%	24.9%	29.70%	25.40%	23.60%	24.80%	25.60%	25.3%	χ² (1, N = 4,001) = 0.17, P = 0.68
Men	(3/28)	(68/273)	(121/407)	(145/571)	(169/715)	(201/812)	(306/1,195)	(1,013/4,001)
Women	89.3%	74.7%	69.8%	74.30%	76.10%	75.20%	73.20%	74.20%
Women	(25/28)	(204/273)	(284/407)	(424/571)	(544/715)	(611/812)	(875/1,195)	(2,967/4,001)
Non-binary, two-spirit, not listed, or prefer not to disclose	0	0.40%	0.50%	0.40%	0.30%	0	1.20%	0.40%
	0	(1/273)	(2/407)	(2/571)	(2/715)	0	(21/1,195)	(21/4,001)
Location
Large city (>200,000)	50.00%	54.2% (148/273)	46.40%	42.60%	40.80%	41.70%	45.10%	44.10%	χ² (1, N = 3,999) = 3.25, P = 0.07
Large city (>200,000)	(14/28)	54.2% (148/273)	(189/407)	(243/571)	(292/715)	(339/812)	(539/1,194)	(1,764/4,000)
Small-medium city (10,000 to 200,000)	10.70%	19.00%	23.80%	29.10%	31.20%	25.20%	25.80%	26.40%
Small-medium city (10,000 to 200,000)	(3/28)	(52/273)	(97/407)	(166/571)	(223/715)	(205/812)	(308/1,194)	(1,054/4,000)
Town or village	35.70%	19.40%	20.90%	20.80%	20.70%	23.30%	20.00%	21.10%
Town or village	(10/28)	(53/273)	(85/407)	(119/571)	(148/715)	(189/812)	(239/1,194)	(843/4,000)
Farm	3.60%	7.30%	8.60%	7.40%	6.70%	9.10%	8.30%	8.00%
	(1/28)	(20/273)	(35/407)	(42/571)	(48/715)	(74/812)	(99/1,194)	(319/4,000)

Reserve	0	0	0.20%	0.20%	0.60%	0.60%	0.80%	0.50%
Reserve	0	0	(1/407)	(1/571)	(4/715)	(5/812)	(9/1,194)	(20/4,000)
Ethnicity
White/Caucasian	92.90%	93.40%	92.00%	89.50%	88.10%	91.30%	89.80%	90.3%	χ² (1, N = 3,992) = 5.14, P = 0.02
White/Caucasian	(26/28)	(253/271)	(368/400)	(511/571)	(630/715)	(743/812)	(1,073/1,195)	(3,604/3,992)
Asian	3.60%	0.40%	1.50%	2.10%	1.30%	0.60%	1.70%	1.40%
Asian	(1/28)	(1/271)	(6/400)	(12/571)	(9/715)	(5/812)	(20/1,195)	(54/3,992)
First Nations/Métis	3.60%	2.20%	3.50%	2.00%	5.10%	5.40%	5.70%	4.50%
First Nations/Métis	(1/28)	(6/271)	(14/400)	(11/571)	(36/715)	(44/812)	(68/1,195)	(180/3,992)
Other	0	4.10%	3.00%	6.50%	5.60%	2.50%	2.80%	3.90%
Other	0	(11/271)	(12/400)	(37/571)	(40/715)	(21/812)	(34/1,195)	(155/3,992)
Employment (most applicable)
Paid employment	67.90%	70.00%	65.40%	64.80%	63.50%	61.50%	68.10%	65.30%	χ² (1, N = 4,000) = 4.62, P = 0.03
Paid employment	(19/28)	(191/273)	(266/407)	(370/571)	(454/715)	(499/812)	(814/1,195)	(2,613/4,001)
Student	3.60%	1.80%	4.20%	7.40%	7.60%	6.90%	6.40%	6.30%
Student	(1/28)	(5/273)	(17/407)	(42/571)	(54/715)	(56/812)	(76/1,195)	(251/4,001)
Unemployed	10.70%	9.20%	11.30%	6.80%	7.40%	7.80%	6.30%	7.60%
Unemployed	(3/28)	(25/273)	(46/407)	(39/571)	(53/715)	(63/812)	(75/1,195)	(304/4,001)
Other (retired, home duties, disability)	17.90%	19.00%	19.20%	21.00%	21.50%	26.80%	19.25%	20.89%
Other (retired, home duties, disability)	(5/28)	(54/273)	(78/407)	(120/571)	(154/715)	(194/812)	(230/1,195)	(835/4,001)
Education
University degree	57.20%	36.20%	35.30%	37.30%	33.40%	33.90%	33.60%	33.7%	χ² (1, N = 4,001) = 1.62, P = 0.20
University degree	(16/28)	(99/273)	(104/407)	(213/571)	(239/715)	(276/812)	(401/1,195)	(1,348/4,001)	χ² (1, N = 4,001) = 1.62, P = 0.20
Relationship status
Married	50.0% (14/28)	48.0% (131/273)	46.2% (188/407)	46.8% (263/562)	45.5% (325/715)	45.2% (367/812)	44.4% (531/1,195)	45.6% (1,819/3,992)	χ² (1, N = 3,992) = 0.93, P = 0.44
Children
Have children	53.60%	63.8% (173/271)	57.6% (234/406)	57.50%	59.00%	58.40%	51.0%	56.4%	χ² (1, N = 3,988) = 13.50, P < 0.001^a
Have children	(15/28)	63.8% (173/271)	57.6% (234/406)	(323/562)	(422/715)	(474/812)	(609/1,195)	(2,250/3,989)	χ² (1, N = 3,988) = 13.50, P < 0.001^a

Note. Bonferroni correction P < 0.006.

^a Over time, there was a lower number of patients reporting they had children. See Supplementary Materials Table 1 showing patient characteristics did not differ by trial.

Clinical Profile at Screening

Table 2 shows baseline symptom severity for patients who started iCBT each year remained stable (Supplementary Material Table 2 also shows the clinical profile was stable across trials). Across years, 83.7% (n = 3,352) of patients had depression and or anxiety in the clinical range and 60.7% (n = 2,290) reported psychotropic medication use in the last 3 months.

Table 2.

Baseline Symptom Severity and Current and Lifetime Mental Health Service Use by Year and Overall for Patients Who Started the Wellbeing Course.

	2013	2014	2015	2016	2017	2018	2019	Total	Significance
Baseline symptom scores
Mean PHQ-9 (SD)	11.86 (5.05)	12.59 (5.67)	12.05 (5.66)	11.79 (5.95)	12.34 (5.92)	12.02 (5.79)	11.91 (5.62)	12.05 (5.76)	F _(6,3995) = 1.03, P = 0.40
Mean GAD-7 (SD)	12.07 (5.66)	12.07 (5.10)	11.13 (5.27)	11.81 (5.20)	12.31 (5.00)	12.05 (5.14)	11.62 (4.95)	11.85 (5.09)	F _(6,3995) = 3.02, P = 0.006
Mean K10 (SD)	26.36 (6.84)	27.82 (7.45)	27.14 (7.30)	27.52 (7.34)	28.46 (7.80)	28.20 (7.60)	27.64 (7.58)	27.84 (7.55)	F _{(6, 3995)} = 2.14, P = 0.04
Mean SDS (SD)	18.11 (8.10)	17.37 (8.00)	17.58 (7.72)	17.11 (7.67)	17.01 (7.65)	17.02 (7.87)	16.89 (7.92)	17.09 (7.81)	F _{(6, 3995)} = 0.56, P = 0.76
Mean PDSS-SR (SD)	9.39 (8.16)	7.22 (6.22)	7.41 (6.21)	7.36 (6.12)	7.57 (5.96)	7.53 (5.92)	7.14 (5.93)	7.38 (6.03)	F _(6,3995) = 1.07, P = 0.38
Mean SPS/SIAS-6 (SD)	12.18 (9.78)	13.19 (10.39)	14.23 (10.24)	14.70 (10.37)	15.01 (10.84)	15.06 (11.29)	14.91 (10.69)	14.71 (10.73)	F _(6,3995) = 1.61, P = 0.14
Symptoms in the clinical range
Depression or anxiety	85.7% (24/28)	84.0% (231/275)	83,3% (339/407)	81.6% (466/571)	84.6% (605/715)	84.4% (685/812)	83.8% (1,002/1,195)	83.7% (3,352/4,003)	χ2 (1, N = 4,003) = 0.26, P = 0.61
Mental health service use (last 3 months)
Receiving other mental health treatment	32.1% (9/28)	49.6% (134/270)	47.2% (191/405)	40.8% (233/571)	43.0% (307/714)	48.9% (397/812)	49.2% (587/1,194)	46.5% (1,858/3,994)	χ2 (1, N = 3,994) = 3.90, P = 0.05
On wait-list for other services	—	—	8.6% (16/187)	9.6% (55/571)	14.4% (103/714)	14.2% (115/812)	10.5% (125/1,194)	11.9% (414/3,478)	χ2 (1, N = 3,532) = 0.16, P = 0.69
Psychotropic medication	42.9% (12/28)	62.4% (171/274)	60.4% (246/407)	57.3% (327/571)	56.2% (401/713)	55.6% (451/811)	56.9% (682/1,194)	57.3% (2,292/3,999)	χ2 (1, N = 3,999) = 1.72, P = 0.19
Mental health service use (lifetime)
Family doctor/walk-in clinic	—	—	—	—	88.6% (109/123)	84.7% (688/812)	86.9% (1,037/1,194)	86.1% (1,834/2,129)	χ2 (1, N = 2,129) = 0.31, P = 0.58
Psychiatrist	—	—	—	—	30.9% (38/123)	30.4% (247/812)	32.5% (388/1,194)	31.61% (673/2,129)	χ2 (1, N = 2,129) = 0.78, P = 0.38

Note. PHQ-9 = Patient Health Questionnaire 9-item; GAD-7 = Generalized Anxiety Disorder Questionnaire 7-item; K10 = Kessler Distress Scale; SDS = Sheehan Disability Scale; PDSS-SR = Panic Disorder Severity Scale Self-Report; SIAS-6/SPS-6 = Social Interaction Anxiety Scale and Social Phobia Scale (SIAS-6/SPS-6). Bonferroni correction P < 0.004.

Primary and Secondary Follow-up Measures

Table 3 shows means, tests of symptom reduction, effect sizes, and percentage reductions for the overall sample and those in the clinical range. Analysis revealed significant symptom reductions on all measures at post-treatment, which were maintained at follow-up. For those in the clinical range, effect sizes were large for all post-treatment and follow-up measures, except for social anxiety, which had a moderate post-treatment effect. A significant portion of patients demonstrated post-treatment recovery and very few experienced deterioration across years (see Table 4).

Table 3.

Treatment Outcomes in Overall Sample.

	Estimated Means, Standard Deviations, and Test of No Change from Pre-treatment			Hedges g Effect Sizes from Pre-treatment		Percentage Change in Estimated Mean from Pre-treatment		Reliable Treatment Outcomes
	Pre-treatment	Post-treatment	Follow-up	to Post-treatment	to Follow-up	to Post-treatment	to Follow-up	Deterioration	Recovery
Overall sample
PHQ-9	12.05 (5.76)	6.43 (5.39) χ² = 1,115.7, P < 0.001	5.86 (5.04) χ² = 1,563.5, P < 0.001	1.01 [0.97, 1.05]	1.14 [1.10, 1.18]	47% [45% to 49%]	51% [50% to 53%]	1.7%	36.3%
GAD-7	11.85 (5.09)	5.91 (4.89) χ² = 1,339.6, P < 0.001	5.21 (4.51) χ² = 2,101.5, P < .001	1.19 [1.15, 1.23]	1.38 [1.34, 1.42]	50% [48% to 52%]	56% [54% to 58%]	2.0%	45.0%
K10	27.84 (7.55)	20.72 (7.73) χ² = 1,074.4, P < 0.001	19.32 (7.63) χ² = 2,169.7, P < 0.001	0.93 [0.89, 0.97]	1.12 [1.08, 1.16]	26% [24% to 27%]	31% [29% to 32%]	—	—
SDS	17.12 (7.81)	10.52 (8.37) χ² = 741.9, P < 0.001	9.15 (8.67) χ² = 972.2, P < 0.001	0.81 [0.78, 0.85]	0.97 [0.93, 1.00]	39% [36% to 41%]	47% [44% to 49%]	—	—
PDSS-SR	7.38 (6.03)	4.51 (4.84) χ² = 444.4, P < 0.001	3.46 (4.23) χ² = 774.6, P < 0.001	0.53 [0.49, 0.56]	0.75 [0.72, 0.79]	39% [36% to 42%]	53% [50% to 56%]	—	—
SIAS-6/SPS-6	14.71 (10.73)	11.15 (9.31) χ² = 474.2, P < 0.001	9.79 (9.34) χ² = 421.1, P < 0.001	0.35 [0.32, 0.39]	0.49 [0.46, 0.52]	24% [22% to 27%]	33% [30% to 36%]	—	—
Clinical sample^a
PHQ-9 (≥ 10, N = 2,607)	15.39 (3.98)	8.09 (5.58) χ² = 1,123.2, P < 0.001	7.12 (5.29) χ² = 1,567.1, P < 0.001	1.51 [1.45, 1.56]	1.77 [1.70, 1.83]	47% [45% to 49%]	54% [52% to 56%]	1.3%	55.7%
GAD-7 (≥ 8, N = 3,068)	13.94 (3.76)	6.86 (4.98) χ² = 1,463.8, P < 0.001	5.96 (4.64) χ² = 2,008.8, P < 0.001	1.60 [1.55, 1.66]	1.89 [1.83, 1.95]	51% [49% to 53%]	57% [56% to 59%]	1.7%	58.7%
K10 (≥ 21, N = 3,279)	30.21 (6.10)	22.05 (7.73) χ² = 1,061.7, P < 0.001	20.45 (7.75) χ² = 2,045.8, P < 0.001	1.17 [1.13, 1.22]	1.40 [1.35, 1.45]	27% [26% to 28%]	32% [31% to 33%]	—	—
SDS excluding lower quartile (≥ 11, N = 3,105)	20.40 (5.27)	12.25 (8.31) χ² = 781.5, P < 0.001	10.46 (8.86) χ² = 1,014.0, P < 0.001	1.17 [1.12, 1.22]	1.36 [1.31, 1.41]	40% [38% to 42%]	49% [47% to 51%]	—	—
PDSS-SR (≥ 8, N = 1,832)	12.91 (3.99)	7.24 (5.18) χ² = 606.4, P < 0.001	5.41 (4.72) χ² = 1,120.2, P < 0.001	1.23 [1.17, 1.29]	1.72 [1.64, 1.79]	44% [41% to 47%]	58% [56% to 60%]	—	—
SIAS-6 ≥7 and SPS-6 ≥ 2 (N = 2,073)	22.59 (8.58)	16.48 (9.23) χ² = 590.2, P < 0.001	14.66 (9.74) χ² = 507.5, P < 0.001	0.69 [0.64, 0.73]	0.86 [0.81, 0.91]	27% [25% to 29%]	35% [32% to 38%]	—	—

^a For each measure, estimated means, effects sizes, and percentage change are calculated based on those who scored in the clinical range on the measure at pre-treatment.

Table 4.

Reliable Change, Program Engagement, Working Alliance, and Treatment Ratings by Year and Overall.

	2013	2014	2015	2016	2017	2018	2019	Total	Significance
Reliable change
PHQ-9 post-treatment recovery	42.0%	39.4%	36.8%	35.9%	37.1%	35.2%	35.7%	36.3%	F _{(6, 2609)} = 0.33, P = 0.92
PHQ-9 post-treatment deterioration	4.7%	1.3%	1.8%	1.4%	1.9%	2.0%	1.4%	1.7%	F _(6,1434) = 0.36, P = 0.90
GAD-7 post-treatment recovery	50.7%	43.7%	39.5%	44.3%	47.5%	45.5%	45.5%	45.0%	F _{(6, 2252)} = 1.00, P = 0.43
GAD-7 post-treatment deterioration	0.8%	1.3%	3.7%	2.1%	2.0%	1.6%	1.7%	2.0%	F _(6,1117) = 0.76, P = 0.60
Program engagement
Accessed > 4 lessons	85.7% (24/28)	85.8% (236/275)	79.4% (323/407)	79.3% (453/571)	78.0% (558/715)	81.8% (664/812)	80.3% (959/1,195)	80.4% (3,217/4,003)	χ2 (1, N = 4,003) = .48, P = 0.49
Accessed 5 lessons	75.0% (21/28)	78.9% (217/275)	70.3% (286/407)	70.6% (403/571)	66.4% (475/715)	71.1% (577/812)	71.3% (852/1,195)	70.7% (2,831/4,003)	χ2 (1, N = 4,003) = 1.21, P = 0.27
Mean number of therapist emails to patient (SD)	9.68 (2.98)	9.44 (2.29)	8.33 (1.98)	7.07 (2.70)	8.15 (2.94)	10.01 (3.14)	11.29 (3.97)	9.43 (3.52)	F _{(6, 3996)} = 152.29, P < 0.001^a
Mean number of emails from patient to therapist (SD)	6.93 (8.02)	4.78 (3.30)	4.13 (3.70)	3.34 (3.44)	3.53 (3.20)	4.09 (3.76)	4.14 (3.57)	3.97 (3.60)	F _{(6, 3996)} = 11.04, P < 0.001^a
Mean number of phone calls (SD)	1.21 (1.69)	0.77 (1.23)	0.88 (1.29)	0.54 (0.89)	0.68 (1.04)	0.96 (1.32)	0.98 (1.21)	0.84 (1.19)	F _{(6, 3996)} = 18.00, P < 0.001^a
Working alliance
WAI-SR total score	—	—	—	43.67 (11.92)	45.36 (10.89)	46.30 (10.56)	47.22 (10.01)	46.19 (10.62)	F _{(3, 1973)} = 7.81, P < 0.001^b
WAI-SR bond score	—	—	—	15.80 (4.76)	16.58 (3.91)	16.88 (3.79)	17.14 (3.47)	16.80 (3.84)	F _{(3, 1966)} = 8.14, P < 0.001^b
WAI-SR task score	—	—	—	14.78 (3.73)	14.89 (3.66)	15.08 (3.58)	15.34 (3.40)	15.11 (3.55)	F _{(3, 1966)} = 2.25, P = 0.08
WAI-SR goal score	—	—	—	13.09 (5.03)	13.84 (4.99)	14.30 (4.57)	14.73 (4.45)	14.25 (4.68)	F _{(63, 1966)} = 8.69, P < 0.001^b
Treatment ratings
Satisfied/very satisfied overall	90.9% (20/22)	84.7% (171/202)	81.3% (222/273)	80.4% (300/373)	86.3% (402/466)	84.4% (487,576)	88.6% (675/762)	85.2% (2,277/2,674)	χ2 (1, N = 2,674) = 3.23, P = 0.07
Satisfied/very satisfied with materials	95.5% (21/22)	93.1% (188/202)	89.0% (243/273)	86.9% (324/373)	90.8% (423/466)	88.7% (511/576)	90.4% (689/762)	89.7% (2,399/2,674)	χ2 (1, N = 2,674) = 0, P = 0.99
Satisfied/very satisfied with telephone calls	71.4% (5/7)	77.3% (85/110)	71.9% (110/153)	67.9% (133/196)	71.2% (183/257)	75.0% (252/336)	74.8% (357/477)	73.2% (1,125/1,536)	χ2 (1, N = 1,536) = 4.93, P = 0.03
Satisfied/very satisfied with emails	90.9% (10/11)	83.7% (169/202)	78.0% (213/273)	74.0% (276/373)	77.0% (359/466)	83.8% (482/575)	84.1% (641/762)	80.7% (2,150/2,662)	χ2 (1, N = 2,622) = 8.344, P = 0.004
Increased/greatly increased confidence	95.5% (21/22)	91.0% (184/202)	86.4% (236/273)	87.9% (328/373)	93.6% (436/466)	90.1% (519/576)	93.0% (709/762)	91.0% (2,433/2,674)	χ2 (1, N = 2,674) = 2.78, P = 0.10
Increased/greatly increased motivation for other treatment	77.3% (17/22)	79.2% (160/202)	75.8% (207/273)	77.2% (288/373)	81.5% (380/466)	83.5% (481/576)	86.9% (636/732)	81.1% (2,169/2,674)	χ2 (1, N = 2,674) = 4.90, P = 0.03
Course was worth the time (%)	100% (22/22)	98.0% (198/202)	94.5% (258/273)	94.6% (353/373)	96.8% (451/466)	95.8% (552/576)	97.2% (741/762)	96.3% (2,575/2,674)	χ2 (1, N = 2,674) = 0.77, P = 0.38
Would recommend course to a friend (%)	100% (22/22)	97.5% (197/202)	93.8% (256/273)	94.9% (354/373)	97.2% (453/466)	97.0% (559/576)	97.2% (741/762)	96.6% (2,582/2,674)	χ 2 (1, N = 2,674) = 2.78, P = 0.10

Note. PHQ-9 = Patient Health Questionnaire-9-item; GAD-7 = Generalized Anxiety Disorder-7-item; WAI-SR = Working Alliance Inventory-Short Report. Bonferroni correction p < .002.

^aPost-hoc comparisons revealed some year-to-year differences in emails sent, emails received, and phone contact. These differences are attributed to trial designs investigating different levels of therapist support. For detailed comparisons by trial, see Supplementary Materials.

^bPost-hoc comparisons revealed the WAI-SR Total, Bond subscale, and Goal subscale were lower in 2016 (year patients offered optional support) than in 2018 or 2019 (years offered once-weekly or greater support). For detailed comparisons by trial, see Supplementary Materials.

Program Engagement

Year-to-year analysis of showed high engagement across measures and years (e.g., 80.4%; n = 3,217 of patients accessed at least 4 lessons; see Table 4). Some significant differences were observed in number of emails and phone calls between therapists and patients over the years and across trials reflecting research trials that varied support levels (see Table 4; Supplementary Material Table 3).

Working Alliance and Patient Satisfaction

Working alliance and satisfaction variables show strong patient satisfaction across all indicators over time (see Table 4). There was a significant increase in patients’ WAI-SR total, bond, and goal scores over the years and related to research trials where there was increased therapist contact (see Table 4, Supplementary Material Table 3).

Secondary Analysis of Healthcare Use

For the sub-sample who completed healthcare questions at screening (n = 655) and follow-up (n = 453), there was no change in the proportion of patients who reported visits to a psychiatrist (16.6% vs. 16.6%), mental health professional (27% vs. 21.6%), medical or health professional (7.2% vs. 4.9%), or having an emergency room/hospital admission (4.7% vs. 2.6%). There was a reduction in visits to family physicians (61.2% vs. 32.5%; χ2 (1, N = 655) = 88.16, P < 0.001) and patients not utilizing any service for mental health concerns (50.3% vs. 24.0%; χ2 (1, N = 655) = 81.82, P < 0.001).

Discussion

This observational cohort study adds to evidence related to patient features, engagement, satisfaction and longitudinal improvements in therapist-assisted transdiagnostic iCBT in routine care in Canada. Consistent with studies on iCBT in routine care in Australia,^10,
33 large effect sizes were found across years on primary measures, distress, and disability that were maintained at follow-up. A considerable number of patients met criteria for recovery (55.7% for depression; 58.7% for anxiety), and deterioration rates were very low (1.3% for depression; 1.7% for anxiety). These findings demonstrate the stability in iCBT symptom improvements over time in routine care and after initial implementation efforts, which has rarely been reported in the literature with such large sample sizes and has not been examined in Canada to date.³⁵

The Online Therapy Unit showed significant growth in the numbers of patients completing telephone screens (average annual increase: 37.7%) and starting iCBT (average annual increase: 35.3%), indicating increasing demand for digital mental health services. This finding supports the Canadian Government announcement to fund virtual mental health tools, especially iCBT. It would be valuable to explore whether increased use of digital mental services is part of an observed trend for increased use of mental health services⁴⁷ or whether increased use of iCBT is associated with generally decreased use of other services. Examining trends suggests that characteristics of those seeking services have remained stable. Patients seeking iCBT show considerable diversity in terms of age (18 to 88) and location of residence (with ∼56% of patients residing outside large cities). Regarding ethnic diversity, there was a trend toward increased iCBT use by patients who identified as First Nations or Métis across the years. Comparison to 2016 provincial statistics⁴⁸ suggests, however, First Nations and Métis people are still under-represented in iCBT (4.5% of the sample compared with 16.3% of Saskatchewan’s First Nations or Métis population). Also noteworthy, across years only about one-quarter of patients identified as male. The under-representation of First Nations or Métis individuals⁴⁹ and males⁵⁰ reflects a broader concern about lower mental health utilization in these populations and is not necessarily specific to iCBT.

Of interest, there is considerable diversity in symptom severity among those seeking iCBT and iCBT programs need to take this into account (∼16% did not have scores in the clinical range; 57.3% of patients were taking medication). Over 50% of patients were referred by physicians, suggesting iCBT is a valuable resource for physicians in managing patient needs. In the subsample of patients who completed healthcare use questions pre and post-ICBT, there was a decreased report of physician visits after completing iCBT (61.2% vs. 32.5%), highlighting the importance of coordinating iCBT with medical care and draws attention to the need for research on how iCBT impacts healthcare use.

Across years, patient engagement and satisfaction were excellent (e.g., 80% review most materials; 96.3% report iCBT was worth their time). There were some differences in the patient/therapist emails, phone calls, and working alliance observed across years, with supplementary analyses suggesting differences reflected research trials that offered varying degrees of therapist support. Importantly, this past research shows increasing therapist support beyond once weekly does not improve effects and in fact is associated with higher costs and operational challenges.^20,21

Comparison with Wellbeing Course in Australia

The sample characteristics are similar to the Australian sample,³³ except a greater proportion of our patients reported concurrent medication use and treatment. Notably, patients in our sample were more likely to complete most treatment materials (80.4% vs. 67.1%), post-treatment measures (70.8% vs. 60.0%), and follow-up measures (60.3% vs. 39.3%). Lower completion rates in the Australian sample may be explained by higher pre-treatment symptom severity or differences in practices. For example, a large proportion of the Australian patients were seeking symptom consultation rather than iCBT, and users were permitted to access iCBT without first speaking to a therapist. A greater focus on consultation and system navigation is a potential avenue for the Online Therapy Unit in the future, especially since this represents an area of growth in Australia. Of note, the Australian group identified an increase in proportion of young adult users, Aboriginal or Torres Strait Islander, and those with anxiety, which we did not identify; these may be trends to watch that could impact programming. Notwithstanding the differences, the similarity in patient characteristics and longitudinal effects across the two countries is striking. The results provide promising evidence that with minor adaptations, iCBT may be used effectively across countries.

Limitations

Post-treatment and follow-up data were missing for approximately 30% and 40% of patients, respectively, although rates of follow-up are higher than what is typical of routine care studies,^9,51
-53 and a conservative approach was used to replace missing data. This study did not include a control group, although previous RCTs on the Wellbeing Course,⁵⁴ and many randomized controlled trials of iCBT have been undertaken.⁶ Self-report measures were used rather than diagnostic interviews and key demographic characteristics (e.g., gender, sexual orientation) were not tracked, so certain groups may be under-represented. The results need to be considered in light of the fact that many patients use other services while using iCBT, and we used a brief measure of healthcare use for a subsample only; as such, it is impossible to fully ascertain whether participating in iCBT is associated with decreased use of other services, including face-to-face CBT, at follow-up. Finally, it is important to note we did not track additional resource use in the current study, and future research should explore how these resources contribute to outcomes.

Strengths and Conclusions

In light of research showing many patients do not receive adequate quality or quantity of treatment⁵⁵ intensified by the COVID-19 pandemic,⁵⁶ the spotlight is on digital mental health services to improve treatment access. This study highlights growth can be expected in iCBT over time, and it is possible to sustain large effects, patient engagement, and satisfaction over time after initial iCBT implementation efforts have been employed. A strength of this study is its large sample size spanning 6 years, and the stability of symptom improvements from post-treatment to 3- to 4-month follow-up, engagement, and satisfaction over time. Publication of large studies like this one assists groups who are planning changes to mental health systems³³ and is important as there is no large health service databases to draw on related to iCBT. The findings highlight iCBT is not a replacement for other services, as ∼20% of patients seeking iCBT are referred to other services, and a high percentage of patients concurrently take medication; as such, iCBT needs to be woven into existing healthcare systems. The study identifies some gaps in service users, suggesting directions for improving reach of iCBT that represent important considerations for others implementing iCBT in Canada. The longitudinal effects observed in this large study may serve as a benchmark for the growing number of iCBT programs that are emerging in Canada.¹⁴ Benchmarking is a common technique in routine practice that will allow groups implementing iCBT programs to compare effects across different programs⁵⁷ and can guide quality improvement efforts (e.g., encouraging programs with lower effects to improve care) as well as funding decisions.

Supplemental Material

Supplemental Material, sj-docx-1-cpa-10.1177_07067437211006873 - Utilization, Patient Characteristics, and Longitudinal Improvements among Patients from a Provincially Funded Transdiagnostic Internet-delivered Cognitive Behavioural Therapy Program: Observational Study of Trends over 6 Years

Supplemental Material, sj-docx-1-cpa-10.1177_07067437211006873 for Utilization, Patient Characteristics, and Longitudinal Improvements among Patients from a Provincially Funded Transdiagnostic Internet-delivered Cognitive Behavioural Therapy Program: Observational Study of Trends over 6 Years by Heather D. Hadjistavropoulos, Vanessa Peynenburg, David L. Thiessen, Marcie Nugent, Eyal Karin, Lauren Staples, Blake F. Dear and Nickolai Titov in The Canadian Journal of Psychiatry

Supplemental Material

Supplemental Material, sj-pdf-1-cpa-10.1177_07067437211006873 - Utilization, Patient Characteristics, and Longitudinal Improvements among Patients from a Provincially Funded Transdiagnostic Internet-delivered Cognitive Behavioural Therapy Program: Observational Study of Trends over 6 Years

Supplemental Material, sj-pdf-1-cpa-10.1177_07067437211006873 for Utilization, Patient Characteristics, and Longitudinal Improvements among Patients from a Provincially Funded Transdiagnostic Internet-delivered Cognitive Behavioural Therapy Program: Observational Study of Trends over 6 Years by Heather D. Hadjistavropoulos, Vanessa Peynenburg, David L. Thiessen, Marcie Nugent, Eyal Karin, Lauren Staples, Blake F. Dear and Nickolai Titov in The Canadian Journal of Psychiatry

Footnotes

Acknowledgments

The authors acknowledge patients, research staff, therapists and managers associated with the Online Therapy Unit at the University of Regina and the Online Therapy Team at the Saskatchewan Health Authority. The authors are indebted to the Saskatchewan Ministry of Health and acknowledge Information Services at the University of Regina, especially Max Ivanov, for his pivotal role in developing the Online Therapy Unit platform.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: H.D.H. is the Executive Director of the Online Therapy Unit.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: H.D.H. is funded by Saskatchewan Ministry of Health to operate the Online Therapy Unit in collaboration with the Saskatchewan Health Authority, who is also provided Ministry of Health funding to employ online therapists. Research on the Wellbeing Course within the Online Therapy Unit resulting in data used in this paper has been funded by the Canadian Institutes of Health Research (293379, 152917), Saskatchewan Health Research Foundation, and Rx&D Health Research Foundation. Funders had no involvement in the design of the paper, collection, analysis, or interpretation of the data. N.T. and B.F.D. are funded by the Australian Government to operate the MindSpot Clinic. N.T. and B.F.D. developed the Wellbeing Course but derive no financial benefit from it.

ORCID iDs

Heather D. Hadjistavropoulos, PhD

Nickolai Titov, PhD

Statement About Data Access

Available by contacting the University of Regina.

Supplemental Material

The supplemental material for this article is available online

References

GBD 2016 Disease and Injury Incidence and Prevalence Collaborators. Global, regional, and national incidence, prevalence, and years lived with disability for 328 diseases and injuries for 195 countries, 1990–2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet. 2017;390(10100):1211–1259.

Sunderland

Findlay

. Perceived need for mental health care in Canada: results from the 2012 Canadian community health survey—mental health. Statistics Canada. 2013;24(9):3–9.

Mojtabai

Olfson

Sampson

, et al. Barriers to mental health treatment: results from the national comorbidity survey replication. Psychol Med. 2011;41(8):1751–1761.

Wind

Rijkeboer

Andersson

Riper

. The COVID-19 pandemic: the “black swan” for mental health care and a turning point for e-health. Internet Interv. 2020;20:100317.

Justin Trudeau, Prime Minister of Canada. Prime Minister announces virtual care and mental health tools for Canadians [News Release]. 2020, May 3. [accessed 2020 Nov 24] https://pm.gc.ca/en/news/news-releases/2020/05/03/prime-minister-announces-virtual-care-and-mental-health-tools

Andersson

Carlbring

Titov

Lindefors

. Internet interventions for adults with anxiety and mood disorders: a narrative umbrella review of recent meta-analyses. Can J Psychiatry. 2019;64(7):465–470.

Andersson

Titov

Dear

Rozental

Carlbring

. Internet-delivered psychological treatments: from innovation to implementation. World Psychiatry. 2019;18(1):20–28.

Romin

Batelaan

Kok

, et al. Internet-delivered cognitive behavioural therapy for anxiety disorders community versus clinical service recruitment: meta-analysis. J Med Internet Res. 2019;21(4):e11706.

Hedman

Ljótsson

Kaldo

, et al. Effectiveness of Internet-based cognitive behaviour therapy for depression in routine psychiatric care. J Affect Disord. 2014;155(1):49–58.

10.

Etzelmueller

Vis

Karyotaki

, et al. Effects of Internet-based cognitive behavioral therapy in routine care for adults in treatment for depression and anxiety: a systematic review and meta-analysis. J Med Internet Res. 2020;22(8):e18100.

11.

Titov

Dear

Staples

, et al. MindSpot Clinic: an accessible, efficient, and effective online treatment service for anxiety and depression. Psychiatr Serv. 2015;66(10):1043–1050.

12.

Kelders

Kok

Ossebaard

Van Gemert-Pijnen

. Persuasive design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res. 2012;14(6):e152.

13.

Andersson

. Internet interventions: past, present and future. Internet Interv. 2018;12:181–188.

14.

Canadian Agency for Drugs and Technologies in Health. Internet-delivered cognitive behavioural therapy for major depressive disorder and anxiety disorders: an environmental scan. Environmental Scan. 2018. [accessed 2020]. https://cadth.ca/internet-delivered-cognitive-behavioural-therapy-major-depressive-disorder-and-anxiety-disorders-1

15.

Hadjistavropoulos

Nugent

Alberts

Staples

Dear

Titov

. Transdiagnostic Internet-delivered cognitive behaviour therapy in Canada: an open trial comparing results of a specialized online clinic and nonspecialized community clinics. J Anxiety Disord. 2016;42:19–29.

16.

Hadjistavropoulos

Nugent

Dirkse

Pugh

. Implementation of Internet-delivered cognitive behaviour therapy within community mental health clinics: a process evaluation using the consolidated framework for implementation research. BMC Psychiatry. 2017;17(1):331.

17.

Hadjistavropoulos

Schneider

Edmonds

, et al. Randomized controlled trial of internet-delivered cognitive behaviour therapy comparing standard weekly versus optional weekly therapist support. J Anxiety Disord. 2017;52:15–24.

18.

Hadjistavropoulos

Schneider

Mehta

Karin

Dear

Titov

. Preference trial of Internet-delivered cognitive behaviour therapy comparing standard weekly versus optional weekly therapist support. J Anxiety Disord. 2019;63:51–60.

19.

Hadjistavropoulos

Peynenburg

Thiessen

, et al. A pragmatic factorial randomized controlled trial of transdiagnostic internet-delivered cognitive behavioural therapy: exploring benefits of homework reflection questionnaires and twice-weekly therapist support. Internet Interv. 2020;22:100357.

20.

Hadjistavropoulos

Peynenburg

Nugent

Karin

Titov

Dear

. Transdiagnostic Internet-delivered cognitive behaviour therapy with therapist support offered once-weekly or once-weekly supplemented with therapist support within one-business-day: pragmatic randomized controlled trial. Internet Interv. 2020;22:100347.

21.

Hadjistavropoulos

Gullickson

Schneider

Dear

Titov

. Development of the internet-delivered cognitive behaviour therapy undesirable therapist behaviours scale (ICBT-UTBS). Internet Interv. 2019;18:100255.

22.

Hadjistavropoulos

Schneider

Klassen

Dear

Titov

. Development and evaluation of a scale assessing therapist fidelity to guidelines for delivering therapist-assisted Internet-delivered cognitive behaviour therapy. Cogn Behav Ther. 2018;47(6):447–461.

23.

Hadjistavropoulos

Williams

Adlam

, et al. Audit and feedback of therapist-assisted Internet-delivered cognitive behaviour therapy with routine care: a quality improvement case study. Internet Interv. 2020;20:100309.

24.

Sundström

Hadjistavropoulos

Wilhelms

Keough

Schaub

. Optimizing Internet-delivered cognitive behaviour therapy for alcohol misuse: a study protocol for a randomized factorial trial examining the effects of a pre-treatment assessment interview and health educator guidance. BMC Psychiatry 2020;20(1):126.

25.

Hadjistavropoulos

. Internet-delivered cognitive behaviour therapy for chronic conditions: comparing low intensity delivery models. ClinicalTrials.gov; 2018: NCT03500237.

26.

Mehta

Hadjistavropoulos

Nugent

Karin

Titov

Dear

. Guided Internet-delivered cognitive behaviour therapy for persons with spinal cord injury: a feasibility trial. Spinal Cord. 2020;58(5):544–552.

27.

Hadjistavropoulos

Schneider

Hadjistavropoulos

Titov

Dear

. Effectiveness, acceptability and feasibility of an internet-delivered cognitive behavioural pain management program in a routine online therapy clinic in Canada. Canadian Journal of Pain. 2018;2(1):62–73.

28.

Friesen

Hadjistavropoulos

Schneider

Alberts

Titov

Dear

. Examination of an in internet-delivered cognitive behavioural pain management course for adults with fibroymyalgia: a randomized controlled trial. Pain. 2017;158(4):593–601.

29.

Dirkse

Hadjistavropoulos

Albert

, et al. Making internet-delivered cognitive behaviour therapy scale for cancer survivors: a randomized non-inferiority trial of self-guided and technician-guided therapy. J Cancer Surviv. 2020;14(2):211–225.

30.

Schneider

Hadjistavropoulos

Dear

Titov

. Efficacy of Internet-delivered cognitive behavioural therapy following an acute coronary event: a randomized controlled trial. Internet Interv. 2020;21:100324.

31.

Owens

Hadjistavropoulos

. Examining the effectiveness of transdiagnostic internet-delivered cognitive behaviour therapy in new mothers experiencing anxiety and depression. ClinicalTrials.gov; 2019: NCT04012580.

32.

Peynenburg

Hadjistavropoulos

. Internet-delivered cognitive behaviour therapy (ICBT) for post-secondary students. ClinicalTrials.gov; 2020: NCT0426458.

33.

Titov

Dear

Nielssen

, et al. User characteristics and outcomes from a national digital mental health service: an observational study of registrants of the Australian MindSpot Clinic. Lancet Digit Health. 2020;2(11):e582–593.

34.

Chiu

Gatov

Vigod

, et al. Temporal trends in mental health service utilization across outpatient and acute care sectors: a population-based study from 2006 to 2014. Can J Psychiatry. 2018;63(2):94–102.

35.

Mohr

Riper

Schueller

. A solution-focused research approach to achieve an implementable revolution in digital mental health. JAMA Psychiatry. 2018;75(2):113–114.

36.

Kroenke

Spitzer

Williams

. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606–613.

37.

Spitzer

Kroenke

Williams

Löwe

. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006;166(10):1092–1097.

38.

Manea

Gilbody

McMilian

. Optimal cut-off score for diagnosing depression with the Patient Health Questionnaire (PHQ-9): a meta-analysis. CMAJ. 2012;184(3):191–196.

39.

Houck

Spiegel

Shear

Rucci

. Reliability of the self-report version of the panic disorder severity scale. Depress Anxiety. 2002;15(4):183–185.

40.

Allen

Newby

Mackenzie

, et al. Internet cognitive-behavioural treatment for panic disorder: randomised controlled trial and evidence of effectiveness in primary care. BJPsych Open. 2016;2(2):154–162.

41.

Peters

Sunderland

Andrews

Rapee

Mattick

. Development of a short form Social Interaction Anxiety (SIAS) and Social Phobia Scale (SPS) using nonparametric item response theory: the SIAS-6 and SPS-6. Psychol Assess. 2012;24(1):66–76.

42.

Kessler

Andrews

Colpe

, et al. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychol Med. 2002;32(6):959–976.

43.

Sheehan

. The anxiety disease. Scribner, (NY): Bantam Books. 1983.

44.

Hatcher

Gillaspy

. Development and validation of a revised short version of the working alliance inventory. Psychother Res. 2006;16(1):12–25.

45.

Karin

Dear

Heller

Gandy

Titov

. Measurement of symptom change following web-based psychotherapy: statistical characteristics and analytical methods for interpreting change. JMIR Ment Health. 2018;5(3):e10200.

46.

Hubbard

Ahern

Fleischer

, et al. To GEE or not to GEE: comparing population average and mixed models for estimating the associations between neighborhood risk factors and health. Epidemiology. 2010;21(4):467–474.

47.

Mojtabai

Form

. Trends in psychological distress, depressive episodes and mental health treatment-seeking in the United States: 2001-2012. J Affect Disord. 2015;174(C):556–561.

48.

Statistics Canada. Focus on geography series, 2016 census. Statistics Canada Catalogue. 2017:98-404-X2016001. [accessed 2020 Dec 5] https://www12.statcan.gc.ca/census-recensement/2016/as-sa/fogs-spg/Facts-PR-Eng.cfm?TOPIC=9&&GK=PR&GC=47

49.

Boksa

Joober

& Kirmayer

. Mental wellness in Canada’s Aboriginal communities: striving toward reconciliation. J Psychiatry Neurosci. 2015;40(6):363–365.

50.

Harris

Baxter

Reavley

Diminic

Pirkis

Whiteford

. Gender-related patterns and determinants of recent help-seeking for past-year affective, anxiety and substance use disorders: findings from a national epidemiological survey. Epidemiol Psychiatr Sci. 2016;25(6):548–561.

51.

Hedman

Ljótsson

Rück

, et al. Effectiveness of internet-based cognitive behaviour therapy for panic disorder in routine psychiatric care. Acta Psychiatr Scand. 2013;128(6):457–467.

52.

Newby

Mackenzie

Williams

, et al. Internet cognitive behavioural therapy for mixed anxiety and depression: a randomized controlled trial and evidence of effectiveness in primary care. Psychol Med. 2013;43(12):2635–2648.

53.

Ruwaard

Lange

Schrieken

Dolan

Emmelkamp

. The effectiveness of online cognitive behavioral treatment in routine clinical practice. PLoS One. 2012;7(7):e40089.

54.

Titov

Dear

Staples

, et al. Disorder-specific versus transdiagnostic and clinician-guided versus self-guided treatment for major depressive disorder and comorbid anxiety disorders: a randomized controlled trial. J Anxiety Disord. 2015;35:88–102.

55.

Vigo

Haro

Hwang

, et al. Toward measuring effective treatment coverage: critical bottlenecks in quality- and user-adjusted coverage for major depressive disorder. Psychol Med. 2020;1–11.

56.

World Health Organization. The impact of COVID-19 on mental, neurological and substance user services. 2020. [accessed 2020 Dec 5] https://www.who.int/publications/i/item/978924012455

57.

Reese

Duncan

Bohanske RT Owen

Minami

. Benchmarking outcomes in a public behavioral health setting: feedback as a quality improvement strategy. J Consult Clin Psychol. 2014;82(4):731–742.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.06 MB

0.22 MB