Are claims of non-inferiority of Internet and computer-based cognitive-behavioural therapy compared with in-person cognitive-behavioural therapy for adults with anxiety disorders supported by the evidence from head-to-head randomised controlled trials? A systematic review

Abstract

Objective:

This review examines the evidence from head-to-head randomised controlled trials addressing whether the efficacy of cognitive-behavioural therapy for anxiety disorders, obsessive-compulsive disorder and post-traumatic stress disorders in adults delivered by computer or online (computer- and Internet-delivered cognitive-behavioural therapy) is not inferior to in-person cognitive-behavioural therapy for reducing levels of symptoms and producing clinically significant gains at post-treatment and at follow-up. A supplementary aim is to examine the evidence for severity as a moderator of the relative efficacy of computer- and Internet-delivered cognitive-behavioural therapy and in-person cognitive-behavioural therapy.

Method:

PubMed, PsycINFO, Embase and Cochrane database of randomised trials were searched for randomised controlled trials of cognitive-behavioural therapy for these disorders with at least an in-person cognitive-behavioural therapy and Internet or computer cognitive-behavioural therapy arm.

Results:

A total of 14 randomised controlled trials (9 Internet, 5 computer) of cognitive-behavioural therapy for social anxiety disorder, panic disorder and specific phobia and 3 reports of effect moderators were included. One study showed a low risk of bias when assessed against risk of bias criteria for non-inferiority trials. The remaining studies were assessed as high or unclear risk of bias. One study found that Internet-delivered cognitive-behavioural therapy was superior and non-inferior at post-treatment and follow-up to group in-person cognitive-behavioural therapy for social anxiety disorder. One study of Internet-delivered cognitive-behavioural therapy for panic disorder showed non-inferiority to individual in-person cognitive-behavioural therapy for responder status at post-treatment and one of Internet cognitive-behavioural therapy for panic disorder for symptom severity at follow-up. Other comparisons (22 Internet, 13 computer) and for estimates pooled for Internet cognitive-behavioural therapy for social anxiety disorder, Internet cognitive-behavioural therapy for panic disorder and computer-delivered cognitive-behavioural therapy studies did not support non-inferiority. Evidence of effect moderation by severity and co-morbidity was mixed.

Conclusion:

There is limited evidence from randomised controlled trials which supports claims that computer- or Internet-delivered cognitive-behavioural therapy for anxiety disorders is not inferior to in-person delivery. Randomised controlled trials properly designed to test non-inferiority are needed before conclusions about the relative benefits of in-person and Internet- and computer-delivered cognitive-behavioural therapy can be made.

Prospero:

CRD420180961655-6

Keywords

Anxiety disorders cognitive-behavioural therapy Internet cognitive-behavioural therapy non-inferiority

Introduction

Mental disorders which are characterised by excessive anxiety either as persistent anxious apprehension or as frequent experiences of acute fear, make up a large proportion of mental health presentation in primary care and carry one of the highest burden of all mental health disorders (Whiteford et al., 2013). These disorders are classified into three broad domains (anxiety disorders, trauma- and stressor-related disorders, obsessive-compulsive and related disorders; Diagnostic and Statistical Manual of Mental Disorders [5th ed.; DSM-5]; American Psychiatric Association [APA], 2013) depending on the nature of the behavioural changes, physical difficulties and cognitive manifestations which accompany the excessive anxiety and fear. While they have distinctive features, there is good evidence that these disorders have common vulnerabilities and share underlying pathological processes which explain how the maladaptive regulation of anxiety and fear develops and is maintained (Barlow, 2002; Brown et al., 2001; Suarez et al., 2009). Cognitive-behavioural therapy (CBT) attempts to alter these underlying maintenance processes and reduce the ongoing influence of the vulnerabilities. Consistent evidence shows that CBT for fear-based anxiety disorders, post-traumatic stress disorders (PTSD) and obsessive-compulsive disorder (OCD) is superior at post-treatment to credible psychological placebos, pill placebos, care-as-usual (CAU) and waitlist comparators (Carpenter et al., 2018; Cuijpers et al., 2016). It is considered a first-line treatment (American Academy of Child and Adolescent Psychiatry [AACAP], 2012; APA, 2009; National Institute for Health and Clinical Excellence [NICE], 2005) and is recommended in preference to medication or to optimise treatment response when combined with medication for some disorders (March et al., 2004; Turner et al., 2018).

These conclusions and recommendations are based on evidence for CBT when delivered in a format where the therapist meets in person with the patient to work through the various components of the therapy. The last few decades have seen a rapid growth in the development of new delivery formats for CBT, in particular the use of computers and the Internet (Firth et al., 2016). Recent clinical practice guidelines for the treatment of anxiety disorders (Andrews et al., 2018a) support the use of digital delivery of CBT and claims that it ‘appears to be equally beneficial’ (Andrews et al., 2018b: 119) compared with CBT with a therapist. While there is evidence that computer- and Internet-delivered CBT (c/iCBT) for anxiety disorders is superior to the passage of time (waitlist control) and some active controls (CAU) in randomised controlled trials (RCTs; Andersson et al., 2014; Carlbring et al., 2018), an essential test for the efficacy of a new treatment or new version of an existing efficacious treatment is how it compares to the established best standard of care, that is, in-person CBT. Unless there are reasons to believe c/iCBT is superior to in-person CBT, studies need to ask if there is evidence that the benefits of c/iCBT are not inferior to the benefits of in-person CBT to a degree more than is offset by any advantages it confers. In other words, the observed relative benefits of in-person CBT over c/iCBT should not be greater than an allowable loss of efficacy for c/iCBT because it may, for example, be more easily accessible, reduce costs or be a better platform for help-seeking for those wanting to preserve their anonymity or to minimise stigma. This allowable loss of efficacy for c/iCBT (non-inferiority margin) sets the limits that if not exceeded establishes c/iCBT to be not inferior to in-person CBT (Schumi and Wittes, 2011).

Existing reviews of the efficacy of c/iCBT for anxiety disorders, PTSD and OCD, which have included c/iCBT compared with in-person CBT outcomes (Andersson et al., 2014; Andrews et al., 2018a; Arnberg et al., 2014; Carlbring et al., 2018), are questionable in regard to the conclusion that c/iCBT is not inferior to in-person CBT. Most notably, they rarely consider study design recommended for testing non-inferiority (CONSORT Criteria Extension for non-inferiority and equivalence randomised trials; Piaggio et al., 2012). These criteria include prior specification of the non-inferiority margin, a test of the efficacy of the active comparator and estimating sample size for a non-inferiority hypothesis and use of a range of trial analyses (per-protocol, intention-to-treat [ITT]). To date, reviews have assessed risk of bias in the included studies as if they aimed to identify treatment superiority and do not evaluate any risk of bias of the studies in respect to inferring non-inferiority. Many reviews (Andrews et al., 2018a; Carlbring et al., 2018) also draw the erroneous conclusion that inferiority or equivalence of c/iCBT compared with in-person CBT is established or indicated because the pooled estimate of the between-group effect (c/iCBT vs in-person CBT) is not significantly different from zero or its 95% confidence interval (CI) crosses zero. It is a misinterpretation to conclude that a failure to find evidence for a difference between two treatments in a trial is evidence that the treatments are equivalent or that the new treatment is not inferior to the comparator (Altman and Bland, 1995; Greene et al., 2008; Piaggio et al., 2012). None of the previous reviews have tested the pooled relative treatments effects using a non-inferiority hypothesis and against a non-inferiority margin. Using proper study designs for assessing non-inferiority protects against the widespread acceptance and dissemination of a new treatment or treatment format which lowers benefits to patients (Rief and Hofman, 2018). Therefore, it is as important to apply these designs when evaluating a new psychosocial treatment for a mental disorder as for any new medication (Greene et al., 2008).

This study uses systematic review methodology to address these limitations by identifying all studies comparing in-person CBT to c/iCBT for anxiety disorders, PTSD and OCD and applying risks of bias criteria relevant for inferring non-inferiority to identified studies. In addition, it assesses the strength of the c/iCBT compared with in-person CBT effects against the non-inferiority hypothesis, H₀: μ (in-person CBT) – μ (c/iCBT) ⩾ Δ, where Δ = non-inferiority margin. This review assesses individual study outcomes in terms of study-defined non-inferiority margin, or if no margin is specified it re-examines the evidence using the most appropriate estimate of this margin obtained from existing research of in-person CBT across many RCTs. In addition to clarifying the current state of the evidence about the relative efficacy of c/iCBT compared with in-person CBT, a secondary aim of the current review is to synthesise any evidence from the included RCTs which addresses whether the severity of the disorder moderates the relative treatment benefits of c/iCBT compared with in-person CBT. Pre-intervention symptom severity is identified as a predictor of response to CBT interventions for anxiety disorders (Turner et al., 2018) and has been suggested as a marker for beginning with a low intensity intervention such as digital CBT in stepped care models of delivery for common mental health problems (Van Straten et al., 2015). In particular, the review examines if c/iCBT is inferior to in-person CBT for patients with severe anxiety disorders but not inferior to in-person CBT for patient with mild to moderate disorders.

Aims

This study evaluates whether CBT delivered by computer or online (c/iCBT) is not inferior to CBT in-person for reducing levels of symptoms and producing clinically significant gains at post-treatment and at follow-up for adults with anxiety disorders, PTSD or OCD. A supplementary objective of the review is to examine the evidence for severity of the disorder as a moderator of the relative efficacy of c/iCBT and in-person CBT for these disorders.

Methods

Study design and protocol

The review was undertaken using the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) guidelines (Moher et al., 2009). A review protocol was developed and registered with Prospero International prospective register of systematic reviews (O’Kearney et al., 2018; Prospero: CRD42018096165).

Criteria for considering eligibility

Types of studies

Randomised control trials with at least a c/iCBT and an in-person CBT arm were included. Studies were included if they were set up to compare c/iCBT to in-person CBT regardless of whether they specify a superiority or non-inferiority of c/iCBT against CBT in-person. Studies may include additional comparators such as, waitlist, CAU or pill or psychological placebo. Any psychological placebo comparator must not include cognitive therapy or exposure strategies.

Types of participants

Adults (age ⩾ 18 years) with any anxiety disorder, PTSD or OCD as the primary diagnosis or high levels of symptoms were included with no exclusions for co-morbidity.

Types of interventions

c/iCBT are interventions which have been developed from existing disorder-specific CBT protocols or principles for the treatment of any anxiety disorder (agoraphobia [PDA], social anxiety disorder [SAD], specific phobia [SP]; panic disorder [PD]; generalised anxiety disorder [GAD]; PTSD; acute stress disorder [ASD], OCD) or from treatment protocols which take a trans-diagnostic approach to fear-based disorders and have been translated to be accessible on a computer or other electronic device either off-line or via the Internet. Included trials could evaluate CBT programmes accessible online, delivered on a desktop or laptop computer or a mobile device in a local setting, for example, primary care practice or at home. c/iCBT studies include those that are completely autonomous and those which include some clinician or assistant contact with the patients either electronically via email, messaging, telephone or face-to-face as long as the primary mode of delivery for all CBT components is by computer or Internet.

In-person CBT is disorder-specific CBT treatment protocols for any anxiety disorder, PTSD or OCD, or CBT protocols which take a trans-diagnostic approach to fear-based disorders in which CBT components are delivered to the patient by the therapist in person. These studies may include some use of electronic contact such as telephone or email as long as the primary mode of delivery is in person. CBT which is delivered by the therapist face-to-face but remotely via telelink, web camera or social media was not included.

Types of outcomes measures

Primary outcomes

Post-treatment and follow-up scores on a primary outcome measure either a disorder-specific symptom severity measure (e.g. Panic Severity Scale; Yale–Brown Obsessive-Compulsive Scale) or a general distress/anxiety measure (e.g. Depression Anxiety Stress Scale; Spielberger Anxiety Inventory).

Post-treatment and follow-up categorical outcomes (e.g. responded; clinically significant improvement, no longer symptomatic) as defined by the study or identified in the study outcomes.

Search methods

We searched the major bibliographic databases (PubMed, PsycINFO, Embase and the Cochrane database of randomised trials) by combining terms (both MeSH terms and text words) indicative of psychological treatment, Internet delivery of psychological treatment and either SAD (social phobia, social anxiety, public-speaking anxiety), GAD (worry, generalised anxiety) or PD with or without agoraphobia (panic, PD), agoraphobia (PDA), PTSD (post-trauma disorders), ASD (acute stress) and OCD with filters for RCTs. We examined the references of earlier meta-analyses on psychological treatments for the included disorders (Andersson et al., 2014; Andrews et al., 2018a; Arnberg et al., 2014; Carlbring et al., 2018). The deadline for the searches was 30 September 2018. We included all trials of a psychological treatment identified as CBT, CT (cognitive therapy) and BT (behaviour therapy) which had an arm where treatment is delivered via the Internet or computer and an in-person CBT arm.

Data collection

Selection of studies

Two reviewers examined identified abstracts for potential inclusion/exclusion and independently screened the full text studies for inclusion. All decisions were recorded and a PRISMA study flow diagram drawn.

Data extraction and management

Two reviewers independently extracted the following study characteristics and outcome data from included studies:

Methods. Reference population; recruitment method, study design (number of arms; simple or cluster randomised; crossover designs), duration, country of study, year of study; outcome reporting; superiority/non-inferiority/equivalence design; inferiority margin (a priori; how determined).

Participants. Number in each group, age, socioeconomic status and sex demographics, how diagnosis was obtained; study inclusion and exclusion criteria; co-morbidity.

Interventions. For c/iCBT amount of therapist assistance; for c/iCBT and in-person CBT total number of modules/sessions; group or individual; total hours; therapist qualification; CBT components included; method (self-directed; therapist assisted; other) and amount of exposure and cognitive therapy. For control comparator – type (CAU; waitlist; no treatment; psychological placebo; pill placebo).

Outcomes. Number lost to outcome assessment; for primary outcomes (mean [SD] post and follow-up for continuous measures for each group using ITT and/or per-protocol N; number with outcome for binary outcomes [and N]; effect sizes – if reported for ITT or per-protocol samples); number of sessions/modules completed.

Other. Funding for trial, conflict of interest; financial and/or non-financial allegiance.

Risk of bias assessment

Two reviewers independently undertook risk of bias assessment for each included study using the Cochrane Collaboration risk of bias tool (The Cochrane Collaboration, 2018) modified for assessing RCTs inferring non-inferiority (Piaggio et al., 2012). In addition to the domains of selection bias, performance bias, detection or measurement bias and reporting bias, the risk of bias assessment focused on whether the design/analysis was congruent with the study question, whether there was a priori setting of an inferiority margin, whether prior evidence was used to set the margin, whether results are reported for the analysis for ITT and per-protocol outcomes, whether the study included a test of the sensitivity of the in-person CBT and whether correct conclusions are draw from the analyses (Piaggio et al., 2012).

Analysis and data synthesis

We report effect sizes or extracted post and follow-up outcomes to estimate between-group effect sizes on the primary outcomes for the c/iCBT against in-person CBT comparison. We report bias-corrected standardised mean difference (SMD; Hedges’ g) for symptom severity outcomes and odds ratio (OR) for categorical outcomes. For studies with a non-inferiority of c/iCBT against in-person CBT hypothesis, both ITT and per-protocol effect sizes are reported when available and compared with the study identified non-inferiority margin (Δ). If no margin is specified, we used the recommendation from the Guidance Statement on Choice of Control Groups in clinical trials (International Conference on Harmonisation [ICH], 2001). These guidelines recommend that the margin should not be greater than the smallest effect size that the active treatment would be reliably expected to have compared with a credible placebo (Greene et al., 2008; Hwang and Morikawa, 1999; ICH, 2001). We set the margin (Δ) as the lower bound of the 95% CI for the pooled effect sizes reported in the most recent high quality meta-analysis of in-person CBT compared with placebo controls for the particular anxiety disorders (SAD, PD, SP), PTSD or OCD (Carpenter et al., 2018). Non-inferiority is established where the upper bound of the 95% CI for the effect size estimate comparing in-person to c/iCBT for an individual study is less than Δ (Althunian et al., 2017; Schumi and Wittes, 2011). The decision to pool study between-group outcomes was based on the number of studies in a comparison and the degree of statistical heterogeneity (95% CIs for I²). We also considered methodological heterogeneity between the studies particularly variability in the studies in their CBT components and degree of therapist involvement in c/iCBT. If pooling is done, random effects analyses were used and the pooled estimates and their 95% CIs were compared to the non-inferiority margin. We undertook sub-group analysis for severity if enough study outcomes were disaggregated by severity of symptoms (participants with mild and moderate versus high levels of symptoms).

Results

Search results

Figure 1 presents the PRISMA flow diagram for the study selection process and the reasons for exclusion. Reports were excluded if they were duplicates of other studies, were reviews, protocols of studies, did not have an in-person comparator, included participants below 18 years of age without disaggregation of the outcomes by age or did not report on any primary outcome. A total of 14 studies were identified for data extraction. Nine studies involved the comparison of Internet-delivered CBT (iCBT) against in-person CBT and five the comparison of computer-delivered CBT against in-person CBT. A total of 13 studies provided between-group effect sizes for a continuous outcome (severity of symptoms) and 11 for a categorical outcome (responder status). Two outputs from the same RCT (Haug et al., 2015; Nordgreen et al., 2016) reported on a categorical and a severity outcome, respectively.

Figure 1.

Flow of studies through the selection process.

Table 1 presents the characteristics of the included studies. The studies reported results from 1135 participants with 885 in the Internet CBT versus in-person CBT studies (n = 458; Internet CBT) and 250 in the computer CBT versus in-person comparisons (n = 135; computer CBT). Three Internet studies examined social phobia, three PD, two combined social phobia and PD and one SP. Of the computer CBT studies, two studied SP, one agoraphobia, one OCD and one mixed phobias including agoraphobia, social and SP. One of the Internet studies included a third arm (waitlist) while all of the computer studies included a control comparator (relaxation, book, waitlist). Of the Internet studies, all except Nordgreen et al. (2016) reported on a symptom severity primary outcome and all except Andrews et al. (2011) and Haug et al. (2015) reported on a categorical primary outcome. All computer studies reported for a symptom severity primary outcome and two (Gilroy et al., 2000; Heading et al., 2001) also reported on a categorical primary outcome. Delivery of in-person CBT was individual in all the studies except in Hedman et al. (2011) and Bergstrom et al. (2010) where group in-person CBT was used.

Table 1.

Characteristics of included studies.

Study ID	i/cCBT (n)	In-person CBT (n)	Other control (n)	AgeM (SD)	% female	Disorder	i/cCBT (modules)	In-person CBT (sessions/hours)	Primary outcomes
Internet
Andersson et al. (2006)	15	15	n/a	25.6 (4.1)	85	Specific phobia	5	1	SPQ; CSI
Andrews et al. (2011)	23	24	n/a	31.9 (8.8)	40.5	Social phobia	6	7	SIAS
Bergstrom et al. (2010)	53	60	n/a	34.1 (9.5)	62	Panic disorder	10	10	PDSS; RS
Botella et al. (2010)	62	36	29 (WL)	24.4 (5.8)	79.2	Social phobia	^a	4–6	SADS; RS
Carlbring et al. (2005)	25	24	n/a	34.2 (6.0)	71	Panic disorder	10	10	BSQ; CSI
Haug et al. (2015)	85	88	n/a	32.4 (8.9)	52	Social phobia/panic disorder	9/10	12	ComS
Hedman et al. (2011)	64	62	n/a	35.3 (11.3)	35	Social phobia	15	15	LSAS; CGI-I
Kiropoulas et al. (2008)	46	40	n/a	38.9 (11.1)	72	Panic disorder	6	12	PDSS; PFS
Nordgreen et al. (2016)	85	88	n/a	32.4 (8.9)	52	Social phobia/panic disorder	9/10	12	RS
Computer
Gilroy et al. (2000)	15	15	15 (relaxation)	33.1 (0.9)	100	Spider phobia	3	3	FQ; RS
Ghosh et al. (1988)	17	14	15 (book)	34.9 (^a)	81	Agoraphobia	8	10	FQ(A)
Griest et al. (2002)	55	55	66 (relaxation)	39 (12)	42	OCD	9	11	YBOCS; CGI
Heading et al. (2001)	13	14	13 (WL)	34.9 (11)	95	Spider phobia	1	1	FQ; RS
Marks et al. (2004)	35	38	17 (relaxation)	38 (12)	69	Mixed phobias	9	6	FQ

BSQ: Body Sensations Questionnaire; CGI: Clinician Global Impression Improved; ComS: Composite Severity Scale; CSI: Clinically Significant Improvement; FQ: Fear Questionnaire; FQ(A): Fear Questionnaire (Agoraphobia); i/cCBT: Internet- or computer-delivered cognitive-behavioural therapy; LSAS: Leibowitz Social Anxiety Scale; OCD: obsessive-compulsive disorder; PDSS: Panic Disorder Severity Scale; PFS: Panic Free Status; RS: Recovered Status; SADS: Social Anxiety Disorder Scale; SIAS: Social Interaction Anxiety Scale; SPQ: Spider Phobia Questionnaire; WL: waitlist; YBOCS: Yale–Brown Obsessive-Compulsive Scale.

Not reported.

Three additional reports (Andersson et al., 2008; Hedman et al., 2012; Nordgreen et al., 2016) were identified that presented data from corresponding included RCTs (Carlbring et al., 2005; Haug et al., 2015; Hedman et al., 2011, respectively) about pre-treatment factors which may differentially predict treatment response for c/iCBT and in-person CBT.

Risk of bias of included studies

The risk of bias summary for the 14 studies is provided in Table 2. In regard to the chief criteria for assessing non-inferiority conclusions, two studies (Bergstrom et al., 2010; Hedman et al., 2011) established a non-inferiority hypothesis and set a non-inferiority margin prior to the data collection. Only Hedman et al. (2011) based this on evidence from in-person compared with placebo RCTs. Bergstrom’s initial choice of ∆ = 0.20 was based on detecting a small effect but then changed to ∆ = 0.50 for power reasons although the study did not use either margin in their analysis. Nine studies were assessed as having high or unclear risk of bias in relationship to including a test of the sensitivity of the in-person CBT while all studies were assessed as having high or unclear risk of bias associated with insufficient analysis to test non-inferiority. Hedman et al. (2011) study was assessed as low risk of interpretation bias for inferring non-inferiority of iCBT and Ghosh et al. (1988), Gilroy et al. (2000), Griest et al. (2002) and Heading et al. (2001) were assessed as low risk of bias when inferring superiority of in-person CBT. All the other studies have a high or unclear risk of interpretation bias.

Table 2.

Risk of bias summary.

	Random sequence generation (selection bias)	Allocation concealment (selection bias)	Blinding of participants and personnel (performance bias)	Blinding of outcome assessment (detection bias)	Incomplete outcome data (attrition bias)	Selective reporting (reporting bias)	Other bias	Non-inferiority Ho and margin	Includes comparator	Proper analysis	Interpretation bias
Andersson et al. (2006)
Andrews et al. (2011)
Bergstrom et al. (2010)
Botella et al. (2010)
Carlbring et al. (2005)
Ghosh et al. (1988)
Gilroy et al. (2000)
Griest et al. (2002)
Haug et al. (2015)
Heading et al. (2001)
Hedman et al. (2001)
Kiropoulas et al. (2008)
Marks et al. (2004)
Nordgreen et al. (2016)

Relative effects of the interventions

We grouped the computer and Internet studies separately in the presentation of each primary outcome because the CBT procedures in the computer studies focused on exposure and were mostly office based while the Internet studies included a wider range of CBT procedures and were home based. In addition, as all the Internet studies had CBT protocols designed as disorder-specific, they were grouped according to disorder (SAD, PD, SP) in the presentation of the results.

Post-treatment outcomes

Figure 2 presents the forest plot for the between-group (in-person CBT – c/iCBT) effect size estimates for symptom severity at post-treatment (Hedges’ g). The sign of the effect sizes have been adjusted to show the strength of the effect in favour of in-person CBT effect as positive. Figure 3 presents the forest plot for effect size estimates for the proportion of participants who were responders to treatment at post-treatment (i/cCBT/in-person CBT, OR) and the 95% CI around these estimates. All outcome data are ITT as per-protocol outcomes could not be extracted reliably from the studies. Pooled effect sizes for the sub-groups are also presented but need to be interpreted with caution because of small sample size in many studies (Lin, 2018), small number of studies in the sub-groups, notable methodological heterogeneity within sub-groups and additional statistical heterogeneity for the SAD Internet studies.

Figure 2.

Forest plot of SMD for comparison of c/iCBT to in-person CBT for severity of symptoms at post-treatment with non-inferiority margin (Δ = 0.26) marked.

Figure 3.

Forest plot of OR for comparison of c/iCBT to in-person CBT far recovered status at post-treatment, with non-inferiority margin (Δ = 1.64) marked.

Severity of symptoms at post-treatment

Of the two studies which assessed outcomes against a study-justified non-inferiority margin, Hedman et al. (2011) found evidence of a superiority of iCBT compared to in-person group CBT (SMD = –0.38, 95% CI = [–0.73, –0.03], Δ = 0.38), while Bergstrom et al.’s (2010) results did not support a conclusion of non-inferiority for iCBT compared to group CBT for PD (SMD = –0.02, 95% CI = [–0.39, 0.34], Δ = 0.20). Heading et al. (2001) found evidence of a superiority of in-person CBT over computer-delivered CBT for spider phobia (SMD = 0.94, 95% CI = [0.13, 1.74]) while Haug et al. (2015) results showed a superior outcome for in-person CBT over the Internet CBT component of stepped care for SAD and PD (SMD = 0.30, 95% CI = [0.0, 0.60]).

Figure 2 includes the larger of the non-inferiority margins (Δ) for symptom severity identified as the lower bound of the 95% CI for the pooled effect sizes reported in Carpenter et al. (2018; for SAD Δ = 0.26; Hedges’ g from 11 studies = 0.48; 95% CI = [0.26, 0.71]; for PD Δ = 0.12; Hedges’ g from four studies = 0.38; 95% CI = [0.12, 0.65]). The upper bounds of the 95% CI for all the studies except Hedman et al. (2011) were greater than Δ. The upper 95% CI of the pooled sub-group estimates for computer studies, Internet SAD studies and Internet PD were also greater than both non-inferiority margins.

Responder status at post-treatment

No study assessed responder status outcome against a study-defined non-inferiority margin for the OR. Hedman et al. (2011) found evidence of a superiority of iCBT compared with in-person group CBT for responder status at post-treatment (OR = 0.64, 95% CI = [0.31, 1.30]). Griest et al. (2002) found evidence of a superiority of in-person CBT over computer-delivered CBT for OCD in terms of proportion of treatment responders (OR = 2.43, 95% CI = [1.13, 5.22]). Nordgreen et al. (2016) provided evidence of a superiority of in-person CBT compared with the Internet CBT component of stepped care for PD for treatment response (OR = 2.34, 95% CI = [1.08, 5.09]) while Andersson et al.’s (2008) results showed a superiority of in-person CBT over Internet CBT for spider phobia in proportion of treatment responders (OR = 6.00, 95% CI = [1.17, 30.72]).

Figure 3 includes the larger of the non-inferiority margins for responder status outcome (Δ) identified from the pooled OR reported in Carpenter et al. (2018; for SAD Δ = 1.64; OR from 11 studies = 3.51; 95% CI = [1.64, 7.53]; for PD Δ = 1.33; OR from four studies = 2.10; 95% CI = [1.13, 3.90]). The upper bounds of the 95% CI for each of the computer studies are greater than Δ. For the Internet SAD studies, Hedman et al.’s (2011) results supported non-inferiority (OR = 0.64, 95% CI = [0.31, 1.30], Δ = 1.64) and for the Internet PD studies Carlbring et al.’s (2005) results supported non-inferiority (OR = 0.27, 95% CI = [0.06, 1.19], Δ = 1.64). The pooled sub-group estimates did not support non-inferiority.

Follow-up outcomes

Figure 4 presents the forest plot for the between-group (in-person CBT – c/iCBT) effect size estimates for symptom severity at follow-up (SMD) and Figure 5 the forest plot for responder status at follow-up (OR) and the 95% CI around these estimates. Because consistent estimates for the effects size of in-person CBT compared with placebos at follow-up are not available, we took a conservative approach to estimating the non-inferiority margins at follow-up and assumed no loss of relative efficacy of in-person CBT compared with placebos from post to follow-up.

Figure 4.

Forest plot of SMD for comparison of c/iCBT to in-person CBT for severity of symptoms at follow-up, with non-inferiority margin (Δ = 0.26) marked.

Figure 5.

Forest plot of OR for comparison of c/iCBT to in-person CBT for recovered status at follow-up, with non-inferiority margin (Δ = 1.64) marked.

Severity of symptoms at follow-up

Nine studies (Andersson et al., 2006; Bergstrom et al., 2010; Botella et al., 2010; Carlbring et al., 2005; Ghosh et al., 1988; Gilroy et al., 2000; Haug et al., 2015; Heading et al., 2001; Hedman et al., 2011) reported between-group effects for symptom severity at follow-up. For the Internet SAD studies, Hedman et al.’s (2011) results supported non-inferiority (SMD = –0.37, 95% CI = [–0.73, –0.02], Δ = 0.26) and for the Internet PD studies, Bergstrom et al.’s (2010) results supported non-inferiority (SMD = –0.19, 95% CI = [–0.51, 0.18], Δ = 0.26). The individual results from the others studies and the pooled estimates do not support non-inferiority (>Δ = 0.26).

Responder status at follow-up

Six studies (Andersson et al., 2006; Botella et al., 2010; Carlbring et al., 2005; Gilroy et al., 2000; Heading et al., 2001; Hedman et al., 2011) reported responder status outcome at follow-up. All except Hedman et al. (2011), which shows a superiority for iCBT over in-person CBT, report upper bounds of the 95% CI around their OR estimates of greater than 4 (>Δ = 1.64).

Severity as a moderator of relative treatment response

Table 3 presents the outcomes for the three studies which report results for moderation effects for disorder severity or co-morbidity. Each of the studies used data from a RCT to examine whether symptom severity and co-morbidity at pre-treatment differentially predicts outcome for iCBT and in-person CBT. Hedman et al. (2012) reported that higher levels of general anxiety were associated with higher social anxiety scores at post-treatment and with higher likelihood of having a diagnosis of SAD at 6-month follow-up in iCBT but did not predict response to in-person CBT. In addition, Hedman et al. (2012) reported that absence of co-morbid depression is associated with better response in the iCBT group at post-treatment but not in in-person CBT. Haug et al. (2015) showed that that there is no difference between outcomes for iCBT and in-person CBT for SAD and PD patients with depression but for those without depression in-person CBT is superior to iCBT. Andersson et al. (2006) presented evidence that for PD patients more severe interference in mobility at pre-treatment predicted poorer treatment response for in-person CBT but not for iCBT while cluster C personality traits predicted a poorer response in iCBT but not in-person CBT.

Table 3.

Studies reporting moderator effects for symptoms severity or co-morbidity on relative benefits of in-person CBT and iCBT and the reported outcomes.

Study	Moderator	Effect
Hedman et al. (2012)	Level of general anxiety	At post: lower levels of general anxiety associated with lower social anxiety scores for iCBT but not in-person CBT (β = 0.23, SE = 1.59, p < 0.001). At 6-month follow-up: lower levels of general anxiety associated with a diagnosis of SAD for iCBT but not in-person CBT (β = 0.34, SE = 0.40, p < 0.01).
	Presence of co-morbid depression	At post: Poorer response in iCBT group, but not in-person CBT.
Haug et al. (2015) (for SAD and PD)	Co-morbid depression	With co-morbid depression: no difference between outcomes for iCBT and in-person CBTWithout co-morbid depression: in-person CBT is superior to iCBT (β = 0.58, SE = 0.26, p = 0.024).
Andersson et al. (2006) (for PD)	Reduced mobility	Poorer treatment response in ACQ, BSQ for in-person CBT (r = –0.61, r = –0.55, respectively), but not for iCBT (r = 0.05, r = 0.04).
	Cluster C personality	Poorer treatment response in ACQ, BSQ for iCBT (r = –0.48, r = –0.39), but not for in-person CBT (r = 0.31, r = 0.20).

ACQ: Agoraphobia Cognitions Questionnaire; BSQ: Body Sensations Questionnaire; CBT: cognitive-behavioural therapy; iCBT: Internet CBT; PD: panic disorder; SAD: social anxiety disorder.

Discussion

For adults with anxiety disorders, PTSD and OCD CBT delivered in person has been shown to be consistently effective compared with credible placebos (Carpenter et al., 2018; Cuijpers et al., 2016). There has been extensive recent efforts to adapt CBT for these disorders so it can be delivered by Internet or computer to increase access, reduce costs and improve acceptability and the fidelity of delivery. This study examined the evidence from RCTs addressing whether the benefits in symptom reduction and clinically significant gains from CBT delivered by computer and online (c/iCBT) are not inferior to those of CBT in-person.

There are two general conclusions from the current review. First, the vast majority of studies from which non-inferiority or equivalence has been inferred have not been properly designed to establish such conclusions. One study (Hedman et al., 2011) showed low risk of bias as a non-inferiority trial while all other studies were not consistent with the recommended CONSORT criteria (Piaggio et al., 2012). Most critically, these studies did not establish a proper non-inferiority hypothesis and did not set a non-inferiority margin. Their conclusions often repeat the widespread problem (Altman and Bland, 1995; Greene et al., 2000) of misinterpreting a failure to reject a null hypothesis as evidence of equivalence or non-inferiority. Second, the results of 12 out of 13 studies for symptom severity at post-treatment and of 11 out of 13 for responder status at post-treatment do not support a conclusion of non-inferiority when the effect size estimates from the studies are tested against the established lowest estimate of the true effects of in-person CBT against credible placebos (Carpenter et al., 2018). Hedman et al. (2011) found superiority and non-inferiority of iCBT for symptom severity and responder status at post-treatment for SAD while Carlbring et al. (2005) found evidence of non-inferiority of iCBT for PD for responder status at post-treatment. Hedman et al. (2011) for SAD and Bergstrom et al. (2010) for PD reported evidence supporting non-inferiority of iCBT at follow-up. None of the pooled estimates for the computer-delivered studies and for Internet studies for SAD and PD support a conclusion of non-inferiority. It is important to note that the effects for some studies and for the pooled effects for the in-person CBT compared with computer delivery studies indicate that in-person CBT has superior benefits.

These conclusions strongly indicate that, based on evidence from head-to-head RCTs, it is premature to suggest that the treatment effects of CBT for anxiety disorders delivered by Internet or computer are equivalent to or not less than those of therapist-delivered CBT, or that Internet CBT with minimal therapist contact is an equivalent first-line treatment to in-person CBT (Andersson et al., 2014; Andrews et al., 2018a, 2018b; Carlbring et al., 2018). As Internet delivery in particular offers attractive advantages in terms of dissemination, access and cost, it is very important to properly assess the extent of any treatment benefits which may be given up in using the new mode of delivery relative to in-person CBT. Our review indicates that with one exception (Hedman et al., 2011), this has not been addressed in properly conducted non-inferiority RCTs of CBT for anxiety disorders, PTSD or OCD. The need for these studies in regard to Internet CBT is further underlined by the rapid expansion of applications aimed at translating CBT to digital platforms in the absence of a regulatory framework for ensuring quality and protecting consumers.

The studies identified for our analysis were of Internet CBT which involved none or minimal therapist guidance and computer CBT which used intensive exposure in a few sessions. Internet CBT ranged from 6 to 15 modules with therapist involvement ranging from none to minimal (therapist feedback by email; 10 minutes by phone after each module or a single psychoeducation session). Modules included a variety of CBT components focusing on psychoeducation and instructions in exposure and cognitive therapy while computer-delivered CBT were usually intensive exposure focused treatments of one to three sessions. There has been a recent increase in studies which report on blended Internet CBT and in-person delivery which while predominantly digitally delivered involve significantly more therapist input than the studies included in the current review. As far as we are aware, none of these have been RCTs which compared a more blended mode of delivery with traditional in-person CBT for anxiety disorders. The overall conclusions from the review are also restricted by the features of the included studies. Most of the evidence comes from studies of patients with social anxiety, PD and SP. We identified only one study of OCD patients using computer delivery and none for PTSD or GAD. Two studies which showed non-inferiority (Bergstrom et al., 2010; Hedman et al., 2011) used group in-person CBT and one (Carlbring et al., 2005) used individual in-person CBT.

We identified three studies which provide evidence from RCTs about factors which may predict a differential treatment response to in-person compared with Internet- or computer-delivered CBT. Hedman et al. (2012) findings are consistent with the suggestion that pre-treatment disorder severity and co-morbidity may predict a poorer outcome in iCBT but findings from Haug et al. (2015) with regard to co-morbid depression and Andersson et al. (2006) for higher functional impairment are not consistent with this suggestion. Higher level of cluster C personality traits were associated with poorer treatment outcome in iCBT but not in-person CBT (Andersson et.al. 2008). These data are difficult to integrate and suggest considerable additional work is required to provide more confidence about inferring for which patients with anxiety disorders Internet CBT should be recommended when in-person CBT is available. Given the small number of studies and their small samples, the most productive direction for this work might be by merging person level data from existing studies which have compared in-person CBT to Internet CBT rather than using aggregated data in meta-regression. As attempts to use technologies in psychological treatments increase, providing high quality evidence about factors which may moderate the relative efficacy of in-person CBT and the new modes of delivering CBT is of critical importance to optimise outcomes for individuals and maximise the overall effectiveness of the health care system.

There are a number of potential limitations and caveats which are important in assessing the review’s conclusions. First, only head-to-head comparisons were included in the review because these are the most direct tests with the highest validity for drawing inferences about relative efficacy. We acknowledge that there are other RCTs of digitally delivered CBT for anxiety disorders which support its efficacy relative to waitlist and treat-as-usual comparators. Indirect comparisons of these effects to results of similar in-person CBT studies, however, have limited utility for inferring non-inferiority or equivalence because it is not possible to estimate the impact of variations in how the trials were implemented on possible real differences in the efficacy of the two modes of delivery. Second, because only one study properly specified a non-inferiority margin, our statistical tests of the evidence for non-inferiority are post hoc and may introduce bias. We offset the potential bias of post hoc analysis by using a strongly empirical approach to setting the margins based on formal statistical considerations using evidence about the consistent performance of in-person CBT against credible placebos. This is the recommended approach (ICH, 2000; Schumi and Wittes, 2011) and the margins used can be easily verified (Carpenter et al., 2018). We compensated for the conservative nature of this approach by using the larger of the relevant disorder-specific estimates and by not reducing this margin further by applying the preserved fraction as is often recommended (Althunian et al., 2017). The risks in using the margins adopted here include problems in the constancy of the effect as the efficacy of in-person CBT measured in RCTs has decreased over time, the limited precision in the pooled OR estimates from the Carpenter et al. (2018) study and the lack of provision for quantifying the benefits attributable to the new mode of delivery. It is arguable that because of the benefits digitally delivered CBT provides in terms of cost, access and ease of use, it is reasonable to accept a larger non-inferiority margin. It is important, however, for researchers to be aware that in making this decision that there needs to be a compelling clinical rationale provided supported by the evidence of the strengths of these advantages. At the same time, our review highlights the lack of a proper rationale for inferring non-inferiority in most of the head-to-head trials as well as the challenges of future studies aimed at evaluating the efficacy of new modes of delivery of CBT. Other limitations to the review come from the features of the studies included in the review. As noted above, there are limitations to conclusions because of the small range of disorders studied, the small sample sizes, the exclusion of Internet and computer CBT which had more than minimal clinician assistance, variability in the type of in-person CBT which includes group and individual delivery and lack of a consistent way of defining responder status in the included studies.

Overall, this review shows that the current best evidence does not support claims that CBT for anxiety disorders adapted to be delivered by Internet or computer is not inferior to in-person delivery. At best, the evidence from RCTs is inconclusive in regard to whether these modes of delivery are ‘equally beneficial’ (Andrews et al., 2018b) because only one properly designed study has been conducted. It is recommended that future studies improve the rigour of the evaluation of Internet- or computer-delivered CBT compared with in-person CBT for anxiety disorders because establishing non-inferiority of a new treatment or mode of treatment delivery helps clinicians and policy makers ensure that a less efficacious treatment is not accepted as the standard of care.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Richard O’Kearney

References

Althunian

de Boer

Groenwold

, et al. (2017) Defining the noninferiority margin and analysing noninferiority: An overview. British Journal of Clinical Pharmacology 83: 1636–1642.

Altman

Bland

(1995) Absence of evidence is not evidence of absence. British Medical Journal 311: 485.

American Academy of Child and Adolescent Psychiatry (AACAP) (2012) Practice parameters for the assessment and treatment of children and adolescents with obsessive compulsive disorder. Journal of the American Academy of Child and Adolescent Psychiatry 51: 98–113.

American Psychiatric Association (APA) (2009) Practice guideline for the treatment of patients with panic disorder. The American Journal of Psychiatry 155: 1–34.

American Psychiatric Association (APA) (2013) The Diagnostic and Statistical Manual of Mental Disorders, 5th Edition. Arlington, VA: APA.

#Andersson

Carlbring

Grimlund

(2008) Predicting treatment outcome in internet versus face to face treatment of panic disorder. Computer in Human Behavior 24: 1790–1801.

*Andersson

Carlbring

Holmström

, et al. (2006) Internet-based self-help with therapist feedback and in-vivo group exposure for social phobia: A randomized controlled trial. Journal of Consulting and Clinical Psychology 74: 677–686.

Andersson

Cuijpers

Carlberg

, et al. (2014) Guided internet-based vs. face-to-face cognitive behaviour therapy for psychiatric and somatic disorders: A systematic review and meta-analysis. World Psychiatry 13: 288–295.

Andrews

Bell

Boyce

, et al. (2018a) Royal Australian and New Zealand college of psychiatrists clinical practice guidelines for the treatment of panic disorder, social anxiety disorder and generalised anxiety disorder. Australian and New Zealand Journal of Psychiatry 52: 1109–1172.

10.

Andrews

Cuijpers

Craske

, et al. (2018b) Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: An updated meta-analysis. Journal of Anxiety Disorders 55: 70–78.

11.

*Andrews

Davies

Titov

(2011) Effectiveness randomised controlled trial of face to face versus internet cognitive behaviour therapy for social phobia. Australian and New Zealand Journal of Psychiatry 45: 337–340.

12.

Arnberg

Linton

Hultcrantz

, et al. (2014) Internet-delivered psychological treatments for mood and anxiety disorders: A systematic review of their efficacy, safety, and cost-effectiveness. PLoS ONE 9: e98118.

13.

Barlow

(2002) Anxiety and Its Disorders: The Nature and Treatment of Anxiety and Panic, 2nd Edition. New York: Guilford Press.

14.

*Bergstrom

Andersson

Ljótsson

, et al. (2010) Internet-versus group-administered cognitive behaviour therapy for panic disorder in a psychiatric setting: A randomised trial. BMC Psychiatry 10: 54–64.

15.

*Botella

Gallego

Garcia-Palacios

, et al. (2010) An internet based self-help treatment for fear of public speaking: A controlled trial. Cyberpsychology, Behavior and Social Networking 13: 407–431.

16.

Brown

Campbell

Lehman

, et al. (2001) Current and lifetime comorbidity of the DSM-IV anxiety and mood disorders in a large clinical sample. Journal of Abnormal Psychology 110: 49–58.

17.

Carlbring

Andersson

Cuijpers

, et al. (2018) Internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: An updated systematic review and meta-analysis. Cognitive Behaviour Therapy 47: 1–18.

18.

*Carlbring

Nilsson-Ihrfelt

Waara

, et al. (2005) Treatment of panic disorder: Live therapy vs. self-help via Internet. Behaviour Research and Therapy 43: 1321–1333.

19.

Carpenter

Andrews

Witcraft

, et al. (2018) Cognitive behavioral therapy for anxiety and related disorders: A meta-analysis of randomized placebo-controlled trials. Depression and Anxiety 35: 502–514.

20.

Cuijpers

Cristea

Karyotaki

, et al. (2016) How effective are cognitive behavior therapies for major depression and anxiety disorders? A meta-analytic update of the evidence. World Psychiatry 153: 245–258.

21.

Firth

Torous

Yung

(2016) Ecological momentary assessment and beyond: The rising interest in e-mental health research. Journal of Psychiatric Research 80: 3–4.

22.

*Ghosh

Marks

Carr

(1988) Therapist contact and outcomes of self-exposure treatment for phobias: A controlled study. British Journal of Psychiatry 151: 234–238.

23.

*Gilroy

Kirby

Daniels

, et al. (2000) Controlled comparison of computer aided vicarious exposure versus live exposure in the treatment of spider phobia. Behavior therapy 31: 733–744.

24.

Greene

Morland

Durkalski

, et al. (2008) Noninferiority and equivalence designs: Issues and implications for mental health research. Journal of Traumatic Stress 21: 433–439.

25.

Greene

Concato

Feinstein

(2000) Claims of equivalence in medical research: Are they supported by the evidence? Annals of Internal Medicine 132: 715–722.

26.

*Griest

Marks

Baer

, et al. (2002) Behaviour therapy for obsessive-compulsive disorder guided by a computer or by a clinician compared with relaxation as a control. The Journal of Clinical Psychiatry 63: 138–145.

27.

*#Haug

Nordgreen

Ost

L-G

, et al. (2015) Stepped care versus face-to-face cognitive behaviour therapy for panic disorder and social anxiety disorder. Behaviour Research and Therapy 71: 76–89.

28.

*Heading

Kirby

Martin

, et al. (2001) Controlled comparison of single session treatments for spider phobia: Live graded exposure alone versus computer aided vicarious exposure. Behaviour Change 18: 103–113.

29.

#Hedman

Andersson

Ljotsson

, et al. (2012) Clinical and genetic outcome determinants of Internet- and group-based cognitive behaviour therapy for social anxiety disorder. Acta Psychiatrica Scandinavica 126: 126–136.

30.

*Hedman

Andersson

Ljotsson

, et al. (2011) Internet-based cognitive behavior therapy vs. cognitive behavioral group therapy for social anxiety disorder: A randomized controlled non-inferiority trial. PLoS ONE 6: e18001.

31.

Hwang

Morikawa

(1999) Design issues in non-inferiority/equivalence trials. Drug Information Journal 33: 1205–1218.

32.

International Conference on Harmonisation (ICH) (2001) International Conference on Harmonisation; choice of control group and related issues in clinical trials; availability. Notice. Federal Register 66: 24309–24391.

33.

*Kiropoulas

Klein

Austin

, et al. (2008) Is internet-based CBT for panic disorder and agoraphobia as effective as face-to-face CBT? Journal of Anxiety Disorder 22: 1273–1284.

34.

Lin

(2018) Bias caused by sampling error in meta-analysis with small sample sizes. PLoS ONE 139: e0204056.

35.

March

Foa

Gammon

, et al. (2004) Cognitive-behavior therapy, sertraline, and their combination for children and adolescents with obsessive-compulsive disorder. The Journal of the American Medical Association 292: 1969–1976.

36.

*Marks

Kenwright

McDonough

, et al. (2004) Saving clinicians’ time by delegating routine aspects of therapy to a computer: A randomized controlled trial in phobia/panic disorder. Psychological Medicine 34: 9–18.

37.

Moher

Liberati

Tetzlaff

Altman

, The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6: e1000097. DOI:10.1371/journal.pmed1000097

38.

National Institute for Health and Clinical Excellence (NICE) (2005) Obsessive-Compulsive Disorder: Core Interventions in the Treatment of Obsessive-Compulsive Disorder and Body Dysmorphic Disorder: Clinical Guideline 31. London: NICE.

39.

*Nordgreen

Haug

Ost

L-G

, et al. (2016) Stepped care versus direct face-to-face cognitive behavior therapy for social anxiety disorder and panic disorder: A randomized effectiveness trial. Behavior therapy 47: 166–183.

40.

O’Kearney

Calear

Dawson

, et al. (2018) Internet cognitive behavioural therapies versus in-person cognitive behavioural therapies for adults with anxiety disorders; OCD and PTSD: A protocol for a systematic review. Prospero CRD42018096165.

41.

Piaggio

Elbourne

Pocock

, et al. (2012) Reporting of noninferiority and equivalence randomized trials: Extension of the CONSORT 2010 statement. The Journal of the American Medical Association 308: 2594–2604.

42.

Rief

Hofman

(2018) Some problems with non-inferiority tests in psychotherapy research: Psychodynamic therapies as an example. Psychological Medicine 48: 1392–1394.

43.

Schumi

Wittes

(2011) Through the looking glass: Understanding non-inferiority. Trials 12: 106.

44.

Suarez

Bennett

Goldstein

, et al. (2009) Understanding anxiety disorders from a ‘triple vulnerabilities’ framework. In: Antony

Stein

(eds) Oxford Handbook of Anxiety and Related Disorders. New York: Oxford University Press, pp. 153–171.

45.

The Cochrane Collaboration (2018) Cochrane Handbook of Systematic Reviews of Interventions (Version 6). London: The Cochrane Collaboration.

46.

Turner

O’Gorman

Nair

, et al. (2018) Moderators and predictors of response to cognitive behaviour therapy for pediatric obsessive compulsive disorder: A systematic review. Psychiatry Research 261: 50–60.

47.

Van Straten

Hill

Richards

, et al. (2015) Stepped care treatment delivery for depression: A systematic review and meta-analysis. Psychological Medicine 45: 231–246.

48.

Whiteford

Degenhardt

Rehm

, et al. (2013) Global burden of disease attributable to mental and substance use disorders: Findings from the global burden of disease study 2010. The Lancet 382: 1575–1586.