Abstract
A promising strategy for improving access to treatment for anxiety and depression involves administering psychological treatments via computers or the Internet [1–4]. Such programmes are increasingly common, often based on cognitive behavioural therapy (CBT), and highly structured. Computerized CBT (cCBT) or Internet-based CBT (iCBT) programmes comprise multiple components including online lessons, homework assignments, and may also include regular communication with a therapist via email, telephone, or online forums [5–6]. Recent meta-analyses of iCBT and cCBT for depression [5] and anxiety disorders [7] indicate that such treatments are clinically efficacious and effective. The reported effect size superiority over control conditions is large in anxiety disorders (23 studies, Cohen's d = 1.1) [7], and moderate in clinician guided programmes for depression (d = 0.61) [5]. Significant clinical benefits have also been reported for technician guided iCBT programmes for depression [8] and generalized anxiety disorder [9], and for CCBT programmes for depression [10]. These results raise important questions about the optimum combination of staff resources, treatment components, and procedures required to result in clinically significant improvements, while maintaining both acceptability to consumers and cost effectiveness [11].
The present study is the seventh in a research programme examining the clinical efficacy and effectiveness of an iCBT programme (the Shyness programme) for people with social phobia [12–18]. The six previous trials indicate that the Shyness programme reliably produces significant clinical benefits for people with social phobia when guided by either a clinician or technician, with one study [16] reporting the efficacy of a self-guided version. For the purposes of this paper, self-guided refers to programmes that do not involve any contact or guidance with researchers or clinicians during the duration of the programme [19]. Self-guided iCBT programmes are eminently scalable and cost-effective and may represent an important first step for many people troubled by symptoms, but who are reluctant to seek professional help.
The available data indicate that trials of open access programmes, that is, those available from public websites and that do not involve recruitment into a clinical trial are associated with low completion rates and small clinical benefits [20–21]. Meta-analyses of iCBT, cCBT and written bibliotherapy interventions for depression conclude that the mean effect size superiority over control conditions for unassisted programs is low [3,5,19]. In contrast, unreplicated clinical trials of self-guided programmes for those with symptoms of post-traumatic stress disorder [22], or diagnoses of panic disorder [23], and social phobia [16], have reported clinical outcomes comparable to clinician or technician guided programmes.
The present study extends the results of earlier studies exploring the efficacy of a self-guided version of the Shyness programme. In our first attempt (Shyness 3) [14], 33% of participants completed the self-guided programme and obtained a mean within-group effect size of 0.66 across two measures of social phobia. That version was subsequently enhanced (Shyness 4) [16] with the addition of automatic emails, which were sent to participants when they completed lessons, to remind them of uncompleted lessons, to reinforce their adherence to the programme, and to notify them of new resources. In Shyness 4, 68% of participants completed the programme and using an intention to treat model of analysis obtained a mean within-group effect size of 0.86 across two measures of social phobia, which compares favourably with face-to-face programmes [24–25].
The present study had two aims: to replicate the results obtained in Shyness 4, and, in order to try to improve adherence to the programme, to explore the effect of adding techniques based on motivational interviewing strategies (MI). Motivational interviewing enhances motivation for clinical change by exposing and resolving ambivalence to change. It is a client-centred, directive treatment originally developed in the addictions domain, but has been recently applied to anxiety and depression [26] with encouraging preliminary findings as a pre-treatment for people enrolled in face-to-face cognitive behavioural therapy for generalized anxiety disorder [27] and social phobia [28]. A version of a popular manual for motivational interviewing for anxiety disorders [29] was modified for use in a self-guided online programme to explore the impact of motivational strategies on completion rates and clinical outcomes. MI is usually administered in an interactive manner and relies on empathy, reflective listening and rolling with resistance on the part of the clinician. However, for technical reasons in the present study motivational strategies were administered in the form of written questions appended to online lessons and homework assignments.
We hypothesized that the results of the earlier trial would be replicated, and that people who were provided with motivational strategies would show significantly higher completion rates and greater improvements on measures of social phobia.
Method
Participants
Applicants applied online to www.virtualclinic.org.au during 3 weeks in August/September 2009 (details of participant flow are in Figure 1). Of the 131 who applied, 118 met inclusion criteria used in Shyness 6 [18] and were contacted and administered the social phobia section of the Mini International Neuropsychiatric Interview Version 5.0.0 (MINI) [30] during a telephone interview to determine whether they meet DSM-IV [31] criteria for social phobia. One hundred and thirteen applicants did so and were randomly allocated to the iCBT (n = 56) or iCBT+MS (n = 57) group.

Shyness 7 Study flow chart. PHQ-9, Patient Health Questionnaire – 9 item; MINI, Mini International Neuropsychiatric Interview; DSM-IV, Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition; iCBT, Internet (self-guided) CBT group; iCBT + MS, Internet (self-guided) CBT plus motivational strategies group.
Interventions
The Shyness programme comprises six online lessons; a summary/homework assignment for each lesson; comments by prior participants about each lesson; access to supplementary materials; and automatic emails. Part of the content of each lesson is presented in the form of an illustrated story about a young man with social phobia who, with the help of a clinical psychologist, successfully gains mastery over his symptoms. Lessons 1 and 2 provide education about the symptoms and treatment of social phobia, lesson 3 provides instructions on how to develop an exposure hierarchy and about practising graded exposure, lessons 4 and 5 reinforce principles of graded exposure and demonstrate principles of cognitive restructuring, while lesson 6 comprises information about relapse prevention. Each lesson also includes a downloadable document which summarizes the key messages of that lesson.
All participants completed two additional education lessons before they began the Shyness programme. Participants in the iCBT+MS group saw the same educational lessons as the iCBT group plus additional material appended to the two educational lessons and to the homework assignments for those lessons. This additional material was derived from the manual for motivational interviewing adapted for anxiety by Westra and Dozois (2003) [29]. The common components of education lesson 1 comprised information about the prevalence and impact of common anxiety disorders, while the additional components seen only by those in the iCBT+MS group comprised reflective questions about i) costs and benefits of participating in the Shyness programme, ii) the participant's values and how their symptoms affect attainment of those values, iii) their reasons for participating, iv) how their life might be in five years if they didn't participate in treatment, and v) their personal strengths. The common components of education lesson 2 comprised information about the common symptoms of anxiety disorders, about the principles of cognitive behavioural therapy, and about realistic expectations of treatment. The additional components seen only by the iCBT+MS group comprised reflective questions about i) the participant's goals for participating, ii) obstacles that could affect participation, iii) techniques they would use to overcome obstacles, and iv) reflection on personal strengths. Participants were given a timetable encouraging them to complete the educational lessons within the first two weeks of the programme and then to complete the six lessons of the Shyness programme within the following nine weeks. This allowed participants 11 weeks in total to complete all eight lessons. Participants did not receive any contact with the researchers once the programme began, except for advice about browser connectivity. All participants received approximately 25 automatic email messages during the 11 week programme. These were triggered when they completed a lesson, to remind them to complete a lesson, or to inform them about materials that had become available.
Outcomes
Outcome measures were administered at pre-treatment, one week post-treatment, and 3 months post-treatment. Applicants completed the Social Interaction Anxiety Scale [32] (SIAS); the Social Phobia Scale [32] (SPS); the Patient Health Questionnaire 9-item [33] (PHQ-9), the Kessler 10-item [34] (K-10); and the Sheehan Disability Scales [35] (SDS). Applicants completed a demographic questionnaire and the Credibility/Expectancy Questionnaire (CEQ) [36] to explore pre-treatment differences in expectancies or perception of treatment credibility between groups. Changes in the SIAS and SPS were considered the primary social phobia outcome measures, changes in the PHQ-9, K-10, and SDS were the secondary outcome measures. At post-treatment, participants completed the above questionnaires, a 7-item treatment satisfaction questionnaire based on the Credibility/Expectancy Questionnaire, 3 questions rated on a 10 point scale enquiring how much the educational lessons affected their understanding about their symptoms, their motivation to participate, or their determination to overcome obstacles. Additional questions enquired about the amount of time spent by participants during the programme, and process measures included information about number of log-ins and homework downloads. At 3 month follow up participants completed the primary and secondary outcome measures.
Sample size
The power for the within-group comparisons were estimated based on a conservative effect size of (Cohen's) d = 0.8, and the sample sizes in each group were regarded as sufficient to detect a within-group effect of this size, with additional numbers to hedge against attrition. Given the previous literature we considered a mean standardized difference at or below d = 0.2 as the criteria for equivalence for the main outcome measures, the SIAS and SPS. We were not able to power the study for the reliable detection of a smaller between-group effect.
Randomization
The 113 people accepted into the programme were randomized by NT via a true randomization process (www.random.org) to either the iCBT (n = 56) or iCBT+MS (n = 57) groups (Table 1). Allocation preceded the screening phone call.
Results of outcome measures: Means, standard deviations, confidence intervals and pooled effect sizes (Cohen's d) for each group (intention to treat; last observation carried forward)
Pre, pre treatment; Post, post-treatment; Follow up, 3 month follow up; SIAS, Social Interaction Anxiety Scale; SPS, Social Phobia Scale; PHQ-9, Patient Health Questionnaire 9-Item; K-10, Kessler 10-Item; SDS, Sheehan Disability Scale; iCBT, Internet (self-guided) CBT group; iCBT+MS, Internet CBT (self-guided) plus motivational strategies group; CI, confidence interval.
Statistical methods
Group differences in demographic data, pre-treatment measures, and pre-treatment expectations were analysed with one-way analyses of variance (ANOVAs) and chi-square tests. Two factor scores (credibility and expectancy) were derived from the CEQ as described [36]. All post-treatment and follow up analyses adopt an intention-to-treat (ITT) design where missing data is replaced by the last observation carried forward (LOCF).
Pre- to post-treatment and pre- to follow up changes in participants’ questionnaire scores were analysed using univariate analyses of covariance (ANCOVAs) [37–38]. Effect sizes (Cohen's d) were calculated for within-group changes, based on the pooled standard deviation. The percentage of participants in each group who met criteria for both reliable and clinically significant change on the SIAS was calculated. This was defined as the proportion who met the criteria of statistically reliable change as described in Jacobson and Truax [39] and who also demonstrated a reduction in SIAS scores below a clinically significant cut-off. A reliable change index for the SIAS was calculated separately for each group using their pre-treatment standard deviation, and a test–retest reliability coefficient of 0.93, as reported in the original paper [32]. A clinically significant cut-off was determined on the SIAS by employing the mean SIAS score reported by Mattick and Clarke [32] for their community sample compared to the mean scores for the SIAS reported in the current sample, resulting in a clinically significant cut-off on the SIAS at 36 and below, comparable to that reported by similar research [40].
The study was approved by the Human Research Ethics Committee (HREC) of St Vincent's Hospital (Sydney, Australia) and the HREC of the University of New South Wales (Sydney, Australia). This trial was registered as ACTRN12609001082291.
Results
Baseline data
The mean age of participants was 43.6 years (SD = 14.6), 57 (53%) were men, 51 (47%) were married or in a de facto relationship, 84 (78%) had post-secondary education or training, and 50 (46%) were in full-time employment. Fifty-five (51%) reported having previously talked to a health professional about their symptoms of social phobia, and 17 (16%) reported taking medication for mood and/or anxiety. Fifty-three (49%) participants reported using the Internet more than 10 hours per week, and 65 (60%) reported they were very confident using computers and the Internet. No between-group differences on demographic characteristics, pre-treatment measures or pre-treatment expectations (CEQ factor scores) were observed using one way ANOVAs (F1, 106 = 0.00 – 3.2, p > 0.05) or chi-squared tests (p > 0.05).
Participant flow and completion rates
The flow is shown in Figure 1. One iCBT and four iCBT+MS group participants did not complete the pre-treatment questionnaires or start the programme and were not eligible for analysis, resulting in 55 and 53 participants in the iCBT and iCBT+MS groups, respectively. Thirty-one (56%) iCBT group and 40 (75%) iCBT+MS group participants completed all eight lessons within the required time frame. A chi-squared test confirmed this difference was statistically significant (χ 2 = 4.38, df = 1, 108, p < 0.04).
Post-treatment data was collected from 48 (87%) iCBT group and 51 (96%) iCBT+MS group participants. Follow up data (3 months post-treatment) was collected from 46 (84%) iCBT group and 41 (77%) iCBT+MS group participants. Reasons for participants not completing lessons or questionnaires were not available. In accordance with the ITT and LOCF models, missing data was replaced with the last observed score on that measure.
Post-treatment (12 week) outcomes
Primary and secondary outcome measures
Univariate ANCOVAs on post-treatment SIAS and SPS scores, controlling for pre-treatment scores (see Table 1), did not show a difference between groups in post-treatment SIAS (F1, 105 = 1.33, p<0.25) or SPS (F1, 105 = 3.23, p<0.08) scores. Univariate ANCOVAs on PHQ-9, K-10, and SDS post-treatment scores, while controlling for pre-treatment scores, failed to reveal any significant post-treatment differences between groups for the PHQ-9 (F1, 105 = 1.49, p = 0.22), K-10 (F1, 105 = 0.15, p = 0.7) or SDS (F1, 105 = 0.12, p = 0.73).
Effects sizes
Within-group effect sizes for the outcome measures are included in Table 1. Large pre- to post-treatment within-group effect sizes were found for the iCBT group on the SIAS (1.16), SPS (1.04), and SDS (0.97). Large within-group effect sizes were also found for the iCBT+MS group on the SIAS (1.15), and SDS (0.97). Moderate within-group effect sizes were found on the PHQ-9 (0.57) and K-10 (0.66) for the iCBT group, and on the SPS (0.75), PHQ-9 (0.79) and K-10 (0.74) for the iCBT+MS group.
Clinical significance and reliable clinical change
Based on the criteria for reliable and clinically significant change (statistically reliable change and a post-treatment SIAS score of 36 or below), 38% of iCBT group, and 42% of iCBT+MS group participants were classified as having achieved reliable clinical change at post-treatment.
Treatment satisfaction
Chi-squared tests and one-way ANOVAs failed to reveal any differences between treatment groups’ ratings of overall satisfaction with the programme (p = 0.31) or quality of the treatment lessons (p = 0.39). Overall, participants reported a high level of satisfaction with the overall programme, with 90/99 (91%) reporting being either very satisfied or mostly satisfied, 9/90 (10%) neutral/somewhat dissatisfied, and 0% reporting very dissatisfied. Most participants (90%) rated the quality of the treatment modules as excellent or good, 9/99 (9%) rated them as satisfactory, and 1/99 (1%) as unsatisfactory.
When asked to provide a rating from 1 to 10, where 10 indicates a high level of agreement, the average participant rated the treatment as logical (9/10); they reported feeling confident that the treatment would be successful at teaching them techniques for managing their symptoms (8/10); and they reported a high level of confidence in recommending this treatment to a friend with social phobia (9/10). No between treatment group differences were found on these items (F1, 97 = .01 – 0.22, p > 0.05).
Impact of MS and process questions
One-way ANOVAs failed to reveal any differences between group ratings of how the education lessons had affected their understanding about their symptoms (p = 0.91), motivation to participate (p = 0.67), or their determination to overcome obstacles during the programme (p = 0.57). One-way ANOVAs also failed to reveal any differences in the number of logins (p = 0.7) and homework downloads (p = 0.68). A chi-squared test failed to reveal differences between treatment group reports of how much time they spent working on the programme (p = 0.49).
Follow up (3 month) outcomes
Primary and secondary outcome measures
Univariate ANCOVAs on follow up SIAS and SPS scores, controlling for pre-treatment scores (see Table 1), failed to show a difference between groups in follow up SIAS (F1, 105 = 0.76, p < 0.39) or SPS (F1, 105 = 0.37, p < 0.55) scores. Univariate ANCOVAs on PHQ-9, K-10, and SDS post-treatment scores, while controlling for pre-treatment scores, also failed to reveal any significant follow up differences between groups for the PHQ-9 (F1, 105 = 0.50, p = 0.48), K-10 (F1, 105 = 0.04, p = 0.85) or SDS (F1, 105 = 0.08, p = 0.94).
Effect sizes
Large within-group effect sizes were found for the iCBT group on the SIAS (1.15), SPS (0.97), and SDS (0.98). Large within-group effect sizes were also found for the iCBT+MS group on the SIAS (1.25), SPS (0.88), and SDS (0.95). Moderate within-group effect sizes were found on the PHQ-9 (0.57) and K-10 (0.73) for the iCBT group, and on the PHQ-9 (0.71) and K-10 (0.77) for the iCBT+MS group.
Clinical significance and reliable clinical change
At follow up, 47% of iCBT group, and 38% of iCBT+MS group participants were classified as having achieved reliable clinical change on the SIAS.
Discussion
The present study had two aims: To replicate the results of an earlier trial of a self-guided version of the Shyness programme, and to examine whether the addition of motivational strategies based on motivational interviewing improved completion rates and clinical outcomes. Participants were randomly allocated to complete the Shyness programme plus two educational lessons about anxiety disorders, or to complete the Shyness programme plus the two educational lessons that also included material based on motivational interviewing principles.
These results replicate those from the Shyness 4 study [16] indicating that a self-guided version of the Shyness programme, that is one that did not involve human contact during the programme, resulted in good clinical gains and high levels of acceptability. Large mean within-group effect sizes on the two measures of social phobia were found both at post-treatment for the iCBT (1.1) and iCBT+MS (0.95) groups and at follow up (1.06 and 1.07, for the iCBT and iCBT+MS groups, respectively). These effect sizes compare favourably with those reported for face-to-face treatment [24–25], but are lower than clinician or technician-guided versions of the Shyness programme [18]. Criteria for reliable and clinically significant change at follow up was met by 47% of iCBT and 38% of iCBT+MS group participants, which also compares favourably with face-to-face treatment [40]. Importantly, acceptability ratings were high across groups, with 91% of participants reporting they were very or mostly satisfied with the programme.
These results are consistent with recent studies indicating that self-guided programmes for anxiety disorders are efficacious [22–23]. However, caution in interpreting these results is required as not all studies have found similar patterns, and the reported sample sizes have been relatively small. Another key issue is that the successful studies have all been clinical trials, while open access websites report low completion rates and low outcomes [21]. It is likely that these differences reflect considerably greater motivation in participants who are prepared to apply, and wait for, clinical trials, compared to those who chose open access sites, and direct comparisons are needed to inform interpretation.
No evidence of benefit from the addition of motivational strategies was found. No differences were detected between groups at post-treatment on measures enquiring about the impact of the education lessons on motivation or determination. Consistent with this, no differences were detected between groups at post-treatment or follow up on outcome measures or acceptability. Similarly, the magnitude of differences between the mean within-group effect sizes on the two measures of social phobia at post-treatment and follow up fell below the criteria of a significant difference between groups (specified as 0.2 effect size difference). However, significantly more participants in the iCBT+MS group (75%) completed the eight lessons within the required time frame than participants in the iCBT group (56%), indicating that the motivational techniques may have facilitated completion rates. The recent studies applying motivational interviewing techniques to anxiety and depression [27–28] have included up to four individual 50 min face-to-face sessions of motivational interviewing pre-treatment, a considerably more powerful intervention than the simple static and self-guided questions included in the present study. Against this backdrop, it is encouraging that any effect was observed and we would expect that an interactive version of the self-guided motivational strategies included in this study may result in improvements in outcomes, in addition to improved completion rates.
Limitations
Due to technical problems no direct measures of the effect of the motivation strategies were made directly after the two educational lessons, which would have provided important data about the immediate impact of the MS techniques. These findings need to be independently replicated to further determine their reliability and validity. Manuals for the use of the Shyness programme and software to allow other services to test this programme are being developed.
Conclusions
This study confirms that a self-guided version of the Shyness programme, enhanced with resources and regular reminders can result in completion rates, clinical outcomes, and acceptability ratings comparable with face-to-face treatment [25]. Supplementing the programme with motivational strategies increased completion rates, but not clinical outcomes. It is expected that enhancing these additional components with interactive online media will improve their strength. The potential of self-guided programmes within a carefully constructed stepped-care model is considerable. Careful and independent replication is required. Further research will examine how to facilitate open-access self-guided versions of the Shyness programme, and will directly compare Internet versus. face-to-face treatment.
Footnotes
Acknowledgements
The authors gratefully acknowledge Henny Westra for kindly providing the Motivational Interviewing Adapted for Anxiety Manual (2003). We also wish to acknowledge the participants for their involvement and helpful comments, and the contributions to the development of the VirtualClinic programme by Jason Hando, and Daniel Winter.
