Abstract
Keywords
In a randomized controlled trial (RCT) Titov et al. randomly allocated 105 participants with social phobia to treatment or waitlist control groups [1]. Treatment involved an Internet-based clinician-assisted computerized cognitive behavioural treatment (CaCCBT) programme for social phobia (the Shyness programme) and consisted of four components: completing six lessons online (at www.shyness.tv, now www.shynessclinic.tv); cognitive behavioural homework assignments; participation in an online discussion forum; and regular email contact with a therapist.
Mean within- and between-group effect sizes (Cohen's d) across the two primary outcome measures were 1.15, and 0.95, respectively, with approximately 2½ h therapy contact time per participant. These encouraging effect sizes are comparable to those observed in a high-quality face-to-face group treatment programme for social phobia [2], and are consistent with those reported from other RCTs examining online treatment of social phobia [3], [4], providing preliminary evidence that social phobia can be effectively treated over the Internet with an Australian sample.
Such studies are part of a promising and rapidly growing body of clinical research exploring the use of computer technology to assist in the administration of cognitive behavioural therapy (CBT) programmes. We use the term CaCCBT to refer to clinician-assisted computerized CBT that includes therapist contact, to differentiate it from fully computerized CBT (CCBT) programmes, which do not require therapist involvement, although there is some debate about the best terms to use [5].
A primary reason for exploring the efficacy of CaCCBT programmes is their potential for increasing the capacity of existing mental health services [6], [7]. Results from recent RCTs indicate that CaCCBT with email or telephone contact can result in clinically significant improvements not only in patients with social phobia [1], [3], [4] but also for patients with depression [8] and panic disorder [9–13], at a level similar to that provided by face-to-face clinicians, and often requiring much less therapist contact time. These results suggest that CaCCBT has promise as a tool for increasing the capacity of existing mental health services [14–17] without compromising clinical or cost-effectiveness.
Potential advantages to patients of using Internet-based treatment programmes include reducing barriers to treatment resulting from the tyranny of distance, barriers resulting from stigma or embarrassment about seeking specialist mental health services, or barriers resulting from the usual requirement to attend therapy during standard business hours [7], [18]. In addition, CaCCBT programmes have the potential to provide a consistent level of high-quality content [5]. But before CaCCBT programmes can be confidently added to the armamentarium of existing mental health services their reliability and their clinical and cost-effectiveness in different treatment environments should be carefully evaluated. Systematic research is also required to identify the characteristics of patients likely to benefit from CaCCBT, and to evaluate the effectiveness of CaCCBT relative to other treatment models.
The present study is part of a systematic programme of research exploring the effectiveness of CaCCBT in treating social phobia and other common mental disorders. As indicated here, our preliminary results showed that the Shyness programme (Shyness 1) resulted in substantial changes in standard measures of social phobia, comparable to those obtained in high-quality face-to-face therapy [1]. The present study contributes to this body of research by reporting the results of a replication (Shyness 2) of the procedure used in Shyness 1 with a further 88 participants meeting the DSM-IV-TR diagnosis of social phobia [19].
Method
Recruitment and outcome measures
No significant changes were made either to the inclusion criteria, treatment procedure, or materials used in the original Shyness programme. Applicants who met the inclusion criteria completed the following questionnaires, considered reliable, valid, and appropriate for both clinical and research purposes: Social Interaction Anxiety Scale [20] (SIAS); Patient Health Questionnaire–Nine Item [21] (PHQ-9), Social Phobia Scale [20] (SPS), Kessler-10 [22] (K-10); and the World Health Organization Disability Assessment Schedule (2nd edn) [23] (WHODAS-II). The SIAS and SPS are widely used measures of interaction and performance anxiety, respectively; the PHQ-9 measures symptoms of depression, the K-10 measures psychological distress and the WHODAS-II measures disability. Changes in the SIAS and SPS were considered the primary social phobia outcome measures, changes in the PHQ-9, K-10, and WHODAS-II were the secondary outcome measures. To provide data about behavioural change, participants were also asked to estimate the number of activities they had avoided and the number of events they had attended in the previous 7 days (the tertiary outcome measures). Applicants then completed a questionnaire enquiring about demographic characteristics.
In addition to the questionnaires used in Shyness 1, applicants also completed the Anxiety Change Expectancy Questionnaire (ACES) [24] and the Credibility/Expectancy Questionnaire (CEQ) [25] to explore whether expectancies or perception of treatment credibility are related to outcomes. Both scales have respectable psychometric properties and have been shown to predict outcomes on some measures [24], [25].
As in Shyness 1, applicants with severe symptoms of depression (≥20 on the PHQ-9), or those indicating suicidal ideation were excluded from participating and were encouraged to discuss their symptoms with their general practitioner or with another health professional. A follow-up email was also sent to applicants to reinforce this message, and they were invited to telephone one of the authors (GA) for further information.
Two hundred and eighty people applied (Figure 1), 122 met the inclusion criteria and were invited to complete and return consent forms prior to a telephone interview. Eighteen applicants did not return consent forms and 13 could not be contacted by telephone. Ninety-one applicants completed the telephone interviews and were given the social phobia section of the Composite International Diagnostic Interview Version 3.0 [26], with additional questions relating to other psychiatric disorders to assist with differential diagnosis. In two applicants another disorder was the prime source of difficulty and they were excluded from this study and advised about more appropriate treatment options. One eligible applicant chose not to participate prior to randomization.
2 Study flow chart. CBT, cognitive behavioural therapy; CIDI 3.0, Composite International Diagnostic Interview Version 3.0; PHQ-9, Patient Health Questionnaire–Nine Item. †Depression total score >19 on the PHQ-9, or score on item 9 > 0. ‡Two treatment group participants withdrew before treatment began, but could not be contacted to determine reasons. Five waitlist control group participants did not complete post-treatment questionnaires: two reported they were on extended holiday, and three could not be contacted. §Reasons for not completing or withdrawing included: work commitments (n = 3); relationship breakdown (n = 1); did not think programme was effective (n = 1); thought programme was very effective and did not require additional therapy (n = 1); computer failed (n = 1); no reason given (n = 1).
At post-treatment assessment, participants completed the aforementioned questionnaires and a seven-item treatment satisfaction questionnaire based on the CEQ [25], [27]. The study was approved by the Human Research Ethics Committee (HREC) of the University of New South Wales (Sydney, Australia), and by the HREC of St Vincent's Hospital (Sydney, Australia). This trial was registered as ACTRN12607000235404.
Procedure
Participants and attrition
The 88 people accepted into the programme were randomized via a true randomization process (www.random.org) to either the treatment (n = 43) or waitlist control (n = 45) groups (Table 1). Two treatment group and five waitlist control group participants withdrew before beginning treatment. The mean age of participants was 36.79 years (SD = 10.93), and 37% of the sample were men. Approximately 40% of participants reported having previous treatment, and approximately 26% reported currently taking medication for mood and/or anxiety. No between-group differences were observed (p > 0.05) on any of these variables using t-tests or χ2 tests.
Participant details
†Medication and/or psychological treatment.
Treatment programme
The treatment programme was the same used in the original Shyness programme: six online lessons; homework assignments; participation in an online discussion forum; and regular email contact with a therapist. Part of the content of each lesson was presented in the form of an illustrated story about a young man with social phobia who, with the help of a clinical psychologist, successfully gains mastery over his symptoms. Lessons 1 and 2 provide education about the symptoms and treatment of social phobia; lesson 3 provides instructions on how to develop an exposure hierarchy and about practising graded exposure; lessons 4 and 5 reinforce principles of graded exposure and demonstrate principles of cognitive restructuring; while lesson 6 includes information about relapse prevention. Each lesson begins with a summary of the material described in the previous lesson.
Each lesson also includes a printable summary and homework assignment. Participants were expected to complete each homework assignment and post messages on a secure online discussion forum, using an alias. The forum software was phpBB 2.0.22. The therapist moderated the forum and aimed to respond to postings and to emails within 24 h. After completing each lesson participants were emailed by the therapist. The themes of the therapist's emails varied from encouragement to practise the relevant treatment skills, encouragement to continue participation and to complete lessons and homework assignments, and enquiries about progress.
Treatment procedure
Treatment group participants began the programme shortly after randomization, and were asked to complete the first four lessons within the first 2 weeks, in order to provide more opportunity to practise the graded exposure, cognitive skills and other coping techniques. Participants were asked to complete the six lessons within 10 weeks of starting. Feedback on questions was generally provided within 24 h, reminder emails were sent regularly, and if necessary participants were telephoned by the therapist.
Waitlist control group participants completed questionnaires at the same times as the treatment group. The waitlist control group began treatment shortly after the treatment group completed.
Therapist
One clinical psychologist (NT) provided all clinical contact with participants. A research assistant (GS) provided administrative support to collate data, follow progress and assist in the telephone interviews. The duration of each contact with participants was recorded.
Statistical analyses
Group differences in demographic data and pre-treatment measures were analysed using t-tests and χ2 tests. Changes in participants’ questionnaire scores were analysed using repeated measures analyses of variance, followed by t-tests with Bonferroni-corrected ps. Effect sizes were calculated as Cohen's d, for both within and between groups, based on the pooled standard deviation. All post-treatment analyses involved an intention-to-treat design.
Two factor scores (credibility and expectancy) were derived from the CEQ by subjecting the scores to a linear transformation with a minimum of 1 and maximum of 9, to create factor scores with a minimum of 3 and maximum of 27 as described in [26]. Change scores for each outcome measure were also calculated for each treatment group participant and Pearson product moment correlations were calculated between these and between the two CEQ factors (expectancy and credibility) to explore variables that may affect outcomes.
Results
Pre-treatment differences
No between-group differences in pre-treatment scores were observed using t-tests (t79= − 0.75–0.98, p = 0.33–0.77).
Completion rates
Eight members of the treatment group (20%) failed to complete all six lessons within the required time frame. Of these eight non-completers, one said the programme was not helpful, and one reported they had improved sufficiently. The mean number of lessons completed was 5.5, and 33 participants completed all six lessons. The overall completion rate was 80%.
Although eight members of the treatment group did not complete all six lessons, post-treatment data were collected from 78 participants (38/41 treatment group participants and 40/40 waitlist control group participants). In accordance with the intention-to-treat paradigm, the pre-treatment scores of the three treatment group participants who did not complete the post-treatment questionnaires were replicated as their post-treatment scores.
Primary outcome measures
Repeated measures analyses of variance were performed on the two primary outcome measures; the SIAS and SPS. These analyses indicated significant main effects for time, group, and significant main effects for group×time interactions for both the SIAS and SPS (Table 2). Planned post hoc t-tests with Bonferroni-corrected ps indicated that the treatment group had significantly lower SIAS scores (t79=5.83, p < 0.001) and significantly lower SPS scores (t79=4.93, p < 0.001) than the waitlist control group at the post-treatment assessment.
Results of outcome measures
∗p < 0.05, ∗∗p < 0.01, ∗∗∗p < 0.001.
Secondary outcome measures
Repeated measures analyses of variance indicated significant main effects for time, group, and significant main effects for the group×time interaction for the WHODAS-II (Table 2). A planned post hoc t-test with Bonferroni-corrected ps indicated that the treatment group had significantly lower WHODAS-II scores (t79=3.01, p < 0.01) than the waitlist control group at the post-treatment assessment.
Main effects for time and for the group×time interaction (but not for group) were shown on the PHQ-9, but only a main effect for time was observed for the K-10. This indicates that both groups experienced reductions in severity of depressive symptoms over the treatment period, but that the treatment group did not report more improvements that the waitlist control group (p > 0.05).
Tertiary outcome measures
Repeated measures analyses of variance were also performed on the two tertiary outcome measures; the number of activities participants reported avoiding and the number of events they reported attending in the previous 7 days. These analyses indicated significant main effects for time, group, and significant main effects for the group×time interaction for the avoidance estimates (Table 2). Planned post hoc t-tests with Bonferroni-corrected ps indicated that the treatment group had significantly lower self-reported avoidance of events (t79=3.82, p < 0.001) than the waitlist control group at the post-treatment assessment.
Repeated measures analyses of variance indicated significant main effects for time for the estimated number of events attended in the previous 7 days, but a main effect was not found for group.
Effect sizes
Effect sizes for the primary and secondary outcome measures are included in Table 2. There were large within- and between-group effect sizes for the SIAS (1.21 and 1.29, respectively) and SPS (1.14 and 1.10, respectively). Effect sizes for the secondary outcome measures varied, with moderate within- and between-group effect sizes for the WHODAS-II (0.77, and 0.67, respectively), moderate to small within- and between-group effect sizes for the K-10 (0.61, and 0.35, respectively), and large to small within- and between-group effect sizes for the PHQ-9 (0.81, and 0.32, respectively).
Effect of pre-treatment expectancies and perceptions of credibility
The expectancy factor from the CEQ and the ACES total score were positively correlated with the SIAS change score (r = 0.44, p < 0.01, n = 34), but no other correlations reached the standard of significance (p < 0.01).
Completers versus non-completers
Analyses were conducted in an attempt to identify characteristics of and outcomes for participants who did not complete the Shyness programme. Treatment group participants from Shyness 2 were divided into those who completed (n = 33) and those who did not complete the six lessons (n = 8). Because of the low statistical power inherent in small samples, two similar groups were created from the Shyness 1 treatment group participants (n = 50) and these groups were compared using t-tests and χ2 tests with their counterparts from Shyness 2. No differences were observed between the non-completers for Shyness 1 and 2 on the primary and secondary pre-treatment measures (p > 0.21), and so these participants were collapsed into a non-completer group (n = 19). Similarly, no differences were observed between the completers for Shyness 1 and 2 on any demographic or pre-treatment measures (p > 0.05), so these participants were collapsed into a completer group (n = 72).
Independent sample t-tests were then conducted to determine whether differences in pre-treatment primary and secondary measures could be identified between completers and non-completers. Differences were found between these two composite groups on the PHQ-9 (t89=2.50, p = 0.02) and K-10 (t89=3.09, p = 0.01), indicating that the non-completers had higher pre-treatment scores on the measure of depressive symptoms and the measure of psychological distress, respectively. Independent sample t-tests were also conducted to explore whether completers had poorer outcomes than non-completers, as indicated by change scores. No differences were observed (t89= − 0.24–1.46, p = 0.16–0.85), indicating that non-completers appeared to benefit as much as completers.
Treatment satisfaction
Participants indicated a high level of satisfaction with the overall treatment, with all responding participants reporting being either ‘very satisfied’ or ‘mostly satisfied’. More than 97% of responding participants rated the quality of the treatment modules as ‘excellent’ or ‘good’, and 91% rated the quality of Internet correspondence with the therapist as ‘excellent’ or ‘good’, 6% rated it as neutral or somewhat dissatisfied, while one person indicated that they were ‘very dissatisfied’. With respect to the pace or tempo of treatment, 46% reported that there was either ‘a bit too little’ or ‘much too little time’, 51% indicated that the amount of time was exactly correct, while 3% reported that the tempo was too slow.
When asked to provide a rating from 1 to 10, where 10 indicates a high level of agreement, on average participants rated the treatment as very logical (9/10); they reported feeling very confident that the treatment would be successful at teaching them techniques for managing their symptoms (8/10); and they reported a high level of confidence in recommending this treatment to a friend with social phobia (9/10).
Time/contact events spent per participant
The average therapist time per treatment group participant was 126.76 min (SD = 30.89 min) including monitoring of the discussion forum and feedback. An additional average 25 min per patient was required for administrative purposes, including the diagnostic telephone interview. A review of email and forum contact showed that a total of 915 emails were sent to treatment group participants (mean = 22.32, SD = 4.06), with a mean of 2.36 (SD = 2.06) forum postings in response to each participant's posts (additional postings were made to the entire group). These data do not include time spent on project development.
Discussion
The present study is part of a systematic programme of research exploring the effectiveness of CaCCBT in treating common mental disorders over distance via the Internet. The results indicate that Shyness 2 produced similar results to Shyness 1. That is, using the same treatment procedure, the treatment group in Shyness 2 had significantly reduced symptoms of social phobia (SIAS and SPS) at post-treatment assessment compared to the waitlist control group. Differences between groups after treatment were not observed across measures of depressive symptoms (PHQ-9) or psychological distress (K-10), but a significant difference between groups after treatment was observed on the measure of disability (WHODAS-II). Consistent with these gains, treatment group participants in Shyness 2 also reported significant reductions in frequency of avoidance in the previous week, relative to controls, although this was not reflected in a significant increase in number of events attended.
The pattern of effect sizes across the primary and secondary measures were also consistent with those observed in Shyness 1. Large between- and within-group effect sizes were observed for the SIAS and SPS (within-group effect sizes of 1.21 and 1.14, respectively), with more modest within-group effect sizes for the K-10, WHODAS-II, and PHQ-9 (0.61, 0.77, and 0.81, respectively). Also consistent with the results of Shyness 1, the mean within-group effect size across the five primary and secondary outcome measures was 0.91.
Box 1. Examples of forum postings
…I have found it invaluable … firstly to realize that I'm not alone in my feelings, and also be pro-active (at last) in trying to deal with it in a very supportive environment. What I am doing now which I wasn't before is to be pro-active calling people and arranging social occasions, talking to people at work, including my boss, and generally feeling more confident and less self conscious when I go out. I think most valuable of all is being more equipped to challenge my distorted thoughts, which I realize now have been really crippling.
What I would like to do is be able to take specialist health classes, which I am starting the training for next year. Thanks again to everyone for your honesty and openness, good luck to all in the future
Well, it's a shame its finished really, because I am enjoying the contact with others on this forum. Many thanks to the research team for their encouraging emails and responsive contact. And for all of you for sharing what are inevitably difficult and sensitive feelings about things. I felt better just writing about what I feel are difficult situations – seeing my stuff on the screen distances me from it and puts it in perspective for me. Particularly since I have never told anyone these things before. Seeing what you found difficult was very encouraging to me, because I realized others have the same problems.
As far as progress goes – last week I was asked to go to two Christmas parties. Without thinking much I just said I'd love to go. Now that is completely the opposite of what I have done up to now (which has been to try and get out of an invitation like that). I might feel a bit uncomfortable going to the parties, but I doubt it will bother me much. Because I will have the support of knowing that there are all you folks who understand what it's like to feel shy socially. And also it really doesn't matter much if I do feel a bit shy.
I reckon I might even start to forget about my feelings and enjoy the Xmas parties. …
I have found this course very helpful and the info very relevant. The most helpful thing I've learnt is to be aware of and challenge my negative thoughts. I have tackled a lot of ‘firsts’ recently which in the past I would have declined – attending a wedding as a family (normally my husband or I would stay home with our daughter who has a rare condition and is very challenging) and being told it was great to see us all there together. Also went to my son's school concert and sat near the front! Held a small birthday party at home for my daughter.
I've realized that even though things may not go smoothly, I'm able to handle it better and feel a sense of accomplishment that I didn't back out and things will get easier with practice. Thank you to everyone for your honesty and encouragement and I hope we all continue on our new paths.
Attempts at identifying the effects of pre-treatment expectations were moderately successful. Even with the reduced sample (n = 35), statistically significant correlations (r = 0.43 and r = 0.43) were observed between both the ACES and the expectancy factor of the CEQ (measures of expected change in anxiety symptoms) and the amount of change occurring in the SIAS score from pre- to post-treatment assessment, indicating that pre-treatment expectations could be useful at predicting treatment responders (and in identifying applicants who might benefit from educational or other interventions before participating in treatment programmes). Analyses also indicated that non-completers of Shyness 1 and 2 had significantly higher levels of pre-treatment depressive symptoms and psychological distress than completers. Interestingly, however, non-completers did not differ from completers in the magnitude of change experienced on outcome measures, indicating that even reduced exposure to the treatment programme was beneficial.
The indices of treatment satisfaction obtained in Shyness 2 were consistent with those observed in Shyness 1. That is, the majority of participants reported that the quality of Internet correspondence with the therapist was either excellent or good. Consistent with this, 100% of respondents reported that they were either very satisfied or mostly satisfied with their treatment, and when asked to rate their confidence in recommending the Shyness programme to a friend with social phobia the mean score was 9/10. Further evidence of treatment satisfaction could be noted from participants’ postings in the forum. Examples of representative postings (used with participants’ permission, de-identified, and abridged) are included in Box 1.
Overall, these results indicate that the Shyness treatment programme produced reliable statistical results and was acceptable to participants. Another essential criterion for evaluating the utility of new clinical tools is their comparative effectiveness. Although the Shyness programme has not yet been formally tested against face-to-face therapy, it is possible to compare pre- and post-treatment SIAS and SPS scores and effect sizes from the published results (Table 3) of other RCTs treating social phobia over the Internet [3], [4], and from benchmarking studies with samples from a research unit and a clinical psychology private practice [28], and from a hospital-based specialist anxiety disorders clinic [2]. Table 3 shows that the effect sizes are comparable and, in fact, favour the Internet treatments. Pre- and post-treatment severity for the Shyness RCTs is comparable to the face-to-face clinical studies [2], [28], indicating that similar immediate clinical outcomes were obtained for the face-to-face and the Shyness programmes. As noted in Shyness 1, these findings are particularly encouraging given that the average therapist treatment contact per participant in the Shyness Programme was <3 h, including administration time, despite high levels of treatment satisfaction and good outcome scores.
Pre- and post-treatment means and effect sizes for the SIAS and SPS across studies (ITT)
CaCCBT, clinician-assisted computerized cognitive behavioural treatment; ITT, intention to treat; SIAS, Social Interaction Anxiety Scale; SPS, Social Phobia Scale.
†McEvoy reports pre- and post-treatment data for the completer group, but effect sizes are for the ITT. ‡Sample from research unit; §sample from private practice.
The use of the CaCCBT Shyness programme has several helpful features for a therapist. Email templates are individualized to each participant, allowing the therapist to spend more time addressing complex questions, sending encouraging and reinforcing messages, and responding to forum postings. In addition, the software does much of the background teaching usually required of a clinician, saving additional time. The advantages to participants include that they can review lessons at any time, and can read the materials at their own pace. Finally, clinicians can adjust the number of treatment contacts to the needs of each participant, a feature more difficult to adapt in group treatment programmes.
Two potential threats to validity should be discussed, both of which were raised in Shyness 1. First, the mean within-group effect size benefit across the five primary and secondary outcome measures for the waitlist control group in Shyness 2 was the same as observed in Shyness 1 (0.28 and 0.24, respectively). The placing of research participants on a waitlist control group is an important ethical issue, which these data suggest did not result in any deleterious short-term consequences for participants.
The second threat is the absence of significant differences between groups following treatment on the measure of depressive symptoms (PHQ-9). Again, we suspect that the low pre-treatment mean (in the mild range) is likely to have produced a floor effect. This hypothesis will be explored in a subsequent RCT with a loosening of the inclusion criteria, allowing participants with higher levels of depressive symptoms to participate.
Outstanding issues include the question of longer-term outcomes; 6 month follow ups with the treatment group participants for Shyness 1 and 2 have been scheduled and will be reported in a future paper. A second outstanding issue involves the importance of the therapist; an RCT comparing the Internet Shyness treatment programme for social phobia with a therapist (CaCCBT) versus the Shyness programme without therapist contact (CCBT) is being developed to answer this question. We expect superior results with the version using a therapist, but this needs to be tested.
In conclusion, although considerable research still remains to be conducted, these encouraging results provide further positive data about the potential utility of Internet-based guided self-help programmes for people with common mental disorders who may not otherwise seek or receive treatment.
Footnotes
Acknowledgements
The authors gratefully acknowledge the participants for their involvement and helpful comments. Dr Peter McEvoy and Dr Alison Mahoney kindly provided supervision and encouragement. This project was partially funded by a research grant from the Faculty of Medicine, UNSW.