Abstract
The Internet provides a platform for cost-effective mental health interventions and services. Previous research from our group has shown that MoodGYM, an interactive program that delivers cognitive behaviour therapy (CBT), when compared to an attentionplacebo condition is effective in reducing depression and anxiety symptoms [1]. We have subsequently shown that outcomes for spontaneous users of the site are of the same magnitude as those of trial participants enrolled in our randomized controlled trial [2]. However, approximately 40% of the public registrants to the site fail to complete any of the online assessments, suggesting that the public health impact of the site might be improved if users could be retained on the site for longer periods of time.
The present paper describes outcome and compliance data for public registrants to the MoodGYM site. The aims are twofold: (i) to examine predictors of expected final depression and anxiety scores as a function of characteristics such as gender, number of modules completed, and initial anxiety and depression scores of users; and (ii) to compare user characteristics and outcomes from the originalMoodGYMsite (Mark I)with those of public registrants of the new public version of the site (MoodGYM Mark II). Launched late in 2003, this version of the site was built with features to improve its research capability including: (i) compulsory requirements for completion of ‘core’ assessments before users are able to progress through the site; (ii) an administrator interface to allow trials to be conducted from a number of international sites simultaneously; (iii) improved data collection including the recording of time sequences; and (iv) an automated randomized controlled trial capability allowing users to be randomly assigned to various internal versions of the program to test the effectiveness of its components. For this second aim of the paper, we compare gender, initial depression and anxiety scores, and completion rates for the two site versions to examine whether structural changes to the site resulted in different user profiles.
Method
The 19 607 visitors who registered on the site between April 2001 and September 2003 were sampled, along with 182 participants in the MoodGYM condition of the BlueMood trial [2]. Symptom assessments using Goldberg Depression and Anxiety Scales [3] were repeated within the website intervention to allow the examination of change in symptoms across modules. Outcome variables were gender, initial depression severity scores, number of assessments attempted (maximum number = 5) and symptom levels following intervention. Linear regression analyses were used to develop predictors of anxiety and depression final scores. Differences between the two versions of the site were examined using chi-squared or t-tests. For the comparison between Mark I and Mark II versions of the site, we compared the 19 607 visitors to the original site, with 38 791 users who registered on the Mark II version of the site between September 2003 and October 2004.
Results
Predictors of final anxiety and depression scores for MoodGYM (Mark I)
For 3176 people, it was possible to compare the initial and final completed depression scales, although there were only 138 for whom the profile across all five modules could be traced. A full account of the analysis of the 3176 person dataset is available [2]. Using this dataset and that from the randomized clinical trial (RCT) we showed that there were no differences in outcomes between the RCT participants and those accessing the original MoodGYM website.
The focus of the analysis in the current paper is on predicting the final depression score for different starting levels of depression. In a linear regression where the dependent variable was the final score and independent variables were gender, number of modules completed (treated as three dummy variables), initial depression score and a quadratic function of the initial score, all independent variables and the interaction between initial depression score and number of modules were significant. The findings are illustrated in Fig. 1. Men are predicted to be 0.19 units (SE = 0.095) higher than women on depression, controlling for the initial depression level and number of modules completed. For initial depression scores above 2, it is predicted that the final score will indicate improvement relative to the initial score, the magnitude of the improvement increasing as a function of the number of modules attempted.
Estimated final Goldberg depression score by initial score, number of modules completed and gender. (Derived from general linear model with final score as dependent variable, gender and number of modules completed as fixed factors and linear and quadratic terms for the Goldberg scale as covariates. Data based on 948 men and 2311 women who completed at least two Goldberg depression scores.)
A parallel analysis was undertaken for anxiety. For 1668 people, it was possible to compare the first and last completed anxiety scales, although there were only 76 for whom the profile across all five modules could be traced. The relationship of final score to initial value was examined in a linear regression where the dependent variable was the final anxiety score and independent variables were gender, number of modules (treated as categorical variable), initial anxiety score and a quadratic function of the initial score. All independent variables except gender were significant. The interaction terms between initial anxiety score and number of modules were also significant. For initial anxiety scores above 2, it is predicted that the final score will indicate improvement relative to the initial score, the magnitude of the improvement increasing as a function of the number of modules attempted. Figure 2 illustrates these findings.
Estimated final Goldberg anxiety score by initial score and number of modules completed. (Derived from general linear model with final score as dependent variable, gender and number of modules completed as fixed factors and linear and quadratic terms for the Goldberg scale as covariates. Based on data from 1668 people.)
Differences between Mark I and Mark II
The 19 607 MoodGYM Mark I participants were compared to 38 791 Mark II participants on demographic and compliance variables. Sixtysix per cent of Mark I and 60.8% of Mark II visitors were women. This difference was significant (χ[1] = 157.23, p<0.001). Mark II registrants had higher initial levels of impairment on the Goldberg Depression Scale (M = 5.56 vs M = 5.40), t(36694) = −5.821, p<0.001. Although statistically significant because of the large numbers of registrants, this difference does not represent a clinical difference, with both samples having elevated rates relative to population samples [2].
Compliance data are illustrated in Fig. 3, which show for each version of MoodGYM the number of tests completed by participants once they register on the site. A substantially greater proportion of Mark I participants failed to complete at least one assessment for both depression and anxiety. Forty-six per cent of Mark I users completed one depression test, and only 37.6% completed one anxiety test. The proportion of Mark II users who completed one test was 51% for both depression and anxiety. Requiring compulsory completion of assessments increases ‘compliance’ but dropout rate was high for both Mark I and Mark II. Approximately 16% of Mark I and 18.0% of Mark II users completed only two or more depression assessments, whereas 7.9% of Mark I and 17.7% of the Mark II sample completed only two or more anxiety assessments. The number of depression tests completed differed significantly between the versions (t[57770] = −11.68, p < 0.001), as did the number of anxiety tests (t[57759] = −49.47, p<0.001).
Percentage of participants attempting: (a) Goldberg depression quizzes; and (b) Goldberg anxiety quizzes, in MoodGYM Mark I and MoodGYM Mark II.
Discussion
Findings from this analysis indicate that ‘free range users’ of the online version of MoodGYM Mark I are more likely to have lower depression at the end of the intervention if they are women, have lower initial scores, and complete more module assessments. These dose–response relationships may illustrate the importance of a user's adherence to the site for positive outcomes, although they may also be due to the retention of users who make the biggest gains early. The completion data from MoodGYM indicate that adherence to the full program is poor, with less than 7% of the site users progressing beyond the first two modules in the Mark II site. The remaining proportion of users, the ‘one hit wonders’, drop out early. The addition of compulsory components appears to increase compliance for a second assessment, but does not increase persistence, with approximately the same proportion of users from both sites discontinuing at the same rate at subsequent assessment occasions. The Mark II structure is also associated with a reduced proportion of female users and a (statistically significant) increase in registrants with higher levels of depression.
The analysis of Internet outcome data fromMoodGYM highlights many of the issues that emerge in the analysis of clinical trial data. However, consideration of these issues also questions the usefulness ofmaking direct comparisons of the outcome level and attrition rates of Internet interventions and clinical trials. Outcomes for both Internet and clinical trial interventions will be tied to the motivation, symptom severity and expectations of participants. Patterns of attrition and missing data will also reflect these sample characteristics. Internet sites create the opportunity to either ‘opt in’ or ‘opt out’ of ‘treatment’ easily (through the click of a mouse or press of a key), making them likely to ‘enroll’ diverse individuals with low levels of commitment and little expectation of being ‘helped’. Clinical trials provide infrastructure and positive expectations. These differences question the usefulness of directly comparing rates of adherence or compliance across the two types of interventions. Moreover, selective attrition is difficult to interpret in both clinical and Internet trials, because ‘dropout’ or non-adherence may arise for different reasons and be associated with different outcomes for different individuals. For example, Internet users may ‘drop out’ either because they are dissatisfied with the intervention (real ‘dropouts’) or because the intervention has met their needs (these individuals are labelled ‘attainers’ in e-education environments [4]). Identifying the proportions of these different classes of dropouts, will require different analysis models to those traditionally used, including models which take into account individual trajectories of change based on sample characteristics.
The Internet has the capacity to reach many individuals who may never seek formal treatment for mental health services. It may well be that the Internet's role in disease prevention will be in the delivery of short positive health messages, rather than through the delivery of ‘therapy’ that requires hours of online work. Website adherence or ‘stickiness’ will cease to be an issue for online sites like MoodGYM if we can show that shorter interventions lead to similar health outcomes and that even brief bursts of information lead to increased helpseeking.