Evaluations of Web-Based Psychological Interventions Need to Avoid the Re-Test Artefact

Anthony F. Jorm, ORYGEN Research Centre, Centre for Youth Mental Health, University of Melbourne, Melbourne, Victoria, Australia: Van Vliet and Andrews recently reported an evaluation of a Web-based course for the management of stress in junior high school students [1]. The course involved giving students six lessons in how to recognize and manage stress. To evaluate the impact of the course, students were given questionnaires before and after completing the six lessons and at a 3 month follow up. Psychological distress was found to decrease and life satisfaction to increase. The study also had a smaller non-randomized comparison group that did not show these changes. The difference between the groups was attributed to the effects of the stress management training.

The design of that study has a potential confounding which I have seen in other evaluations of Web-based interventions. In addition to the questionnaires given at pre-test assessment, post-test assessment and at follow up, the students in the Web intervention group completed questionnaires along with each weekly lesson. In other words, the intervention group was tested on many more occasions than the comparison group. There are many studies showing that re-testing of psychological symptoms can in itself produce reduced scores, even though there is no real change in psychological state. This phenomenon has been called the ‘re-test effect’ or ‘re-test artefact’ [2–4].

Some previous studies of Web-based interventions are also subject to this confounding. For example, Christensen et al. carried out a randomized controlled trial of brief and full cognitive behaviour therapy for depression [5]. Participants received varying number of modules of the intervention. Depressive symptoms were measured before and after, as well as at the start of each module. Again, this produces a confounding between the number of modules presented and the number of testing occasions.

The possibility of a re-test effect may not always be apparent with Web-interventions. For example, in some websites (e.g. BluePages, www.bluepages.anu.edu.au) users are provided with an online test of symptoms for self-screening. In trials of such interventions it is possible that there is an unintended re-test artefact, even if repeated testing is not explicitly built into the design [6].

In conclusion, Web-based interventions make it relatively easy to automatically test participants on multiple occasions. This can lead, however, to a confounding between the intervention and number of tests completed. The resulting re-test artefact may lead to an exaggerated estimate of the benefits of the intervention. Researchers in this area need to ensure that the number of testing occasions is equal between intervention and control groups.