The first 30 months of the MindSpot Clinic: Evaluation of a national e-mental health service against project objectives

Abstract

Objective:

The MindSpot Clinic provides online mental health services to Australian adults with anxiety and depression. This paper describes users of MindSpot between January 2013 and June 2015. Outcomes are considered against three key objectives: improving access to mental health services, improving public awareness of how to access services and providing evidence-based treatments.

Method:

Website traffic data were examined to determine patterns of use. Demographic characteristics, past service utilisation and reasons for contacting MindSpot were analysed. Outcomes for patients enrolled in a MindSpot treatment course were also analysed. Primary outcomes were scores on the 9-Item Patient Health Questionnaire, Generalised Anxiety Disorder 7-Item, Yale–Brown Obsessive Compulsive Scale and Post-Traumatic Stress Disorder Checklist–Civilian Version, administered at assessment, post-treatment and 3-month follow-up.

Results:

The website was visited by almost 500,000 Australians, of which 33,990 adults started assessments, and 25,469 people completed assessment and were eligible for analysis. Mean age was 36.4 years (standard deviation = 13.3 years; range = 18–94 years), and 72% were female. The proportion living in rural or remote regions and who identified as Aboriginal and Torres Strait Islander closely matched national statistics. The majority (82%) reported that they were not currently in contact with mental health services. Most patients sought an assessment, information about treatment options, or referral to another service, and only 24% of those completing an assessment commenced a MindSpot treatment course. Of these, large clinical effects (d: 0.7–2.4; average symptom reductions: 25.5% to 61.6%) were found from assessment to follow-up on all outcome measures. Deterioration ranged from 1.0% to 4.3%.

Conclusion:

Based on the number of website visits, completed assessments and treatment outcomes, MindSpot achieved its three programme objectives. This model of service provision has considerable value as a complement to existing services, and is proving particularly important for improving access for people not using existing services.

Keywords

Internet treatment anxiety depression cognitive behaviour therapy service implementation

Introduction

Each year, more than 700 million people are affected by mood and anxiety disorders, which together with other mental and substance use disorders account for 7.4% of the global total burden of disease (Kessler et al., 2009; Whiteford et al., 2013). In a 12-month period, fewer than half of those affected seek or receive evidence-based treatments (Wang et al., 2007; Whiteford et al., 2014). Barriers to treatment include cost, availability of services, limited awareness of illness and the potential benefit of treatment, and stigma (Mojtabai et al., 2011).

The Australian Government has introduced a range of initiatives to reduce these barriers (e.g. Pirkis et al., 2005, 2011). One of the recent initiatives has been the funding of mental health services delivered via the Internet (Australian Government, 2012). These strategies have been initiated following research demonstrating the efficacy of online treatments for high prevalence mental disorders (Andersson and Titov, 2014), and their potential to eliminate some of the barriers to mental health care.

As part of the Australian Government e-Mental Health Strategy (Australian Government, 2012), the MindSpot Clinic was launched in late 2012. MindSpot is aimed at adults with symptoms of anxiety and depression and was designed to provide brief online assessments, information about local mental health services and therapist-guided Internet-delivered cognitive behaviour therapy (iCBT). An evaluation of results from the first year of service reported that MindSpot was used by more than 10,000 people, many of whom had no previous experience with mental health care. The initial evaluation showed that the outcomes of routine treatment provided by MindSpot were comparable to those observed in controlled clinical trials and with benchmarks of face-to face CBT treatments (Titov et al., 2015b).

Internationally, there is increasing interest in the potential of online mental health services (Australian National Mental Health Commission, 2014; Mental Health Commission of Canada, 2014; Ministry of Health New Zealand, 2012; National Institute for Health and Care for Clinical Excellence, 2009), particularly in response to recognition that the burden of common mental health conditions is unlikely to be fully addressed via traditional treatment approaches (Kazdin, 2015). While there are examples of such services providing effective treatments, not all attempts at introducing online treatments have been successful (Gilbody et al., 2015; Kenter et al., 2015), and relatively little is known about the users of online mental health services. Consequently, there are now calls for evaluations of online mental health services to inform policy development (Meurk et al., 2016).

The present study describes visitors to the MindSpot website and the characteristics of patients who used MindSpot during the first 30 months. The study also examines the outcomes of MindSpot against key objectives set by the funding body, the Australian Government. These objectives are (1) to increase access to mental health services, (2) to increase knowledge about mental health conditions and how to access services and (3) to provide evidence-based treatment for anxiety and depression.

Methods

Design and ethical approval

Data about MindSpot users were collected from two sources over the first 30 months of MindSpot services (January 2013–June 2015 inclusive). First, data were collected about Australian visitors to the publicly available MindSpot Clinic website (www.mindspot.org.au), measured using Google Analytics (Clifton, 2010). Visitors to the website were able to review the Terms of Use, which explained that while aggregated visitor data would be used for reporting purposes, individual website visitors would not be identifiable. Website users were able to access information on the MindSpot website without registering. Second, data were obtained from patients who registered to receive services provided by mental health professionals employed by MindSpot. Registered patients were asked to provide consent for their data to be included in evaluations of the service, a process that was approved by the Macquarie University Human Research Ethics Committee.

Website visitors

Google Analytics was used to extract information about visitors to the MindSpot website, in a manner similar to that described in recent reports (Crutzen et al., 2013). The information on the MindSpot website is designed to promote public understanding about how to recognise anxiety and depressive disorders, how to seek help and treatment options. The website currently comprises 21 pages as well as links to assist people to access other (including crisis) services and links to the MindSpot Privacy Policy and related documents. Data of interest included total number of visitors, duration of visit, bounce rate (proportion leaving the website after visiting only the homepage), the pages visited and the type of device used to access the website.

MindSpot was initially promoted using paid advertisements on Google Ads and Facebook Ads. Over the first 30 months, links were established on government and non-government health-related websites. Large-scale promotions were intentionally avoided, in order to control the demand for services while staff were recruited and trained and systems infrastructure was established.

Registered patients

Visitors to the website who registered and completed an assessment were eligible for clinical services. Registered patients met the following criteria: (1) were an Australian resident eligible for publicly funded health services, (2) were aged 18 years and over and (3) reported a principal complaint of anxiety and/or depression. The majority of registered users (97%) gave consent for their de-identified data to be analysed and reported. Those who did not consent were still provided with services, but their data are not reported here.

To register, users created an online account that asked for a name, email address and telephone number. They were then asked to provide demographic and clinical details, including gender, age, address, location (state and postcode), relationship status, education level, employment status, country of birth, indigenous status, duration of symptoms and details of previous and current contact with health professionals about symptoms of anxiety and depression. Patients completed standardised self-report questionnaires asking about the presence of symptoms of specific anxiety disorders, depression and psychological distress. They then nominated their primary reason for using the Clinic, that is, whether they were seeking an assessment to learn about their symptoms and local treatment options, or online treatment at the Clinic, or other reasons. All patients were asked questions about suicidal thoughts, intentions and plans. As described in a recent report of the safety procedures used at the Clinic, a telephone-administered structured risk assessment was performed with all people who reported suicidal intent or plans, and safety plans were developed for all users to assist them to stay safe in the event of an increase in symptoms (Nielssen et al., 2015). People who were acutely suicidal were referred to local mental health services or emergency services, depending on the urgency of the situation.

People who did not complete an assessment were sent information about managing symptoms, contact details for crisis services and were invited to contact MindSpot to discuss their symptoms. People who completed the assessment were invited to discuss their results and provide additional details to a therapist by telephone in order to tailor treatment advice. An assessment report that identified clinically significant symptoms was sent to each patient and, if requested, to a nominated health professional, usually a general practitioner (GP). This report included individually prepared treatment recommendations and advice about accessing relevant national or local mental health services. All patients also received written information about managing symptoms and were encouraged to visit their GP for a physical review and to discuss the results of their assessment.

Patients who chose to participate in a MindSpot online treatment course were then enrolled in the next available course (see below). People were not eligible for online treatment if they were (1) acutely suicidal, (2) already engaged in regular psychotherapy for anxiety or depression or (3) had clinical presentations that were deemed to require comprehensive face-to-face assessment. Examples of the latter included people who had untreated and disabling psychotic symptoms, or endorsed severe symptoms across four or more types of anxiety or depressive disorder, making it difficult to recommend an appropriate treatment approach. Patients with subclinical symptoms were eligible to enrol in treatment courses, as were patients taking psychotropic medication. Clinic services were provided at no cost to patients.

Data were automatically entered into a secure electronic clinical platform operated by MindSpot. In 0.5% of cases, the assessment questionnaires were completed by telephone and the data were entered into the electronic platform by the therapist.

MindSpot online treatment

The iCBT treatment courses used at MindSpot were developed and validated in a series of randomised controlled trials at the Macquarie University online research clinic, the eCentreClinic (www.ecentreclinic.org). People interested in receiving iCBT from MindSpot were offered one of four courses, depending on their age and symptom profile: the Wellbeing course for adults aged 18 years and over with symptoms of anxiety or depression (Dear et al., 2015; Titov et al., 2013, 2015a); the Wellbeing Plus course for older adults aged 60 years and over with symptoms of anxiety and depression (Titov et al., 2016), the obsessive-compulsive disorder (OCD) course for adults aged 18 and older with symptoms consistent with OCD (Wootton et al., 2013) and the post-traumatic stress disorder (PTSD) course for adults aged 18 years and over with symptoms consistent with PTSD (Spence et al., 2014). Courses consisted of four to six lessons delivered over 8 weeks. Each lesson comprised a series of slides that presented the principles of CBT via text and images. Details regarding the structure and content of the courses are provided elsewhere (Titov et al., 2015b).

Outcome measures

Primary outcome measures for the Wellbeing and Wellbeing Plus courses were the 9-Item Patient Health Questionnaire (PHQ-9) (Kroenke et al., 2001) and the Generalised Anxiety Disorder 7-Item Scale (GAD-7) (Spitzer et al., 2006). The PHQ-9 consists of nine items measuring symptoms of major depressive disorder, consistent with the Diagnostic and Statistical Manual of Mental Disorders, 4th ed. (DSM IV). Scores range from 0 to 27, with a total score of 10 or more indicating a diagnosis of depression. The PHQ-9 has good internal consistency and is sensitive to change (Kroenke et al., 2010). The GAD-7 consists of seven items and is sensitive to the presence of GAD, social phobia and panic disorder (Spitzer et al., 2006). Scores range from 0 to 21, and a total score of 8 or more indicates the likely presence of an anxiety disorder (Kroenke et al., 2007). The GAD-7 also has good internal consistency and good convergent and divergent validity with other anxiety and disability scales (Kroenke et al., 2010). The primary outcome measure for the OCD course was the Yale–Brown Obsessive Compulsive Scale–Self Report (YBOCS-SR) (Baer, 1991), which measures the severity of obsessions and compulsions. The YBOCS-SR consists of 10 items, with scores ranging from 0 to 40. Scores of 16 or more indicate clinically significant symptoms. The primary outcome measure for the PTSD course was the Post-Traumatic Stress Disorder Checklist–Civilian Version (PCL-C) (McDonald and Calhourn, 2010). The PCL-C consists of 17 items, with scores ranging from 17 to 88. A score of 45 or more on that scale indicates clinically significant symptoms.

The Kessler 10-Item Scale (K-10) was used as a secondary outcome measure to assess general psychological distress (Kessler et al., 2002). Scores range from 10 to 50, with scores of 22 or more associated with the presence of anxiety and depressive disorders (Australian Bureau of Statistics [ABS], 2007).

Each week patients completed the GAD-7, PHQ-9 and single-item measures enquiring about personal safety and treatment satisfaction. The K-10, YBOCS-SR and PCL-C were administered as part of a larger battery of questionnaires at assessment, mid-treatment, post-treatment (primary endpoint) and 3-month follow-up (secondary endpoint). Questionnaires were self-report measures administered online. These questionnaires appeared automatically after login each week and again at post-treatment and 3-month follow-up.

Statistical analyses

Generalised estimating equation (GEE) models were used to examine changes in symptom measures over time (Hubbarb et al., 2010). Consistent with the principles of intention-to-treat analyses, missing data were imputed using separate GEE models. An unstructured working correlation matrix and maximum likelihood estimation were used, and gamma distribution with a log link response scale was specified to address positive skewness in dependent variable distributions.

Clinically significant change was calculated in two ways: percentage symptom changes from baseline, and within-group Cohen’s d effect sizes based on the estimated marginal means derived from the GEE model. For the Wellbeing and Wellbeing Plus courses, clinical deterioration was defined as an increase in scores ⩾6 for the PHQ-9, and ⩾5 for the GAD-7. For the OCD course, deterioration was defined as an increase ⩾6 on the YBOCS-SR; and for the PTSD course, deterioration was defined as an increase ⩾13 on the PCL-C (Titov et al., 2015b). Treatment acceptability and satisfaction were assessed using chi-square analyses. Data were analysed using SPSS version 21.0.

Results

Patient flow

In the 30-month period from 01 January 2013 to 30 June 2015, the MindSpot website was accessed by 458,921 Australian visitors, who made a total of 713,622 visits. A total of 33,990 Australians registered as users and commenced an online assessment (see Figure 1). Of these, 2210 (6.5%) were excluded from further analyses as they were less than 18 years of age (3.4%), did not consent for analysis of their data (3.0%) or the data were unreliable due to system error (0.1%). A further 6311 (18.6%) did not complete the assessment and were sent information about managing symptoms, contact details for crisis services and were invited to contact MindSpot in the future.

Figure 1.

Patient Flow.

Of the 25,469 registered users who completed an assessment, 19,320 (75.9%) received an assessment report, information about available services and were encouraged to access those services. A total of 6149 (24.1%) individuals who completed an assessment received an assessment report and enrolled in a MindSpot treatment course.

Website use

Visitors spent an average of 122 seconds on the MindSpot website, and looked at an average of 2.2 pages. The bounce rate (i.e. the percentage of visitors who navigated away from the site after viewing only one page) was 58.2%, although this figure includes people who went directly from the homepage to begin to register as a MindSpot user, or logged on via the homepage to access course material. With regards to frequency of visits, 64.9% (297,840 visits) of users visited the site on one occasion, 14.7% (134,923 visits) on two occasions and 20.4% (280,560 visits) on three or more occasions. Pages containing information about MindSpot services were viewed most frequently (54.9% of page views), followed by pages that included multimedia content or online quizzes (31.5%). Pages containing information about symptoms of anxiety and depression were viewed less frequently (13.3%), followed by pages concerned with Terms of Uses, Privacy Policy and similar information (0.3%).

Most users (49%) accessed the website using a desktop device, with the remainder using either a tablet (25.9%) or mobile phone (23.6%). The amount of time (and number of pages visited) for each device was 143 (2.5 pages), 95 (1.9 pages) and 100 seconds (2.0 pages) for the desktop, tablet and mobile users, respectively.

Patient characteristics

Demographic characteristics of patients completing an assessment, or assessment and treatment, are shown in Table 1. Where relevant, a comparison with national statistics is provided. The mean age of registered patients was 36.4 years (standard deviation [SD] = 13.3 years), and those who enrolled in treatment were slightly older. Approximately 72% of patients were female, which is higher than the national average for those with an anxiety or depressive disorder, and there were higher levels of tertiary qualifications, but also higher levels of unemployment, compared with national statistics. The proportion born overseas, identifying as indigenous (Aboriginal and Torres Islander), or living in rural or remote regions, closely matched national statistics.

Table 1.

Demographic characteristics of eligible individuals contacting the MindSpot Clinic between 01 January 2013 and 30 June 2015.

	Assessment (n = 19,320)	Assessment + Treatment (n = 6149)	Total (n = 25,469)	National estimates
Gender
Female	72.0% (13,905/19,320)	71.1% (4372/6149)	71.8% (18,277/25,469)	62%^a
Age (years)
Mean (SD)	35.4 (13.0)	39.8 (13.3)	36.4 (13.3)
Range	18–94	18–88	18–94 years
Cultural background
Born in Australia	75.8% (11,073/14,614)	74.9% (3310/4419)	75.6% (14,383/19,033)	72.3%^b
Aboriginal/Torres Strait Islander	2.7% (406/15,166)	1.6% (63/5001)	2.3% (469/20,167)	3.1%^a
Locality
Major city	68.7% (12,368/18,000)	68.2% (4056/5950)	68.6% (16,424/23,950)	71.9%
Inner regional	19.7% (3540/18,000)	19.3% (1148/5950)	19.6% (4688/23,950)	18.1%
Outer regional	9.4% (1686/18,000)	10.0% (597/5950)	9.5% (2283/23,950)	10.0%^c
Remote/very remote	2.3% (406/18,000)	2.5% (149/5950)	2.3% (555/23,950)	10.0%^c
Relationship status
Married/de facto/in a relationship	59.4% (11,171/18,796)	65.4% (3929/6011)	60.7% (15,099/24,807)
Separated/divorced/widowed	11.5% (2157/18,796)	12.2% (735/6011)	11.7% (2892/24,807)
Single	29.1% (5468/18,796)	22.4% (1347/6011)	27.5% (6815/24,807)
Education
High school or less	23.2% (4317/18,581)	17.0% (1011/5961)	21.7% (5328/24,542)
Apprenticeship/certificate/diploma	40.0% (7424/18,581)	37.2% (2216/5961)	39.3% (9640/24,542)	64%^d
University degree	36.8% (6840/18,581)	45.9% (2734/5961)	39.0% (9574/24,542)	64%^d
Employment
Employed full or part time	53.9% (10,415/18,774)	58.8% (3614/6016)	56.6% (14,029/24,790)
Unemployed	12.7 (2444/18,774)	11.1 (680/6016)	12.6% (3124/24,790)	11%^b
Parenting/home duties	9.3% (1799/18,774)	9.6% (589/6016)	9.6% (2388/24,790)
Student	13.4% (2590/18,774)	8.4% (519/6016)	12.5% (3109/24,790)
Disabled	5.0% (947/18,774)	4.4% (266/6016)	4.9% (1213/24,790)
Retired	3.1% (579/18,774)	5.8% (348/6016)	3.7% (927/24,790)

SD: standard deviation.

National Survey of Mental Health and Wellbeing (Australian Bureau of Statistics [ABS], 2007).

Labour Force, Australia, Catalogue No. 6202.0 (ABS, 2015).

General Social Survey, Summary Results, Catalogue No. 4159.0 (ABS, 2010).

Year Book Australia (ABS, 2012).

Baseline symptoms and treatment history

Details of symptoms and treatment history are shown in Table 2. Only 9.7% of patients did not meet clinical cut-offs on the PHQ-9 or GAD-7. At assessment, patients reported that symptoms of anxiety or depression had been present for a mean of 8.6 years (SD 10.5). Nearly one-third of patients (29.8%) reported thoughts of suicide in the past week, and 2.8% reported both suicidal thoughts and intentions or plans. When contacted, 0.6% were unable to guarantee their safety in the next 24 hours and were referred to crisis services (Nielssen et al., 2015). Approximately one-third of patients (34.7%) reported they had never previously spoken to a health professional about symptoms of anxiety or depression, and only 18.1% were currently engaged in mental health treatment with a health professional, although 30.0% of respondents reported that they were taking prescribed psychotropic medication. Less than half (46.5%) of patients reported having a current GP that they spoke with about mental health.

Table 2.

Baseline symptoms, treatment history and reasons for contact.

	Assessment (n = 19,320)	Assessment + Treatment (n = 6149)	Total (n = 25,469)
Assessment scores
PHQ-9 at assessment (observed mean and SD)	15.8 (6.2)	13.8 (6.0)	15.3 (6.2)
GAD-7 at assessment (observed mean and SD)	12.8 (5.1)	12.1 (4.9)	12.6 (5.1)
K-10 at assessment (observed mean and SD)	32.5 (7.5)	30.3 (6.9)	32.0 (7.5)
Clinical symptoms
Above clinical cut-offs at assessment (PHQ-9 ⩾ 10 and/or GAD-7 ⩾ 8)	90.8% (17,339/19,091)	88.7% (5357/6038)	90.3% (22,696/25,129)
Above clinical cut-off at assessment (K-10 ⩾ 22)	91.8% (17,735/19,320)	89.0% (5463/6139)	91.1% (23,198/25,459)
Below clinical cut-offs at assessment (PHQ-9 and GAD-7)	9.2% (1752/19,091)	11.3% (681/6038)	9.7% (2433/25,129)
Duration of symptoms, suicidal thoughts and intentions
Duration of symptoms in years (mean and SD)	8.2 (10.2)	9.5 (11.5)	8.6 (10.5)
Suicidal thoughts within previous week	33.1% (4362/13,166)	25.9% (1227/4939)	29.8% (5389/18,105)
Suicidal thoughts and intention for self-harm	3.3% (439/13,166)	1.5% (73/4939)	2.8% (512/18,105)
Reported treatment history
No previous mental health treatment	36.8% (6720/18,273)	28.2% (1603/5680)	34.7% (8323/23,953)
Current mental health treatment	18.3% (3477/18,979)	17.5% (1062/6068)	18.1% (5119/28,300)
Has current GP, and talks with GP about mental health	43.8% (798/1820)	56.0 (291/520)	46.5% (1089/2340)
Current psychotropic medication	29.2% (5460/18,698)	32.3% (1926/5957)	30.0% (7386/24,655)
Main reason for using MindSpot
Assessment and information	63.0% (8428/13,395)	40.9% (2057/5027)	56.9% (10,485/18,423)
Treatment	30.5% (4089/13,395)	54.9% (2760/5027)	37.2% (6849/18,423)
Other	6.5% (878/13,395)	4.2% (210/5027)	5.9% (1088/18,423)
Main reason for using MindSpot instead of face-to-face services
Reasons due to privacy, anonymity, stigma, or that face-to-face treatment is too confronting	40.4% (4788/11,859)	32.4% (1067/3294)	38.6% (5855/15,153)
Reasons due to lack of local services, expense of services, or difficulty accessing therapy during business hours	33.7% (3992/11,859)	40.6% (1339/3294)	35.2% (5331/15,153)
No perceived need for face-to-face treatment	15.2% (1802/11,859)	15.4% (508/3294)	15.2% (2310/15,153)
Other	10.8% (1277/11,859)	11.5% (380/3294)	10.8% (1830/16,982)

PHQ: 9-Item Patient Health Questionnaire; SD: standard deviation; GAD-7: Generalised Anxiety Disorder 7-Item Scale; K-10: Kessler 10-Item Scale; GP: general practitioner.

With regards to the main reason for using MindSpot, 56.9% reported they were seeking assessment and information about local services, a further 37.2% reported they were considering online treatment at MindSpot and 5.9% reported other reasons. When asked about their main reasons for not choosing face-to-face services, 38.6% reported reasons relating to concern about anonymity and privacy; 35.2% reported reasons relating to convenience, availability and the cost of face-to-face services; 15.2% reported they did not believe they required face-to-face treatment; and the remainder (10.8%) cited other reasons.

Treatment outcomes

A total of 70.5% of patients (4338/6149) completed treatment. Post-treatment symptom questionnaires were completed by 66.1% of patients (4064/6149), and 3-month follow-up questionnaires were completed by 43.1% of patients (2652/6149). Completion rates by course are shown in Figure 1.

Treatment outcomes, including means, percentage reductions and effect sizes, are shown in Table 3. Analysis of the entire sample revealed significant main effects over time on the PHQ-9 (Wald’s χ² = 14,606, p < 0.001), GAD-7 (Wald’s χ² = 12,475, p < 0.001) and K-10 (Wald’s χ² = 16,647, p < 0.001). Pairwise comparisons showed significant improvements from baseline to post-treatment (ps < 0.001) and from baseline to 3-month follow-up (ps < .001). Analyses of outcomes by course revealed highly consistent results, with significant main effects over time observed for all courses on all measures (Wald’s χ² range = 78–14,706, ps < 0.001). For all courses, scores on each outcome measure improved significantly from assessment to post-treatment and from assessment to 3-month follow-up (ps < 0.001).

Table 3.

Treatment outcomes overall and by course.

	Estimated means			Percentage changes		Reliable deterioration		Effect sizes
	Assessment	Post-treatment	3-month follow-up	Assessment to post-treatment	Assessment to 3-month follow-up	Assessment to post-treatment	Assessment to 3-month follow-up	Assessment to post-treatment	Assessment to 3-month follow-up
Overall (n = 6149)
Depression (PHQ-9)	13.8 (6.0)	6.9 (4.8)	6.5 (4.1)	49.8% [48.9–50.6%]	52.8% [52.0–53.5%]	2.1% (132/6149)	2.4% (132/6149)	1.3 [1.2–1.4]	1.4 [1.3–1.5]
Anxiety (GAD-7)	12.1 (4.9)	6.0 (3.9)	5.4 (3.3)	50.2% [49.4–51.0%]	55.0% [54.3–55.6%]	2.7% (167/6149)	2.9% (181/6149)	1.4 [1.3–1.5]	1.6 [1.5–1.7]
Distress (K-10)	30.3 (6.9)	22.0 (6.9)	19.9 (5.7)	40.9% [40.0–41.8%]	51.2% [50.5–51.9%]	–	–	1.2 [1.1–1.3]	1.6 [1.5–1.7]
Wellbeing (n = 5427)
Depression (PHQ-9)	13.9 (6.0)	7.0 (4.8)	6.6 (4.1)	49.2% [48.3–50.1%]	52.3% [51.6–53.1%]	2.2% (121/5427)	2.5% (133/5427)	1.3 [1.2–1.4]	1.4 [1.3–1.5]
Anxiety (GAD-7)	12.2 (4.9)	6.1 (3.9)	5.5 (3.3)	49.8% [48.9–50.7%]	54.6% [53.9–55.3%]	2.7% (148/5427)	3.0% (163/5427)	1.4 [1.3–1.5]	1.6 [1.5–1.7]
Distress (K-10)	30.5 (6.8)	22.2 (6.9)	20.1 (5.7)	40.3% [39.4–41.2%]	50.9% [50.1–51.6%]	–	–	1.2 [1.1–1.3]	1.7 [1.6–1.8]
Wellbeing plus (n = 516)
Depression (PHQ-9)	12.7 (6.0)	5.3 (4.1)	5.3 (3.8)	58.0% [55.1–60.7%]	58.4% [55.8–60.9%]	1.0% (5/516)	1.6% (8/516)	1.4 [1.3–1.6]	1.5 [1.3–1.6]
Anxiety (GAD-7)	10.5 (5.0)	4.5 (3.5)	4.0 (2.8)	57.4% [54.4–60.2%]	61.6% [59.2–63.9%]	1.6% (8/516)	1.9% (10/516)	1.4 [1.2–1.5]	1.6 [1.4–1.7]
Distress (K-10)	28.1 (6.9)	19.1 (6.2)	17.9 (5.2)	49.4% [46.3–52.3%]	56.5% [54.0–58.9%]	–	–	1.4 [1.2–1.5]	1.7 [1.5–1.8]
OCD (n = 69)
OCD (YBOCS-SR)	24.0 (6.7)	17.9 (6.7)	17.4 (4.9)	25.5% [18.7–31.8%]	27.5% [22.5–32.1%]	4.3% (3/69)	2.9% (2/69)	0.9 [0.6–1.3]	1.1 [0.8–1.5]
Depression (PHQ-9)	11.3 (6.7)	5.8 (4.1)	7.1 (4.8)	49.0% [39.7–56.8%]	37.0% [26.4% 46.1%]	–	–	1.0 [0.6–1.3]	0.7 [0.4–1.1]
Anxiety (GAD-7)	13.2 (5.3)	7.1 (4.6)	7.5 (3.9)	46.2% [37.3–53.9%]	43.4% [36.0–50.0%]	–	–	1.2 [0.9–1.6]	1.2 [0.9–1.6]
Distress (K-10)	28.7 (8.3)	21.5 (7.7)	21.5 (5.3)	38.8% [28.8–48.0%]	38.5% [31.7–45.0%]	–	–	0.9 [0.5–1.2]	1.0 [0. 7–1.4]
PTSD (n = 137)
PTSD (PCL-C)	62.5 (12.2)	42.9 (13.3)	36.4 (9.3)	43.0% [37.9–47.7%]	57.4% [53.8–60.6%]	2.9% (4/137)	2.9% (4/137)	1.5 [1.3–1.8]	2.4 [2.1–2.7]
Depression (PHQ-9)	15.6 (6.6)	8.3 (5.0)	6.9 (3.6)	46.8% [41.2–51.9%]	56.0% [52.1–59.7%]	–	–	1.3 [1.0–1.5]	1.6 [1.4–1.9]
Anxiety (GAD-7)	13.0 (5.1)	7.1 (4.2)	6.0 (3.4)	45.2% [39.5–50.3%]	53.6% [48.9–57.8%]	–	–	1.3 [1.0–1.5]	1.6 [1.3–1.9]
Distress (K-10)	32.4 (8.4)	24.4 (6.8)	20.8 (5.4)	35.9% [30.7–40.8%]	51.8% [47.6–55.7%]	–	–	1.1 [0.8–1.3]	1.6 [1.4–1.9]

PHQ: 9-Item Patient Health Questionnaire; GAD-7: Generalised Anxiety Disorder 7-Item Scale; K-10: Kessler 10-Item Scale; OCD: obsessive-compulsive disorder; YBOCS-SR: Yale–Brown Obsessive Compulsive Scale–Self Report; PTSD: post-traumatic stress disorder; PCL-C: Post-Traumatic Stress Disorder Checklist–Civilian Version.

Observed means based on actual scores at that time point. When data were not available, the most recent sessional data were used. For K-10 and PCL-C percentage change calculations, baseline scores were standardised to zero. Standard deviations are shown in rounded parentheses for the means, and 95% confidence intervals are shown in square parentheses for the percentage changes and effect sizes.

There was a significant main effect of time on the YBOCS for the OCD course (Wald’s χ² = 140, p < 0.001), with significant improvements from assessment to post-treatment and assessment to follow-up (ps < 0.001). For the PTSD course, there was a significant effect of time on the PCL-C (Wald’s χ² = 534, p < 0.001), with improvements from assessment to post-treatment and assessment to follow-up (ps < 0.001).

Clinical significance and deterioration

Percentage changes in symptoms, within-group effect sizes and reliable deterioration are shown in Table 3. Symptom reductions of at least 43.0% at post-treatment and 52.3% at follow-up occurred on primary measures for the Wellbeing, Wellbeing Plus and PTSD courses. Lower percentage reductions were observed for patients in the OCD course, with reductions in YBOCS-SR scores of 25.5% at post-treatment and 27.5% at follow-up. Similarly, effect size estimates were large (⩾0.7) but were greater for those in the Wellbeing, Wellbeing Plus and PTSD courses (ds = 1.1–2.4) compared to the OCD course (ds = 0.7–1.2. Deterioration rates ranged from 2.2% to 3.0% for the Wellbeing course, 1.0% to 1.9% for Wellbeing Plus, 2.9% to 4.3% for the OCD course and 2.9% for the PTSD course.

Therapist time, patient activity and treatment satisfaction

An average of 25 minutes of therapist time was required for assessment. For treatment, therapist time and patient activity are summarised in Table 4. Results indicate a high level of course engagement as measured by total number of logins, total time spent logged into the course and number of additional materials downloaded. The average amount of therapist time per patient per course ranged between approximately 2 and 3 hours. This included all contact with patients, preparation time for each patient including reading and responding to messages, administration and supervision time during treatment and during follow-up.

At post-treatment, 63.0% (3871/6149) and 64.3% (3954/6149) of patients answered the first and second treatment evaluation questions, respectively. Of these, 97.6% (3778/3871) reported they would recommend the course to others, and 95.5% (3778/3954) reported that participating in the course was worth their time.

Table 4.

Course engagement

	Wellbeing	Wellbeing Plus	OCD	PTSD
Total number of logins^a
Mean (SD)	14 (10.2)	19 (15.1)	16 (9.1)	21 (17.4)
Range	1–98	1–109	1–45	1–87
Total online course time^a
Mean (SD)	316 minutes (291.6 minutes)	396 minutes (305.9 minutes)	308 minutes (247.6 minutes)	403 minutes (330.2 minutes)
Range	4–1799	11–1524	32–1242	9–1379
Resource downloads
Mean (SD)	12 (10.0)	20 (17.1)	13 (10.2)	19 (14.6)
Range	0–173	0–144	0–50	0–60
Therapist contact during treatment^a
Mean (SD)	131 minutes (80.4 minutes)	153 minutes (90.3 minutes)	183 minutes (119.2 minutes)	194 minutes (114.0 minutes)
Range	12–417	12–405	17–392	10–398
Emails sent to patients
Mean (SD)	33 (12.5)	30 (12.1)	37 (17.0)	34 (14.3)
Range	2–252	2–136	4–134	10–138

OCD: obsessive-compulsive disorder; PTSD: post-traumatic stress disorder; SD: standard deviation.

Outliers removed.

Discussion

Data from the first 30 months of service show that the MindSpot Clinic successfully achieved its key objectives of increasing access to mental health services, increasing knowledge about mental health services and providing effective online treatment to a large number of patients. Most patients had symptoms of both depression and anxiety in the clinical range, and most had been troubled by symptoms for several years.

The MindSpot website was visited by almost 500,000 Australians, more than one-third returned to the site on at least one further occasion and more than 30,000 Australian adults registered with MindSpot. The proportion of users from each of the states and territories, rural or remote regions, indigenous Australians and those born overseas closely matched national statistics, which showed that MindSpot was used by a national audience. Some differences did emerge, including the disproportionate number of females and higher rates of tertiary qualifications and unemployment compared to the wider population.

Most patients (82%) were not in contact with mental health services, and 35% had never spoken with a health professional about their symptoms. Moreover, less than half had a GP with whom they would discuss their mental health. Consistent with this, the wish for anonymity and concerns about privacy, availability, cost and convenience were cited as reasons for seeking care from MindSpot instead of existing face-to-face services. These data confirm that MindSpot has had some success in making care available for hard-to-reach groups, and highlights the importance of allowing people to self-refer. However, MindSpot has not been as successful in attracting men to treatment, indicating the need for ongoing efforts to engage this group.

The large number of people using MindSpot is also notable, especially in light of the limited promotion of the service. It is highly likely that a large and sustained promotional campaign would considerably increase public demand for MindSpot services. The level of interest and engagement in the service despite limited promotion confirms the level of public interest and need for online mental health services.

MindSpot also achieved the objective of increasing knowledge about mental health conditions and access to services. Most people reported that their primary reason for using MindSpot was to obtain an assessment and information about treatment options (including face-to-face services). Many people were unaware of the existence of local mental health services or how to access them, indicating the role of online services in helping people navigate the mental health system.

Only 37% of people using MindSpot reported that they were primarily seeking online treatment. While the conversion rate is similar to that reported by other online services (AL-Asadi et al., 2014; Klein et al., 2011), we were initially surprised to find that people with symptoms in the clinical range were not necessarily seeking treatment. While low rates of treatment seeking are often reported in epidemiological surveys (Wang et al., 2007), feedback from our patients revealed that the opportunity to self-refer for an assessment and to discuss the findings and treatment options with a qualified therapist was itself a valuable goal. This confirms our earlier observations that the demand for online mental health services extends beyond providing treatment (Titov et al., 2015b). These results support the view that online mental health services should include components of education, assessment, triage and support to arrange urgent help for people in crisis, referral, as well as treatment, in a similar way to existing mental health services.

The treatment outcomes confirm that MindSpot achieved its objective of providing evidence-based treatment to a large number of people. We found large improvements in symptoms of anxiety and depression across all treatment courses, with improvements sustained at 3-month follow-up. The levels of improvement in symptoms were consistent with those obtained in the first year of the service (Titov et al., 2015b). Importantly, for all courses, the proportion of patients meeting criteria for deterioration was low (1.0–4.3%) and satisfaction rates were high. These large symptom reductions were achieved with an average of 2–3 hours of therapist time, indicating the cost-effectiveness of this model of care.

The magnitude of improvement replicates results from randomised controlled trials, and is also consistent with reports from other specialist iCBT services, confirming that iCBT can be successfully provided as part of routine clinical care (e.g. Hedman et al., 2013; Mewton et al., 2012; Ruwaard et al., 2012). The results continue to compare well with benchmarks of outcomes from face-to-face CBT treatments (Cahill et al., 2010; Cuijpers et al., 2008; Stewart and Chambless, 2009), and compare favourably with other initiatives that involve the large-scale implementation of psychological treatments (Gyani et al., 2013; Richards and Borglin, 2011).

Clinical implications

Critically, many people accessing MindSpot are either not using existing mental health services or are unlikely to do so, showing the important role that online services have to play in addressing gaps in public mental health services. This also raises important questions about how online and existing mental health services can be most effectively integrated. While there are some encouraging examples of how this might occur (Hadjistavropoulos et al., 2016), the challenges are considerable and not all attempts at combining online services with existing services have been successful (Gilbody et al., 2015; Kenter et al., 2015). The ability to identify models of best practice for combining iCBT and face-to-face services is limited by the small number of published reports, differences between services in characteristics and preferences of patients and referrers, the quality of the iCBT interventions offered and differences in study design and methodology. Factors that may affect the uptake of online mental health services include the context in which they are introduced, for example, whether they already have established and effective models of care and may not require, or value, iCBT and may even see iCBT as a threat. However, in locations where mental health services do not exist, where populations are geographically dispersed and where patients can self-refer to the service, the acceptability and engagement with iCBT is likely to be higher.

A clear benefit of iCBT for anxiety and depression is the comparatively small amount of therapist time required to provide effective treatment for most patients. MindSpot therapists took about a quarter of the time required for equivalent face-to-face care, and the preference of many patients for small amounts of therapist contact suggests the average therapist time required could be further reduced. This will help realise the potential of providing clinically and cost-effective services to large numbers of people, thus producing considerable benefit at a population health level.

Another clinical implication is the effect of routine measurement of outcomes. This has facilitated trials of changes to the treatment model used at MindSpot. Measurement of patient outcomes has also guided the supervision and training of individual therapists, whose performance can be compared to their peers. The routine measurement of clinical outcome is used in large initiatives in other countries (Richards and Borglin, 2011), but is yet to find its way into mainstream mental health care. The experience of the MindSpot Clinic is that the routine use of data could greatly improve the efficiency and outcomes of other mental health services.

Limitations and strengths

A limitation of the website data is the possibility that some people accessed the website from different devices, resulting in double-counting. The proportion who did so cannot be determined and the reported figure of almost 500,000 unique visitors is probably an over-estimate. Even taking this into consideration, only a very small proportion of website visitors finally chose to access online treatment. We suspect a significant proportion were unfamiliar with online mental health services and were reluctant to use the service for this reason. However, the number of people registering to use MindSpot is increasing, indicating acceptability is also increasing.

Also, the comparisons between MindSpot patients and official statistics are descriptive and did not involve formal analyses. A more detailed analysis may reveal other differences between MindSpot patients and the wider population. Other limitations include the absence of a control group, which would have shown the rates of natural remission, and the effect of missing data, which includes the 3% of patients who declined permission to analyse their data, the 19% who did not complete assessment and the proportion of patients who did not complete questionnaires at post-treatment (34%) and follow-up (57%). However, the effect of missing data at post-treatment and follow-up has been mitigated by the weekly collection of data during treatment and by appropriate statistical modelling.

The main strength of this study is the availability of comprehensive data on a large consecutive sample collected over 30 months as part of routine clinical care. A second strength is the regular measurement of symptoms to monitor treatment effects. Moreover, the treatment results from 30 months closely match those reported for the first 12 months of MindSpot, indicating the robustness of the Clinic’s procedures and the clinical effects. These data indicate that the MindSpot Clinic model has been highly successful. Key contributors to this success include the systematic development both of online treatments in a large number of controlled trials and of online procedures for assessment, triage, referral, treatment and risk management (Nielssen et al., 2015). Other key contributors include staff who are deeply committed to improving access to evidence-based care and advice from the people who use MindSpot, which guides the ongoing evolution of MindSpot.

Conclusion

The MindSpot Clinic is achieving its objectives of increasing access to mental health services, increasing knowledge about symptoms and treatment options and providing effective online treatments. Online services are not a panacea, and are not designed to replace face-to-face services, especially for the seriously mentally ill and those in crisis. However, we are convinced of the potential benefits of online services as part of comprehensive mental health services. Their particular value is to address unmet needs at a population health level, particularly for those not accessing existing services and who are unlikely to do so.

Footnotes

Acknowledgements

The MindSpot Clinic is funded by the Australian Government. The authors gratefully acknowledge the patients for allowing the use of their data as well as the efforts of Macquarie University, the MindSpot Clinic and eCentreClinic staff in launching and operating the Clinic.

Declaration of Conflicting Interests

Dr Titov and Dr Dear are authors and developers of the treatment courses used at the MindSpot Clinic but derive no personal or financial benefit from them. Dr Titov, Dr Dear, Dr Bennett-Levy, Dr Rapee and Dr Klein are funded by the Australian Government to develop and provide a free national online and telephone-delivered treatment service. Dr Nielssen is a member of a Lundbeck advisory board for an antipsychotic medication. The other authors report no financial relationships with commercial interests.

Funding

This study was investigator initiated. It was funded by Departmental funds from the Faculty of Human Sciences, Macquarie University.

References

AL-Asadi

Klein

Meyer

(2014) Posttreatment attrition and its predictors, attrition bias, and treatment efficacy of the anxiety online programs. Journal of Medical Internet Research 16: e232.

Andersson

Titov

(2014) Advantages and limitations of Internet-based interventions for common mental disorders. World Psychiatry 13: 4–11.

Australian Bureau of Statistics (ABS) (2007) National Survey of Mental Health and Wellbeing: Summary of Results. Available at: www.ausstats.abs.gov.au/ausstats/subscriber.nsf/0/6AE6DA447F985FC2CA2574EA00122BD6/$File/43260_2007.pdf

Australian Bureau of Statistics (ABS) (2010) General Social Survey: Summary Results, Australia (ed. Statistics ABo). Canberra, ACT, Australia: ABS.

Australian Bureau of Statistics (ABS) (2012) Year Book Australia. Canberra, ACT, Australia: ABS.

Australian Bureau of Statistics (ABS) (2015) Labour Force. Canberra, ACT, Australia: ABS.

Australian Government (2012) E-Mental Health Strategy for Australia (ed. Ageing DoHa). Canberra: Australian Government. Available at www.health.gov.au/internet/main/publishing.nsf/content/7C7B0BFEB985D0EBCA257BF0001BB0A6/$File/emstrat.pdf

Australian National Mental Health Commission (2014) The National Review of Mental Health Programmes and Services. Sydney, NSW, Australia: National Mental Health Commission.

Baer

(1991) Getting Control: Overcoming Your Obsessions and Compulsions. London: Penguin Books.

10.

Cahill

Barkham

Stiles

(2010) Systematic review of practice-based research on psychological therapies in routine clinic settings. British Journal of Clinical Psychology 49: 421–453.

11.

Clifton

(2010) Advanced Web Metrics with Google Analytics. Indianapolis, IN: Wiley Publishing.

12.

Crutzen

Roosjen

Poelman

(2013) Using Google analytics as a process evaluation method for Internet-delivered interventions: An example on sexual health. Health Promotion International 28: 36–42.

13.

Cuijpers

van Straten

Andersson

et al . (2008) Psychotherapy for depression in adults: A meta-analysis of comparative outcome studies. Journal of Consulting and Clinical Psychology 76: 909–922.

14.

Dear

Staples

Terides

et al . (2015) Transdiagnostic versus disorder-specific and clinician-guided versus self-guided internet-delivered treatment for generalized anxiety disorder and comorbid disorders: A randomized controlled trial. Journal of Anxiety Disorders 36: 63–77.

15.

Gilbody

Littlewood

Hewitt

et al . (2015) Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial): Large scale pragmatic randomised controlled trial. British Medical Journal 351: h5627.

16.

Gyani

Shafran

Layard

et al . (2013) Enhancing recovery rates: Lessons from year one of IAPT. Behaviour Research and Therapy 51: 597–606.

17.

Hadjistavropoulos

Nugent

Alberts

et al . (2016) Transdiagnostic Internet-delivered cognitive behaviour therapy in Canada: An open trial comparing results of a specialized online clinic and nonspecialized community clinics. Journal of Anxiety Disorders 42: 19–29.

18.

Hedman

Ljótsson

Kaldo

et al . (2013) Effectiveness of Internet-based cognitive behaviour therapy for depression in routine psychiatric care. Journal of Affective Disorders 155: 49–58.

19.

Hubbarb

Ahern

Fleisher

et al . (2010) To GEE or not to GEE: Comparing population average and mixed models for estimating the associations between neighborhood risk factors and health. Epidemiology 21: 467–474.

20.

Kazdin

(2015) Evidence-based psychotherapies II: Changes in models of treatment and treatment delivery. South African Journal of Psychology 45: 3–21.

21.

Kenter

RMF

van de Ven

Cuijpers

et al . (2015) Costs and effects of Internet cognitive behavioral treatment blended with face-to-face treatment: Results from a naturalistic study. Internet Interventions 2: 77–83.

22.

Kessler

Aguilar-Gaxiola

Alonso

et al . (2009) The global burden of mental disorders: An update from the WHO World Mental Health (WMH) surveys. Epidemiology Psychiatry 18: 23–33.

23.

Kessler

Andrews

Colpe

et al . (2002) Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychological Medicine 32: 959–976.

24.

Klein

Meyer

Austin

et al . (2011) Anxiety online – A virtual clinic: Preliminary outcomes following completion of five fully automated treatment programs for anxiety disorders and symptoms. Journal of Medical Internet Research 13: e89.

25.

Kroenke

Spitzer

Williams

(2001) The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine 16: 606–613.

26.

Kroenke

Spitzer

Williams

et al . (2007) Anxiety disorders in primary care: Prevalence, impairment, comorbidity, and detection. Annals of Internal Medicine 146: 317–325.

27.

Kroenke

Spitzer

Williams

et al . (2010) The patient health questionnaire somatic, anxiety, and depressive symptom scales: A systematic review. General Hospital Psychiatry 32: 345–359.

28.

McDonald

Calhourn

(2010) The diagnostic accuracy of the PTSD Checklist: A critical review. Clinical Psychology Review 30: 976–987.

29.

Mental Health Commission of Canada (2014) E-Mental Health in Canada: Transforming the Mental Health System Using Technology. Ottawa, ON, Canada: Mental Health Commission of Canada.

30.

Meurk

Leung

Hall

et al . (2016) Establishing and governing e-Mental health care in Australia: A systematic review of challenges and a call for policy-focussed research. Journal of Medical Internet Research 18: e10.

31.

Mewton

Wong

Andrews

(2012) The effectiveness of internet cognitive behavioural therapy for generalized anxiety disorder in clinical practice. Depress Anxiety 29: 843–849.

32.

Ministry of Health New Zealand (2012) Rising to the Challenge: The Mental Health and Addiction Service Development Plan 2012–2017. Wellington, New Zealand: Ministry of Health.

33.

Mojtabai

Olfson

Sampson

et al . (2011) Barriers to mental health treatment: Results from the National Comorbidity Survey Replication. Psychological Medicine 41: 1751–1761.

34.

National Institute for Health and Care for Clinical Excellence (2009) Depression in adults: Recognition and management. Available at: www.nice.org.uk/guidance/CG90 (accessed 26 November 2015).

35.

Nielssen

Dear

Staples

et al . (2015) Procedures for risk management and a review of crisis referrals from the MindSpot Clinic, a national service for the remote assessment and treatment of anxiety and depression. BMC Psychiatry 15: 304.

36.

Pirkis

Harris

Hall

et al . (2011) Evaluation of the better access to psychiatrists, psychologists and general practitioners through the Medicare Benefits Schedule Initiative, 22 February. Centre for Health Policy, Programs and Economics, University of Melbourne, Melbourne, VIC, Australia.

37.

Pirkis

Hickie

Young

et al . (2005) An evaluation of beyondblue, Australia’s National depression initiative. International Journal of Mental Health Promotion 7: 35–53.

38.

Richards

Borglin

(2011) Implementation of psychological therapies for anxiety and depression in routine practice: Two year prospective cohort study. Journal of Affective Disorders 133: 51–60.

39.

Ruwaard

Lange

Schrieken

et al . (2012) The effectiveness of online cognitive behavioral treatment in routine clinical practice. PLoS ONE 7: e40089.

40.

Spence

Titov

Johnston

et al . (2014) Internet-based trauma-focused cognitive behavioural therapy for PTSD with and without exposure components: A randomised controlled trial. Journal of Affective Disorders 162: 73–80.

41.

Spitzer

Kroenke

Williams

et al . (2006) A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine 166: 1092–1097.

42.

Stewart

Chambless

(2009) Cognitive-behavioral therapy for adult anxiety disorders in clinical practice: A meta-analysis of effectiveness studies. Journal of Consulting and Clinical Psychology 77: 595.

43.

Titov

Dear

Johnston

et al . (2013) Improving adherence and clinical outcomes in self-guided Internet treatment for anxiety and depression: Randomised controlled trial. PLoS ONE 8: e62873.

44.

Titov

Dear

Staples

et al . (2015a) Disorder-specific versus transdiagnostic and clinician-guided versus self-guided treatment for major depressive disorder and comorbid anxiety disorders: A randomized controlled trial. Journal of Anxiety Disorders 35: 88–102.

45.

Titov

Dear

Staples

et al . (2015b) MindSpot Clinic: An accessible, efficient and effective online treatment service for anxiety and depression. Psychiatric Services 66: 1043–1050.

46.

Titov

Fogliati

Staples

et al . (2016) Treating anxiety and depression in older adults: Randomised controlled trial comparing guided v. self-guided Internet-delivered cognitive-behavioural therapy. British Journal of Psychiatry Open 2: 50–58.

47.

Wang

Aguilar-Gaxiola

Alonso

et al . (2007) Use of mental health services for anxiety, mood, and substance disorders in 17 countries in the WHO world mental health surveys. The Lancet 370: 841–850.

48.

Whiteford

Buckingham

Harris

et al . (2014) Estimating treatment rates for mental disorders in Australia. Australian Health Review 38: 80–85.

49.

Whiteford

Degenhardt

Rehm

et al . (2013) Global burden of disease attributable to mental and substance use disorders: Findings from the Global Burden of disease study 2010. The Lancet 382: 1575–1586.

50.

Wootton

Dear

Johnston

et al . (2013) Remote treatment of obsessive-compulsive disorder: A randomized controlled trial. Journal of Obsessive-Compulsive and Related Disorders 2: 375–384.