Abstract
Keywords
In July 2001, the Australian Government funded the Better Outcomes in Mental Health Care (BOiMHC) programme in response to low treatment rates for common mental disorders. A key feature of the BOiMHC programme is its Access to Allied Psychological Services (ATAPS) component, which involves a series of projects run by Divisions of General Practice (Divisions). The ATAPS projects enable GPs to refer patients with high prevalence disorders (e.g. depression and anxiety) to allied health professionals (predominantly psychologists) for low-cost mental health care [1]. Most commonly this care takes the form of cognitive behavioural therapy (CBT), which has been shown in recent meta-analyses by Haby et al. [2] and Lynch et al. [3] to be effective for patients with these types of disorders. This care is typically delivered in six individualized sessions, with an additional six (or sometimes 12) sessions made available if this is considered warranted after a review by the referring GP. Up to 12 group sessions are also available [1].
Various policy changes have occurred during the life of the ATAPS projects which have had an impact on their nature and direction. In 2006, the Australian Government introduced the Better Access programme, which facilitates equivalent access to mental health care from similar providers, but does so via a series of Medicare Benefits Schedule (MBS) item numbers rather than through Divisional projects. The ATAPS projects have continued to run alongside the Better Access programme in their original form [4], but several sub-programmes have been or are being introduced that focus on particular at-risk populations (e.g. women with perinatal depression, people at risk of suicide, people at risk of homelessness, people affected by the Victorian bushfires, people in remote locations, children with mental disorders) and/or use specific modalities of service delivery (e.g. telephone-based or web-based CBT). The original ATAPS arrangements are now called Tier 1 services, and the sub-programmes are called Tier 2 services. Tier 3 services, which will provide flexible care packages to people with severe mental illness, are currently being developed [5]. The term ‘tier’ has been used elsewhere to denote other concepts in the Australian mental health sector (e.g. levels of system performance in the National Mental Health Performance Framework [6]), but is used in the manner described here for the remainder of this paper.
Tier 1 services remain the ‘core business’ of the ATAPS programme. Currently, 110 Divisions are running ATAPS projects which are delivering these services. We have been evaluating the Tier 1 ATAPS projects since their introduction, and have confirmed that they are being delivered in the way that was originally intended, under a range of models [7,8]. We have also shown that their uptake has been high in both urban and rural areas, and that they have reached a broad range of patients [4,7,9–12]. In addition, we have demonstrated that they have produced positive outcomes both for providers [13] and for patients [11,14].
To date, our evaluation of the ATAPS programme has not explored whether the Tier 1 ATAPS projects have achieved better outcomes for certain groups of patients or under particular treatment circumstances. This is an important question because it has the potential to refocus the Tier 1 projects if required, and to inform the direction of the Tier 2 and Tier 3 projects. The current paper examines patient-related and treatment-related predictors of outcomes, drawing on data from a purpose-designed minimum dataset.
Method
Divisions running ATAPS projects are required to enter de-identified patient-level and session-level data into a web-based minimum dataset which we developed for the purposes of the evaluation. The patient-level information includes variables describing patients’ socio-demographic and clinical characteristics at referral, as well as pre- and post-treatment scores on standardized outcome measures. The session-level information includes variables that quantify the amount of treatment provided for a given patient, describe the nature of that treatment, and indicate whether the patient received care free of charge or made a co-payment.
Two points should be noted about the outcome data available in the minimum dataset. Firstly, when the ATAPS projects were originally funded, Divisions were instructed that they should encourage providers to use a standardized outcome measure to chart patients’ progress, but the specific measure was left to each Division's discretion. As a consequence, over 20 different instruments are being used by Divisions, with some Divisions using more than one outcome measure for the same patient [14,15]. These Division-level choices about outcome measures mean that there is no particular relationship between an individual patient's presenting problem and the outcome measure that may be used to assess his or her progress. Secondly, the minimum dataset did not originally include outcome data fields; these were only added in mid 2005 [14,15].
We extracted data for the period from the beginning of January 2006 to the end of June 2010 in order to ensure that outcome data were routinely available. We conducted preliminary descriptive analyses of the potential predictor variables, and the results are presented as simple frequencies and percentages. We then used paired t-tests to examine the difference between mean pre- and post-treatment scores on the range of outcome measures, excluding patients who did not have a ‘matched pair’ of pre- and post-treatment scores, and excluding outcome measures for which there were fewer than 200 observations in order to ensure sufficient precision. Finally, we conducted a linear regression analysis using scores on the most commonly used outcome measure as the outcome of interest, and the full range of predictor variables as covariates. Because responses within Divisions were likely to be correlated, variance was calculated using cluster-robust standard errors. Pre-treatment scores were included as a covariate. The effect of categorical predictors was assessed using the joint Wald test.
Results
In total, 150 954 referrals were made via the ATAPS projects during the period of interest. These referrals resulted in 113 107 patients receiving an episode of care (i.e. between one and 18 sessions); in a relatively small proportion of cases (7%), the patient was not unique (i.e. he or she received more than one episode of care). Pre- and post-treatment outcome data were available for 16 700 patients (15% of the 113 107). The majority of these patients (60%) had outcome data from one measure only, but 13% had data from two, 21% from three, 5% from four and 1% from five.
Socio-demographic, clinical and treatment profiles
Table 1 shows the socio-demographic and clinical profiles of all patients and those for whom outcome data were available. Note that the totals do not always equal 113 107 or 16 700 because data on given variables were missing from the minimum dataset. In the main, the profiles for the two groups of patients were similar. Over two thirds of the patients in both groups were female, and two fifths were aged between 25 and 44 years. Two thirds of the patients in both groups were on low incomes, despite being relatively well educated (with over a quarter completing tertiary education). Most commonly, the patients in both groups had a diagnosis of depression or anxiety or both. Just under half of the patients in both groups had no previous history of mental health care.
Socio-demographic, clinical and treatment profiles of patients receiving care through the ATAPS projects, 1 January 2006 to 30 June 2010
aWith or without alcohol and drug use disorders, psychotic disorders, and/or unexplained somatic disorders.b Alcohol and drug use disorders, psychotic disorders, unexplained somatic disorders, and/or unknown or missing diagnoses.c Includes behavioural interventions and/or cognitive interventions, with or without diagnostic assessment, psycho-education, relaxation strategies, skills training and/or interpersonal therapy.d Excludes behavioural interventions and/or cognitive interventions, and includes diagnostic assessment, psycho-education, relaxation strategies, skills training and/or interpersonal therapy.
Table 1 also shows the treatment profiles for all patients and those for whom outcome data were available. Patients in the latter group were less likely to have small numbers of sessions, and more likely to have received CBT in at least one session. Roughly equivalent proportions of both groups (less than one fifth) made a co-payment at any session; the vast majority in both groups received all sessions free of charge.
Changes on outcome measures from pre- to post-treatment
Table 2 shows the mean difference in scores on the nine most commonly used outcome measures: the Beck Anxiety Inventory (BAI) [16], the Behaviour and Symptom Identification Scale 32 (BASIS-32) [17], the Beck Depression Inventory (BDI) [18], the Depression Anxiety Stress Scales (DASS) [19], the Global Assessment of Functioning (GAF) [20], the General Well-Being Index (GWBI) [21], the Hospital Anxiety and Depression Scale (HADS) [22], the Health of the Nation Outcome Scales (HoNOS) [23], and the Kessler 10 (K-10) [24]. With the exception of the DASS, the mean differences were based on total scores; in the case of the DASS, the mean differences were based on scores for each of the three sub-scales because a total score on this instrument is not regarded as meaningful [19]. Across all measures, the mean difference was statistically significant and indicative of clinical improvement.
Pre- and post-treatment outcome scores on available outcome measures for patients receiving care through the ATAPS projects, 1 January 2006 to 30 June 2010
∗p <0.001
Predictors of improvement on outcome measures
Table 3 considers predictors of outcome as assessed by the most common measure used by Divisions, the K-10. As noted in Table 2, the K-10 is a patient-rated measure which assesses non-specific psychological distress and yields a total score of 10 to 50. A low score indicates low levels of stress and a high score indicates high levels of stress, so a positive difference between pre- and post-treatment scores indicates improvement [24].
Coefficients and 95% confidence intervals predicting change in Kessler-10 scores
Analysis of the residuals from the fitted model (not shown) supported the assumptions of constant variance and normally distributed errors. Table 3 shows that after adjusting for clustering by Division and controlling for the other variables in the model, outcome on the K-10 was associated with age, level of income, level of education, previous receipt of mental health care, number of sessions, treatment received and pre-treatment K-10 score. Outcome was not associated with gender, diagnosis or co-payment.
To be more specific, those in the older age group (≥ 65 years) showed an incremental improvement of 1.79 points on the K-10 over and above their younger counterparts (those aged < 25 years). Those on relatively higher incomes had a level of improvement that was 1.55 points higher than those on low incomes. Those who had completed high school to some level or had completed a tertiary qualification experienced improvements of between 1.36 and 1.58 points higher than those who had not completed high school. Those who had not received previous mental health care showed levels of improvement that were 1.38 points higher than those who had done so. Those who received 13–18 sessions of care showed lower levels of improvement than those who received six sessions (1.5 points lower). Those who had no categorized treatment fared relatively poorly, showing levels of improvement that were 2.37 points lower than those who received CBT. Finally, improvements rose as a function of pre-treatment K-10 scores, doing so at a rate of 0.53 points per each additional one point increase on the pre-treatment score.
Discussion
Limitations
Several caveats should be taken into account before interpreting the above findings. Firstly, the proportion of patients for whom pre- and post-treatment outcome data were available was less than optimal at 15%. It is possible that this may have introduced a systematic bias, such that those for whom no outcome data were available may have had poorer outcomes (if, for example, outcome data were unavailable because they dropped out of treatment). The profile of the patients for whom outcome data were available was fairly similar to that of all patients, although they tended to have fewer sessions and be less likely to receive CBT, which may have had an impact on their outcomes. It was not possible to compare the two groups on all relevant variables, and there may have been additional differences which could have influenced their outcomes. For example, it was not possible to gauge the presenting level of severity of symptoms for the total sample (because this would have relied on pre-treatment data being available from a single outcome measure). We would contend that our findings still stand, but recognize that they may present a somewhat over-optimistic picture.
Secondly, our analysis focused primarily on adults. Only 10 252 (9%) of the 113 107 patients receiving care through the ATAPS projects in the period of interest were aged under 18. The only child-specific outcome measure used by the ATAPS projects is the Strengths and Difficulties Questionnaire (SDQ), and fewer than 200 individuals had pre- and post-treatment outcome data on this measure. For this reason, no conclusions can be drawn about outcomes for children.
Thirdly, our examination of potential predictors of outcome was restricted to those variables which were available through the minimum dataset. These were reasonably comprehensive, but did not include some of the contextual factors that may have a significant impact on how well a patient responds to treatment (e.g. his or her support networks). In addition, they did not include alternative forms of therapy that the patient may have been receiving alongside care provided through the ATAPS projects. Ideally, it would have been desirable to control for medications that a patient may have been taking, since this may have influenced his or her outcomes. Data on medication use collected via the minimum dataset were not sufficiently detailed to permit this refinement to the analysis.
Fourthly, our reliance on data that were routinely collected from providers meant that we had no way of validating the available information against other data sources. There is no prima facie reason to suspect that providers were not providing accurate information but, for example, we were unable to confirm that when they said they were providing CBT that they were actually doing so.
Finally, and probably most importantly, it has been beyond the scope of our ATAPS evaluation work to include any sort of comparison group because of the widespread uptake by Divisions of the ATAPS projects. Without a control group it is not possible to determine the degree of improvement that might have occurred in the absence of treatment, although it is unlikely that this would match the magnitude of improvement shown here. It is also worth reiterating that patients typically experienced high levels of psychological distress prior to treatment, indicating that the ATAPS projects are reaching people with severe symptomatology. This group may have disorders that would be less likely to resolve if left untreated than their counterparts with moderate or mild symptomatology. Nonetheless, it is not possible to conclusively infer that the observed clinical improvements were due to the ATAPS projects.
Interpreting the findings
Despite the above limitations, the current study shows that people who receive care through the ATAPS projects are achieving positive outcomes, irrespective of the way in which these outcomes are being measured. These outcomes are not only statistically significant, they are clinically significant too. Many patients shift from quite extreme levels of symptomatology to much lower ones. Taking data from the K-10 as an example, the mean pre-treatment score of 31.29 is indicative of a severe mental disorder (range: 30–50), whereas the mean post-treatment score of 22.68 is suggestive of mild mental disorder (range: 20–24) [24,25]. According to the 2007 National Survey of Mental Health and Wellbeing, the mean K-10 score for the general adult population is 14.50 [26].
Some ATAPS patients appear to do better than others, although even for the latter the level of improvement is high. In terms of socio-demographic characteristics, older patients and patients who are of relatively higher socio-economic status make the greatest gains. The former finding requires further exploration; it is not intuitively clear why older patients would fare better than their younger counterparts. The latter finding is consistent with the contention that CBT may be more successful with patients who are relatively well educated (because it relies on a certain degree of conceptual ability and language skills), although other studies have produced equivocal findings in this regard. For example, a 2002 review by Hamilton and Dobson [27] found little evidence for such a relationship, whereas a more recent study by Myhr et al. [28] found that employment status was predictive of positive outcomes from CBT.
In terms of clinical characteristics, those who have no previous history of mental health care show greater levels of improvement than those who have previously received treatment. One interpretation of this finding might be that a considerable proportion of those who are new to the system may have had difficulties accessing services in the past, and these people may be particularly likely to be compliant with treatment now that they have been given the opportunity to access care. Independent of this finding, those with comparatively higher pre-treatment K-10 scores [24] (i.e. worse baseline manifestations of psychological distress) demonstrate greater levels of improvement than those with lower pre-treatment scores. This finding is consistent with a recent study by Prytys et al. [29] which found that those who were above clinical cut-offs on given measures of depression benefited more from CBT workshops than those who scored below this threshold at presentation. Our finding is at odds, however, with the previously mentioned review by Hamilton and Dobson [27] which found that, on balance, individuals with more extreme symptoms of depression appeared to be less responsive to CBT. One explanation for the pattern we observed may be that those with higher original scores may have greater opportunities to improve before they hit a ‘floor’ score. Another explanation may be that, arguably, they have more ‘invested’ in treatment.
The findings regarding the relationship between treatment variables and outcome indicators provide indicative evidence that the ATAPS projects are offering an appropriate model of treatment. Some have suggested that the recommended list of therapies available through ATAPS is too prescriptive, and that the focus on evidence-based care (particularly CBT) potentially excludes other useful strategies. The fact that those who receive CBT have better outcomes than those who receive unknown or unclassified treatment provides suggestive, though not conclusive, evidence that the overarching treatment approach of ATAPS is suitable. Some have also argued that six sessions of care are insufficient, but the current analysis would suggest that they are not only sufficient but perhaps optimal in many cases; greater numbers of sessions are associated with poorer outcomes, even after taking into account factors such as baseline levels of morbidity (as assessed by the K-10 [24]) and diagnosis. These findings are broadly consistent with conventional wisdom regarding the evidence base for CBT. The above-mentioned meta-analyses by Haby et al. [2] and Lynch et al. [3] demonstrated that CBT was an effective treatment for anxiety and depressive disorders, and the one by Haby et al. showed that there was no significant effect of increasing weeks of treatment.
Implications for the ATAPS projects
Our findings provide some evidence to suggest that the Tier 1 ATAPS projects are achieving the aim encompassed in the title of the overarching programme: better outcomes in mental health care. Although these outcomes are more consistently achieved for some groups than others, the benchmark is sufficiently high that even the groups that fare relatively less well are still showing strong improvement in absolute terms. There does not appear to be a case for restructuring the Tier 1 projects, although consideration might be given to ways of optimizing outcomes for all groups. Similarly, there does not appear to be evidence to suggest that particular groups who are targeted by the Tier 1 projects might be better served by the more tailored Tier 2 or Tier 3 projects. Essentially, the findings are supportive of retaining the status quo with respect to Tier 1 projects.
From an evaluation perspective, more comprehensive collection and collation of outcome data would be desirable, as would rationalizing the number of outcome measures used by Divisions (perhaps to the most common two, the K-10 [24] and the DASS [19]). Strengthening the ATAPS outcome data collection in this way would enable us to make more definitive statements about the achievements of the projects.
Conclusions
Patients who receive care through the ATAPS projects are making considerable clinical gains. A range of socio-demographic, clinical and treatment-based variables are associated with the levels of outcomes achieved, but improvements are still substantial even for those in the relatively disadvantaged groups. Divisions have worked hard to tailor their ATAPS projects to local needs [8], so it is perhaps not surprising that our findings are positive.
Footnotes
Acknowledgements