Does methodology influence the outcome of psychotherapy randomised controlled trials: A look at Australian and New Zealand Studies

How psychotherapy works remains uncertain. There are two often sharply divergent opinions. One posits specific treatment effects for patients with specific diagnoses. The other argues for the widespread operation of common factors such as therapist/patient alliance, therapist’s allegiance to a theoretical orientation and general effects such as the mobilisation of hope and expectation. This uncertainty is reinforced by psychotherapy randomised controlled trials (RCTs) sometimes reporting equal efficacy but sometimes reporting that one psychosocial treatment is superior to another. These conflicting findings have in turn often been ascribed to methodological issues.

To discuss these questions, I have reviewed all psychotherapy RCTs published in the Australian and New Zealand Journal of Psychiatry (ANZJP) in 2013 and 2014. Five studies were published. There are four initial observations. First, agreeing with the inconsistent literature, three RCTs (Jordan et al., 2014; Malbos et al., 2013; Turner et al., 2013) report no difference in outcome between psychological interventions (the ‘negative’ studies), while the remaining two (Battersby et al., 2013; Tan and King, 2013) report a significant effect favouring the evaluated treatment (the ‘positive’ studies). Second, all RCTs report improvements in all groups enrolled in the studies no matter which arm they were randomised to. Third, the subjects in each trial are from very different groups ranging from patients with schizophrenia to patients with cardiac disease and depression. Fourth, the length and type of treatment vary significantly as does the length and type of control treatment. Despite all being psychotherapy RCTs, the studies are, in fact, not particularly comparable.

Do these methodological differences influence the outcome of the RCTs? On the face of it they appear to. First, all patients improved suggesting that natural remission, symbolic treatment effect or some common factor was at least partially responsible for the observed changes. This finding suggests that large sample sizes may be needed to demonstrate differential efficacy. The two larger studies (Battersby et al., 2013; Tan and King, 2013) while still modest in size were the ones that reported positive effects. Second, making the control treatment as different as possible from the evaluated treatment may be more likely to lead to measurable differences; in the positive studies, wait-list control (Battersby et al., 2013) and physical exercise (Tan and King, 2013) were used. The studies who added cognitive therapy to one arm but provided some baseline treatment to both groups (Malbos et al., 2013; Turner et al., 2013) or compared two structured psychotherapies (Jordan et al., 2014) reported no difference. Third, using subjects with more chronic psychopathology may be more likely to lead to measurable differences. Patients with schizophrenia attending vocational training or day rehabilitation programmes were used in one of the positive studies (Tan and King, 2013). The other enrolled Vietnam veterans with alcohol use disorder and a chronic condition (Battersby et al., 2013). In contrast, the negative studies used outpatients with depression (Jordan et al., 2014; Malbos et al., 2013) and agoraphobia (Turner et al., 2013). Fourth, having a longer treatment was associated with positive findings. The patients with schizophrenia received 60 hours of cognitive remediation or physical exercise (Tan and King, 2013), while the Vietnam veterans were given 9 months of a multi-faceted intervention (Battersby et al., 2013). In contrast, the therapy consisted of 6 seminars in the cardiac patients with depression (Turner et al., 2013), 10 sessions in the patients with agoraphobia (Malbos et al., 2013) and up to 12 sessions of cognitive behavioural therapy (CBT) or metacognitive therapy (MCT) in the depressed patients (Jordan et al., 2014).

In summary, when attempting to show efficacy in a psychotherapy RCT, there are a number of methodological factors that may be worth considering given the trials published locally. All patients are likely to improve regardless of what intervention they receive. This may be less pronounced in patients with more severe and chronic conditions, so using such patients may be an advantage. Large sample size is also (always) desirable. The control treatment should be as different as possible from the evaluated treatment, and longer treatments may be more likely to show a differential effect. These factors make conducting and funding psychotherapy RCTs time-consuming and expensive, so it is little wonder that evidence-based psychotherapy treatment remains limited and at times contradictory.

These five studies do not tell us how psychotherapy works. However, they indicate the importance of common therapeutic factors in all treatments. They also reveal how methodological variables including subject characteristics, control treatment choice and length of treatment are likely to have an influence on outcome. The positive studies suggest that cognitive remediation is better than physical exercise in improving cognitive functioning and that the Flinders Programme was superior in reducing alcohol use disorders identification test (AUDIT) scores than usual care. One could validly argue that neither control conditions were bona fide treatments in that neither delivered a treatment with specific therapeutic actions that were consistent with the rationale presented to the patient. One did not deliver a treatment at all. (It is interesting to note that the exercise treatment did improve physical fitness significantly more in the control group than in those who received cognitive remediation (Tan and King, 2013)). The ‘positive’ studies might support the hypothesis that psychotherapy is better than no psychotherapy but not the concept of specificity, since specific treatments were not compared. The three ‘negative’ studies suggest that bona fide treatments all have similar efficacy again not supporting the concept of specificity. Overall, there is nothing in these studies to support rejecting the null hypothesis that specific ingredients of a psychosocial treatment are not remedial for the disorder. The Dodo bird strikes again.