An axeman in the cherry orchard: Early intervention rhetoric distorts public policy

In psychiatric as well as other reform processes, logic and scientific evidence are necessary but insufficient. Rhetoric, marketing, effective networking, altruistic promotion of a vital public health issue, economic arguments and a confluence of common interests have fuelled the momentum and are vital for real reform to take root.

McGorry et al. (2005: s2)

Recently, I have provided detailed critical analyses of sections of the early intervention in psychosis (EI) literature, and concluded that there is evidence of selective reporting likely to substantially distort conclusions regarding efficacy and cost effects (Amos, 2012a, b, c, d, e). I make specific points regarding inadequate methodology and unsustainable conclusions within and between research articles and commentary. In their replies, neither Yung (2012b) nor Friis et al. (2012) address a single one of these methodological concerns. Instead, they suggest that my analyses are personal criticisms (Yung, 2012b) and attacks based on an unstated agenda (Friis et al., 2012). In this paper, I note the errors in these papers, and demonstrate why it is important that methodological inadequacies be identified.

In defence of selective analysis

Professor Yung acknowledges that I have argued her approach lacks scientific rigour, takes issue with my identification of an incorrect reference, and attributes to me the accusation that she is wrong to use a conference abstract rather than peer-reviewed literature. Although I have suggested a lack of scientific rigour, it is not clear how this constitutes a personal attack. Professor Yung does not show why, if it is important that the OPUS group report a functional improvement, it is not important that the Treatment and Intervention in Psychosis Study (TIPS) group found a functional worsening. Nor is it shown why, if it is important that the TIPS group found a symptomatic improvement, it is not important that the OPUS group found no such improvement (Yung, 2012b).

It is instructive that in response to my conclusion that her analysis lacks scientific rigour, Professor Yung refers to Norman et al. (2011) as evidence of positive EI outcomes without noting that this study compared outcomes within the Canadian PEPP study to published results of the Scandinavian OPUS study. That is, not only is there no randomisation, no case–control, and no matching, there is a post-hoc comparison between outcomes of two unrelated studies in different countries with no identification of possible sources of bias. To respond to an analysis suggesting insufficient scientific rigour with this paper supports the original conclusion.

I did not suggest Professor Yung was wrong to refer to a conference abstract by the TIPS group (McGlashan et al., 2011), but that the sceptic might wonder why she preferred this type of evidence to the authors’ peer-reviewed research (e.g. Larsen et al., 2011). I noted that she had neglected to refer to another abstract in the same issue by the same authors which indicated that early detection of psychosis (ED) slightly reduced psychopathology but doubled the rate of severe functional deficit (Friis et al., 2011), another example of selective report.

I am mystified that Professor Yung would object to my identification of an error in her reference to the abstract (McGlashan et al., 2011). As this was a reference to one of two similar abstracts by the same set of authors in an unindexed 327-page document, the fact that Professor Yung’s reference had the wrong volume and title made the reference difficult to locate. I note that Professor Yung’s reply repeats the error (Yung, 2012b).

Selective report of results

Examining the paper (Hegelstad et al., 2012) generated from the abstract (McGlashan et al., 2011) illustrates the danger of relying upon an abstract rather than peer-reviewed literature. As I have demonstrated elsewhere (Amos, 2012a, c, d, e), Hegelstad et al. (2012) show the same selective bias as Professor Yung, not analysing results which suggest worse outcomes as a result of ED, while emphasising positive results, using different measures at different times.

The clarity of Friis et al.’s (2012) reply is impaired by the fact that they appear to be referring to an earlier draft of my Viewpoint than the final published form; nevertheless, I believe it is reasonable to expect the TIPS group and Professor Yung to acknowledge that contrary to their false statements, I have correctly identified an error in Professor Yung’s reference to McGlashan et al. (2011), and analysed both abstracts.

As Friis et al. (2012) note, my analysis of Professor Yung’s argument is selective. However, as I was demonstrating selective use of research, not assessing the relative merits of different sides of the EI debate, this is not a fault, but the appropriate focus. Nevertheless, it is interesting that Friis et al. (2012) acknowledge that they reported two sets of results from the same experiment at the same conference: one with positive results, the other with negative results. Professor Yung refers to the positive abstract (McGlashan et al., 2011); she does not refer to the negative abstract (Friis et al., 2011). This raises the question why the authors split the results this way. Yung (2012a) demonstrates that it facilitates the selective reporting of evidence.

The presentation of TIPS’ results changes between abstract and article. In particular, the negative abstract notes that ED patients had double the proportion of non-recovered patients with no capacity for independent living (21% vs 11%, significance unreported) (Friis et al., 2011). In their peer-reviewed paper, Hegelstad et al. (2012) report a related figure (patients living independently, 78% to 62%). They do not report the comparison suggesting that double the proportion of ED patients had no capacity for independent living. They dismiss the lower rate of independent living in the ED group because living independently was not correlated with their novel measure of recovery. This is selective in two ways. It does not acknowledge that there are different measures of function, and the ED group appear better than the treatment as usual (TAU) group on one functional measure (employment), and worse on another (independent living). It also ignores the fact that while recovery might measure a high level of function, an inability to live independently appears at least prima facie a good measure of an extremely poor level of function. By dismissing the independent living results, Hegelstad et al. (2012) do not address the paradox of why ED would lead to both extremely good and extremely poor outcomes in different subsets of patients.

Both Yung (2012b) and Hegelstad et al. (2012) argue that the 10-year TIPS results are affected by selective attrition of ED patients with better outcomes at 5 years. However, neither group acknowledge that the ED patients who were not followed up at 10 years had significantly longer median duration of untreated psychosis (DUP) than those who were followed up. It is logically impossible for there to have been, at the same time, selective attrition of ED patients with better outcomes caused by reduction of DUP, and selective attrition of ED patients with increased DUP. If these authors maintain that the results at 10 years are due to selective attrition of ED patients with better 5-year outcomes, it would seem logical that they accept that selective attrition of ED patients with longer DUP might be the cause of the symptomatic advantage of the ED group throughout the study. It would be interesting to see data examining the relative DUP of ED patients lost to follow-up at earlier time points in this regard.

Resurrecting the TIPS hypothesis

Hegelstad et al. (2012) do not address their original reasons for establishing the TIPS trial. The explicit justification for the trial was to disentangle the causal effects of DUP from confounding factors in the outcomes of psychotic illness (Melle et al., 2004). Reporting 1-year results they noted that while positive symptoms, general symptoms, and time to remission were no different between the groups, negative symptom differences remained significant. They stated that if earlier treatment has no effect on the progression of psychotic illness, symptom outcomes should converge (Larsen et al., 2006). At 10-year follow-up, they do not acknowledge the actual convergence; thus, they do not consider the implications for the original hypothesis, that reducing DUP causes improved outcomes, rather than just being associated with them (Hegelstad et al., 2012).

In addition, Hegelstad et al. (2012) do not consider the most obvious alternative hypothesis, that reducing the threshold for entry into treatment recruits a group of patients with an enduring pre-intervention tendency towards less severe psychosis. Several paradoxes in their research naturally resolve with this alternative. They acknowledge that their intervention reduced the threshold for entry into treatment irrespective of the DUP (Hegelstad et al., 2012). ED patients showed less psychopathology at baseline and shorter DUP, but a shorter DUP was correlated with worse outcome measures at baseline (Melle et al., 2008). At 1-year follow-up they reported identical survival curves for time to remission of ED and TAU patients (Larsen et al., 2006). They also reported that DUP did not predict outcomes at 2- and 10-year follow-up, while coming from the ED area did predict outcomes (Hegelstad et al., 2012; Melle et al., 2008).

It has been proposed that psychosis forms a continuum in the general population comprising those with psychotic disorder and a much larger group with less severe psychotic experiences who behave similarly to psychotic disorder (Linscott and van Os, 2012). It appears plausible that lowering the threshold for entry into treatment would recruit from people along the borderline between stable psychotic experiences and frank psychotic disorder, with less severe psychopathology. Such an effect appears better matched to the results reported by the TIPS group. In particular, it could explain why coming from the ED area predicted recovery, while DUP did not.

Expert opinion and funding of EI

Mihalopoulos et al. (2012) are another EI group who have recently demonstrated a strong response to critical analysis. They do not attempt to refute my very specific conclusion that all the cost-effectiveness analyses of early-intervention programmes selected a positive measure from amongst multiple possible comparisons, increasing the likelihood of bias (Amos, 2012b). Instead, they suggest that by identifying this source of potential bias arising from poor methodology, I have made a strong accusation of professional misconduct.

This misrepresentation of my position occurs in response to a systematic review which concludes that there is no compelling evidence that EI reduces costs (Amos, 2012b). In their reply, Mihalopoulos et al. (2012) refer to the National Health and Hospitals Reform Commission (NHHRC) Report (Bennett, 2009) as the basis for expanded funding for EI in the 2011 Australian Federal Budget. This report based its recommendations, including expansion of the EPPIC framework for EI, on an editorial by McGorry et al. (2007) and discussion paper by Hickie (2008). McGorry et al. (2007) point to research by the TIPS, OPUS, and Lambeth Early Onset (LEO) groups to conclude that reducing DUP will almost certainly improve outcomes, and that there is a critical period for psychotic illness within which intensive treatment can change long-term outcomes from gradual deterioration to stable improvement. The NHHRC Report’s expectations about the return from expenditure directly reflect McGorry et al.’s (2007) optimistic analysis, which is not supported by the research cited.

OPUS and LEO

The OPUS and LEO groups report the effects of intensive treatment of patients with first-episode psychosis with reversion to TAU after 2 years (Bertelsen et al., 2008; Gafoor et al., 2010). The OPUS group randomised 547 patients with first-episode schizophrenia spectrum diagnoses (International Classification of Diseases (ICD)-10) to EI or TAU (Bertelsen et al., 2008). At 2-year follow-up there were small advantages for the EI group on positive and negative symptom scales and the Global Assessment of Functioning (GAF). At 5 years there were no differences on these primary measures or remission/course of illness; there were also no differences in the secondary measures of depression, substance abuse, suicidal behaviour, work/education, use of antipsychotic medication, or hospitalisation in years 3–5. Fewer EI patients were living in supported housing (4% vs 10%). The authors concluded ‘the benefits of the intensive early-intervention program after 2 years were not sustainable, and no basic changes in illness were seen after 5 years from the start of the program’ (pp. 769–770).

The LEO group randomised 144 patients with first- or second-episode psychosis to EI or TAU. EI patients averaged about 2.5 years with the specialist service before transition to other care. At 18-month follow-up they found small advantages for the EI group on number of admissions, more employment, and better quality of life. At 5-year follow-up with 99 patients they noted that their results concorded with the OPUS study in finding no significant differences on primary or secondary outcome measures, although they also noted limited power to detect real differences (Gafoor et al., 2010). While Bertelsen et al. (2008) accept the evidence of their own study, Gafoor et al. (2010) were equivocal, concluding ‘the extent to which [the] benefits [of EI] persist in the longer term remains unclear’ (p. 375).

Duration of untreated psychosis and the critical period

As can be seen, the main studies in the EI literature, which have been used to justify increased expenditure on EI, provide evidence that reducing the DUP does not improve long-term outcomes, and do not support the hypothesis that intensive treatment in a critical period improves prognosis. Three main results suggest that while intensive treatment can improve outcome measures while intensive treatment continues, there is no evidence of the lasting advantages which would be expected if the disease process was altered by EI:

• reducing DUP does not change time to remission (Larsen et al., 2006);

• DUP is negatively correlated with outcomes at time of detection, and does not predict outcomes at 2- and 10-year follow-up (Hegelstad et al., 2012; Melle et al., 2008);

• early intensive treatment for 2 years does not improve outcomes at 5 years (Bertelsen et al., 2008; Gafoor et al., 2010).

Some have suggested that EI programmes should be expanded from 2 years to 5 years or longer (McGorry et al., 2007; Yung, 2012a). There is some reason to believe that the small improvements sustained while intensive intervention continues might also be maintained over longer periods. However, the evidence does not suggest that extending the intensive intervention period would change the course of psychotic illness.

Evidence and rhetoric

The tendency to selectively report and liberally interpret results from the EI literature that I have described above is illustrated in a recent editorial on these pages. During a passionate debate, McGorry (2012) claimed that research by the OPUS and LEO groups supports the disease-modifying effect of EI (p. 315). He refers to Bertelsen et al. (2008), who explicitly rule out a disease-modifying effect (see above), and the initial results of the LEO team (Craig et al., 2004; Garety et al., 2006), rather than the more recent LEO results (Gafoor et al., 2010) which confirm the absence of long-term improvements of the OPUS team. Professor McGorry is one of two leading psychiatrists referenced in the NHHRC’s Report (Bennett, 2009), which Mihalopoulos et al. (2012) identify as the basis for the expansion of Federal funding of EI. Given the weight of evidence in the opposite direction, it would be beneficial for Professor McGorry to demonstrate how the research to which he refers suggests a disease-modifying effect of EI.

Trust and scientific rigour

Despite being accused of personal incivility and an unstated agenda, in a spirit of scientific detachment I have given detailed examples of selective analysis and report which is likely to misrepresent the evidence regarding early intervention in psychosis. I call for similarly detached responses which examine the identified issues. Whatever the response, the best antidote for the expert reader is to read the references and draw their own conclusions. As the NHHRC Report illustrates (Bennett, 2009), the funding of mental health services is heavily influenced by expert opinion. Robust debate and critical analysis is the only way to ensure that non-expert decision-makers can place their trust in the advice they receive regarding allocation of resources to competing mental health programmes. If their peers do not identify flaws and insist upon high standards from EI proponents, the non-expert bodies who rely upon their advice will continue to make crucial decisions on grounds that can be questioned.