Burning the straw men: Intellectual integrity in the early intervention debate

Mihalopoulos et al. (2012) describe the review of economic evaluations of early intervention in psychosis (EI) in Amos (2012a) as unfair, unbalanced, and factually incorrect. Below, I demonstrate that these criticisms are based on misunderstanding and misinterpretation. Before descending into the minutiae, it is important to examine the implications of their article. The authors include the first or senior author of six of the nine reviewed articles, as well as senior EI research figures. It seems reasonable to assume that if there is significant research going on regarding EI, or economic evaluations of it, this group is ideally placed to identify it and argue its merits.

It is therefore significant that Mihalopoulos et al. (2012) do not demonstrate that economic evaluations of EI have shown reduced costs. They suggest McCrone et al. (2009) might have been added to the review, while ignoring that these data have been considered in McCrone et al. (2010). They do not argue that the addition would have changed the conclusions of the study. They do not identify any works that support a different conclusion. Their failure to even attempt to show why the overall conclusions are incorrect concedes the point apparent in the ‘Cost effect’ and ‘Clinical effect’ columns of Table 1 (Amos, 2012a). The literature does not support a different conclusion.

Mihalopoulos et al. (2012) conclude by attributing to Amos (2012a) the ‘strong accusation’ that the authors he reviews have engaged in professional misconduct. They base this extremely serious claim on my analysis that the reviewed papers report the positive results from amongst multiple possible comparisons without correction. Cursory examination of Amos (2012a: p. 728) reveals that the reviewed papers report the cost-effectiveness of EI in improving five different clinical and functional measures. All select from multiple comparisons. Not one reports the same cost-effectiveness measure as any of the others. Carr’s (2012: p. 787) commentary on my review appears apposite: ‘The probability that research findings are not true increases with…the use of multidimensional outcomes with ‘cherry-picking’ of the best results’.

Thus Mihalopoulos et al. (2012) not only do not acknowledge the obvious possibility of bias, they suggest that by identifying it, I have accused my colleagues of professional misconduct. Although I clearly have not done this, I am concerned that the authors appear to prefer not to identify systematic methodological weaknesses. The end result of this can be seen in Hegelstad et al. (2012), the 10 year follow-up to the five year results of the TIPS group represented in the review by their five year results (Larsen et al., 2011).

Five years after a public health intervention which successfully reduced the duration of untreated psychosis, TIPS claimed to have demonstrated sustained clinical and functional benefit based on small relative improvements on the negative, depressive and cognitive scales (PANSS), and more frequent contact with friends (Larsen et al., 2011). They did not analyse the fact that the EI group spent 50% more time in hospital, simply claiming policy differences between the treatment and control regions. Having identified a confound significant enough to increase average patient hospitalisation from 30 to 45 weeks over 5 years, they did not consider whether it might also affect symptom scores. They did not explore the absence of differences in remission rates in their discussion.

At 10 years, Hegelstad et al. (2012) were disappointed that the evidence from symptom scores had reversed. All the measures in the original study now either indistinguishable or favouring the control group (they simply didn’t report hospitalisation), they constructed an entirely new measure. The only result in their study that suggested an advantage for the treatment group at 10 years was a measure they created at 10 years. As at 5 years, they did not consider whether measures favouring the control group might reflect a real advantage for that group. The potential for bias becomes obvious by considering whether they would have constructed a new measure if remission rates had favoured EI.

An editorial accompanying Hegelstad et al. (2012) concludes ‘depending on how you evaluate the impact of rater non-blindness, logical consistency of effects, and models of controlling for multiple comparisons, the advantages of early-detection programming may or may not remain’ (Cannon, 2012: p. 347). This type of study might find it more difficult to be published in a literature more interested in methodological rigour. As I show, the evidence of bias Mihalopoulos et al. (2012) detect in Amos (2012a) comprises misinterpretations and misunderstandings. I do not claim that EI is not effective in the long term, or that it has no economic benefits, only that these conclusions are not well established.

Analytic competence and intellectual integrity

Although they do not explicitly identify factual errors versus unbalanced conclusions, Mihalopoulos et al.’s (2012) analysis demonstrates that what they believe to be error arises only from their poor understanding of the research. They suggest that Amos (2012a) is in error to note a change in the parameter of transition to psychosis used to construct the model of Valmaggia et al. (2009). They say that an estimate of transition to psychosis by different authors in different countries does not invalidate the estimate used in the model. This demonstrates that they are unaware that the original estimate is based upon work by Yung et al. (2003, 2004), and that these same authors later changed their estimate (Yung et al., 2008). They have concluded that Amos (2012a) is in error reflecting perhaps their poor understanding of the published literature.

Similarly, Mihalopoulos et al. (2012) contradict Goldberg et al. (2006), who conclude that ‘time series analysis of our data did not confirm that the reduction in costs was clearly the result of the introduction of the [EI] service’ (p. 902). Mihalopoulos et al. (2012) suggest that the latter authors, despite not finding that EI freed up beds, were arguing that ‘beds freed by [EI] patients are available to be alternatively used’ (p. 809). Nowhere does Amos (2012a) claim that these authors did not include behavioural measures, and refers directly to such measures in Table 1.

Some of Mihalopoulos et al.’s (2012) points may be rooted in personal sensitivity to impersonal criticism (p. 809): ‘Amos also describes the retrieval of patient files in the Mihalopoulos et al. study as ‘ad hoc’. This is an unfairly disparaging description and one designed to seed doubt in readers’ minds about the rigour of this study.’

An ad hoc measure is one where the method was ‘made for purpose’ after the original design, such as file retrieval 5 to 8 years after the initial intervention. Ad hoc measures are a potential source of bias. Again, Mihalopoulos et al. (2012) appear to experience the identification of possible bias as an unfair attack, even when accurate.

Factual inaccuracies

Mihalopoulos et al. (2012) make the serious accusation of factual inaccuracies (p. 808):

Amos begins his review by claiming that the expansion of EIP services in Australia is responsible for the contraction of other services, specifically general practitioner managed psychological interventions. No evidence for this assertion, which has been made previously by others, is provided.

The authors state I have not provided evidence for an assertion I have not made. Moreover, they do not identify which others have previously made this assertion. The relevant passage in Amos (2012a: 720) reads:

Recent events in Australia demonstrate the competition for resources within mental health. Despite resistance from the medical profession, funding of GP managed psychological interventions has been significantly reduced at the same time as large financial resources have been steered towards EI.

This juxtaposes two events occurring at the same time with different outcomes for different groups with no suggestion of a causal relationship. The statement is accurate, unbiased, and relevant, as it identifies a set of resource-limited systems in which winners and losers are determined by economic considerations.

Again introducing a bias that does not exist in the original, Mihalopoulos et al. (2012) propose (p. 808):

…because some critics in response to the reforms underway in Australia have questioned the evidence showing that better functional outcomes are achieved by specialised EIP services, Amos asserts that early psychosis researchers have tacitly accepted these critiques and sought to buttress the case with cost-effectiveness data.

Compare this with my statement (Amos, 2012a: 719): Faced with ongoing scepticism about clinical outcomes, proponents have argued that EI programmes should be implemented to save mental health resources to be used in other areas (McGorry, 2011).

I have not argued that EI researchers have accepted the criticisms of others. I have also provided a reference to the arguments of a strong EI advocate to illustrate my meaning. Mihalopoulos et al. (2012) appear to commit the error they accuse me of.

Basic accounting principles

Mihalopoulos et al. (2012) do not acknowledge the accounting points in the review. In a public mental health system where clinician resources are constrained by caseloads and paid for at the level of individual workers, cost attribution based upon individual instances of care introduces substantial distortion.

Imagine two services, A and B, each managing 100 outpatients with caseloads of 10:1 and 25:1. On average, service A patients consume 10% of their clinician’s contact time, service B patients 4%. Service A requires 10 clinicians, B only 4. Assuming a base wage of $100,000, the annual cost of clinician care per patient in service A and B is $10,000 and $4000, respectively. The yearly wage bill of service A is $1,000,000; of service B it is $400,000. These costs do not change with variation in number of contacts. All the reviewed literature calculate unit costs, as described by Mihalopoulos et al. (2012); it is because they calculate costs in this way that their estimates are distorted. If average service A and B patients each have 10 1-hour appointments per year, the unit-cost approach calculates the cost to the services to be identical. However, costs remain $6,000 higher for the average patient in service A regardless of the number of contacts.

The authors fail to acknowledge a related potential distortion in McCrone et al. (2010). Amos (2012a) notes that the provision of an out-of-hours service is likely to increase costs. While Mihalopoulos et al. (2012) insist that the paper included all costs, the failure to adequately describe how they did this does not allow the review to come to the same conclusion. A unit-cost approach to an out-of-hours service is particularly prone to distortion because it is likely that much of the time is spent without patient contact. If a weekend clinician spends an 8-hour day without contact, none of the cost of providing the service is measured by the unit-cost approach.

It is curious that Mihalopoulos et al. (2012) identify a lack of attention to the concept of opportunity cost as a weakness of Amos (2012a). Of course, the reason that it is not discussed in the review is because the concept is universally absent from the reviewed literature. A review of a different set of articles in which one or more papers considered opportunity cost or in any way tried to measure or analyse these concepts might have usefully included them. Nevertheless, much criticism of EI refers to opportunity cost in non-technical language. Castle (2012), for example, questions why resources are sequestered for particular populations when the same techniques can be shown to be effective for all patients.

Mihalopoulos et al. (2012) do not consider cited evidence that attempts to reduce total costs by reducing hospitalisation have failed. Although they acknowledge that short-term cost savings are unlikely from EI, they also state (p. 809):

… a bed not used for an EIP patient will be available to be used for another patient or purpose, with benefits associated with such utilisation. If the bed is not used, the variable costs associated with that bed are released as financial cost savings. In the long-term all resources are variable and there can be planned contraction of service use.

This is fine as theory. However, Amos (2012a) refers to evidence that past attempts to achieve cost savings by reducing hospitalisation rates, motivated by such theories, have led to increased total costs. Again, Mihalopoulos et al. (2012) do not provide evidence for their assertions.

Minor points

Mihalopoulos et al. (2012) find it unreasonable to detect the possibility of bias in Mihalopoulos et al. (2009). This is a historical case-control study, with large attrition differences, where most of the observed effect is due to a subgroup, and the outcomes are grossly different from a comparison study with more rigorous design (Bertelsen et al., 2008) as well as the rest of the literature. The study is an ad-hoc addition to a pre-existing experiment (eg it was not part of the original design). It is therefore reasonable to conclude that the study involves multiple potential sources of bias.

Discussing Valmaggia et al. (2009), Mihalopoulos et al. (2012) misinterpret the statement that subsequent research by the team that justified an initial parameter had ‘invalidated the assumptions’ of the model as ‘invalidating the model’ (p. 810). They also do not acknowledge that I had in fact praised the model for its sensitivity analysis, which allowed it to survive invalidated assumptions, by comparison with Serretti et al. (2009), where the absence of a sensitivity analysis meant that the model was invalidated along with its assumptions.

When I propose that the evidence for the long-term efficacy of EI is not compelling, they accuse me of asserting that ‘EIP services do not work over the longer term’ (p. 810). The latter is in reference to a reviewed article which itself speculates that the long-term impact of EI ‘may not be substantial’ (McCrone et al., 2010: 382). Where I note that the use of patient report measures ‘reduc[ed] the accuracy of their cost estimates’, they report this as ‘Amos argues that McCrone et al. did not accurately assess the costs associated with EIP services’ (p. 809). This clearly alters the meaning, and to proffer it as evidence of bias appears to contradict McCrone et al. (2010) themselves, who list patient report as a limitation of their study (p. 381).

It should be clear that Mihalopoulos et al. (2012) have attempted to create the impression of a lack of balance in my review by removing the qualified, measured language I use and substituting dogmatic, absolute statements whose accuracy varies from misleading to mistaken.

Concessions

Mihalopoulos et al. (2012) do identify one error: Phillips et al. (2009) should be identified as CMA rather than CEA in Table 3 (Amos, 2012a). It is also true that I was unable to recruit a second reviewer. I approached many people, including the Clinical Director of the EI service in my district. The director indicated that, interesting as the paper would be, given the ongoing tender process for funding for EI projects, it might be discreet to leave the name of the EI service off the paper. The current circumstances might support that director’s wisdom.

Conclusions

In sum, then, Mihalopoulos et al. (2012) suggest that Amos (2012a) is unbalanced because it effectively identifies multiple sources of bias in the articles it reviews, suggests sources of error in the calculation of outpatient costs, and expresses doubt that savings can be realised in practice, based on cited evidence. However, the authors fail to cite or analyse any evidence for the proposition that EI costs less than treatment as usual, and argue that identifying possible biases in their research is akin to accusing them of professional misconduct. Consequently, Hegelstad et al. (2012) is presented as an example of the results of this approach to research.