Counting the cost: Commentary on a systematic review of cost of early intervention in psychosis

It has been claimed that most published research findings are false (Ioannidis, 2005). The probability that research findings in medicine are not true increases with:

• small sample sizes and consequent low statistical power;

• small effect sizes;

• greater number and lesser selection of relationships tested in a given study, as in exploratory research or ‘fishing expeditions’;

• greater flexibility in design, definitions, choice of outcomes and modes of analysis used, such as in the use of multidimensional outcomes with ‘cherry-picking’ of the best results;

• greater financial and other interests or prejudices in the particular research field, including an investigator’s strong commitment to their own beliefs and findings; and

• ‘hot topics’ with numerous scientific teams working in the same area where there may be intense competition to achieve impressive results (Ioannidis, 2005).

Finally, the problem of undetected bias – in study design, data collection, analysis strategies and presentation of findings, as in selective or distorted reporting – increases the likelihood that the results reported are false. Replication, of course, is the key to ultimately homing in on whether a finding is true or false, but replication studies are not attractive to discovery-oriented scientists or granting bodies, although they are essential to the advance of science.

The foregoing is not intended to be used here as a framework within which to comment on the 11 individual studies reviewed by Amos in this issue of the Journal (Amos, 2012) as to whether their findings are likely to be true. Rather, it is intended as a reminder of the continuing need for critical evaluation of research findings, and that all published studies ought to be viewed through the prism of doubt, lest one embrace every new finding as revealed truth.

In the paper by Amos (2012) we can see a similarly critical stance with respect to the literature on the costs of early intervention (EI) in psychosis. Amos (2012) places his critique within a framework for economic evaluations of health care programs developed in the USA by the Panel on Cost-Effectiveness in Health and Medicine (Weinstein et al., 1996), as subsequently elaborated for economic evaluations more broadly by Drummond et al. (2005). Amos (2012) then assesses each of nine economic studies selected (two of the original 11 were quasi-experimental studies that measured clinical effects only and not costs) against 10 key elements of economic evaluation derived from the above sources. There were only two randomized controlled trials (RCTs), four studies used historical controls and two were cost-effectiveness modelling studies. The main criticisms of the literature reviewed were:

• a lack of evidence for the clinical superiority of EI;

• evidence of bias that was not accounted for;

• failure to correct for multiple comparisons;

• conceptual errors in calculating costs and supposed cost reductions;

• methodological problems of small sample sizes, absence of power analyses, and weaknesses in identifying the costing viewpoint with a general neglect of the societal perspective; and

• design problems such as use of post hoc historical and regional case controls.

Clearly, the case for cost-effectiveness of EI for psychosis has not been made, and similarly the argument that cost savings from EI would be available for reinvestment in other services is not based on credible evidence. This does not mean that EI for psychosis should be abandoned, nor that cost advantages of EI can be ruled out. It means that the quality and strength of the evidence concerning the relationship between the costs and the utility of EI is lacking. How can this gap in our knowledge base be filled?

First, methodologically rigorous studies especially designed to measure the costs and effects of EI need to be conducted. It is not adequate simply to add to a study, which has been designed for another purpose, cost collection methods in an opportunistic, post hoc way. Cost data collection needs to be built in from the outset as a primary variable of interest, tied to a central hypothesis, and not treated as an afterthought. Power calculations and sample size determination should be conducted in a way that is appropriate to the hypothesis being tested, and the RCT should be taken as the gold standard in comparing costs and effects of two or more interventions.

Second, sound cost data collection requires specifying the perspective taken (government and/or societal), and outcomes measurement ought to include an appropriate index of health utility assessed over a suitable length of time (5 years minimum). Amos (2012) mentions several other critical issues for future research, including the measurement and analysis of bias where it cannot be avoided or minimized, and sensitivity analyses, to which could be added the requirements of RCTs, such as adequate randomization, protection of the blind features of the trial, minimizing and dealing with attrition, ensuring treatment fidelity, and so on.

Given all these requirements for reliable and valid determination of the costs and effects of EI, is it all just too difficult to be pursued? Can we settle for quasi-experimental designs, convenience sampling and flawed cost estimations? Well, no, not if we really want to find the truth. Science is hard work, and we will not learn the truth through small-scale, poorly designed or methodologically flawed studies, however plentiful they may be. For comparison of EI against a credible alternative or ‘treatment as usual’, at least one large, well-designed and executed study with one successful independent replication ought to be sufficient or, alternatively, several smaller-scale, good quality studies of sufficiently compatible designs to enable meta-analysis. Further, within this context, rigorous economic protocols for measurement of incremental cost-effectiveness ratios are required to determine whether EI represents ‘value for money’. Without studies like these, the truth about the costs and the effects of EI for psychosis will remain unknown.

See Review by Amos, 2012, 46(8): 719–734