Selection bias in the economic evaluation of crisis accommodation for patients with severe mental illness

To the Editor

Siskind et al. (2013) describe an economic evaluation of a crisis accommodation service, established to provide an alternative to hospitalisation for patients with severe mental illness in a metropolitan catchment in Queensland, Australia. While the article is valuable in its description of a possible solution to a significant problem, there are reasons to question the accuracy of the economic evaluation. Most troubling, in describing the process of selecting controls, the authors state that ‘To reduce the risk of [the different rate of admission in the prior year between the control and intervention group] confounding the analysis … a subgroup of controls was randomly selected from the control group pool so as to contain the same proportion of persons with admissions in the year prior to the index date as did the AtH participant group’ (p.669).

It is impossible to take a random selection from a control group with the aim of selecting a subset which matches some arbitrary characteristic, such as the proportion of admissions in the prior year. Some forcing function must be applied. In this case, as the authors were selecting from a group with a 24.4% rate of admissions in the prior year to obtain a subset with a 47.2% rate, their selection was biased to select more severely affected controls, or to exclude less severely affected controls. Moreover, the selection bias specifically involves the variable – hospitalisation – most likely to increase cost. There are traces of this bias in the higher baseline acuity of controls on the Health of the Nation Outcome Scales (HoNOS), and particularly on the higher proportion of clinically significant problems with living conditions in the prior year.

It is also unclear why the economic and clinical analyses are limited to the index episode, when the authors report that in the year after the intervention, treatment patients were significantly worse than controls on rates of hospitalisation, outcome scores, problems with living conditions, longer duration of case management and readmission to the crisis accommodation.

An implicit assumption of the paper is that the use of alternatives to hospitalisation does not lead to worse outcomes than hospitalisation. While the comparison is not made in the paper, the fact that baseline HoNOS scores were worse for controls, while post-treatment HoNOS scores were better for controls, is at least suggestive of an improvement with hospitalisation for controls not achieved by the treatment group.

Therefore, while Siskind et al. (2013) provide a valuable description of a useful program, without further information their economic evaluation must be considered invalid. The design of their experiment systematically increases the control costs, they do not include all relevant economic and clinical information in their analysis, even of the data they report, and they do not analyse the clinical implications of the intervention.