Abstract
Brann and Coleman [1] have provided a thoughtful analysis of some of the issues involved in the assessment of change using the Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA), one of the outcome measures nationally mandated in Australia and New Zealand.
In discussing their results using the Reliable Change Index (RCI) they propose several caveats. One of these is that a 95% level of confidence may be unrealistically high for routine clinical purposes. There are in fact several examples of a less restrictive level (90%) being used in various areas of health, including neurology and neuropsychology. Despite the RCI being the most widely used method of estimating personal change [2], obstacles and misunderstandings persist. Zahra and Hedge [3] identified (i) that popular statistics packages don't compute it, and (ii) the frequent non-availability of normative statistics. In fact, the computation is relatively straightforward, and the norms issue is not a limitation to the use of the RCI, but only to the estimation of Significant Change. Contrary to assumptions in certain quarters, it is quite possible to use the former without the latter: as Jacobson and Revensdorf [4] have clearly stated ‘RC has nothing to do with clinical significance’ (p.136).
Brann and Coleman also refer to ‘statistical and logical problem’ when, in the course of an episode of care, adjacent pairs of intermediate assessments are not statistically different, yet the difference between first and last is. But taking a case where there is consistent improvement of 3 points between intake, first review, second review, and discharge, each change taken in isolation may well fall short of some criterion of significance or reliability, yet the change of 9 points over the whole episode will exceed their own (and others' [5]) threshold for real change. Small intermediate changes summing to a large overall effect are commonly encountered and pose no special problems.
The authors express the concern that a finding of no reliable change between two assessment occasions is likely to be disheartening to the stakeholders. Any small change, whether expressed in raw units, percentage, or effect size, etc. might suffer from this effect. The issue here is that the methods employed to understand change for the purposes of research or service evaluation may have to be different from those required when communicating with consumers and family members. Generally, discussions focused on an individual consumer will rely on detailed ratings of problems, rather than aggregated, summative, scores. Furthermore, it has been noted that, in some circumstances, no change or minimal change may be a good outcome [6].
Finally, the paper questions the inclusion of consumers with zero initial ratings (i.e. no problem) in the computation of group change averages, on the grounds that these individuals cannot improve, and the subsequent mean change will underestimate the benefit conferred on those who do start with some degree of problem. The case for retaining low or zero scores is, firstly, that outcome measures are not solely about quantifying change; they are also relevant to profiling the problems of those accepted into treatment, and to this extent, it is important to know that there are low prevalences of certain problems among them. Secondly, if the measures reveal that there are large numbers of consumers accepted into treatment with very low overall severity levels, questions may be legitimately asked about intake criteria and practices. Thirdly, while low scorers cannot get much better, they can certainly get a lot worse. The proportion of consumers who deteriorate in psychotherapy has been estimated at between 5% and 10% [7], and, according to national figures available through the Australian Mental Health Outcomes and Classification Network, 13.5% of HoNOSCA total scores show worsening between start and end of episodes in both community and acute inpatient settings. Any generalization about the degree of benefit attributable to services needs to offset the ‘deteriorators’ against the ‘improvers’.