Abstract
Australia, like many other western countries, has increasingly recognized the importance of the concept of recovery in the treatment of mental illness. Slade and colleagues have compiled several excellent overviews of perspectives on recovery, noting that there has been a gradual shift from an emphasis on ‘clinical’ or ‘service-based’ definitions of recovery to ‘personal’ or ‘user-based’ definitions [1, 2]. The former are located within a medical model and relate to sustained remission, typically evidenced by reduction of symptoms and/or improvements in functioning. The latter have emerged from the ever-strengthening consumer movement in mental health, and draw on the documented ‘life journeys’ of people experiencing mental illness. These accounts share in common a theme which forms the basis of the alternative definition of recovery which suggests that recovery is much more than the absence of symptoms and functional impairment, and is more akin to a change in outlook that is related to leading a meaningful, purposeful life, with or without ongoing episodes of illness [3–5]. At most, the typically-sought-after reduction in symptoms and improvement in functioning might be thought of as clinical recovery, whereas the more nuanced attitudinal change can be considered as personal recovery.
Australia's Fourth National Mental Health Plan makes explicit reference to the concept of recovery, with the first of its five priority areas being ‘Social Inclusion and Recovery’ [6]. It outlines five indicators against which to measure desired change in this priority area, namely: participation rates by people with mental illness of working age in employment; participation rates by young people aged 16–30 with mental illness in education and employment; rates of stigmatizing attitudes within the community; percentage of mental health consumers living in stable housing; and rates of community participation by people with mental illness [6]. These indicators are laudable, but by themselves may not be sufficient to glean an accurate picture of the extent to which the Fourth National Mental Health Plan's recovery-related outcomes are achieved over the next five years. In general, they relate most closely to sectors outside the mental health sector, and do not gauge the performance of mental health services themselves in offering recovery-orientated care. In addition, they may be of less relevance to recovery than to the, albeit related, notion of social inclusion. They are also relatively blunt, offering only an aggregated picture which may not provide an accurate gauge of the nuanced stages of recovery for individuals.
A number of quite specific instruments designed to measure recovery exist, and it is likely that some of these may have utility in measuring the recovery-related outcomes articulated in the Fourth National Mental Health Plan. The current review aimed to identify these and evaluate their potential for routine use in Australian public sector mental health services. It concerned itself with: (i) instruments that could be useful to individual consumers, carers and service providers in monitoring recovery status and change, and in detecting opportunities for improving aspects of recovery by the individual by their own means and/or collaboratively with the support of family, friends and services; and (ii) instruments designed to measure the recovery orientation of services that could be useful to consumers, carers, service providers and managers in co-operatively monitoring the status and change of the recovery orientation of particular teams or services, in comparing services, and in detecting opportunities for improving the recovery orientation of services and improving aspects of recovery for consumers of those services.
Method
Our identification of potential instruments drew on a series of existing reviews of recovery measures conducted by the Human Services Research Institute in Boston in the USA [7-9]. We conducted a further search of MEDLINE and PsycINFO to identify any additional instruments, using the following search string: ((‘mental’ OR ‘psychiatr∗’) AND ‘recovery’)). We also made contact with various international colleagues working in the field to seek information on any additional developments in the area.
We used a hierarchical, criterion-based approach to assess whether given instruments might be candidates for measuring recovery in a routine fashion in Australian public sector mental health services. In the case of instruments designed to measure recovery at an individual level, the criteria were as follows:
explicitly measures domains related to personal recovery;
is brief and easy to use (≤ 50 items);
takes a consumer perspective;
yields quantitative data;
has been scientifically scrutinized;
demonstrates sound psychometric properties (e.g. of internal consistency, validity, reliability and sensitivity to change);
is applicable to the Australian context; and
is acceptable to consumers.
In the case of instruments designed to assess the recovery orientation of services, the following, somewhat less strict criteria were applied:
Measures domains directly relevant to the recovery orientation of services;
Is manageable and easy to use in terms of administration (≤ 100 items);
Has undergone appropriate processes of development, piloting and documentation, and ideally been scientifically scrutinized;
Includes a consumer perspective;
Is applicable to the Australian context;
Is acceptable to consumers.
In both cases the criteria were used in a hierarchical fashion to exclude instruments. This meant that an instrument that was excluded on the basis of one of the early criteria was not assessed against any subsequent criteria. The order of each hierarchy flowed from high-level criteria about the concept of recovery through mid-level criteria about practical issues and perspective to low-level criteria about relevance and acceptability. Had the criteria been ordered differently, the end result would have been the same in each case in the sense that the same final instruments would have remained in contention. Excluded instruments would have been rejected at different points in the hierarchy, however.
Results
Overview of identified instruments
Together, the Human Services Research Institute's reviews identified 31 instruments that have been developed to evaluate recovery. Through our own search strategy we identified a further two instruments, bringing the total number to 33. Of these, 22 are designed to measure individuals’ recovery and 11 are designed to assess the recovery orientation of services (or providers). None of the instruments measures both individual's recovery and the recovery orientation of services.
Instruments designed to measure individuals’ recovery
Table 1 profiles the 22 instruments designed to measure individuals’ recovery, describing them in terms of the perspective they take, the domains they assess, and their item structure. As a general rule, these instruments are concerned with mental health service consumers as a broad group, and do not target specific sub-groups of consumers.
Profile of instruments designed to measure individuals’ recovery
†Note that this instrument was developed for a specific study designed to conceptualize recovery, and was not named by its developers. We have named it on the basis of its content and aims, for the purposes of this review.
Criterion 1: Explicitly measures domains related to personal recovery
Table 1 shows that 20 of the 22 instruments explicitly measure domains related to personal recovery. The exceptions are the Recovery Attitudes Questionnaire (RAQ-16; RAQ-7), which measures attitudes to recovery more generally, and the Reciprocal Support Scale (RSS), which measures mutual support from the perspective of consumers taking part in a specific recovery-orientated programme. These instruments are excluded from further consideration.
Criterion 2: Is brief and easy to use (≤50 items)
Table 1 shows that the majority of the remaining 20 instruments satisfy this criterion. The exceptions are Ohio Mental Health Consumer Outcomes System (OMHCOS), the Peer Outcomes Protocol (POP) and the Recovery Measurement Tool (RMT). These instruments are excluded from further analysis on the grounds that they are too long, leaving 17 instruments.
Criterion 3: Takes a consumer perspective
Table 1 shows that all of the remaining 17 instruments take a consumer perspective, with the exception of the Milestones of Recovery Scale (MORS), which is a provider-rated measure of a consumer's level of recovery. The MORS is excluded from further consideration, leaving 16 instruments.
Criterion 4: Yields quantitative data
Table 1 shows that most of the remaining 16 instruments yield quantitative data. There are two exceptions, however. These are the Rochester Recovery Inquiry (RRI) and the Recovery Interview (RI), which both employ open-ended questions which generate qualitative data. Such data could be extremely informative for some purposes – for example, in encouraging dialogue between consumers and providers – but could not realistically form part of a suite of measures designed to assess progression through stages of recovery for large numbers of individuals across large numbers of service settings. For this reason, the RRI and the RI are excluded from further consideration as candidate instruments for measuring recovery in a routine fashion, leaving 14 instruments.
Criterion 5: Has been scientifically scrutinized
A number of the remaining instruments have not been subject to scientific scrutiny, having never been published as peer-reviewed journal articles. Specifically, nine of the remaining instruments fall into this category. The Crisis Hostel Healing Scale (CHHS), the Agreement with Recovery Attitudes Scale (ARAS), the Consumer Recovery Outcomes System (CROS), the Relationships and Activities that Facilitate Recovery Survey (RAFRS) and the Mental Health Recovery Star (MHRS) were the outputs of specific projects and have only ever been presented in the form of unpublished reports or manuals; none of them has undergone further testing since their development. The Personal Vision of Recovery Questionnaire (PVRQ) emerged from a doctoral thesis published in 1998; its development and psychometric properties have never been published in the scientific literature, although a small number of peer-reviewed journal articles have made reference to the instrument in the context of evaluations of programmes (e.g. Hutchinson et al.[36]). The Mental Health Recovery Measure (MHRM), the Multi-Phase Recovery Scale (MPRS) and Self Identified Stage of Recovery (SISR) all flowed from peer-reviewed journal articles articulating models of recovery [37, 38], but the instruments themselves have not actually been published in the scientific literature, except in the context of acting as a comparator for other instruments (e.g. Andresen et al. [30]). Excluding these nine instruments leaves five for consideration against the remaining criteria.
Criterion 6: Demonstrates sound psychometric properties (e.g., of internal consistency, validity, reliability and sensitivity to change)
Table 2 summarizes the psychometric properties of the remaining five instruments. Specifically, it considers their:
Internal consistency (i.e. the extent to which items that reflect the same construct yield similar results);
Validity (i.e. the extent to which the instrument measures what it intends to measure);
Reliability (i.e. the extent to which the instrument gives stable, consistent results);
Sensitivity to change (related to both validity and reliability – an instrument that is both valid and reliable, and which demonstrates change over time, can be regarded as being sensitive to change).
Psychometric properties of instruments meeting Criteria 1–5
Four of the five remaining instruments have been shown to have relatively sound psychometric properties, although the Stages of Recovery Instrument (STORI) and the Recovery Process Inventory (RPI) have undergone less testing than the Recovery Assessment Scale (RAS) and the Illness Management and Recovery (IMR) Scales. Notably, none of these instruments has been assessed in terms of its sensitivity to change, which is important in the context of routine measurement of the recovery process at an individual level. Further work is required in this regard, but none of these instruments is excluded on the basis of this criterion.
The final remaining instrument, the Recovery Orientation (RO), has undergone minimal psychometric testing since its sound internal consistency was established when it was developed in 2005. Its validity, reliability and sensitivity to change have not been assessed, despite its having been available for more than four years. The RO is excluded from further consideration.
Criterion 7: Is applicable to the Australian context
Of the four remaining instruments, the Stages of Recovery Instrument (STORI) is most immediately applicable to the Australian context, having been developed here. The other three instruments might require at least minor modifications to their language to make them applicable to public sector mental health services in Australia. For example, one of the questions on the Illness Management and Recovery (IMR) Scales asks, ‘how involved are you in consumer run services, peer support groups, Alcoholics Anonymous, drop-in centers, WRAP (Wellness Recovery Action Plan), or similar self-help programs? Australian examples of self-help programmes would need to be considered here. Further exploration is required in this regard, but none of the remaining four instruments is excluded on the basis of this criterion.
Criterion 8: Is acceptable to consumers
All of the four remaining instruments were developed in consultation with consumers. The original Recovery Assessment Scale (RAS) was based on narrative analysis of four consumers’ recovery stories, and revised with input from an independent group of consumers. The original items for the Illness Management and Recovery (IMR) Scales were generated by one group of consumers and providers, and feedback on these items was obtained from a second group of consumers and providers. The draft and final versions of the Stages of Recovery Instrument (STORI) were piloted with groups of consumers and consumer-researchers. The Recovery Process Inventory (RPI) was developed with input from four consumer focus groups, and piloted and revised with input from individual consumers.
The involvement of consumers in the development of the four instruments is positive and none of these instruments is excluded on the basis of this criterion. However, further work is required to determine the broader acceptability of these instruments to consumers.
Criterion 9: Promotes dialogue between consumers and providers
There is an argument that instruments that are completed via a discussion between a provider and a consumer are more likely to promote dialogue between the two than instruments that are completed by the consumer in isolation. This collaborative process of completing the instrument might alleviate any problems with language, particularly for consumers who might otherwise struggle to understand the meaning of particular items. More importantly, it might facilitate providers’ engagement with consumers, improve providers’ recognition of the consumer's recovery process, and create further dialogue about individual consumers’ concerns.
The Recovery Process Inventory (RPI) was explicitly designed to be completed via an interview between the consumer and provider. The Recovery Assessment Scale (RAS) can be completed via an interview, or can be self-completed by the consumer. The Illness Management and Recovery (IMR) Scales and the Stages of Recovery Instrument (STORI) were designed for consumer self-administration, but it is possible that they could be adapted for interview administration if required. All four instruments remain in consideration for routine use as measures of individual recovery.
Summary
Figure 1 summarizes the instruments that met criteria at each level of the hierarchy, and shows that by the end of the elimination process they were reduced to four: the Recovery Assessment Scale (RAS); the Illness Management and Recovery (IMR) Scales; the Stages of Recovery Instrument (STORI); and the Recovery Process Inventory (RPI).
Summary of individual-level instruments meeting criteria at each level of the hierarchy.
All are worthy of consideration as candidate instruments for routinely assessing recovery at an individual level in Australian public sector mental health services, but it is likely that all would require some further development and testing for the Australian context.
Instruments designed to measure the recovery orientation of services
Table 3 profiles the 11 instruments designed to assess the recovery orientation of services (or providers), describing them in terms of the domains they assess and their item structure.
Profile of instruments designed to assess the recovery orientation of services (or providers)
Criterion 1: Measures domains directly relevant to the recovery orientation of services
Table 3 shows that all but three of the 11 instruments measure domains are directly relevant to the recovery orientation of services. The exceptions are the Recovery Knowledge Inventory (RKI), the Staff Attitudes to Recovery Scale (STARS) and the Recovery Promoting Relationships Scale (RPRS) which assess the knowledge of and attitudes towards recovery of individual providers. Although this is important, it is beyond the scope of service-level assessments that could potentially be introduced into routine practice. Excluding the RKI, the STARS and the RPRS reduces the number of potentially eligible instruments to eight.
Criterion 2: Is manageable and easy to use in terms of administration (≤ 100 items)
Table 3 shows that three of the remaining eight instruments contain more than 100 items. These are the Recovery Enhancing Environment Measure (REE), the Recovery Based Program Inventory (RBPI) and the Magellan Recovery Culture Report Card (MRCRC). Excluding the REE, the RBPI and the MRCRC on the grounds of their length leaves five potential candidate instruments.
Criterion 3: Has undergone appropriate processes of development, piloting and documentation, and ideally been scientifically scrutinized
The documentation surrounding one of the remaining five instruments designed to assess the recovery orientation of services is insufficient to make a judgement about the appropriateness of its development. The Recovery Oriented Service Evaluation (AACP ROSE) was reported by personal communication only for inclusion in Campbell-Orde's 2005 review [8], and has undergone no further development since this time [9]. This instrument is excluded from further examination.
By contrast, the documentation relating to the remaining four instruments is quite comprehensive. The Recovery Self Assessment (RSA) and the Recovery Promotion Fidelity Scale (RPFS) both underwent appropriate processes of item development (drawing on stakeholders’ input) and testing (using techniques such as concept mapping, principal components analysis and factor analysis), and both have been published in peer-reviewed journals [47, 54]. The Recovery Oriented Systems Indicators Measure (ROSI) also underwent an appropriate process of item development (involving consumers) and testing of both the factor structure of the instrument and issues to do with its implementation. The Recovery Oriented Practices Index (ROPI) underwent a similar development process [52]. Although the ROSI and the ROPI have not been published in the peer-reviewed literature, they are the subject of comprehensive, publicly available technical reports and conference presentations [46, 52]. Although none of these instruments appears to have undergone much additional psychometric testing beyond the initial development phase, all are retained for analysis against subsequent criteria.
Criterion 4: Includes a consumer perspective
All of the four remaining instruments include a consumer perspective. The Recovery Self Assessment (RSA) and the Recovery Promotion Fidelity Scale (RPFS) both draw on the views of consumers, as well as the views of service managers/administrators, providers, and family members or carers. The Recovery Oriented Systems Indicators Measure (ROSI) seeks input from consumers and providers, and supplements this with administrative data. The Recovery Oriented Practices Index (ROPI) involves conducting interviews with consumers, family members or carers, service managers and service providers, and carrying out a document review. None of the remaining four instruments is excluded on the basis of the consumer perspective criterion.
Criterion 5: Is applicable to the Australian context
All of the remaining four instruments were developed in the USA, but there is no prima facie reason why they could not be adapted to the Australian setting. Consideration would need to be given to some of the terminology that relates to the US mental health system, and translating it to the Australian system. For example, the Recovery Oriented Practices Index (ROPI) uses terms like ‘representative payee’ and ‘outpatient commitment’, which are not used here.
There are precedents for adapting at least some of these instruments to cross-national settings. For example, the Recovery Oriented Practices Index (ROPI) formed the basis of the Scottish Recovery Instrument (SRI) [53]. All four instruments are retained for analysis against the final criterion.
Criterion 6: Is acceptable to consumers
All of the four remaining instruments were developed in consultation with consumers. Development of the Recovery Oriented Systems Indicators Measure (ROSI) began with a series of structured focus groups with consumers about what helps and what hinders mental health recovery [55]. Development of the Recovery Oriented Practices Index (ROPI) began with the establishment of a working group with consumer representation to examine existing tools and evidence on recovery and to consider what form the tool should take [52]. Early development of the Recovery Self Assessment (RSA) and the Recovery Promotion Fidelity Scale (RPFS) relied more on recourse to the scientific literature [47, 54], but draft versions of all four instruments were extensively piloted for feedback with consumers before they were finalized.
Summary
Figure 2 summarizes the instruments that met criteria at each level of the hierarchy, and shows that by the end of the elimination process they were reduced to four: the Recovery Oriented Systems Indicators Measure (ROSI); the Recovery Self Assessment (RSA; the Recovery Oriented Practices Index (ROPI); and the Recovery Promotion Fidelity Scale (RPFS).
Smmary of service-level instruments meeting criteria at each level of the hierarchy.
All are worthy of consideration as candidate instruments for routinely assessing the recovery orientation of Australian public sector mental health services, but it is likely that all would require some further development and testing for the Australian context.
Discussion
Macro issues to consider in assessing recovery
Before discussing the findings of the current review in detail, it is worth reflecting on some of the macro issues associated with measuring recovery in the Australian context. Firstly, consideration needs to be given to the differences between measuring recovery (or the recovery orientation of services) and reductions in symptomatology or increases in levels of functioning (or services’ ability to foster these). Although recovery may be associated with these sorts of clinical improvements, it may be completely independent of them. Measuring recovery alongside these clinical constructs is important from the perspective of broadening the range of individual-level outcomes and service-level processes that equate to good quality care.
Secondly, it should be noted that at an individual level the measurement of recovery is relevant across the lifespan and across phases of illness and episodes of care. The majority of the instruments designed to assess individuals’ recovery that were identified in the current review explicitly or implicitly focus on recovery for adults with severe and persistent mental illness. Only a few of the identified instruments have been used to measure broad aspects of recovery across a range of age groups and with deteriorating disorders such as dementia. The Scottish version of the short-listed Recovery Oriented Practices Index (ROPI), the Scottish Recovery Instrument (SRI), is one notable exception and was used this way in a pilot in five Health Board Areas in Scotland [53]. The applicability of existing recovery instruments to people from culturally and linguistically diverse backgrounds, people with comorbid mental health and drug and alcohol problems, and other groups with particular needs, also warrants further exploration.
Thirdly, there is a lack of consistency in the way in which recovery is viewed in mental health circles. Although there are several commonly used definitions of the term ‘recovery’, including those cited in the introduction, these have not been operationalized particularly satisfactorily. This lack of clarity about what the term ‘recovery’ means in practice may explain the variability in the domains measured by the instruments we identified in the current review. The developers of different instruments may have made divergent assumptions about the salient factors which contribute to the core processes of recovery.
Finally, the review was undertaken on the assumption that if Australia were to embrace the notion of routinely measuring recovery at an individual level or at a service level it would be preferable to draw on existing validated instruments. Although the review succeeded in identifying eight instruments that show potential, it may still be the case that these instruments are not ideal for the Australian context and that developing a locally specific instrument is seen as desirable.
Interpreting the findings of the current review
As noted, our analysis has identified eight instruments that might be potential candidates for routine use in measuring recovery in Australian public sector mental health services: four that are designed to assess individuals’ recovery: Recovery Assessment Scale (RAS), Illness Management and Recovery (IMR) Scales, Stages of Recovery Instrument (STORI), and Recovery Process Inventory (RPI); and four that are designed to assess the recovery orientation of services: Recovery Oriented Systems Indicators Measure (ROSI); Recovery Self Assessment (RSA); Recovery Oriented Practices Index (ROPI); and Recovery Promotion Fidelity Scale (RPFS).
We would recommend that a series of steps be followed to refine this list further. All of these steps should involve extensive consultation with key stakeholders, particularly consumers. They should also involve collaboration with the developers of the relevant instruments. As noted above, it is conceivable that the final conclusion of these deliberations about the instruments might be that none is ultimately suitable for the Australian context. This should not be ruled out as a possibility.
As the first step, a decision needs to be made about whether the emphasis should be on the measurement of individuals’ recovery or on the measurement of the recovery orientation of services, or both.
Secondly, and depending on the outcome of the earlier decision, nuances about which aspects of recovery to measure at the individual level or the service level will need to be further explored. For example, in the case of measures of individuals’ recovery, if emphasis was to be given to assessing consumers’ progress through stages of recovery, the Stages of Recovery Instrument (STORI) would be selected as the instrument of choice. By contrast, if multiple perspectives on recovery were considered important, the Illness Management and Recovery (IMR) Scales would be selected on the grounds of their having a consumer version and a provider version.
Thirdly, further development and testing of the instruments will be required for the Australian context. This process will require careful examination of the individual items, to ensure that they are linguistically and culturally appropriate to Australia. The psychometric properties of the instruments designed to measure individuals’ recovery have been fairly well established (although less so in the case of sensitivity to change than some other parameters), and several of these instruments are now being used as evaluation tools, but if individual items are refined for the Australian context further field testing and psychometric evaluation may be required. The psychometric properties of the instruments designed to measure the recovery orientation of services have been less well tested, and there is a need to establish their validity and reliability, particularly as they relate to the Australian context. A question that remains regarding their validity, for example, is whether consumers in services that rate well on their recovery orientation experience greater rates of individual recovery than those in services that rate poorly [56].
Finally, consideration will need to be given to issues related to the administration of the chosen instruments. In the case of the individual-level instruments, it would presumably be the case that the process would be incorporated into that surrounding the National Outcomes and Casemix Collection (NOCC) [57]. A number of issues would have to be resolved before this could occur, including whether individual measures of recovery should complement or replace the existing consumer self-report measures in the NOCC suite. Either way, there would be significant implications for training and information infrastructure. Particular efforts might be needed to encourage good data quality since the existing consumer self-report measures in the NOCC suite have typically experienced low return rates. There are also questions as to how a selected recovery instrument might fit with the existing NOCC protocol, such as whether it would be applicable to collection occasions, service settings and age groups. The candidate instruments have not been considered in this context to date, and consequently further exploration would be required in this regard.
In the case of the service-level instruments, consideration would have to be given as to how the selected instrument would align with the National Standards for Mental Health Services [58]. Over and above this, thought would need to be given to some of the additional administrative complexities associated with the service-level instruments. For example, it is often not entirely clear which specific stakeholders should complete them (i.e. which individual consumer(s), which individual provider(s), which individual service manager(s)), nor how the views of different stakeholders should be weighted if there are divergent views.
Conclusion
It is apparent that there are several recovery instruments available which show some promise for use in the routine measurement of recovery in Australian public sector mental health services. Further work is required to determine which, if any, of the existing instruments might best be used for this purpose; the possibility that modifications to existing instruments or the development of new instruments might be required should not be ruled out. It might be desirable to invest in two instruments: one designed to measure individuals’ recovery and one designed to measure the recovery orientation of services. If Australia were to go down this path, it would make sense to align indicators in each as far as possible, and to ensure that they were consistent with existing endeavours aimed at monitoring and improving recovery focused aspects of service quality.
Footnotes
Acknowledgements
The authors would like to acknowledge the assistance of Marianne Farkas, Mike Slade, Julie Williams and Lindsay Oades in preparing this paper.