Abstract
Objective:
A self-report instrument of mental health recovery is needed both to facilitate collaborative, recovery-oriented practice and measure recovery-focused outcomes. The Recovery Assessment Scale – Domains and Stages (RAS-DS) has been developed to simultaneously fulfill these goals. The aim of this study was to test the feasibility and measurement properties of the RAS-DS.
Method:
Feasibility was examined by 58 consumer-staff pairs volunteering from 3 non-government organisations. Consumers completed the RAS-DS, discussed it with staff, and then both completed Usefulness Questionnaires. The psychometric properties were examined using Rasch analysis with the data from these consumer participants and from additional participants recruited from two Partners in Recovery programs (N=324).
Results:
Over 70% of consumers reported taking 15 minutes or less to complete the RAS-DS and rated the instrument as easy or very easy to use. Qualitative data from both consumers and staff indicated that, for most, the RAS-DS was an easy to use, meaningful resource that facilitated shared understandings and collaborative goal setting. However, for a very small number of consumers, the instrument was too confronting and hard to use. Rasch analysis demonstrated evidence for excellent internal reliability and validity. Raw scores were highly correlated with Rasch-generated overall scores and thus no transformation is required, easing use for clinicians. Preliminary evidence for sensitivity to change was demonstrated.
Conclusions:
The results provide evidence of the feasibility and psychometric strengths of the RAS-DS. Although further research is required, the RAS-DS shows promise as a potential addition to the national suite of routine outcome measures.
Introduction
Adoption of a recovery-oriented approach to practice has become a prominent feature of mental health policies and system development plans internationally (e.g., Australian Health Ministers’ Advisory Council’s National Mental Health Working Group, 2009; Department of Health, 2007). Recovery from mental illness, as defined by people with lived experience, is different from cure or symptom amelioration and refers to “a way of living a satisfying, hopeful and contributing life even with limitations caused by illness. Recovery involves the development of new meaning and purpose in one’s life as one grows beyond the catastrophic effects of mental illness” (Anthony, 1993: 12). Recovery involves an individualised journey that each consumer embarks upon. The role of service provision is to be recovery-oriented in order to best facilitate or support consumers on their journey. Recovery-orientation involves a service culture that is based upon a belief that recovery is possible and indeed likely for people experiencing mental illness. Based on this belief, practices are strengths-based and foster hope, choice, self-determination, peer-to-peer relationships and positive risk-taking. Primary value is placed on each consumer or service user’s perspective (e.g., Slade et al., 2014; Piat and Lal, 2012; Amering and Schmolke, 2009; Piat et al., 2010).
Whilst policies, plans and directives around recovery-orientation abound, there appears to be a lag in practice implementation or systemic change on the ‘ground’ (Piat and Lal, 2012; Slade et al., 2014). A growing body of literature reports innovative and recovery-oriented initiatives, however many services have been described as “simply repackaging old wine in the new bottle of recovery language” (Davidson et al., 2005: 481). The systemic dependence on measures of symptomology, social disability and service-use has been identified as one of the barriers to adoption of a recovery-oriented approach to practice internationally (Thornicroft and Slade, 2014).
In Australia, the outcome measures used routinely across state mental health sectors are predominantly symptom/illness or functioning focused (e.g., Health of Nations Outcome Scale (Wing et al., 1998)). In New South Wales, the only self-report measure included in the suite is the Kessler-10 (Andrews and Slade, 2001). However, consumers dislike its negative orientation and are reluctant to use it, as seen in low completion rates (Pirkis and Callaly, 2010).
Assessments (i.e., the data collected) set the tone for service delivery and influence service planning. Service providers, increasingly called upon to demonstrate effectiveness, are almost compelled to focus treatment on the outcomes they will measure. If the data collected focus on symptoms, then intervention is likely to focus on ameliorating symptoms rather than on recovery defined as living a good and productive life even in the presence of symptoms. Similarly, if data are measured only from the perspectives of clinicians, then service planning is unlikely to include consumers as equal partners. A structure that enables consumers to participate actively in assessment is a fundamental first step toward recovery-oriented practice. Active participation of consumers in their own assessment and treatment planning facilitates self-direction and empowerment, and these are repeatedly described as central recovery processes for the individual (Bird et al., 2014).
Recovery-oriented outcome measurement is required within the suite of instruments routinely used in mental health systems (Burgess et al., 2011; Andresen et al., 2010; Thornicroft and Slade, 2014). Since no one knows better than the person experiencing the illness about its effects, a psychometrically-sound self-report measure of recovery is crucial for services to be recovery-oriented.
A second, but related, major challenge with outcome measurement in Australian mental health services is variable completion rates, which severely restrict the capacity to measure service-level outcomes. Whilst Australia has been at the forefront of international efforts to embed routine assessment into practice (Burgess et al., 2012), completion rates even of mandated outcome measures, remain variable (Kightley et al., 2010; Pirkis and Callaly, 2010). Slade et al. (1999) have suggested that feasibility of an instrument is critical to its uptake and use in routine outcome measurement. Feasibility includes: brevity; simplicity; relevance; acceptability; availability and value. Variable completion rates may well be due to a lack of perceived relevance and poor acceptability of current outcome measures (Patterson et al., 2006; Trauer et al., 2009). Both staff and consumers report that they complete forms for ‘bureaucrats’ rather than for their contribution to practice (Andrews and Page, 2005; Lakeman, 2004; Happell, 2008). The perceived lack of relevance to practice has been associated repeatedly with poor ‘compliance’ in outcome measurement (Patterson et al., 2006; Trauer et al., 2009; Trauer et al., 2006). Clearly there is a need for recovery-focused instruments to measure outcomes that are relevant, meaningful and practically useful to both consumers and staff.
Existing measures of recovery
Over the past 15 years, researchers have worked to establish a sound measure of mental health recovery. In keeping with the principles of the recovery paradigm, recovery-based instruments are invariably self-report measures. To date no instrument has reached the status of a gold standard. Over the last few years a number of authors (Burgess et al., 2011; Shanks et al., 2013; Sklar et al., 2013) have reviewed existing recovery measures. With differing evaluation priorities and criteria, each reviewer reached slightly different conclusions about which measure was most useful or psychometrically strong. All reviews, however, highlighted that: a) limited ongoing assessment and testing of measures had been carried out for most measures after their initial development; b) the Recovery Assessment Scale (RAS) (Corrigan et al., 2004) was the most frequently used and psychometrically tested measure, and c) further work was required to identify a gold standard measure of recovery. The UK-developed Recovery Star (Dickens et al., 2012), not available at the time of the Australian review, has been found to be a useful tool to facilitate collaboration. However, it lacks evidence for the psychometric properties required for outcome measurement (Killaspy et al., 2012). Burgess et al. (2012) concluded that a new or modified recovery instrument might be required for use in Australian mental health services.
Background development of the RAS-DS
With the aim of developing an instrument that is both practically useful/feasible and psychometrically sound, we began an iterative process of development. Consumers partnering in our research selected the RAS as their preferred recovery measure (Hancock et al., 2012b). However, after testing, we found several measurement problems with the original instrument (Hancock et al., 2011). We set about reducing these with the RAS-DS. Specifically, we removed redundant and ill-fitting items deemed not to be part of the construct and modified the rating category structure. To address a ceiling effect, we conducted focus groups with consumers who had maximum scores on the RAS and created new items that targeted later stage recovery-based achievement including (a) accepting your illness and gaining control over symptoms, (b) self- love and optimism, (c) doing things for and experiencing pleasure, (d) contributing through meaningful activity, (e) having a diversity of friendships, (f) being needed and valued by others and (g) coming to terms with family relationships (Hancock et al., 2012a; Hancock et al., 2011). We then commenced data collection with a first version of the RAS-DS.
Early on in data collection, qualitative data regarding its feasibility or usefulness revealed the need for an additional point in the rating scale. We suspended data collection, re-defined the scale and piloted our new descriptors with a small group of consumers and staff. Once they indicated increased satisfaction with the modified scale, we re-commenced data collection. The revised RAS-DS has 38 items that consumers rate on a 4-point scale: “untrue”; “a bit true”; “mostly true” and “completely true”.
In this paper we examine its feasibility and psychometric properties. If the RAS-DS was found to be both feasible and psychometrically sound, it would be a potential candidate self-report measure of recovery for use in routine outcome measurement and to enhance consumer voice in the assessment process. Specifically we examined the following attributes of the RAS-DS:
Feasibility (brevity; simplicity; relevance and acceptability) as reported by consumers and staff.
Measurement properties, including internal validity, internal reliability and responsiveness to change.
Methods
The results reported here emerged from two studies approved by the Human Research Ethics Committee of the University of Sydney. In Study 1, we explored feasibility or practical usefulness of the RAS-DS together with consumer–staff dyads from three large non-government organisations (NGOs) in two Australian states: New South Wales and Queensland. Each of these organisations provided a wide range of services to individuals living with mental illness in both rural and urban contexts. In Study 2, we explored the measurement properties of data gathered with the RAS-DS, utilising assessment data collected in Study 1 and additional data from two large Partners in Recovery (PIR) programs in New South Wales. PIR programs are a federally-funded initiative to provide care coordination and support to individuals living with severe and persistent mental illness. The two PIR programs use the RAS-DS as a routine outcome measure and provided de-identified data to the study.
Procedures and data analysis
Study 1: Feasibility
Participants
Fifty-eight consumer-staff dyads participated in Study 1. As some staff partnered with more than one consumer, there were only 39 staff participants. Demographic data for participants is provided in Table 1.
Description of consumer participants.
missing RAS-DS data from 3 participants means that only 55 data sets used in Study 2; bsome participants had more than one diagnosis; cRAS-DS Mean Measure Score differed significantly between NGO and PIR groups (t=5.1, p<.001).
PTSD = Post Traumatic Stress Disorder, OCD = Obsessive Compulsive Disorder, Unspecified = unspecified or ‘mental illness’ with no specific diagnosis given.
Data collection
Consumers who gave informed consent completed the RAS-DS and then discussed their responses with their primary staff member. Both consumers and staff completed an open question about the overall usefulness of the RAS-DS. In addition, each consumer selected the best response to questions about time to complete (“15 minutes or less” or “more than 15 minutes”) and ease of completion (from “very hard” to “very easy”).
Data analyses
Responses from categorical questions were calculated as percentages. Comments were coded using constant comparative analysis (Glaser and Strauss, 1968; Charmaz, 2014). This is an inductive method in which codes are developed directly from the data rather than fitting the data into pre-existing codes. Each small chunk of data (word, phrase or sentence) was closely compared to those following to identify underlying concepts or codes. Codes were then compared to each other, and similar codes were grouped into themes. The first and third authors independently coded the full data set, then met to compare and discuss the codes and to find consensus.
Study 2: Measurement properties
Participants
For this study, 298 consumer participants provided 324 data sets. All participants completed the RAS-DS once and 26 participants completed the RAS-DS a second time after 3 months. Participants included 55 of the 58 consumers from Study 1 and 243 consumers from PIR. Participant details are provided in Table 1. Across all collected demographic details there were minimal differences between groups. RAS-DS mean scores did however differ between groups. The PIR group had a significantly lower mean than the NGO group.
Data analyses
We used Rasch analysis to examine evidence for validity and reliability of data gathered with the RAS-DS. Rasch analysis is based on two core assumptions (Bond and Fox, 2007; Wright and Stone, 1999). As they apply in this study: all people are more likely to affirm items that are easier to endorse and people who are more recovered are more likely than people who are less recovered to affirm items that are hard to endorse. All Rasch analysis procedures were conducted using the program Winsteps (Version 3.75.1, Linacre, 2014) and all other statistical procedures were performed using SPSS (Version 17.0). There was a small amount of missing data in the dataset. Where this was the case, raw scores were averaged from the available items. Rasch analysis procedures are very robust to missing data (Bond and Fox, 2007), so no special management or preparation was required.
Internal validity
Rasch analysis provides evidence for a number of different aspects of construct validity. Firstly, we examine point-measure correlations for each item to insure they were positive. Second, we examined goodness of fit statistics for each item (expressed in mean squares [MnSq]) to determine the extent to which data from each item conformed to the expectations of the Rasch model. MnSq values of ≤ 1.5 were considered acceptable (Wright and Linacre, 1994). Third, we examined the extent to which the overall spread of items and matched the “recovery levels” of the sample to ensure there were neither gaps which would reduce measurement precision nor item redundancy and that the mean of person measure scores was close to the mean of item measures.
Reliability
The Winsteps program provides evidence for several aspects of internal reliability. Two reliability indices, person and item, describe measurement precision; ≥ .80 was deemed acceptable (Fisher, 2007). A person separation statistic provides evidence of the number of levels of recovery represented in the data. To conclude that measure differences are the result of real differences in recovery and not measurement error, the separation should be ≥ 2.0 (Fisher, 2007).
Category structure
If Rasch assumptions are met, then people scoring “1” on any item would have an average instrument measure lower than those scoring “2” and so on. Ideally, scores are spread across the range and all categories are used regularly (n ≥ 10). If any of these was violated, then we considered collapsing scoring categories until the assumptions were met.
Correlations between Rasch-calibrated measure score and raw scores
For the RAS-DS to have maximal clinical utility, scores would be able to be totaled and interpreted without the need for transformation or Rasch-based calibrations. To examine whether this was possible with the RAS-DS we explored a range of indicators: Pearson’s Product Moment and Spearman Rank Order correlations between raw scores and Rasch-calibrated measure scores and the magnitude of changes in rank between raw-score and Rasch-calibrated measure scores.
Data analyses
Ability to detect change over time
We used a paired t-test to examine whether RAS-DS scores were typically higher at Time 2 than at Time 1 in a subgroup of participants (n=26) who completed the RAS-DS twice. Time 2 measures were completed approximately three months following Time 1 measures. The α value was set at .05. If the paired t-test indicated that Time 2 measures were typically higher than Time 1 measures, this would provide some initial evidence of the potential of the RAS-DS to measure change over time.
Results
Study 1: Feasibility
Brevity and simplicity
The majority of consumers (n = 41, 71%) reported that the RAS-DS took 15 minutes or less to complete. Most rated it as “easy” or “very easy” to use (n = 43, 74%). Seven consumers (12%) responded in the space between the “easy” and “hard”. Only 8 (14%) reported that the RAS-DS was “hard” or “very hard”; most of those explained the difficulty as stemming from the challenging nature of the questions (e.g., “a bit personal and confronting,” “making deep decisions,” “having to give a great deal of thought to some questions”) rather than problems with the instrument.
Relevance and acceptability
The themes in the data from consumers and staff regarding relevance and acceptability are provided in Table 2. These are exemplified with quotes. All themes were positive except for the last three. Numbers of staff and consumers who gave equivalent comments are included to provide a sense of frequency. Of the total number of comments made, 80% of consumer comments and 81% of staff comments were positive.
Relevance and acceptability.
Study 2: Measurement properties
Internal validity
Point-measure correlations for all items were positive, ranging from .42 to .70. Item fit statistics are shown in Table 3. Fit statistics for 36 of the 38 items from the RAS-DS were within the acceptable range (i.e., MnSq ≤ 1.5). The two items with poorer fit statistics were “I have friends who have also experienced mental illness” and “I have friends without mental illness.” Outfit MnSq values for these items were 1.97 and 1.64 respectively.
RAS-DS Item Measure and Fit Statistics.
Items presented in original order. M = measure, SE = standard error, MNSQ = mean square.
The mean measure score for people was 52.8 and the mean measure for the items was 50 indicating a good match between item difficulty and the level of people’s self-rated recovery. Figure 1 illustrates the relationship between scores of the people and item difficulty. Figure 1 also shows a good representation of items in a 2 standard deviation band from the mean of the items. However the recovery level of 14.5% of the sample is greater than the difficulty of the hardest item. Similarly, the overall scores of 9.3% of participants are lower than the difficulty level of the easiest item. Measure scores for the most and least recovered individuals are less precise than those in the middle because there are no items exactly at their levels.
Hierarchy of People and Items.
Reliability
Participant and item reliability indices were .93 and .98 respectively. Cronbach’s α score (calculated using raw scores) is .96. The person separation index was 3.53, suggesting the RAS-DS can separate people into approximately five statistically distinguishable groups (Wright and Masters, 2002).
Category structure
Scoring categories (i.e., 1, 2, 3, 4) progressed in an orderly fashion for all items. Apart from infrequent use of the lowest category (“untrue”) in some of the “easiest” items (e.g., “It is important to have healthy habits”), all categories were typically used with sufficient frequency. The overall percentages of use for each category were well balanced: 1 (16%), 2 (29%), 3 (31%) and 4 (24%).
Correlations between Rasch-calibrated measure score and raw scores
The Pearson Product Moment and Spearman Rank Order correlations between raw scores and Rasch-calibrated measure scores were r = .95, p < .001 and rs = 1.00, p < .01, respectively. A very small proportion (11 of 324, 3.4%) of rankings changed more than 3 places when raw scores were converted to Rasch-calibrated measure scores. Considered together, these results suggest that raw scores can be directly interpreted in the clinical setting without the need for transformation. However, it is recommended that raw scores are converted to Rasch-calibrated measure scores prior to further instrumentation analysis of the RAS-DS to ensure optimal accuracy (Linacre, 1998).
Ability to detect change over time
The paired t-test demonstrated that RAS-DS Rasch-adjusted measure scores at Time 2 (M = 54.8, SD = 10.2) were significantly higher than at Time 1 (M = 52.0, SD = 9.8), t = 2.49, p = 0.020.
Discussion
The purpose of this study was to examine the feasibility of use and the measurement properties of data gathered with the RAS-DS, an instrument which has been developed over several iterations from the original RAS (Corrigan et al., 2004). Results presented in this paper provide solid initial evidence for both aspects of the RAS-DS.
Most consumers and staff found the RAS-DS to be an acceptable and relevant instrument. Some commented on its value for facilitating recovery-focused goal setting. Both consumers and staff highlighted the new knowledge and/or awareness that it provided individually and as a partnership. There were, however, some consumers who did not find the RAS-DS relevant or acceptable. These consumers found it either too easy because their recovery was progressing well, or too confronting and personal. This raises two questions or points of discussion. Firstly, is it possible to find any single measure that will meet every individual need and fit every individual’s understanding of his or her experience of mental illness and recovery? The literature describes recovery as a deeply personal and unique process and perhaps therefore there will be no single measure that will be acceptable to all. The addition of harder items might make the instrument more relevant for those further along their recovery journey; however this may well make the instrument even less acceptable for those who found it too hard and confronting. A repeated discussion in the recovery literature on options and choice comes to mind (e.g., Davidson et al., 2006). Second, some consumers said that completing the RAS-DS made them sad and teary. Acknowledging the importance of choice, should we avoid sad and teary? Perhaps this means that we are in fact asking people about things that really matter to them? Perhaps rather than avoiding asking these questions, the challenge is for those supporting consumers to listen, empathise and facilitate them to consider ways to work on what really matters to them?
The instrument was found to be feasible for most in terms of brevity and simplicity. There were however some consumers who took longer than 15 minutes to complete the RAS-DS. For most this was because they had to think deeply about their responses and we consider this to be a positive reason as long as they found it useful overall. For some however, again a small number, it was hard to concentrate and to read all of the items. While we have worked to keep the language as user-friendly as possible, we understand that for some people, having someone to read the statements out to them while they assign the scores will aid completion. This form of administration aids simplicity when needed but, as with other self-report measures, to aid validity and reliability, we caution against “helicoptering” (i.e., hovering over consumers as they rate) or re-wording or re-interpreting the statement. Consumers should be encouraged to rate the statement based on what it means to them (Coombs, 2005).
If, as (Slade et al., 1999) suggested, feasibility is critical to the uptake of measures, then the RAS-DS should be easy for mental health services to adopt. It is freely accessible, (
Examination of the measurement properties of the RAS-DS undertaken as part of this project has provided promising initial evidence for its suitability as a measure of mental health recovery. Examination of internal validity suggests that items are all related to the underlying construct of recovery. Recovery has been repeatedly described in the literature as a multi-dimensional journey that is unique for each individual. Indeed the title of this instrument includes the word ‘domains’ to reflect the breadth of the construct of recovery that has been described by consumer participants during the iterative cycles of instrument development (Hancock, 2012a). Consumers described functional, personal, social and clinical aspects. We began our testing of the instrument by assuming that while each individual is unique, the RAS-DS comprises items that describe 4 aspects or domains of a broader unidimensional hierarchy of mental health recovery. We then tested that hypothesis and found good evidence that it is true. Of course, no construct is perfectly unidimensional but the evidence for multiple dimensions found was actually relatively weak given the good fit of the items. Rather, our analyses demonstrate that these four domains are contained within a larger single recovery construct.
Most items were used consistently by consumers. The two items which demonstrated poorest “fit” to the expectations of the Rasch model (“I have friends who have also experienced mental illness” and “I have friends without mental illness”) were added recently as a result of a study that explored the perspectives of consumers considered to be at “later stages” of their recovery (Hancock et al., 2012a). Given that consumers identified these as important aspects of their conceptualisation of recovery, and given the ‘border-line” nature of the fit statistics meaning that they do not denigrate the data, we believe it would be premature to suggest that these be removed.
Results from the investigation of rating scale structure and reliability indicators also suggest that the RAS-DS is useful in distinguishing differences in recovery between more and less recovered individuals. The term ‘stages’ is included in the title of this instrument as a result of previous work aiming to rectify the lack items addressing later stage recovery achievements (Hancock, 2012a) in the original RAS (Corrigan, 2004). The hierarchy of items and the reliability of the rating scale structure demonstrate that there are earlier and later recovery achievements. However, as can be seen by the essentially normal curve of people distributions in Figure 1, there is a fluidity of stages rather than distinct or discrete steps. This reflects the broader and fluidic stages of recovery that are described in previous literature (e.g., Andresen et al., 2006). Even with the fluidic nature of these stages, our data demonstrates that there were 5 stages or groups of relative recovery within our sample.
Although only preliminary, the examination of change in RAS-DS scores over time is also promising. In a previous iteration of the instrument, results of an examination of test-re-test reliability with a small number of people demonstrated that when changes occurred, the direction of changes were consistent with reported life-changes (Hancock, 2012b). Additionally, in this study, the good item fit statistics suggests that changes are a result of changes in participants’ self-reported measure of recovery rather than instrument instability. However, further examination is required to ensure that this change shown over time is the result of actual change in level of recovery rather than any measurement error caused by scale instability.
Considered together, these results suggest that the RAS-DS holds great promise as an instrument to measure mental health recovery. If further testing continues to demonstrate the feasibility of the RAS-DS and the soundness of the data it generates, it would be a strong candidate for routine adoption within the Australian mental health system.
One final point deserves discussion. As can be seen in Figure 1, a number of items are at the same level of difficulty (i.e., redundancy; Baghaei, 2008). If the aim was to develop the briefest possible measure of recovery that generated psychometrically strong data, these items should be examined for potential removal. The development of a ‘short-form’ version of the RAS-DS that can be used as a screening tool with the long-form reserved for therapeutic purposes is one potential future endeavour. Perhaps more valuable would be the development of a computer assisted version of the RAS-DS that would enable the richness of the domains of recovery to be maintained, but reduce burden by sparing consumers from completing items that were not relevant to their current stage of recovery. However, caution needs to be taken in attempts to reduce burden. As demonstrated in the qualitative, first component of this study, the majority of consumer participants did not find the RAS-DS burdensome. Further, both consumers and staff valued the opportunity to reflect and discuss their positive achievements to date as well as the possible successes ahead. For now we have chosen to retain all the items and will continue to explore the possibility of additional items at the most difficult and least difficult ends of the hierarchy.
There are limitations with this study and further work is required. In terms of the psychometric testing, this study has focused upon the internal workings of the measure. Whilst it may be challenging given there is no gold-standard measure of recovery, future work is required to examine aspects of convergent as well as divergent validity. Study examining further aspects of validity such as criterion and known-groups would also contribute to our understanding of the RAS-DS and its capacity to measure the recovery construct. In relation to the sample, all of the data have been collected from consumers and staff working within the non-government or community-managed mental health sector. Additional study is needed to ensure findings are replicated when the RAS-DS is used within other service sectors such as in-patient contexts. Further, data were all collected within the New South Wales and Queensland states of Australia and we did not record the cultural background of participants. Testing in other states and within other countries is needed to ensure that data gathered in other places are equally robust and to examine how culture affects use of the RAS-DS. Finally, the small data set used to examine sensitivity to change shows positive trends over a relatively short period of time. This however requires further testing.
Conclusions
These promising results suggest that the RAS-DS is an efficient and effective instrument that (a) generates valid and reliable scores reflecting mental health recovery and (b) promotes understanding and partnerships between consumers and staff as a basis for collaborative intervention planning. While further testing is warranted, it appears that the RAS-DS can make an important contribution to service delivery in mental health.
Footnotes
Acknowledgements
We thank the mental health consumers and staff who not only completed the RAS-DS but provided the rich expert knowledge required to understand the relevance and acceptability of the RAS-DS. We thank the Richmond Fellowship Queensland, the Schizophrenia Fellowship of New South Wales, RichmondPRA and both the Hunter and South Western Sydney PIR programs for their participation.
Declaration of interest
The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.