Abstract
Keywords
The Second National Mental Health Plan [1] promotes patient involvement in all stages of service delivery, and the facilitation of access to ‘mainstream’ services with an emphasis on community-based care. The Plan also aims to introduce evidence-based practice and the requirement of outcome measurement in mental health services. To meet the aim of evaluating service provision, the processes of psychiatric rehabilitation need to be operationalised [2]. A key component in the process of rehabilitation is the assessment of the needs of patients. However, there is little consensus on how need should be defined and measured [3].
Most methods of needs assessment are based on the assumption that needs can be measured objectively [4]. However, the perception of need is influenced by the values and expectations of the person making the judgement [5]. Cultural, socioeconomic and other factors affect perceived quality of life and the prioritisation of needs [6,7]. Furthermore, the values of an individual or society may change with time, with new knowledge or with the situation [8]. This led Slade [4] to describe need as a ‘socially negotiated’ concept, and Brewin et al. [9] to claim that there can be no objective definition of need.
Ideological differences between the helping professions are another source of variation in the assessment and prioritisation of needs [10]. For example, social workers focus more on psychosocial problems and environmental interventions, while Community Psychiatric Nurses (CPNs) are oriented more towards health aspects and individual intervention [10]. On the other hand, psychiatrists and psychologists were found to rate more symptoms and disabilities than were CPNs [11]. Community care provides more opportunity for assessments to be conducted by professionals from diverse backgrounds. Therefore, the importance of consensus on the definition of need is essential to the evaluation of service provision [7].
Traditionally, needs assessment methods have been based on the clinician's judgement [12]. However, the views of the clinician frequently do not reflect those of the patient (see [13–18]), and poor agreement has been found between ‘clinically oriented’ and ‘patient-centred’ needs assessment methods [19,20]. Since current policy calls for patients and carers to be involved in all stages of treatment and rehabilitation planning [21], it is essential that assessments accurately reflect the expressed needs of the individual. Magi and Allander [5] claim that total congruence between self-assessed and other-assessed need is not possible. Furthermore, Comtois et al. [18] claim that congruence is not necessary because the emphasis of rehabilitation should be on cooperation and the therapeutic alliance. Integrating patient and clinician views into a single rating can result in the loss of information [17], whereas assessment that records patient-identified and clinician-rated needs separately can act as a catalyst for discussion [16].
Ideally, needs assessment should be carried out regularly as part of clinical practice [4,9,19]. Where assessments are to be used for purposes such as service evaluation it is important that the measures can be incorporated as a meaningful and useful part of the clinical program [22–25].
The present study examines the interrater reliability of the Camberwell Assessment of Need Short Appraisal Schedule (CANSAS), a structured interview in which staff, patient and carer views of need can be recorded separately. It is a short version of the Camberwell Assessment of Need (CAN), which was designed specifically to inform clinical practice and to serve as a service evaluation tool [26]. The CAN was designed to be quick and easy to learn and use by a wide range of professionals, in order to encourage regular, systematic needs assessments [26].
The CAN and CANSAS consist of 22 items of need. The CAN has four sections for each item. Section 1 is scored by rating 0 (no need), 1 (met need due to help given), 2 (unmet need) and 9 (not known). Sections 2–4 assess the amount and type of help needed and received. The CAN met with some resistance as a routine measure because of its length [Slade M: personal communication, 15 April 1999]. Therefore, the CANSAS consists only of Section 1, the identification of needs and level of need.
The interrater reliability of the CAN was investigated by Phelan et al. [26]. Sixty patients and their key workers were interviewed separately. One researcher acted as interviewer and one as observer, each independently rating the responses. Correlations of summary scores indicated high interrater reliability for patient ratings (r = 0.99) and for staff ratings (r = 0.98). Good agreement was found on individual items, with percentage agreement ranging from 88.0% to 100% for staff ratings, and 81.6% to 100% for patient ratings. Kappa coefficients for interrater reliability ranged from K = 0.74–1.00 for staff ratings, and from K = 0.65–1.00 for patient ratings.
To assess interrater reliability under suboptimal conditions, Hansson, Bjorkman and Svensson recruited final year psychiatric nursing students with minimal training on the CAN to act as raters [27]. Fifteen pairs of raters were used, with an interviewer and an observer rating the responses. Interrater reliability was found to be very good on all sections of the CAN, with percentage agreement of over 80% in 91% of the ratings, and only two instances of below 70% agreement. However, Hansson et al. [27] and Wiersma et al. [19] pointed out that, by using different sections of the CAN, unmet need could be defined in a number of ways. Furthermore, using different options resulted in the identification of different areas of need. To the authors' knowledge, no known studies examining the interrater reliability of the CANSAS have been published. Although the reliability of the CAN under certain conditions has been demonstrated, it cannot be assumed that these results generalise to the CANSAS. The omission of sections 2–4 may result in less information being available to the rater. The interrater reliability studies of the CAN discussed above used two raters of similar backgrounds [26,27]. This study assessed the interrater reliability of the CANSAS using raters from different backgrounds and with different levels of experience with the CANSAS. Such a scenario is a more valid representation of clinical settings within public mental health systems.
Method
Participants
Patients of a psychiatric rehabilitation unit in regional New South Wales, Australia who were in contact with the service over the 10-week period of data collection, were approached by care coordinators to have an observer present during an otherwise routine assessment. Thirty-two patients participated, 18 day patients: 13 male, five female, mean age = 38.11 (SD = 7.71) years; and 14 inpatients: 12 male, two female, mean age 37.9 (SD = 7.32) years. Day patients are patients who attend a day program and also receive individual outpatient services. Twenty-seven of the 32 participants had a clinical diagnosis of schizophrenia. Rater 1 and Rater 2 were two psychiatric nurses who were care coordinators for the day patients and inpatients, respectively, and used the CANSAS in practice. Rater 3 was the first author, a fourth-year psychology student with minimal training on the CANSAS.
Procedure
The CANSAS was assessed for interrater reliability on staff and patient ratings. The version used included four examples serving as ‘anchor points’ for each item, which are not on the original CANSAS, but were adapted from the CAN. The raters formed three interviewer-observer dyads: Rater 1 and Rater 3; Rater 2 and Rater 3; and Rater 1 and Rater 2. The interviewer and observer completed both the staff and patient ratings.
Results
Agreement between raters on items of need
Agreement on each item of need was determined for staff and patient ratings. Percentage complete agreement and K coefficients are presented in Table 1. Percentage agreement ranged from moderate to high, with higher agreement on patient ratings than on staff ratings for most items. Kappa coefficients ranged from low to high. For the patient ratings, coefficients ranged from K = 0.39 to K = 1.00, with 64% of the items above K = 0.70. Coefficients for staff ratings were lower, ranging from K = 0.20 to K = 1.00, with 36% of items above K = 0.70. Notably, items 10 and 11 (safety to self and safety to others) had very low K coefficients (K = 0.20) in the staff ratings.
Agreement between raters on each item for staff and patient ratings
Agreement between raters on level of need
The mean of each level of need for staff and patient ratings are presented in Table 2. Interviewers identified more areas of need on the staff ratings than observers (t = 2.56, df = 31, p < 0.02). Correlations between raters on levels of need for staff and patient ratings are shown in Table 3. Correlations were generally high, with the exception of staff ratings of met need (r = 0.53). The correlation on staff ratings of met needs was particularly low for inpatients (r = 0.28). The results suggest disagreement on whether an identified need is met or unmet.
Mean of need levels for interviewers and observers
Correlation between raters on level of need
Discussion
Overall agreement on areas of need was moderate to very high, however, agreement was lower on staff ratings than on patient ratings. Low K coefficients for some items indicated discrepancies in rating the level of need. Of particular interest are the low K coefficients for staff ratings of need on ‘safety to self’ and ‘safety to others’. Patients often responded to these questions with an answer such as: ‘I sometimes think about it, but I wouldn't actually do anything’. It was found that the clinician's personal knowledge of the patient was very influential in rating such a response. The levels of agreement were not as high as those found by Phelan et al. [26] or Hansson et al. [27], highlighting the differences between raters from different backgrounds.
Differences were found in the number of needs identified in the staff ratings. Although the ratings of total needs were highly correlated, discrepancies were found in staff ratings of met need. That poor agreement between raters occurred on the staff ratings indicates that the differences were not in the raters' interpretations of the patient's response. Rather, it suggests that raters are drawing on other information in making an assessment, and/or using different concepts of need. Factors such as the rater's values, or information of which the rater is aware, but which is not expressed in the interview could influence the rating given. These findings are limited by the fact that only three raters were involved, and that the second rater acted only as an observer. However, they demonstrate that other sources of information are drawn on in determining a rating [Lambert G, Caputi P, Deane F: unpublished data].
While raters identify the same areas of need, there is disagreement as to when a need is to be considered ‘met’. A patient who is attending a day program may have goals that could be considered additional to met needs, and these goals may also be scored as ‘met’ or ‘unmet’. On the other hand, a need that is met during hospitalisation may or may not exist when a patient leaves. This leads to the question of whether different definitions of need are implicitly applied for day patients and inpatients. These issues highlight the need for careful training on the definition to be applied.
Insufficient scoring options could also contribute to discrepancies in the rating of met need. Interrater reliability may be improved by increasing the number of levels of need, allowing for greater discrimination. This could have the added benefit of addressing the criticism that CANSAS is insensitive to change [22], thereby enhancing its potential for use as an outcome measure in addition to its current use in treatment planning and case load management.
Conclusions and recommendations
Agreement on the identification of an area of need was generally high. However, discrepancies were found in staff ratings of met need. The interrater reliability of the CANSAS suffers because of the vagueness of the conceptual definition of need, and hence its measurement. The definition used must suit the purposes of the assessment and, for comparative purposes, must be universally applied. Therefore, a definition must be sought which is unambiguous and which also satisfies clinical needs. This would also inform clinicians of the most appropriate sources of information to use when making a rating. That differences between raters were greater on staff ratings than on patient ratings suggests that raters use information other than the patient's responses in assessing the level of need. One approach would be to identify the sources of information that the clinician uses in assessing needs, and to incorporate these into the definition for the purposes of training [Lambert G, Caputi P, Deane F: unpublished data].
Ambiguity in the rating of the level of need may also be alleviated by increasing the number of scoring options, allowing for greater discrimination. This may also enhance the scale's sensitivity to change, and hence its utility as an outcome measure. Further research would be required to determine the optimal scale, including the issue of whether a categorical or continuous scale would be most appropriate.
The results have implications for Australian mental health services with the current policy to introduce national standards for service provision, outcome measures and evidence-based practice [1]. With the focus on community-based care and access to mainstream services, the results of this study highlight the importance of the selection of measures that demonstrate interrater reliability, and the thorough training of professionals from diverse backgrounds to ensure that the measures provide reliable, valid and useful data.