Abstract
There is a need for a model of disability which can be utilized as an objective tool while at the same time acknowledging that disability is by nature subjective [1]. The assessment of disability is an essential part of clinical assessment complementing the diagnostic formulation. It is well recognized that people with mental illness and particularly those with psychotic disorders suffer from significant disability, which goes beyond activity limitations due to impairments in body functions. People with psychotic disorders – including schizophrenia and related disorders, bipolar affective disorder, depression with psychotic features, delusional disorders and other nonaffective psychotic illness – also suffer at the social level as a result of restrictions in their participation, and this is heavily influenced by environmental factors [2]. In the recent Australian National Survey of Mental Health and Wellbeing, 47% of the patients living with psychotic disorders were rated as seriously impaired in their ability to function in social and occupational domains and 59% were found to have evidence of impairment in their social functioning over the past year [3].
In this article the use of the ICIDH-2 (International Classification of Functioning and Disability), developed by the World Health Organization (WHO) as an instrument to measure disability is assessed in a group of patients with psychotic disorders.
The development of the ICIDH-2
Disability is measured in various ways, to inform decisions about health care provision and clinical management. The following discussion focuses on the development of instruments used to measure disability which have been developed by the WHO.
In 1980 the WHO published the influential International Classification of Impairments, Disabilities and Handicaps (ICIDH) [1]. It was designed to incorporate a social perspective in the assessment of disability. The ICIDH defines disablement according to levels of impairment, disability and handicap. The ICIDH assumes that the social environment is fixed and does not incorporate an assessment of the social barriers and facilitators and other environmental factors which may significantly affect the overall disability of the patient.
McFarlane (1988) commented on the applicability of the ICIDH to psychiatry [4]. He noted the instrument's limitations in the assessment of psychological consequences of illness. It has also been demonstrated there are more disadvantages associated with psychiatric illness than the actual needs linked to psychiatric impairments using the ICIDH model would predict [1].
The ICIDH-2 was developed to provide an assessment of disability associated with illness irrespective of diagnosis, using a biopsychosocial model, as well as the assessment of environmental factors and their role in disability [5]. The ICIDH-2 integrates four dimensions of disability, including structural and functional impairments, activity limitations, participation restrictions and environmental factors. Each dimension is conceptualized as an interaction between intrinsic features of the individual and the social and physical environment, rather than as a one–way interaction as implied in the ICIDH framework [5].
The ICIDH-2 incorporates the principle of universalism: the understanding that the dimensions of disability exist on a continuum with ability. A basic tenet of the ICIDH-2 is that it is a classification of human functioning which can be applied to all patients and not only those with identified disabilities or particular diagnoses [1].
Description of the dimensions
The four dimensions are:
1. Impairments with body functions and structure which include impairments in psychological, sensory, voice and other body functions as well as any anomaly, defect, loss or significant deviation in anatomical body structures.
2. Activity limitations which are concerned with the actual performance of activities by the individual in the tasks of learning, communication, movement, self-care as well as domestic, interpersonal and major life activities.
3. Participation restrictions which assess the involvement of an individual in particular areas with reference to whether this involvement is restricted by environmental factors. Participation restrictions are assessed in a number of settings including personal maintenance, mobility, exchange of information, social relationships, home life and assistance to others, education, employment, economic life and community life.
4. Environmental factors which provide an assessment of contextual factors in the individual's physical, social and attitudinal environment, which represent the background of an individual's life. These factors include products and technology, natural and man-made environment, support and relationships, attitudes, values and beliefs of individuals and society, community services, and systems and policies.
The ICIDH-2 document defines the dimensions of disability and the individual areas within each dimension, which are referred to as alphanumeric codes. The ICIDH-2 Checklist, which is used alongside the ICIDH-2 document, provides a list of major codes of the ICIDH-2 document and is designed to elicit and record information according to the four dimensions. The rating score for each code is referred to as a qualifier. A uniform rating scale is used for each qualifier to describe the extent or magnitude of the problem in each dimension as follows: no, mild, moderate, severe and extreme or complete problem [5].
The development team anticipate that the ICIDH-2 will provide an international standard in the assessment of disability [6]. The Beta-2 field trials, in which we were involved, were conducted in different settings in order to establish its global usefulness and the ICIDH-2 is in the process of revision. We used the ICIDH-2 Beta-2 draft (July 1999) version in our study which was conducted in January 2000. The ICIDH-2 is now in the form of a prefinal draft published in December 2000. Whilst the Beta-2 draft has been modified, both versions use the same framework of assessing disability according to the four dimensions discussed earlier. The prefinal draft has incorporated the dimensions of activity limitations and participation restrictions into a common list, although the rating of problems according to these two dimensions remains distinct. The uniform rating scale used to describe problems in each dimension of the Beta-2 draft is also preserved in the prefinal draft. Certain definitions of the codes in the ICIDH-2 document have also been revised in the prefinal draft.
Aims
We aimed to assess the feasibility and reliability of the ICIDH-2 instrument as a clinical tool in the assessment of disability.
Method
The design was a cross-sectional study of inpatients in the acute psychiatric inpatient setting based on the ICIDH-2 Beta-2 Field Trial Study 3: Feasibility and Reliability for Cases and Case summaries [5].
Setting
The study took place at the Clarendon Area Inpatient Unit of St Vincent's Mental Health Service, Melbourne, an acute adult mental health service with 16 inpatient beds.
St Vincent's Mental Health Service Melbourne is an area mental health service established in September 1995 to provide specialist mental health services to adults aged between 16 and 64 years in the cities of Yarra and Boroondara (a population of approximately 216 000). St Vincent's Mental Health Service is affiliated with the University of Melbourne Department of Psychiatry and operates from four sites: a 39-bed inpatient unit, a 20-bed residential rehabilitation unit and two community mental health services, each with approximately 400 registered patients.
Participants
Our study was part of an international multicentre field trial coordinated by the WHO, in which the number of 20 patients was stipulated. Furthermore for our own purposes it was a pilot study to determine the potential use of the instrument in further disability measurement research. For these reasons we chose to study the application of the ICIDH-2 in a small group of 20 patients.
Our 20 participants were inpatients who were available for selection and recruited for the study over a 2-week period, with the selection criterion being that they had been diagnosed with a psychotic disorder.
From a group of 28 potentially eligible patients admitted to the inpatient unit in the study period, one was unavailable for interview and seven patients were not assessed because they were expected to be discharged within the next 5 days.
The 20 patients ranged in age from 19 to 64, with a mean age of 37.1 years. The group comprised 14 males and six females. Eighteen of the 20 patients had a primary diagnosis of schizophrenia, and two had a diagnosis of bipolar affective disorder – manic episode.
Materials
Evaluations were conducted using the ICIDH-2 Beta-2 instrument, and assessments were recorded using the ICIDH-2 Checklist (version 2.1a, clinician form). Information was collected from the patients’ medical records and from a semi-structured interview based on the checklist lasting approximately 30 minutes. Demographic information and psychiatric and medical diagnoses were recorded for each patient.
Design
The assessment interviews were conducted over a two week time period. Prior to this, both interviewers familiarized themselves with the ICIDH-2 document and the checklist. A joint assessment was performed for all 20 patients, and evaluations using the checklist were made independently.
As the assessment was part of a clinical assessment, with the modification of another clinician being present, informed consent to the assessment was obtained verbally from each patient. Written consent was not considered necessary by the hospital research and ethics committee, given that the study was based on an extension of the clinical interview and also the fact that participants were all patients under the direct care of either interviewer.
Data analysis
In accordance with the field trial protocol, six codes were selected from each dimension leading to a subgroup of 30 codes from the entire checklist. This selection reflected those codes which were clinically relevant to the group of patients and which we felt most confident in applying to the patients.
Raw agreement was calculated as the percentage of cases that received exactly the same rating in the two assessments.
Weighted kappas were calculated for each of the chosen codes as a chance-corrected measure of inter-rater agreement [7]. Weighted kappas allowed for the degree of discrepancy in the scores; greater discrepancies, such as mild impairment and severe impairment, were penalized more than lesser discrepancies, such as moderate impairment and severe impairment. P-values were determined to estimate the significance of the weighted kappas. The ‘rule of thumb’ measure, as described by Landis and Koch [8], and which allows for the interpretation of kappa, was used.
Results
Feasibility
We found that the ICIDH-2 could be applied in the description and measurement of disability in this group of patients. We were able to evaluate and record information about the patients’ disabilities according to the four dimensions based on our clinical interviews and the patients’ records. We were often prompted to explore additional and relevant areas of the patients’ experience. Our impression is that the checklist recorded a more comprehensive and systematic assessment of disability than otherwise included in the clinical evaluation.
Although initially time-consuming the instrument became easier to use with practice. Ratings were recorded using the ICIDH-2 Checklist after the assessment interview. The interview to elicit information would take approximately 20 to 30 minutes and the ICIDH-2 Checklist could be completed in approximately 20 minutes following the interview.
Inter-rater reliability
It can be seen from Table 1 that overall our reliability results were modest with a significant number of weighted kappa results below 0.5.
Inter-rater reliability assessment
Part 1a: Impairments with body functions
We expected our level of agreement to be relatively high in this dimension, given that the codes relate to mental functions routinely assessed on clinical examination; however in several codes our kappas were lower than we would have expected (Table 1).
For example, there was only fair agreement for the code b155 (Emotional functions) with a weighted kappa of 0.33. This can be explained by the subjective manner in which the code is defined in the ICIDH-2 draft [5]. The clinician is asked to make a subjective assessment as to whether there is a mild, moderate, severe or complete impairment, with reference to the definition of the code in the ICIDH-2 document [5]. This text provides a definition of the code and a list of inclusion and exclusion criteria but has no description of anchor points for the rating values or qualifiers.
Part 1b: Impairments with body structures
The assessment of impairments with body structures was based on our observations of the patients and their medical records which included information about their physical health. The raw agreements were high: in 80% or more of cases for all of the codes analysed, the two raters gave identical qualifiers (Table 1). The low weighted kappas calculated for those cases in which the raters agreed there was no impairment gives an impression of low agreement across several codes. This is an acknowledged artefact in the application of Cohen's weighted kappa.
Part 2: Activity limitations
This dimension is important as it encompasses various activities in which an individual's performance may be limited by ‘negative symptoms’ of schizophrenia. Our level of agreement ranged widely from poor to almost perfect.
Two codes in this dimension were of particular interest. First the code a660 (Activities of assisting others) refers to various activities involved in helping others [5]. While the performance of basic activities would be limited in those with marked disability, we expected this code may discriminate helpfully in the assessment of those with mild to moderate disability, as in order to be able to assist others one's own functional status would need to be at a relatively high level. Our level of agreement was moderate with a weighted kappa of 0.49.
For a740 (Maintaining complex interactions) [5], our raw agreement was 45% and weighted kappa was –.06; hence our level of agreement was poor. This may be explained by the subjective nature of the definition of the interactions considered in this code, which may have led to a difference in the way the two raters approached the assessment.
Part 3: Participation restrictions
The levels of agreement for codes in the dimension of participation restrictions ranged from moderate to substantial (Table 1).
The code p910 (Participation in community life) [5] was of particular interest. We expected this to be useful in the assessment of restrictions individuals with psychotic disorders may face in participation in community activities. Our agreement was moderate, with a weighted kappa of 0.44.
Part 4: Environmental factors
Each environmental factor was rated according to whether it was considered to be either a barrier or a facilitator for the individual, and also according to severity. For most of the codes analysed our level of agreement was moderate to substantial.
The stigma experienced by patients with psychotic disorders is multifactorial. That component which can be considered as resulting from the individual's social and attitudinal environment is referred to in the code e440 (Societal attitudes) [5]. For many of the patients in our study societal attitudes were described as being a significant barrier, and an important factor contributing to their social marginalization.
The dimension of environmental factors includes several codes which we found difficult to differentiate. For example the instrument makes a distinction between ‘Individual attitudes’ (e410) and ‘Individual values’ (e420), as well as ‘Societal attitudes’ (e440) and ‘Societal values’ (e450) affecting a patient [5]. In addition a distinction is made between services – including communication, transportation, legal, social security, health and employment services – and systems or policies affecting the provision of these services. In the setting of our study this distinction appeared redundant.
Discussion
Feasibility
We found the use of the ICIDH-2 instrument to be a valuable exercise in the qualitative assessment of disability. The dimensions are defined in a meaningful way which facilitates assessment using a biopsychosocial approach. McFarlane identified that the separation of disability and handicap in the original ICIDH model was ambiguous and required value judgements to be made on the individual's personal and social norms [4]. This is overcome in the ICIDH-2 by considering the individual and social aspects of disability according to the four dimensions outlined, enabling an overall assessment of the individual's level of functioning.
Nevertheless the instrument is somewhat unwieldy. McFarlane's criticism of the ICIDH, that its complexity and length reduced its usefulness in the clinical setting [4], remains relevant in our appraisal of the ICIDH-2.
One of the problems we encountered was the definition of several codes in a subjective manner, making the task of assigning a numerical qualifier for each code difficult. Particularly in the dimensions of activity limitations, participation restrictions and environmental factors, the rater needs to use a significant degree of subjective interpretation in order to determine firstly the nature of each factor affecting the individual, and secondly the degree to which each factor contributes to disability.
In Part 5 of the ICIDH-2 Checklist, contextual information may be recorded by the rater to describe personal factors which appear to affect the individual's level of functioning; however, an area which is not assessed is the personal experience of patients and their level of satisfaction, taking into account a subjective assessment of the individual's own experiences of their disability. In the assessment of disability in patients with psychotic disorders there is a need to integrate the measurements of disability as assessed by clinicians with the experience of patients. The WHO Disability Assessment Schedule II (WHO DAS II) is based on the concepts integral to the ICIDH-2 framework. While the ICIDH-2 is used by clinicians, the WHO DAS II is a questionnaire developed to assess the nature of disability directly from the patient's responses [9]. This instrument can be considered to be complementary to the ICIDH-2 as well as to the WHO Quality of Life instrument (WHOQOL) which assesses individuals’ perceived quality of life in the context of the society and culture in which they live [10].
Reliability
There are several instances in which ambiguity in the definition of the codes may explain our overall modest level of inter-rater agreement. The inclusion of anchor points to assist in the rating process would improve reliability.
We also felt that it may be worthwhile to revise the rating scale used; as previously stated the ICIDH-2 uses a uniform scale in rating each code. In many cases the decision to assign a score for a mild or moderate impairment was too fine-grained to be realistic and associated with a low level of confidence in applying the code to an individual case. An alternative rating scale may improve the reliability of the ICIDH-2, in which there are fewer categories more broadly defined as follows: 0 = no impairment, 1 = mild to moderate impairment, 2 = severe to complete impairment.
Nevertheless there are several codes for subjective areas in the assessment of disability in which our level of inter-rater agreement was relatively good.
Finally the ICIDH-2 is designed to be an instrument that can be used by clinicians who are familiar with the background to the development of the ICIDH-2 and with the ICIDH-2 document, and this procedure of familiarization with the instruments forms the training process. While the principles can be grasped readily from reading the document, some formal training may be appropriate in order to enable standardization of the rating procedure.
Limitations
Our reliability assessment of the ICIDH-2 is limited by the fact that only 20 patients were assessed, of whom all were hospitalized with a diagnosis of a psychotic disorder. Patients with other diagnoses were not studied. It should be noted that our study is part of a multicentre international field trial coordinated by the WHO to assess the feasibility and reliability of the ICIDH-2, across a range of settings and disorders. Furthermore the ICIDH-2 is now in the form of a prefinal draft. While there are some differences between the Beta-2 draft used in this study and the prefinal draft, which are summarized earlier, our assessment of the instrument's application still applies to the current version.
Conclusions
The ICIDH-2 represents a significant conceptual move forward in the assessment of disability experienced by patients in general and by patients with psychotic disorders in particular. It is an attempt to move beyond focusing treatment on the improvement of target symptoms and behaviours, and to incorporate a broader holistic approach to addressing all issues under the umbrella of disability.
Several modifications are needed, however, before the ICIDH-2 could be used reliably in patients with psychotic disorders. The use of anchor points for each dimension or code would assist the rating process. An alternative rating scale in which categories for rating are more broadly defined may also improve reliability without necessarily compromising the validity of the assessment. Formal training may be necessary to enable standardization of the rating process. In addition, there is a need to augment measurements by clinicians with the experience of patients, using complementary instruments assessing self-rated disability and quality of life.
Footnotes
Acknowledgements
The authors wish to thank Tom Trauer and Tony Pinzone for assistance with data analysis.