Defining Disability in Psychosis: Performance of the Diagnostic Interview for Psychosis-Disability Module (DIP-DIS) in the Australian National Survey of Psychotic Disorders

Abstract

Objective: We aimed to use data from the Australian Survey of Mental Health and Wellbeing to examine the psychometric properties of the Diagnostic Interview for Psychosis-Disability Module (DIP-DIS). The DIP-DIS is a semi-structured questionnaire specially designed to assess disability associated with psychotic disorders.

Method: The psychometric properties of the instrument were determined by examining its inter-rater reliability, internal structure, as well as its criterion and discriminant validities.

Results: Analysis shows: (1) that it can be rated reliably by trained interviewers, (2) that the items are complementary but tap a number of different domains, (3) that four factors account for over 66%% of the variance, and (4) that it is sensitive to differing clinical populations with expected differences in level of disability. Ratings on the DIP-DIS bore significant relationships with a criterion measure of quality of life.

Conclusions: The DIP-DIS has encouraging psychometric properties for cross-sectional assessment of disability and may be useful in future studies of disablement associated with psychosis. Future work should examine its sensitivity to change.

An important aim of the component of the Australian National Survey of Mental Health and Wellbeing, termed the Study of Low Prevalence (Psychotic) Disorders (LPD) [1], was to assess impairment in role functioning, determine level of independence in activities of daily living, assess subjective experience of quality of life, and evaluate deficits in supportive social networks of people with experience of psychosis. The conceptual framework for this assessment was grounded in the rehabilitation model [2] with its emphasis on disability as restriction or lack of ability in role performance. However, this framework was broadened to incorporate notions of the consequences of mental illness embodied in the recovery model [3],[4], and to include more subjective outcomes [5],[6]. We describe the performance of a measure of disability designed for the study which takes account of limitations of existing measures [7],[8], such as specificity to particular disorders or narrowness of scope [9].

Method

The study was conducted in Brisbane, Canberra, Melbourne, and Perth. A full description of the coverage of the survey is given elsewhere [1]. Research teams at each site received institutional ethics approval for the project at the site.

Instruments

In deciding on the best way to assess disability associated with the disorders of interest, a working group (composed of H. Herrman, O. Gureje and T. Trauer) identified some criteria. These included (1) acceptability to largely community-dwelling respondents, (2) applicability to the clinical population of interest, (3) relative comprehensiveness to cover the main areas of disability experienced by people with psychosis, (4) inclusion of the perspective of the patient and clinician, and (5) brevity (could be administered in 10–15 min). The objective was to have an instrument that would measure deviations from norms in the areas of personal and social activities and manifest behaviour that could be attributable to illness and not to lack of personal or social opportunities. An important consideration during this preparatory work was the overlapping yet distinct domains of objective role functioning and subjective quality of life within the overall concept of disability. It was obvious to the working group that a measure of disability must be sufficiently comprehensive in order to be valid. In order to ensure such comprehensiveness and relevance of the measure, the working group organized meetings with mental health professionals from diverse backgrounds and with mental health consumers. These meetings served to identify core areas of disability associated with the experience of psychosis. At such meetings, a facilitator explained the aims of the project and the particular interests of the investigators to obtain data on the consequences of a psychosis experience to the sufferer. When participants identified various areas of such consequences, a discussion was then held to harmonize the views and present options by which the identified domains could be parsimoniously measured. Once a listing of these areas of disablement had been made, the working group then identified items from existing tools that could best elicit each of the areas.

The resulting disability module formed a part of the Diagnostic Interview for Psychoses (DIP), a specially designed semi-structured interview linked with a computerized diagnostic system that allowed for the generation of diagnosis according to a number of classificatory systems, including ICD-10 [1], [10].

The DIP-Disability Module (DIP-DIS) consists of a core made up of an abridged and modified version of the World Health Organization Disability Assessment Schedule (WHO/DAS) [11]. This was supplemented by sections of the Lancashire Quality of Life Profile [12] and the Current Social Contacts Scale described by Tucker [13]. Eleven items were selected from these sources to reflect the range of disabilities associated with the experience of psychosis and reviewed during consultative meetings with consumers, carers and mental health professionals. After a series of semi-structured probes to establish relevant life circumstances and opportunities and details of functioning of the patient in that particular area, the interviewer then made a rating that utilized all of the information. Ratings were made in the areas of participation in household activities, socializing, social withdrawal, quality of interpersonal relationships, occupational performance, interests and information, intimate relationships, sexual relationships, and self-care. Other than quality of interpersonal relationships which was scored as 0 (no deterioration perceived), 1 (deterioration due to subject's health or loss of interest), 2 (deterioration due to other people's loss of interest), and 4 (improvement perceived), other items were scored 0 (no dysfunction), 1 (obvious dysfunction) and 2 (severe dysfunction). A rating of 8 was made if the item was impossible to assess and 9 if not applicable. Two other items asked respondents to rate their satisfaction with their own independence and with life on a 4-point scale: mainly satisfied, mixed or often dissatisfied, mostly seriously dissatisfied, and uncertain or impossible to rate.

For the assessment of criterion validity, the WHOQOL-Bref [14], an abbreviated version of the 100-item World Health Organization Quality of Life instrument was used. The WHOQOL-Bref, designed from an international data pool of which Melbourne was a part, is a 26-item questionnaire with demonstrated reliability and validity. For the present analysis, the DIP-DIS has been compared with the four domain scores derived from the WHOQOL-Bref: physical health, psychological, social relationships, and environment.

Participants

The examination of the internal structure and of the constituent factors of the instrument was conducted on data obtained from the entire sample of 980 participants across the four sites of the survey. The discriminant validity was established from data obtained from participants in Melbourne (n = 324) because of the large sample of respondents living in marginal accommodation at this site. (Marginal accommodation included halfway homes, rooming houses, hostels, night shelters, etc.). Criterion validity was examined on the data obtained from a sample of 140 participants with a diagnosis of psychotic disorder in Melbourne who took part in an add-on study in which the WHOQOL-Bref was a constituent instrument. Interrater reliability was examined using data obtained from five raters independently rating 10 videotaped interviews of participants with psychotic disorders. Those involved in every phase of the project received a complete description of the study after which they provided a written consent to participate.

Analysis

Interrater reliability was tested with intraclass correlation. The internal structure of the DIP-DIS was examined using bivariate correlation and by principal components analysis with varimax rotation for factor structure. Internal consistency was examined with the Cronbach alpha coefficient [15]. We used correlational analysis to examine criterion validity and ANOVA to evaluate the ability of the instrument to discriminate between three groups of respondents with expected differences in level of disability. All analyses were conducted with the use of the SPSS statistical package. All test results are two-tailed.

Results

Interrater reliability

The results of interrater reliability are shown in Table 1. The data for this exercise were derived from independent ratings by five research interviewers of 10 videotaped interviews. Most of the items achieved good to excellent reliability. The exceptions being intimate relationships and sexual relationships.

Table 1.

Interrater reliability

Internal structure

Table 2 shows the results of analysis examining the item intercorrelations. The Table lists correlation coefficients ≥ 0.30. The highest correlation coefficient (0.67) is between socializing and social withdrawal, two items with complementary meaning and which were also successive items on the questionnaire. Every item correlated with at least one other but with participation in household activities and socializing showing the largest range of associations with other items.

Table 2.

Item intercorrelation analysis (coefficient values > 0.30)

In general, the pattern of intercorrelation suggests that the questionnaire items are not redundant but complimentary. The entire module of 11 items has a Cronbach α-coefficient of.69. However, when the two items with low interrater reliability were removed (intimate relationships and sexual relationships), the value rose to.77 which is a good value for internal consistency [15] and confirms that, while the items may reflect a number of different conceptual features, there is an underlying commonality.

Principal components analysis

Data collected on the entire 980 respondents were submitted to principal components analysis with varimax rotation. Four main factors were derived (Table 3) with all the 11 items entered in the equation contributing. The first factor could be termed ‘domestic, social and occupational roles’, the second ‘psychosexual relationship and self care’, the third ‘subjective experience of satisfaction’, the fourth ‘gregariousness’. The four factors accounted for a total variance of 66.2%%.

Table 3.

Factor structure of the ∗DIP-DIS (Factors with Eigenvalues > 1)

While the ‘domestic, social and occupational roles’ factor is clearly the most important, the substantial variances contributed by the other factors would support a view of the multi-dimensional nature of disability in this population.

Criterion validity

Table 4 shows the results of analysis examining the criterion validity of the module. (Negative correlation coefficients in the table indicate that a higher level of impairment in the DIP-DIS area of functioning is associated with a lower quality of life as rated on the WHOQOL-Bref). For this exercise, the WHOQOL-Bref was used as the external criterion and the analysis was conducted for the sub-set of the sample on whom both the DIP-DIS and the WHOQOL-Bref were rated (n = 140). The two measures of subjective satisfaction (satisfaction with own independence and satisfaction with life) were consistently associated with the four domains of the WHOQOL-Bref. Other than these two items, only socializing, social withdrawal, and sexual relationships showed significant correlations with the domain of psychological health on the WHOQOL-Bref. A summated score of the DIP-DIS was computed, consisting of all items but excluding those that would a priori disadvantage some of the patients or are otherwise not applicable to some: participation in household activities, sexual relationships, and overall performance in main occupation (occupational performance). This summated score correlated with all the four domains of WHOQOL-Bref. If the four domains of the WHOQOLBref are regarded as an adequate way of summarizing quality of life, then it would appear that the essence of quality of life in the LPD Study is captured by items dealing with subjective experience of satisfaction (with independence and life) and by the summated score of the DIP-DIS.

Table 4.

Criterion validity (Criterion: WHOQOL-BREF; Correlations ≥ 0.30)

Discriminant validity

The sampling frame in Victoria, consisting as it did of a substantial number of respondents residing in marginal accommodation, provided an opportunity to examine the discriminant validity of the DIP-DIS. Thus, an examination was made of the ability of the instrument to discriminate between respondents with complete data on the relevant sections of the DIP-DIS drawn from mainstream psychiatric services (mainly patients attending outpatient clinics, n = 150), patients receiving exclusive follow-up care from general practitioners (GPs) or private psychiatrists (n = 36), and respondents residing in marginal accommodation for the homeless (and not receiving mainstream psychiatric services, n = 103). The performance of the three groups of respondents on the summated disability score (computed as described above) were: patients with GPs/private psychiatrists 2.8 (sd 2.3), those in mainstream services 3.8 (2.9), and those in marginal accommodation 5.2 (2.8). These values were significantly different (F = 12.7, d.f. 2,286, p < 0.001) and followed gradations in the expected direction; with the patients with GPs/private psychiatrists being least impaired, and the residents of marginal accommodation being the most disabled.

Discussion

The DIP-DIS was designed to be brief and yet comprehensive in its coverage of the common forms of disablement among individuals with psychosis, thus addressing some of the limitations of existing measures. In this regard, there is a need for a good condition-specific health-related quality of life measure for use with persons with psychosis, that focuses on areas of functioning relevant to persons with psychosis and takes account of the likely extent of impairment experienced. The design of such a measure presents challenges, given the scope and multidimensional nature of disablement in persons with psychosis as illustrated by the framework of the International Classification of Impairment Disability and Handicap (ICIDH-2) [16]. Nevertheless, our examination of the properties of the DIP-DIS is promising. The report from research workers indicated that, in general, most respondents found it acceptable and easy to follow.

The result of the interrater exercise was generally encouraging. The poor reliability of the items relating to sexual and intimate relationships may relate to the sensitive nature of these items for most respondents. Indeed, these were items for which interviewers often recorded an inability to rate because of inadequate information or perceived non-relevance to individual respondents, thus attenuating the sub-sample on which analysis was conducted. The pattern of inter-correlation among the items suggests that the items were more complementary than redundant. However, the pattern also suggests that social relationships, as rated on the items ‘participation in household activities’ and ‘socializing’, have a particular primacy in the conceptualization of disablement associated with psychosis. The results of the factor analysis confirm the observation that the items were complementary. Four main factors, accounting for 26.6%%, 15.8%%, 12.6%%, and 11.2%% of the total variance, respectively, were derived. This suggests that all items are relevant to the DIP-DIS total score. Nevertheless, it would appear from the factor variances that performance in the domestic, social and occupational spheres accounts for most of the variability on the module. Taken together with the items encompassed in the second factor, these items cover a range of role functions that would traditionally be viewed as the core of notions of disability. It is noteworthy that the third and fourth factors identified might be equated with satisfaction, and enrichment/empowerment (in the sense that good interpersonal relationships and the active pursuit of interests may lead to these endpoints), respectively. Each of these features has been noted by Anthony [3] and others [4], [17] to be a part of the process of recovery from mental illness, and as such, our findings lend some support to this recovery model.

The results of the examination of the criterion validity were interesting. While it was reassuring that the summated score of the DIP-DIS correlated with the four domains of the WHOQOL-Bref, it was revealing that the items that most consistently correlated with the four domains were those dealing with respondents’ subjective assessment of satisfaction with own independence and with life. It seems clear from these observations that while the concept of quality of life is integral to that of disability, the latter is nevertheless broader than the former. While patients’ own evaluation of their own quality of life is essential [18], [19], an assessment of disability must go beyond such subjective evaluation. In general, this finding would suggest that interviewer ratings of disability using the DIP-DIS were supported by patients’ assessment of their own quality of life, emphasizing the link between the two [20], [21].

One of the limitations of this assessment of the performance of the DIP-DIS is the lack of data on sensitivity to change. Such data would be necessary to demonstrate whether any objective change in patients’ circumstances would be reflected in a change on their score on the DIPDIS. However, an indication that the instrument may possess this feature is provided by the results of the discriminant validity analysis. The significant differences, in the expected direction, in scores between patients recruited from GP/Private psychiatrists’ clinics, those from mainstream public psychiatric facilities, and those who were residing in marginal accommodation, would suggest that the instrument is sensitive to differing clinical populations with expected differences in level of disability.

In conclusion, the DIP-DIS has adequate psychometric properties for the objectives of the survey for which it was designed. Composed of 11 items, it is brief in administration (taking less than 15 min to complete) and, by integrating items from many sources, it provides a more comprehensive coverage of the core domains of disability than the existing instruments from which it was derived. For example, unlike the DIP-DIS, the WHO-DAS does not have items for rating subjective quality of life and the Lancashire Quality of Life Profile does not assess impairment in role functioning. Its shortcomings include the fact that it was based entirely on the information supplied by respondents and only on a single cross-sectional assessment. Longitudinal assessments of functioning, together with information supplied by key informants may both supplement and enrich an evaluation of the impact of psychotic disorders, as may data derived from observational assessments.