Abstract
It is estimated that 17.7% of Australians experience anxiety, affective or substance-use disorder in a 12 month period. This figure is thought to rise beyond 20% when disorders such as psychosis, neurasthenia, personality disorder and cognitive impairment are taken into account [1]. Some of these people will also have a significant hearing loss.
The Australian Institute of Health and Welfare, based on results from the Wilson et al. study [2], reported that as many as 17% of Australians have a ‘level of impairment likely cause some difficulty in speech discrimination’ [3]. The incidence of hearing loss in Aboriginal communities has been reported to be as high as 60% in some regions of Australia [4]. In all, these statistics indicate that approximately 3 000 000 Australians experience some level of hearing loss.
The incidence of severe to profound pre-lingual deafness in developed countries is generally reported to be approximately 1 per 1000 people. A similar incidence rate has been found for Australia, but the figure was as high as 3.46 per 1000 prior to 1974 due to a rubella epidemic and as low as 0.37 per 1000 by 1980 [5].
Epidemiological research into the incidence of mental illness in Deaf communities (defined as deaf people who use Australian sign language (Auslan) as their first language and who identify as members of a deaf community) is equivocal, but much of the literature agrees that risk factors for childhood neglect and the physical and sexual abuse of deaf children is much higher than for hearing children [6–9]. Despite these risk factors, one of the largest published epidemiological studies found very few significant differences between hearing and deaf populations in the incidence of mental illness in a sample of >500 case records for deaf people and hard-of-hearing people over a 5 year period [10]. More recent research with psychiatric inpatients in North America found that psychosocial functioning of deaf and hearing people was similar, but that deaf people were less likely to be diagnosed with a substance abuse or psychotic disorder, and were more likely be diagnosed with a mood, anxiety, personality or developmental delay disorder [11].
British research by deaf psychologist Dr Sharon Ridgeway using a British sign language version of the General Health Questionnaire (GHQ) found that >38% of a random non-clinical sample of 97 deaf people had scores indicating psychological distress [12]. Similar results were found in research using a New Zealand sign language version of the GHQ with a sample of 200 deaf people [13].
A recent European study also found significantly more symptoms of depression and anxiety in deaf samples [14] although, notably, only 20 of the 431 deaf people accessed the Norwegian sign language versions of the Symptom Checklist. Although these results are concerning, it should also be noted that there are no published deaf norms for sign language versions of the GHQ or other outcome measures. The World Health Organization–Quality of Life, Brief Symptom Inventory and GHQ have also been translated into a local sign language in Austria [15] but normative data have not been reported. Without normative data researchers cannot be confident that the results are not a consequence of linguistic or cultural phenomena. Further, using psychometric tools in their original written language format, with deaf people whose preferred language is a sign language, is a practice that has been widely criticized [16–18].
The relationship between early-onset hearing loss and compromised literacy outcomes is widely acknowledged [19]. Numerous studies report that the average English reading levels for adults with early-onset or pre-lingual hearing loss is well below that of hearing peers [19–21]. An Australian review by Power and Leigh suggested that the average reading level for deaf adults is often in the lower primary school age range [19]. Consequently, the use of routine written English, pen-and-paper assessment instruments is, in many cases, not appropriate for use with Deaf patients [17].
Even when tests are administered with the assistance of a sign language interpreter, there are concepts, English idiom and constructs that are language-based, to which Deaf patients may have never been exposed [17, 22]. Even though recommended practice requires the use of qualified sign language interpreters in mental health appointments, test administration requires standardization that cannot be achieved unless the test is presented in standardized format, such as recording an equivalent sign language translation on DVD. Add to this the fact that culturally deaf people have different cultural norms, social traditions, values and belief systems and it becomes clear that the norms and benchmarks for many routine psychometric instruments are not going to be valid or reliable with this patient group. That is, in the development and assessment of instruments it is important to recognize that the language differences are only one element of the cultural milieu.
Deaf people who use Auslan as their first language and have a preference for socializing with other Auslan users, often identify as members of the Deaf community, regardless of the level of hearing loss [23]. The Deaf community is increasingly recognized as a culturally and linguistically diverse group with distinct norms, social behaviours, values and belief systems. The extent of deafness and the audiological diagnosis play a less important role in a person's sense of belonging to the Deaf community or their experience of Deaf culture [24]. Once clinicians begin to appreciate deafness as a cultural experience it becomes obvious that many of the standard assessment tools have both cultural and linguistic biases and limitations. For many Australian mental health staff, however, the impact of culture on the psychometric assessment of deaf people has not been fully considered [25, 26].
To date, the Youth Self Report (YSR; translation from English to Auslan) is the only Auslan mental health tool that has published data on reliability [27]. Construct validity and norms comparing clinical and non-clinical samples, however, have not yet been established. Although the YSR Auslan version provides a promising starting point for clinical work with adolescents, there are currently no clinical tools available for working with Deaf adults in Australia. Mental health service delivery could be greatly improved if clinicians were able to assess the efficacy of their interventions with the Deaf community using a valid and reliable Auslan instrument.
The current research presents normative and clinical data for the Outcome Rating Scale (ORS)-Auslan, a measure of global functioning for use in mental health settings with Deaf patients. Given that there are no existing measures with published norms in Australia, a global functioning measure is an appropriate starting point from which to develop more specific diagnostic and psychometric tests, and has the broadest relevance to a wide range of mental health settings accessed by Deaf people. The hypotheses tested in the paper are: (i) that the ORS-Auslan can distinguish between clinical and non-clinical groups; (ii) that the ORS-Auslan has good internal reliability; (iii) that the ORS-Auslan demonstrates construct validity; and (iv) that the ORS-Auslan has a high level of acceptability and is user-friendly for Deaf patients.
In addition, preliminary normative data for an Auslan version of the Depression Anxiety Stress Scale-21 (DASS-21) are presented and because there are no other Auslan instruments available, these data form the basis of tests for construct validity.Figure 1.
Boxplots of the distribution of scores for the Outcome Rating Scale (ORS)-Australian sign language in clinical and non-clinical samples. (––), Cut-off score of 23.
Method
Participants
Participants were Deaf Auslan users aged ≥18 years. Advertisements in Deaf community publications, word of mouth, and emails were used to access the community sample in Brisbane, Queensland. In total, 55 Deaf people participated in the data collection for the non-clinical sample. A series of data collections was held at community venues. Each session saw attendance of between six and 15 Deaf people. At each session the chief researcher and a qualified interpreter presented the attendees with consent forms and an outline of the research purpose. All participants were required to sign consent confirming that they were over 18 years of age. The clinical data were gathered from 44 Deaf people in first-session presentations to outpatient and community mental health settings over an 18 month period. Ethics approval was granted by the university hosting the research and the external mental health services providing treatment to the clinical sample.
Instruments
The ORS is a four-item visual analogue measure [28] used to assess the same domains as the widely used Outcome Questionnaire 45 (OQ-45) [29]. Lambert et al. describe these domains as subjective discomfort, interpersonal relations, and social role performance. The ORS has an additional fourth domain that taps overall functioning. Miller et al. describe the four ORS items as (i) individual functioning; (ii) interpersonal functioning; (iii) social role functioning; and (iv) overall functioning [28]. The scale is a measure of general functioning taken from the perspective of the patient.
The original English version of the ORS has demonstrated acceptable reliability and validity and is a suitable brief alternative to the much longer OQ-45 [28]. The ORS has been used with large numbers of patients from diverse mental health settings and a range of presentations including anxiety, depression, substance-use disorders, chronic mental illness and work and family problems [30]. In addition to the broad application of this measure, the visual nature of the ORS lends itself to translation into sign language.
The ORS asks people to respond to each item with regard to how they have been feeling over the past week. Each of the four items has a 10 cm line with a visual anchor at each end: the right-hand end has a smiley face icon indicating high scores for the domain, and the left-hand end has an unhappy icon indicating low scores. People are asked to place a mark somewhere along the line to indicate how they have been feeling in respect to each domain. Scoring is done by measuring in millimetres where the patient's mark is along the line. Across the four items, this scoring method yields total possible scores between zero and 40, where lower scores are indicative of more distress. The authors of the original ORS determined a clinical cut-off score of 25 in adult hearing populations.
The ORS-Auslan version was produced by the commonly used process of translation and back-translation [31–34] and recorded onto a DVD. Two professionally accredited interpreters, two native Deaf signers and a bilingual hearing native signer who was blind to the original version reviewed the final ORS-Auslan 3.5 min DVD version. All parties agreed that the ORS-Auslan was equivalent to the original ORS-English. Each item is explained in sign language and a pause is provided for people to place their mark on the line after viewing the signing of each item. The ORS-Auslan response form differs only marginally from the English version by the addition of happy/unhappy face anchors at the poles of each item. The wording and scoring for both measures is in all other respects the same.
The DASS-21 is a short version of the 42-item self-report measure of depression, anxiety and stress [35]. In a similar manner to the ORS, the DASS-21 also focuses on experiences over the past week. There are no published data on the psychometric properties of the DASS-21-Auslan version and there are no published norms for DASS-Auslan scores with Deaf people.
The DASS-21-Auslan version was produced as part of a masters thesis in 2002 using the appropriate translation and back-translation method [36]. There are a number of language differences between the English version and the Auslan version based on interpretation from written English to visual–spatial mode. For example, the original DASS-21 item “I felt scared without any good reason” is signed as “I felt scared, but nothing was there to frighten me”. Similar adjustments have been made to each item to accommodate the visuospatial nature of Auslan. Scoring for the Auslan version is the same as the English version and uses a 4 point Likert scale of ‘never’, ‘sometimes’, ‘often’ and ‘always’.
In addition, the DASS-21-Auslan version has an added category labelled ‘not clear’ to indicate that the meaning of the signed item was not clear. The ‘not clear’ response category was scored as missing data because this information was not the focus of the current research.
Procedure
Historically, deaf people have been the subject of research by hearing people that has frequently resulted in negative and painful outcomes, and resulted in less trustful attitudes toward hearing researchers [7]. Consequently, it was important to explain to participants that the purpose of this research was to assess the credibility of these Auslan measures and not the mental health status of participants. In recognition of this research emphasis and to reassure participants of anonymity, it was optional for people to complete demographic data on gender and date of birth. Subsequently, there are more missing data for these variables in the non-clinical sample. Participants viewed the DVD versions of the ORS-Auslan and the DASS-21-Auslan and then recorded their responses on written English paper forms for each instrument at each session. That is, participants were able to follow Auslan instructions from the DVD in order to record their responses in written form by either marking along the line for the ORS-Auslan or circling on the Likert scale for the DASS-21-Auslan.
The clinical data collection was drawn from a sample of people attending three different mental health settings (private practice, public mental health outpatient facility and university counselling and family therapy centre) for the first time. Each of these mental health settings was trialling the use of the ORS-Auslan as a quality measure for their work with Deaf patients and agreed to provide first-session scores, age and gender data for the current research.
All patients were >18 years of age and were presenting as new patients or as new presentations. The ORS-Auslan was administered at the beginning of the first session. There were no exclusion criteria based on diagnoses, reason for presentation, referral method, or medication use. The resulting clinical group was a diverse sample representing a range of presentations including, anxiety, depression, chronic mental illness, schizophrenia, family and work problems. The clinical sample did not complete the DASS-21-Auslan because it was deemed too onerous to ask patients who were seeking mental health services to complete an additional assessment measure that takes ≥15 min to administer.
Statistical analysis
Data were analysed using SPSS (SPSS, Chicago, IL, USA). There was no substitution for missing data. In both the clinical and the non-clinical samples, total scores were calculated only for fully completed forms that had no missing subscale data. From the clinical sample of 44 ORS-Auslan forms only one was incomplete, leaving a final clinical sample for the ORS-Auslan total scores of 43.
In the non-clinical sample of 55 forms, ORS-Auslan total scores were calculated from the 49 fully completed forms. For the DASS-21-Auslan there were 37 forms that yielded a DASS total score. Similarly, subscales scores were computed for forms with no missing subscale data, yielding subscale scores for depression (n = 42), anxiety (n = 47), and stress (n = 49).
Results
Demographic data
Demographic data for the samples are presented in Table 1. Across both samples respondents ranged from 22 to 82 years of age.
Demographic data
Normative data
The ORS-Auslan normative data meet the assumption of normality required for t-tests and display markedly similar trends to the original English ORS instrument. The means for the ORS-Auslan were not significantly different from the original English ORS in either the clinical (t = − 0.701, NS) or non-clinical (t = − 0.771, NS) samples.
A two-tailed t-test comparing the non-clinical and clinical mean scores on the ORS-Auslan found that the clinical sample had a significantly lower mean (t = 4.446, p < 0.0001), indicating lower levels of well-being in the clinical sample.
To assess the clinical efficacy of the ORS-Auslan and to determine a clinical cut-off score using the means and standard deviations from both samples, where clinical and non-clinical samples are available and their distributions overlap, the clinical significance formula from Jacobson and Truax [37] was applied.
In the interests of consistency and comparability the same method that the authors of the original scale used to determine the cut-off is used in this Auslan context. Establishing a cut-off has psychometric value in that this score can be used for further tests of concurrent validity, although the purpose of the scale is not diagnosis, but rather to monitor changes in levels of functioning. The formula is:
Further, the distinctiveness of the samples must also be established when applying the Jacobsen and Truax formula, in the form of a significant t-test and a Cohen's d > 0.5 [38]. Using the statistical program G-Power [39] the current data set has Cohen's d = 0.93 (p < 0.05; power = 0.99), confirming the viability of the sizes of the samples for being able to determine the cut-off (Figure 1). In conjunction with the significant t-test results in Table 2 it is clear that the data reflect distinct samples. Using the Jacobsen and Truax method a clinical cut-off score of 23 for the ORS-Auslan was determined.
Auslan and English versions of the ORS (mean±SD)
Auslan, Australian sign language; ORS, Outcome Rating Scale.
Both the English and Auslan ORS measures reflect similar data trends, indicating the integrity or robustness of the scale across samples and settings. It is important, however, to remember that direct comparisons about the general functioning of Deaf and hearing people cannot be made and that the normative data for the Deaf sample are intended to stand alone (Table 2).
Reliability
Internal consistency was assessed using Cronbach's alpha. Specifically, the Cronbach's alpha coefficient is a reflection of the number of items and the average correlation between items [40]. Subsequently, scales that include more items are more likely to yield higher alpha coefficients. The potential impact of the number of items is particularly pertinent to the current analysis, in which the scale has only four items.
The Cronbach's alpha in the non-clinical sample was 0.85 and in the clinical sample it was 0.63. The development of clinical Auslan measures is still at the early stages of research and subsequently ‘reliabilities of.60 or.50 will suffice’ [41]. Given the low number of items and the heterogenous nature of the clinical sample, these results show that the ORS-Auslan demonstrates adequate internal consistency.
Construct validity
Concurrent validity was explored using an Auslan version of the DASS-21 in the non-clinical sample. The DASS-21 is a measure of subjective distress that yields an overall total scale score as well as subscale scores for depression, anxiety and stress [42]. Pearson's correlations were computed between total scores and subscales. Note that high levels of functioning are reflected by higher scores on the ORS-Auslan, while higher scores on the DASS-21-Auslan reflect higher levels of distress, hence the negative correlations. The ORS-Auslan total score correlated significantly with the DASS-21-Auslan total score r = − 0.59 (p < 0.0001) and with each of the DASS-21-Auslan subscales (Table 3). Significant correlations were also evident between the individual items of the ORS-Auslan and the subscales of the DASS-21-Auslan. Moderate to high significant correlations contribute solid evidence of concurrent validity for the ORS-Auslan.
Correlations between ORS-Auslan and DASS-21-Auslan (non-clinical data set)
Auslan, Australian sign language; DASS-21, Depression Anxiety Stress Scale-21; ORS, Outcome Rating Scale.∗p < 0.05, ∗∗p < 0.01.
Concurrent validity was assessed by exploring the ability of the ORS-Auslan to discriminate between clinical and non-clinical samples. A χ2 test of independence was performed to establish whether the clinical cut-off of 23, as determined by the current data set, could discriminate between the two sources of data, namely the clinical and non-clinical samples [43]. The pattern of occurrences between the clinical cut-off score and the source of the data collection was significant (χ2=7.245, p = 0.007). This finding, in conjunction with significant t-tests reported in Table 2, provides evidence that this instrument can distinguish between clinical and non-clinical samples.
Acceptability
Overall, ORS-Auslan had a completion rate of 93% compared with 63% for the DASS-21-Auslan, indicating a high degree of compliance and acceptability for the ORS-Auslan measure. In addition, the ORS-Auslan takes approximately 4 min to view and the DASS-21-Auslan takes approximately 15 min.
DASS-21-Auslan preliminary data
Table 4 summarizes the descriptive data for the DASS-21-Auslan collected from the non-clinical sample of Deaf people with reference to the original DASS-21 data [44] in a large non-clinical sample of hearing people. Higher scores on the DASS-21 reflect more distress.
Descriptive statistics for DASS-21-Auslan
Auslan, Australian sign language; DASS-21, Depression Anxiety Stress Scale-21.
†Normative data for the original DASS-21 with a hearing sample is sourced from [44].
Discussion
The aim of the present study was to examine the psychometric properties of the newly developed ORS-Auslan. The results are consistent with the hypothesis that the ORS-Auslan can distinguish between clinical and non-clinical presentations, has acceptable reliability and validity and is a user-friendly instrument for Deaf patients given the cultural and linguistic needs of this patient group.
Given the solid correlations with the DASS-21-Auslan, the ORS-Auslan also provides a valid and reliable instrument that can be used in future translation research to establish construct validity. This has been a key factor impeding Australian research efforts at developing culturally appropriate mental health instruments. In addition to these findings, the current research presents encouraging preliminary data on DASS-21-Auslan. An important consideration is that the aforementioned results should not be interpreted in relation to existing hearing norms for the DASS. The DASS-21-Auslan data, however, do provide important psychometric information for future research on psychological assessment in Deaf communities.
Increasingly, research is reporting on the acceptability of instruments in clinical settings. The usefulness of any measure is limited by the likelihood that patients will complete it. In the present study the ORS-Auslan had a completion rate of 93% compared with 63% for the DASS-21-Auslan. The likelihood that an instrument will be completed is an important consideration when deciding on an outcome measure.
With mounting pressure for routine outcome assessment in clinical settings, the ORS-Auslan provides clinicians with a culturally appropriate instrument to monitor progress with Deaf patients. The development of the Auslan version of the ORS provides clinicians with valid and reliable instrument to monitor the efficacy of interventions and an opportunity to address the void of research in the area of evidence-based practices for Deaf people who experience mental illnesses. Use of a validated, standardized Auslan version of the global and relatively visuospatial ORS also minimizes the impact of biases from English idiom, culturally bound constructs that may be unfamiliar to Deaf people [17].
The differences in the cut-offs between ORS-Auslan and ORS-English, although small, are consistent with other studies using sign language versions of English instruments that have shown the Deaf community to have higher levels of mental health problems when compared with data from hearing people [12–14]. The present study, however, goes further, by placing these results in the context of Deaf norms from a clinical and non-clinical sample, rather than making comparisons between Deaf results and hearing norms, which is potentially problematic. The fact that Deaf people have different cultural norms, social traditions, values and belief systems that influence the norms and benchmarks for many routine psychometric instruments, impacts on their reliability and validity for Deaf people. The current study complements the YSR, the only previously existing Auslan mental health tool that has published data on reliability [27], and extends that work by presenting evidence of construct validity and norms comparing clinical and non-clinical adult samples on a measure of global functioning.
The Deaf community is underserved in terms of culturally appropriate mental health services [12, 25], and many of the extant tools available to clinicians are inappropriate [22]. The present study has sought to address this need as well as expand the appreciation of clinicians of Deafness as a cultural experience, with the subsequent need for assessment tools that avoid both cultural and linguistic biases and limitations. Mental health service delivery could be greatly improved by clinicians being able to assess the efficacy of their interventions with the Deaf community using the aforementioned valid and reliable ORS-Auslan instrument.
Although the current findings provide a promising and necessary start to the evaluation of outcomes for Deaf patients, there are some important limitations that require further attention. First, with regard to reliability, the ORS-Auslan needs to be assessed for test–re-test reliability and sensitivity to change. Second, a larger sample size would provide further support for the psychometric properties reported in this paper. Last, a pilot study using the ORS-Auslan in a clinical setting would provide valuable information on the application of this instrument in routine outcome assessment.
