Abstract
Keywords
Despite decades of structural reform in Australian mental health services, social and economic marginalization are significant problems for many people living with mental illness. In Australia in 2003, approximately 16% of community residents with schizophrenia were employed compared to 77% of healthy adults [1], a pattern consistent with recent US figures [2]. Employment is a major contributor to mental well-being, along with education, unpaid work, caring for others and inclusion in everyday social activities. The World Health Organization International Classification of Functioning, Disability and Health advocates the terms ‘activity’ (the execution of a task or action by an individual) and ‘participation’ (involvement in a life situation) for these aspects of functioning [3].
Improving activity and participation is a focus of recent mental health policy. The Council of Australian Governments aims to increase ‘…the ability of people with a mental illness to participate in the community, employment, education and training’ [4]. Mental Health Plans for Queensland, Victoria and New South Wales (NSW) include a focus on improved employment and community participation [5–7].
Hence a brief domain-specific measure of vocational activity and community participation is needed. This paper presents the development of the APQ6, a measure designed to complement clinical assessment, to support dialogue between consumers and clinicians about recovery goals [8–11], and to allow monitoring of change over time at both individual and aggregated service levels.
The APQ6 complements measures currently used by Australian mental health services. The National Outcomes and Case-mix Collection includes clinician-rated scales such as the Health of the Nation Outcomes Scale [12] and Life Skills Profile [13], as well as self-report measures such as the Kessler 10 Plus, Behaviour and Symptom Identification Scale 32 and Mental Health Inventory [14–16]. None provides direct domain-specific measures of participation in formal education, competitive employment or other social activities.
Measures for routine use in community mental health services should be brief, inexpensive, valid, reliable, and sensitive to change, yet be simple to administer, score and interpret and require minimal training [17]. Measures of community participation should consider all major domains of role functioning, not only employment. Recovery-oriented measures should enable consumers to report their actual community activity, their satisfaction with their current activities, their participation goals, and their desire to change their level of activity within a specific domain.
To our knowledge there are no measures of community activity and participation that fit these specifications. The Multidimensional Scale of Independent Functioning and Socially Valued Role Classification Scale are domain-specific tools for specialist rehabilitation, but are lengthy and require substantial training [18,19]. Comprehensive checklists such as the Camberwell Assessment of Need [20] and Quality of Life Self-Assessment [21] identify only life domains in which the person reports current difficulty, rather than positively establishing the extent of activity and participation within each life domain.
Methods
Development of the APQ6
The measure was designed for use with adult consumers of ambulatory mental health services. It focuses on vocational, educational and social participation in the past week. The time frame and other features of the draft questionnaire were drawn from existing Australian Bureau of Statistics (ABS) surveys: the Australian Census 2006 [22], Survey of Disability Ageing and Carers 2003 [23], General Social Survey 2006 [24], and the continuous Labour Force Survey [25]. The APQ6 addresses the following domains.
Employment
This is defined as any paid work for at least 1 h in the last week, or unpaid work in a family business. The threshold is that adopted by the ABS in line with international definitions of labour force activity [25]. The extent of participation is assessed in terms of the number of hours employed in the past week.
Seeking employment
This asks whether the person was actively seeking employment in the past week, following the definition of ABS Labour Force Surveys. People who are not employed or actively looking for work are defined as ‘not participating in the labour force’. Together these three mutually exclusive subgroups (employed, looking for work, and not participating in the labour force) determine standard labour force statistics: participation rate, employment rate, and unemployment rate [26].
Unpaid work
Participation in unpaid work as a carer and in other socially valued roles provides individual rewards to the person and the wider community, sometimes as important as those flowing from employment. Carer responsibilities may also limit an individual's capacity to participate in activities such as work or training. Therefore asking about unpaid work is important in understanding a person's current and potential activity. Following the Survey of Ageing, Disability and Carers 2003 [23] this question distinguishes between carer roles due to ageing and long-term illness.
Education and training
This question is based on items in the Australian Census 2006 [22]. It asks about participation in school, University, TAFE or informal settings. Together with information about labour force activity, this defines overall participation in vocational activity (formal study, vocational training or employment).
Social and community participation
This question is based on social capital items within the ABS General Social Survey 2006 [24]. That survey includes too many specific activities for inclusion in a brief measure. Therefore a subset of measures was chosen to capture common activities that can be easily identified, and are relevant to clinical discussion and recovery planning. These include visiting relatives or friends, going out for a meal or entertainment, or participating in organized religious, sporting, arts or other interest group activities.
Readiness to change
This question was drawn from the five-level ‘stages of change’ framework [27] and was included to help clinicians respond to people at different stages of interest in changing their level of activity and participation.
A draft version of the measure was developed for NSW Health by the first author. A steering group to oversee the measure's development included representation from consumers (NSW Consumer Advisory Group), clinical rehabilitation services within two NSW Area Health Services (Hunter New England, South Eastern Sydney Illawarra) and researchers from the Queensland Centre for Mental Health Research who were independently testing this measure. A pilot was conducted with 29 consumers of four NSW vocational rehabilitation services. Qualitative feedback was used to make minor revisions prior to conducting reliability trials.
Test-retest reliability
Trials were conducted simultaneously by teams in New South Wales and Queensland during 2008. Because clinical services in rural or regional areas often rely on telephone contact with consumers, test– retest reliability was examined for both face-to-face and telephone interviews.
Short cycle (<6 day) test-retest reliability was assessed because this is most relevant to construct validity; activity and participation may vary from 1 week to the next. The NSW and Queensland studies used the same version of the measure, but their methods differed in some respects.
New South Wales study
This trial was conducted at psychiatric rehabilitation and community mental health services in regional (Hunter New England Area) and metropolitan (Bondi Junction Community Health, South Eastern Sydney Illawarra Area) settings. The study was approved by the human research ethics committee of each AREA HEALTH SERVICE.
During the study period (March–June 2008) services invited participation by all consumers aged 18–64 attending for outconsumer appointments or receiving clinical telephone contact. There were no exclusion criteria. No remuneration was provided. For each paired administration the APQ6 was administered using the same mode (face-to-face or telephone interview) and by the same clinician. The clinician was known to the consumer. For face-to-face administration the consumer was given a copy of the APQ6 to complete, with support by the clinician if needed. For telephone administration the APQ6 was read aloud without the consumer having a written copy.
A second interview was scheduled within 6 days. Following this consumers were asked to identify the most difficult questions, rate their level of confidence in their answers and provide feedback on further change to the measure.
Queensland study
Suitable candidates were identified from a pool of participants in a large-scale US–Australian collaboration [28], which had recently completed diagnostic assessments of 816 participants from the greater Brisbane region. A subsample of 374 candidates was identified from among subjects who had consented to be contacted for further research. Eligible candidates were systematically contacted until 60 verbally consenting persons meeting inclusion criteria completed two telephone interviews. Inclusion criteria were (i) a DSM-IV research diagnosis of schizophrenia or schizoaffective disorder; (ii) age 18–64 years; and (iii) had previously undertaken, or had indicated an interest in, formal study or training or employment. Exclusion criteria were (i) current hospital admission or acute illness; (ii) not contactable or unable to complete interview arrangements; or (iii) lack of informed verbal consent. Ethics approval was obtained from the West Moreton and Darling Downs Human Research Ethics Committee. Ethics approval and informed consent permitted access to diagnostic records.
Candidates were contacted by telephone by one of two researchers. In all cases except one the same researcher completed both interviews. The first interview sought verbal consent and if provided gathered brief sociodemographic characteristics. Administration of the APQ6 followed, along with four other brief measures being developed that are being reported separately. The second telephone interview, repeating administration of all measures, was conducted between 2 and 5 days later. Mean duration between interviews was 3 days (SD = 1.2, range = 2–5 days). Upon completion of the second interview the participants were mailed an AUD$50 gift voucher.
On completion of the measures, participants were asked to identify the most difficult or challenging questions, their level of confidence in the answers given and to provide feedback on how to improve the measure. Participants were also asked about significant changes in their circumstances between the first and second interviews. Interviewers were asked for feedback on their experience in administering the instrument.
Analyses
The minimum sample size required (n = 60) was conservatively calculated using estimates of the minimum correlation size (r = 0.4) that can be detected with ≥80% power. Reliability was assessed using Cohen's kappa, Spearman or Pearson correlation coefficients or intra-class correlation (ICC) coefficients, as indicated by the nature of the data. Cohen's kappa >0.8 are considered to represent very good agreement; 0.6–0.79 as good; 0.4–0.59 as moderate; 0.2–0.39 as fair, and <0.2 as poor.
Responses to the APQ were considered at both an overall question level (e.g. overall pattern of work or social participation) and at an individual item or subquestion level (e.g. which specific social activities were recorded). Both may be relevant depending on the application of the measure. Individual clinical care requires a detailed approach focusing on the number and pattern of diverse activities. In contrast, use of the APQ6 at service or system level requires calculation of simplified participation rates based on aggregated responses (e.g. overall rates of employment or training). Therefore reliability was considered at both question and item level.
First, kappa was calculated for each item independent of other items within the same question. For questions assessing hours of participation (questions 1a, 3a, 4a, 5a), consistency was measured using ICCs.
The APQ6 includes several questions allowing more than one item to be endorsed, such as question 3 (unpaid work) and question 5 (social and community participation). There are limitations in the use of kappa for this type of question [29], therefore an additional analysis coded all responses into a single value to preserve unique combinations. For instance, if responses 2, 3 and 5 were all selected, the response was coded as 235. The calculation of kappa requires that each response option must be available on both interview occasions. Therefore when a response option was not endorsed on both occasions the affected cases were excluded from analysis. Table 1 shows the number of response pairs included in analysis for each question. Analyses were conducted using SPSS version 17.0 (SPSS, Chicago, IL, USA).
Test–retest reliability for questions
APQ6, Activity and Participation Questionnaire. †Subjects interviewed face to face and by telephone. ‡Cases excluded from analysis due to asymmetry of responses between interview occasions. ∗∗∗p < 0.001.
Results
New South Wales study
Sixty-nine individuals provided paired interviews. Three were excluded due to a period of >6 days between administrations and three were excluded because the measure was repeated by different clinicians, leaving 63 valid interview pairs. Of these 39 (62%) were conducted face to face and 24 (38%) were conducted by telephone. For telephone interviews, all questions were completed by each participant. In personal administration, however, not all questions were completed. Complete responses ranged from 62 of 63 participants for question 1 (employment), to 52 of 63 for question 3 (unpaid work).
Characteristics of the sample are summarized in Table 2. Where demographic or diagnostic data were missing, clinicians were contacted in an attempt to obtain the missing information. Most participants (41, 65%) were receiving treatment for a psychotic disorder, but diagnostic information was not provided for 10 persons (16%). Seventy-five per cent (n 5 47) were under 45 years of age. There were approximately equal proportions of men and women, distributed across inner city, regional and rural locations.
Subject characteristics
Test-retest reliability was first examined at the item level (Table 3). Agreement at the item level was mostly very good (κ 5 0.8–1) or good (κ 5 0.6–0.79). Items with lower reliability included domestic work for household (0.21), visiting relatives or friends (0.49), visiting a restaurant or club (0.47), church or religious activities (0.57), and readiness to change (0.55). Kappas for personal interviews (n 5 39) were consistent with those from telephone administrations (n 5 24) on all items.
Test–retest reliability for each item in the question
APQ6, Activity and Participation Questionnaire; −, statistic not calculated because no respondents endorsed this item on at least one of the interview occasions; n1, no. respondents endorsing item on both interview occasions; n2, no. valid response pairs for analysis. †Subjects interviewed face to face or by telephone. ∗p < 0.05; ∗∗p < 0.01; ∗∗∗p < 0.001.
Responses to questions requiring estimation of hours spent in activities during the preceding week were more variable (Table 4). For these, reliability estimates (ICC) ranged from 0.71 (question 5, social participation) to 0.99 (question 4, Study).
Test–retest reliability using ICC for questions with duration of participation
ICC, intra-class correlation. †Subjects interviewed face to face and by telephone. ∗∗p < 0.01; ∗∗∗p < 0.001.
At the question level (Table 1) kappa was good-very good for questions 1–4 (paid work, seeking employment, unpaid work, education and training), but lower for question 5 (social and community participation, κ 5 0.52) and question 6 (readiness to change, κ 5 0.55). Spearman's correlation coefficients (0.91, 0.81, 0.76, 0.97, 0.55 and 0.46 for the six questions, respectively) were significant at the 99.9% confidence level and followed a similar trend to the kappa coefficients. Kappas for personal interviews were consistent with those from telephone administration at the question level.
Queensland study
Eighty-two participants were contacted as likely to meet all eligibility criteria. Of these, interview pairs were completed with 62 (76%). Nine people (11%) declined to participate. A further 11 people (13%) agreed to participate but were not available to complete both interviews during the study period.
Two of 62 cases were removed prior to analysis due to poor credibility of information. Response credibility was rated to three levels by interviewers on completion of each interview, and only two cases were considered ‘definitely not credible’ due to obvious and consistent response bias (e.g. responding to questions in exactly the same way without any sign of giving each independent thought).
Clinical and demographic information is summarized in Table 2. Fifty-five persons (92%) had a principal diagnosis of schizophrenia and five had a diagnosis of schizoaffective disorder (8%).
At the item level (Table 3) agreement was good-very good for most items. Less reliable items were those rating unpaid work and other community activities. For estimates of the number of hours spent in activities (Table 4), reliability was good-very good for all questions except question 3, hours worked in unpaid work (ICC 5 0.43). At the question level (Table 1) kappa was good-very good for all questions except question 3 (unpaid work, κ 5 0.59) and readiness to change (κ 5 0.56). Spearman's correlation coefficients (0.71, 0.65, 0.71, 0.97, 0.65 and 0.69 for the six questions, respectively) were statistically significant (p, 0.01), and followed a similar pattern to the kappa coefficients.
User utility and consumer acceptability
Of 63 NSW consumers, 39 provided feedback on their experience of completing the measure. Most reported that the measure took, 10 min to complete. All rated themselves as fairly confident (59%) or very confident (41%) in their answers. Seventy-eight per cent reported that none of the APQ6 questions was difficult to answer. When asked to identify the most difficult question, 31% named question 6 (readiness to change), 19% named question 5 (social and community participation), and 10% named question 3 (unpaid work). Some respondents provided suggestions for revised wording.
Of 60 Queensland participants, 40 reported difficulty estimating the time they spent on activities. Queensland interviewers reported that the measure had good clinical acceptability. It was simple to administer verbally; question 6 (readiness to change) seemed the most challenging due to the length of the response options.
Discussion
The present study examined the APQ6's feasibility; whether it was applicable, acceptable and practical in the setting for which it was designed [30]. Consumer feed back was mainly positive. Questions derived from population surveys were well received by consumers in telephone interviews and as a self-report questionnaire. Minimal staff training and support were required.
The study examined construct validity through the key indicator of test-retest reliability. This was in the moderate-very good range and compared favourably with other routinely used mental health outcome measures [31]. These findings were consistent across both samples and were not degraded by telephone administration. Where the measure was self-completed, minimal supervision was required. Although the rate of missing data was higher when administered in this mode, overall reliability was good.
A measure that is sensitive to change will show reduced test-retest reliability when real change occurs between administrations. At the second APQ6 administration participants in the Queensland study were asked if there had been any significant changes in their circumstances in the days since the first APQ6 administration. Twelve participants described significant changes between the first and second interview, such as changing their hours of work or study, joining a special interest group or spending more time with their child. Other respondents may have changed their activity levels but did not volunteer this information. Thus the estimates here are conservative.
A strength of the APQ6 is its very good test-retest reliability for participation in work and education. Reliability was lower for social participation. This may reflect concerns raised by consumers about the larger number of options in these questions and the wording of some specific items. The lower reliability may also reflect the more fluid nature of social participation, which may change substantially over a few days, compared to structured work or study commitments. Social participation is also likely to be less concretely defined and therefore more difficult to recall and quantify than employment or formal study. Reliability was also lower for the readiness to change question. This may reflect problems with applying the theoretical framework [32,33] to a different field, or the complexity of wording in the original draft.
This study had several limitations. Diagnostic information was not available on the entire NSW sample. The Queensland sample, however, was well defined and yielded similar results to the NSW sample. In the Queensland sample the majority of recruitment and interviewing was conducted during office hours using persons drawn from a larger genetics study of people with schizophrenia. Therefore the interviewed sample may differ from those currently in contact with mental health services, and may include fewer full-time students and workers.
Most persons at both sites had a diagnosis of a psychotic disorder. It was not considered necessary to represent community residents with less severe mental illnesses, because the cognitive and perceptual impairments associated with schizophrenia spectrum disorders represent the clinical features most challenging to reliability.
Limited training was provided to interviewers, and no inter-rater comparisons were conducted. This is not usually required for a self-report measure, but it is possible that different interviewers or clinicians may achieve different results. Further investigation of interview conditions is warranted to determine whether a formal interview (by telephone or in person) is necessary compared to unsupervised self-report.
The measure was not intended for people currently confined in a hospital or forensic unit, or who are not residing in the community. The measure has focused on activities relevant to adult consumers in the usual working age range, and is not designed for use with children, adolescents or older persons.
The APQ6 measure has been modified in response to these findings. The distinction between out-of-home and in-home social activity was removed, and an option of Internet-based social activity included. For all items, instructions were modified to consider a typical week if the past week was not typical. Hours of activity was removed from question 3 (unpaid work) and instructions clarified for all questions so that activity is estimated to the nearest whole hour. For question 6, the previously complex language based on ‘stages of change’ was replaced with a simplified question modelled on the work of Arroll et al. [34]. This asks whether the consumer is interested in increasing their level of activity in any of the domains, and if so whether they want service help with that. Testing of the impact of these minor changes based on reliability results has not yet been conducted.
Scoring of the APQ6 is being further developed to take into account the need for a measure that can be used both in individual clinical care and as an aggregate measure for reporting at service or system level. Work, education and socialization are independent but related dimensions of activity. Individual clinical care requires detailed information about a person's overall pattern of activity within and between these domains. For aggregate reporting, scoring will focus on the major domains of employment and social participation. Questions 1 (current employment) and 2 (work seeking) allow classification of individuals as either employed, looking for work or not in the labour force. Question 4 (study) allows dimensional categorization into formal study, informal study and no study. Social activity (question 5) may be aggregated by number or hours of participation.
Combining these dimensions (work, study, social activity) into a single aggregate score is not currently proposed because there is no basis to ascribe a relative value to each of these dimensions. Total hours of participation in all activities may provide a simplified composite measure.
Further refinement of the APQ6 may be needed. Assessment of concurrent validity is currently being undertaken, and testing of sensitivity to change is required. The suitability of the measure in other service settings, for example in non-government or primary care settings has not yet been assessed.
The APQ6 has been made available as an optional module within the NSW statewide mental health clinical documentation suite and is currently being used in routine practice by some mental health rehabilitation services. Changes have been made to NSW information systems to make results of the measure more available for analysis and reporting. Trials of the APQ6 are under way at two sites to assess utility in acute community mental health settings. Further dissemination will occur in 2010. The measure and instructions are available from the corresponding author on request.
Footnotes
Acknowledgements
Thanks to consumers at all sites, participating clinicians at NSCC, SESI and HNE Area Health Services, Susan Palmer (NSW Consumer Advisory Group), Marianna Wong, Victoria Bowrah, Hanna Shaw, Assoc. Professor Barry Frost, Adrienne Morris, Assoc. Professor Beth Kotze, Jo Summers, Lynelle Richards and the MGS2 research team. The Queensland study was funded by the Australian Government Department of Health and Ageing.