A review of social inclusion measures

Abstract

Background:

Social inclusion is crucial to mental health and well-being and is emphasised in Australia’s Fourth National Mental Health Plan. There is a recognition that a measure of social inclusion would complement the suite of outcome measures that is currently used in public sector mental health services. This paper is an initial scope of candidate measures of social inclusion and considers their suitability for this purpose.

Methods:

We identified potential measures through searches of PsycINFO and Medline and a more general Internet search. We extracted descriptive and evaluative information on each measure identified and compared this information with a set of eight criteria. The criteria related to the measure’s inclusion of four domains of social inclusion outlined in Australia’s Fourth National Mental Health Plan, its usability within the public mental health sector and its psychometric properties.

Results:

We identified 10 candidate measures of social inclusion: the Activity and Participation Questionnaire (APQ-6); the Australian Community Participation Questionnaire (ACPQ); the Composite Measure of Social Inclusion (CMSI); the EMILIA Project Questionnaire (EPQ); the Evaluating Social Inclusion Questionnaire (ESIQ); the Inclusion Web (IW); the Social and Community Opportunities Profile (SCOPE); the Social Inclusion Measure (SIM); the Social Inclusion Questionnaire (SIQ); and the Staff Survey of Social Inclusion (SSSI). After comparison with the eight review criteria, we determined that the APQ-6 and the SCOPE–short form show the most potential for further testing.

Conclusions:

Social inclusion is too important not to measure. This discussion of individual-level measures of social inclusion provides a springboard for selecting an appropriate measure for use in public sector mental health services. It suggests that there are two primary candidates, but neither of these is quite fit-for-purpose in their current form. Further exploration will reveal whether one of these is suitable, whether another measure might be adapted for the current purpose or whether a new, specifically designed measure needs to be developed.

Keywords

Social inclusion measure

Background

People with mental health problems are one of the most socially excluded groups in society (Social Exclusion Unit, 2004). While social exclusion is known to be a risk factor for the development of mental health problems (Bertram and Stickley, 2005), social inclusion can have protective benefits, ameliorating the negative effects of stress, and contributing to mental illness recovery (Harrison and Sellers, 2008). Once established, mental illness in turn can have a detrimental effect on social competence, confidence and self-esteem (Anthony, 1993; Borba et al., 2011; Hooley, 2010) and can reduce opportunities for social interaction and participation in all aspects of life (Hooley, 2010; Social Exclusion Unit, 2004), creating a maintaining cycle of social isolation that is seen most vividly in those with chronic mental illness (Anthony, 1993; Borba et al., 2011; Hooley, 2010). For this reason, Australia’s Fourth National Mental Health Plan (Australian Health Ministers, 2009) emphasises social inclusion for people with a mental illness, with the first of its five priority areas being ‘Social Inclusion and Recovery’.

Despite increasing recognition of the importance of ‘social inclusion’ to mental health and well-being, the precise meaning of the term is the subject of some debate (Marino-Francis and Worrall-Davies, 2010). Different commentators have offered different definitions (see Box 1 for examples) that vary widely (Morgan et al., 2007). However, a number share commonalities in their focus on the importance of those with mental illness having access to, and participating in, all opportunities and choices afforded to other people (Bates and Repper, 2001; Mental Health Commission, 2009; Slade, 2009) and the inclusion of both an objective and a subjective element (Huxley et al., 2006; Le Boutillier and Croucher, 2010; Morgan et al., 2007). The objective element relates to the extent to which the individual participates in various life domains and is often measured by counting time spent participating in activities within the community (e.g. ‘Number of days in the past week in which participated in employment’) (Lloyd et al., 2008) and the number of activities participated in and/or the number of social contacts available (e.g. ‘Total number of people in support network “when feeling down in the dumps” ’) (Lloyd et al., 2008). The subjective element refers to whether the individual feels that their participation matches his or her preferences; this is generally measured by assessing the person’s satisfaction with his or her experience (e.g. ‘How do you feel about your range of opportunities to access suitable accommodation?’) (Huxley et al., 2012) or a desire for change (e.g. ‘Are you interested in increasing your participation in the following: employment, unpaid, education and training?’) (Stewart et al., 2010).

Box 1.

Selected definitions of social inclusion

Social inclusion is …

… about each person taking part in society and having control over his or her own resources. It is also about a community that cares for its members, makes them feel welcome and is willing to adjust to fit their various needs (Marino-Francis and Worrall-Davies, 2010)

… the extent to which people are able to exercise their rights and participate, by choice, in the ordinary activities of citizens (Mental Health Commission, 2009)

… a person’s right to participate as an equal citizen in all the opportunities available, employment, education and other social and recreational activities (Slade, 2009)

… full access to mainstream statutory and post-16 education, open employment and leisure opportunities alongside citizens who do not bear these (mental illness) labels (Bates and Repper, 2001)

… a virtuous circle of improved rights of access to the social and economic world, new opportunities, recovery of status and meaning, and reduced impact of disability (Sayce, 2001)

Those life domains that are seen as essential to social inclusion are also not well established (Morgan et al., 2007). However, there is some degree of acceptance of the importance to mental health of access to employment and/or education (Department of Health, 2001; HM Government, 2010; Social Inclusion Unit, 2011), stable housing (Harvey et al., 2002; Huxley et al., 2006), community participation (Harvey et al., 2002; Social Inclusion Unit, 2011) and social networks (Harvey et al., 2002; Huxley et al., 2006). Consequently, Australia’s Fourth National Mental Health Plan outlines five indicators against which to measure desired change in the area of social inclusion: (1) participation rates by people with mental illness of working age in employment; (2) participation rates by young people aged 16 to 30 with mental illness in education and employment; (3) percentage of mental health consumers living in stable housing; (4) rates of community participation by people with mental illness; and (5) rates of stigmatising attitudes within the community (Australian Health Ministers, 2009). Note that, given this is a government initiative, these domains reflect indicators that can be measured at the population level, rather than the individual level; hence the exclusion of social networks. Measuring levels of social inclusion for individuals accessing mental health services can help to measure progress across Australia on this important component of Australia’s mental health strategy. It can also help services to identify whether their practices are promoting social inclusion as a key component of recovery and measurement can promote discussion between individual service providers and mental health service users about strategies to promote social inclusion.

In addition to striving to perform well against the above indicators, the Fourth National Mental Health Plan commits to measuring outcomes for consumers using public sector mental health services (Australian Health Ministers, 2009). The current suite of outcome measures, which includes the Health of the Nation Outcomes Scale (HoNOS) (Wing et al., 2000), the Mental Health Inventory (MHI) (Veit and Ware, 1983), the Behaviour and Symptom Identification Scale 32 (BASIS-32) (Eisen et al., 1986) and the Kessler-10 Plus (K-10+) (Centre for Population Studies in Epidemiology, 2002), is fairly clinical in focus and emphasises reductions in symptomatology and improvements in levels of functioning (Pirkis and Callaly, 2010). A national protocol specifies those measures that should be collected within particular mental health settings and at which time points. For example, the HoNOS is collected for all adults in inpatient, community residential and ambulatory settings at admission, review and discharge from mental health care (Australian Mental Health Outcomes and Classification Network, 2005).

There is recognition that a measure of social inclusion may need to be added to this suite. Such a measure should reflect the first four indicators described above (the fifth needs to be gauged through community surveys) and should include some of the more subjective components of social inclusion (Le Boutillier and Croucher, 2010). The process for collection of social inclusion data would need to be developed in consideration of the measure chosen for this purpose and specified in the national protocol.

A suitable measure for standard use would need to meet certain psychometric criteria as well as being usable within community mental health services. Psychometrically, the measure must be valid, reliable and sensitive to change (Stewart et al., 2010). In order to increase the likelihood of completion, the measure should also be brief, inexpensive, simple to administer, score and interpret (Stewart et al., 2010), preferably be completed by consumer self-report, and be acceptable to mental health consumers.

The imperative to identify a quality measure of social inclusion for potential use in public sector mental health services led us to review the existing measures. We aimed to identify available individual-level candidate measures of social inclusion, to describe their characteristics and to undertake a preliminary examination of their potential for routine use in the current context.

Method

We searched PsycINFO and Medline for articles published between January 2010 and the end of January 2012, using the terms (‘social inclusion’ OR ‘community participation’ OR ‘social capital’ OR ‘social isolation’) AND (‘mental health’) AND (‘measure’). We also conducted a general Internet search, via Google, using the term ‘social inclusion measure’.

We then used the measure names as search terms in PsycINFO and Medline to identify any papers that outlined further psychometric testing of the social inclusion measures identified in the initial search. Where the actual measures were not publicly available, we wrote to the corresponding author to request a copy.

We used eight criteria to review the quality and utility of the candidate social inclusion measures:

measures multiple domains of social inclusion, including employment, education, housing and community participation;

measures both objective and subjective components of social inclusion;

is self-completed by the consumer;

yields qualitative data (not excluding measures that also yield qualitative data);

is relatively brief (50 items or less);

has tested usability with mental health consumers;

is applicable to the Australian context;

has sound, established, psychometric properties.

To compare the measures using this standard set of criteria, we extracted descriptive information on each measure identified through the search. We only considered the published form of each measure in comparison with the criteria, inclusive of all scales and items, and we only considered the published mode of administration. Research evidence suggests that altering the tested mode of administration can affect a measure’s validity and outcomes (Bowling, 2005) and shortening measures alters its psychometric properties such that the shortened form then requires its own psychometric testing (Coste et al., 1997).

We extracted any available information on the psychometric testing of each measure. Specifically, we identified any resulting indicators of validity (i.e. the extent to which they measure what they purport to measure) (Greenhalgh et al., 1998) and reliability (i.e. the extent to which they give stable, consistent results) (Greenhalgh et al., 1998). Specifically, we examined construct validity (which involves conceptually defining the construct to be measured and assessing the internal structure of its components and the theoretical relationship of its items and subscale scores) and concurrent validity (which pits the instrument against a comparable measure at the same point in time). We operationalised reliability in terms of internal consistency (i.e. the extent to which items that reflect the same construct yield similar results) and test-retest reliability (i.e. the degree of agreement when the same measure is completed by the same person at two different points in time). We also considered each measure’s sensitivity to change. Sensitivity to change is related to both validity and reliability: a measure that is both valid and reliable and which demonstrates change over time can be regarded as being sensitive to change. We also attempted to identify whether the measure had undergone testing with mental health consumers and whether it had been tested within Australia.

Results

Search results

We identified the following 10 candidate individual-level measures of social inclusion:

Activity and Participation Questionnaire (APQ-6) (Stewart et al., 2010);

Australian Community Participation Questionnaire (ACPQ) (Berry et al., 2007);*

Composite Measure of Social Inclusion (CMSI) (Lloyd et al., 2008);*

EMILIA Project Questionnaire (EPQ) (Ramon et al., 2009);*

Evaluating Social Inclusion Questionnaire (ESIQ) (Stickley and Shaw, 2006);*

Inclusion Web (IW) (Hacking and Bates, 2008);

Social and Community Opportunities Profile (SCOPE) (Huxley et al., 2012);

Social Inclusion Measure (SIM) (Huxley et al., 2012; Secker et al., 2009);*

Social Inclusion Questionnaire (SIQ) (Marino-Francis and Worrall-Davies, 2010);*

Staff Survey of Social Inclusion (SSSI) (Dorer et al., 2009).*

Those marked with an asterisk were not named by their creators, usually because they were developed with the purpose of evaluating a given service and there was no explicit intention that they might have ongoing use. We have named them, based on the terminology used about them by their creators, in order that we could readily make reference to them in the remainder of this review.

More detail about each of these measures is provided in Table 1. The measures identified have been developed relatively recently in countries that have a current emphasis on social inclusion, notably Australia and the United Kingdom. The candidate measures cover a range of domains related to social inclusion, often including those emphasised in the Fourth National Mental Health Plan (especially employment, education and community participation) (Australian Health Ministers, 2009), and often focus on both objective and subjective experiences. Some (e.g. the CMSI) draw on questions from related instruments and/or national surveys; others (e.g. the IW) were developed for a specific study purpose. With the exception of the ACPQ, all were explicitly developed for use with people with mental illness. They vary in length: the APQ-6 is the shortest, with a maximum of 14 possible items, and the SCOPE–long version is the longest, with 121 items. The majority of measures elicit responses in the form of quantitative data; only the EPQ generates qualitative data. Almost all seek responses directly from consumers; only the SSSI uses staff as informants. The measures also represent a mix of self-report and interviewer-administered instruments, and the latter are sometimes explicitly designed to promote dialogue between service providers and consumers (e.g. the IW) (Berry et al., 2007; Dorer et al., 2009; Hacking and Bates, 2008; Huxley et al., 2012; Lloyd et al., 2008; Marino-Francis and Worrall-Davies, 2010; Ramon et al., 2009; Secker et al., 2009; Stewart et al., 2010; Stickley and Shaw, 2006).

Table 1.

Profile of social inclusion measures.

Measure	Domains	Types of participation / social inclusion factors	Number of items / time taken	Scoring	Administration	Usage
Activity and Participation Questionnaire (APQ)	Report actual activity Satisfaction with activities Participation goals Desire to change level of activity	Employment Seeking employment Unpaid work Education and training Social and community participation Readiness to change	14 possible items (some items may be skipped depending on response to initial questions) < 10 minutes to complete	Participation measured using hours Employment scored categorically Readiness to change allocated to a stage of change based on response; this would require training to score	Self-report Telephone or face-to-face interview	Support clinician–consumer discussions about social inclusion
Australian Community Participation Questionnaire (ACPQ)	Informal social connectedness Civic engagement Political participation	Contact with immediate household, extended family, friends and neighbours Social contact with workmates Organised community activities Giving money to charity Voluntary sector activity Adult learning Religious observance Active interest in current affairs Expressing opinions publicly Community activism Political protest	• 67 items	7-point Likert scale Scoring protocol not specified	• Self-report	Not designed for clinical use Developed for use with general population Not tested with people with a mental illness
Composite Measure of Social Inclusion (CMSI)	Socially valued role functioning Social support Absence of stigma experiences Integration in the rehabilitation community Integration in the wider community	Home duties and self-care Caring for others Engagement in rehabilitation Formal study or approved training Competitive employment	Five domains by 15 levels 9 items on stigma experiences rated on 5-point Likert scale 20 items on community integration	Classification table used to create a SRCS role classification score (uses weekly hours of participation, performance standard, support needed to perform role)	Face-to-face interview First interview ≈ 42 min Second interview ≈ 33 min	• Mental health service users
EMILIA Project Questionnaire (EPQ)	Education Training Employment Meaningful unpaid activities Social networks	As reported on left	10 questions that promote reflection on consumers’ life over the past year and the coming year	Not scored – qualitative only Responses analysed using thematic analysis	• Self-report either written or verbally and tape-recorded	• Clinical use with mental health service users
Evaluating Social Inclusion Questionnaire (ESIQ)	The community Relationships Official services	Community Leisure Education Work Housing Freedom to express beliefs Social life Stigmatisation Treatment by services Friends, family and neighbours Fulfilment of potential	18 items ≈ 20 minutes	7-point Likert scale Suggested that scale be used qualitatively rather than as quantitative measure	Semi-structured interview Originally developed as a self-report but in testing determined that it was better to be used as a semi-structured interview-guide	• Mental health service users
Inclusion Web (IW)	People (personal relationships) Places (institutions that matter to the individual)	Employment Education Volunteering Arts and culture Faith and meaning Family and neighbourhood Sport and exercise Services	• Information about participation in 16 areas (left) charted visually	Count of activities, total people, total places ‘Clockspread’ total Scoring software available Trained administrator converts visual map to summary score above	Clinician and consumer discussion Map of consumer’s network of ‘places’ and ‘people’ is developed	• Facilitation of discussion between mental health service user and clinician
Social and Community Opportunities Profile (SCOPE)	Perceived opportunities Satisfaction with opportunities Subjective well-being	Leisure and participation Housing and accommodation Safety Work Financial situation Self-reported health Education Family and social relationships	Long version: 121 items Consumer completion ≈ 37 min Short form: 48 items Student completion ≈ 9 min	Varied, includes 5- and 7-point Likert scales and categorical (e.g. yes/no) ‘check-box’ responses Responses should be compared with national averages rather than aggregated to measure inclusion	Self-report Interview	General population Mental health service research As an outcome measure in mental health services
Social Inclusion Measure (SIM)	Social isolation Social relations Social acceptance	Building social capital Social acceptance Neighbourhood cohesion Security of housing tenure Leisure and cultural activities Citizenship	19 items Refers to last 3 months	4-point Likert scale Total: sum of items Subscale scores: social isolation, social relations and social acceptance	15 minutes Administration method not specified but probably self-report	• Mental health service users
Social Inclusion Questionnaire (SIQ)	Social relationships Sense of community Mental health services used	Feeling accepted by: neighbours and community and involved in leisure activities and satisfied with friends and mental health workers Seeking and being involved in groups outside mental health	23 items 30 minutes for consumer completion Up to 1 hour if clinician-assisted by reading the questions to the consumer	5-point Likert scale No sub-scales yet determined	Self-report Interview	• Mental health service users
Staff Survey of Social Inclusion (SSSI)	Employment Education Volunteering Arts Faith and culture activities Sport and exercise Local neighbourhoods Day centres Contact with family and friends	Education Employment Day centres Sports Faith Arts Local facilities Family and friends	Staff estimation of time spent in activities over a 7-day period Each activity allocated a level of social inclusion Requires staff training to administer	Activities completed by consumer are allocated to a domain by the staff member Time spent calculated and ranked to one of three social inclusion levels Results could be used at individual or service level	• Staff-completed	• Mental health service users

SRCS: Socially Valued Role Classification Scale.

Comparison of measures with criteria 1 to 7

We initially considered the usability of the measures by comparing the attributes of each measure with the first seven criteria. We then considered the psychometric properties of the measures (criterion 8). Comparing each measure with the first seven criteria, we found that:

The ACPQ, SIM and SIQ measure too few (two domains or less) of the domains of interest in the Fourth National Mental Health Plan (Australian Health Ministers, 2009).

The ACPQ, IW and SSSI focus on the objective components of social inclusion and do not pay sufficient heed to the subjective experiences of the consumer.

The CMSI, IW and ESIQ are administered as face-to-face interviews, rather than through self-report, limiting their usability in public mental health.

The qualitative nature of the EPQ means that although it may be useful at the individual level and may promote discussion between the service provider and the consumer, it is unlikely to generate information that can be aggregated across consumers for the purposes of monitoring broader changes in social inclusion.

The long version of the SCOPE is too long at 121 items, although the shorter, 48-item version may still be of use; the CMSI is also too long, taking approximately 40 minutes to complete the structured interview.

More than half of the measures have undergone scrutiny from the point of view of their acceptability to users (the APQ-6, CMSI, ESIQ, SCOPE, SIM, SIQ and SSSI). In the main, this has occurred in the context of their development and has sometimes resulted in modifications to questions or response sets. In all cases, feedback from users about these measures has been positive (Dorer et al., 2009; Huxley et al., 2012; Lloyd et al., 2008; Marino-Francis and Worrall-Davies, 2010; Secker et al., 2009; Stewart et al., 2010; Stickley and Shaw, 2006). The ACPQ has not been tested with mental health service users and is designed for the general population, and the acceptability of the EPQ and the IW is also not outlined in the literature.

Only the APQ-6, the ACPQ and the CMSI have undergone development and testing within Australia.

Considering comparison of the candidate measures with the first six criteria together, the APQ-6 and the SCOPE–short form are the only measures to meet all six criteria. The APQ-6, however, does not measure the domain of housing (included in criterion 1), but does have the advantage of having been developed and tested for use in the Australian context (criterion 7), while the SCOPE was developed and tested in the UK.

Comparison of measures with criterion 8: psychometric properties

Table 2 summarises existing information about the psychometric properties of each measure that has undergone some form of psychometric testing. Most of the measures have undergone some psychometric testing, albeit not extensively. The exceptions are the ESIQ, the SSSI and the EPQ. The EPQ, as noted above, elicits only qualitative information and is therefore not amenable to psychometric testing.

Table 2.

Psychometric properties of remaining social inclusion measures.

Measure	Validity		Reliability		Sensitivity to change
	Construct validity	Concurrent validity	Internal consistency^a	Test-retest reliability^b
Activity and Participation Questionnaire (APQ-6)	Reported to be ‘good’, on the basis of sound test-retest reliability (see right) and positive consumer feedback; but, it has not been evaluated independently of these properties (Stewart et al., 2010).	Not reported, but see discussion regarding the Composite Measure of Social Inclusion (CMSI) (Lloyd et al., 2008), below.	Not reported	Moderate to very good test-retest reliability (kappa = 0.52–0.96; r = 0.55–0.97) over periods of up to 5 days (n = 129 mental health service users) (Stewart et al., 2010).	Not reported
Australian Community Participation Questionnaire (ACPQ)	Good: 14 types of participation that underpin the ACPQ constitute best-fitting model for the 67 items as assessed by exploratory factor analysis. This solution tested through one-factor congeneric models for each of the factors, and most of the models were fitted with minor modification or no modification (Berry et al., 2007).	Reasonable: seven of the APQ’s 14 types of participation negatively correlated with general psychological distress as assessed by the Kessler-10 (K-10) (Kessler et al., 2002); nine types of participation measured were significantly independently related to distress (Berry et al., 2007).	Not reported	Not reported	Not reported
Composite Measure of Social Inclusion (CMSI)	Not reported	Not reported; but, concurrent validity of the SRCS, of which the CMSI is partly comprised, has been examined (Harris et al., 2011a). SRCS items show moderate to very good associations with some (though not all) relevant items on the APQ-6 (Stewart et al., 2010) and the Work-related Self-efficacy Scale (WSS-37) (Waghorn et al., 2005), but poor correlations with relevant items on the Education-related Self-efficacy Scale (ESS-40) (Harris et al., 2011b).	Acceptable to good (α = 0.74–0.85) (Lloyd et al., 2008).	Socially valued role functioning and social support items suggest sound test-retest reliability (kappas not reported, r = 0.36–0.96 and r = 0.43–1.00, respectively) at 24–96 hours (n = 26 mental health service users); majority of stigma experiences and community integration items do too (kappas not reported, r = 0.63–0.89 and r = 0.41–0.91, respectively, with removal of items with non-significant correlations) (Lloyd et al., 2008).	Not reported
EMILIA Project Questionnaire (EPQ)	Not reported	Not reported	Not reported	Not reported	Not reported
Evaluating Social Inclusion Questionnaire (ESIQ)	Not reported	Not reported	Not reported	Not reported	Not reported
Inclusion Web (IW)	Examined in context of assessing the coherence of the overall measure of clockspread. There were significant correlations for people and places in all domains except those of arts and culture and faith and meaning, suggesting that the notion of clockspread makes sense (Hacking and Bates, 2008).	Not reported	Not reported	Not reported	Demonstrated to be sensitive to change when tested on consumers receiving standard and enhanced services. Consumers demonstrated small to medium improvements on almost all domains in terms of both people and places, and in the overall clockspread score (Hacking and Bates, 2008).
Social and Community Opportunities Profile (SCOPE)–long version	Not reported	Shown to assess concepts that overlap with, but are not identical to, participation (as assessed by the ACPQ) (Berry et al., 2007) and social capital (as assessed by the Resource Generator-UK; RG-UK) (Huxley et al., 2012; Webber and Huxley, 2007). Two subscales correlate with measures of social capital and community participation (r = 0.33–0.48 and r = 0.42–0.42, respectively, p < 0.01).	Questionable to acceptable (α = 0.60–0.75) (Huxley et al., 2012).	Not reported	Under investigation
Social and Community Opportunities Profile (SCOPE)–short form	Not reported	See above	Questionable: acceptable (α = 0.62–0.77) (Huxley et al., 2012). Inter-item correlation = 0.25; long and short scales correlate at r = 0.88–0.92.	Good to very good (kappa = 0.66–0.97; r = 0.62–1.00) over a 2-week period (n = 119) with university students (Huxley et al., 2012).	Under investigation
Social Inclusion Measure (SIM)	Support comes from test of unidimensionality which found that each of the three scales (social isolation, social relations and social acceptance) correlate well with each other (r = 0.52–0.70, p < 0.001), and very highly with the overarching model (r = 0.78–0.91, p < 0.001) (Secker et al., 2009).	Shown to correlate with the Clinical Outcomes in Routine Evaluation (CORE, a measure of mental health status) (Core System Group, 1998) (r = 0.58, p < 0.001) and an adapted empowerment measure (Schafer, 2000) (r = −0.62, p < 0.001).	Good (α = 0.85) (Secker et al., 2009).	Not reported	Not reported
Social Inclusion Questionnaire (SIQ)	Support comes from a factor analysis which revealed seven factors that underpinned the concept of social inclusion. Three items cross-loaded on more than one factor and it was suggested that these be removed from the measure (Marino-Francis and Worrall-Davies, 2010).	Not reported	Good (α = 0.80) (Marino-Francis and Worrall-Davies, 2010).	Moderate or better: over a 2-week period (n = 51) 17/23 items had kappas of > 0.4 (acceptable) and 5/23 had a kappa of > 0.6 (good) (Marino-Francis and Worrall-Davies, 2010). The overall kappa range was 0.12–0.82 (poor to very good).	Not reported
Staff Survey of Social Inclusion (SSSI)	Not reported	Not reported	Not reported	Not reported	Not reported

SRCS: Socially Valued Role Classification Scale.

In the case of internal consistency, a Cronbach’s α of ≥ 0.90 is regarded as excellent, 0.80–0.89 as good, 0.70–0.79 as acceptable, 0.60–0.69 as questionable, 0.50–0.59 as poor and < 0.50 as unacceptable (Streiner, 2003).

In the case of test-retest reliability, kappas ≥ 0.81 are regarded as very good, 0.61–0.80 as good, 0.41–0.60 as moderate, 0.21–0.40 as fair and ≤ 0.20 as poor (Cohen, 1960; Landis and Koch, 1977).

Collectively, those which have been tested have displayed a largely sound performance in terms of some or all of the following: construct validity, concurrent validity, internal consistency and test-retest reliability. The least tested property is sensitivity to change; this has only been explored for the IW, which was shown to be capable of identifying incremental improvements in social inclusion in a sample of mental health service users (Hacking and Bates, 2008). It is important to note that while some psychometric properties of the CMSI as a complete measure are noted in Table 3, the Socially Valued Role Classification Scale (SRCS), which comprises a large portion of the CMSI, has also undergone independent psychometric testing (Waghorn et al., 2007). This testing provides further evidence for the construct validity, test-retest reliability and sensitivity to change of some scales of the CMSI, though not of the measure as a whole.

Table 3.

Summary of attributes of social inclusion measures.

Attribute	Measure
	APQ-6	ACPQ	CMSI	EPQ	ESIQ	IW	SCOPE	SIM	SIQ	SSSI
Measures multiple domains related to social inclusion	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Specifically considers domains emphasised in the Fourth National Mental Health Plan:
Employment	Yes	No	Yes	Yes	Yes	Yes	Yes	No	No	Yes
Education	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	No	Yes
Housing	No	No	No	No	Yes	No	Yes	No	No	No
Community participation	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Measures objective and subjective dimensions of social inclusion	Yes	No	Yes	Yes	Yes	No	Yes	Yes	Yes	No
Was developed for use with people with mental illness	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Is brief (≤ 50 items)	Yes	No	No	Yes	Yes	Yes	Long–No Short-Yes	Yes	Yes	Yes
Self-report	Yes	Yes	No	Yes	No	No	Yes	Yes	Yes	No
Yields quantitative data	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes
Demonstrates sound psychometric properties:
Construct validity	Yes	Yes	Unknown	N/A	Unknown	Yes	Unknown	Yes	Yes	Unknown
Concurrent validity	Yes	Yes	Yes	N/A	Unknown	Unknown	Yes	Yes	Unknown	Unknown
Internal consistency	Unknown	Unknown	Yes	N/A	Unknown	Unknown	Unknown	Yes	Yes	Unknown
Test-retest reliability	Yes	Unknown	Yes	N/A	Unknown	Unknown	Yes	Unknown	Yes	Unknown
Sensitivity to change	Unknown	Unknown	Unknown	N/A	Unknown	Yes	Unknown	Unknown	Unknown	Unknown
Is applicable to the Australian context	Yes	Yes	Yes	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown
Is acceptable to users	Yes	Unknown	Yes	Unknown	Yes	Unknown	Yes	Yes	Yes	Yes

APQ-6: Activity and Participation Questionnaire; ACPQ: Australian Community Participation Questionnaire; CMSI: Composite Measure of Social Inclusion; EPQ: EMILIA Project Questionnaire; ESIQ: Evaluating Social Inclusion Questionnaire; IW: Inclusion Web; SCOPE: Social and Community Opportunities Profile; SIM: Social Inclusion Measure; SIQ: Social Inclusion Questionnaire; SSSI: Staff Survey of Social Inclusion.

Table 3 distils the above information about each measure and indicates whether the measure meets each criteria.

While no measure meets all eight of the criteria, both the APQ-6 and SCOPE–short form meet the majority of the criteria. Neither of these measures yet demonstrates all desirable attributes, partly because their psychometric properties require further testing, but also because the APQ-6 does not consider the domain of housing and because the SCOPE–short form has yet to be tested for Australian use. Furthermore, the authors recommend that the scores on the SCOPE–short form are best used to compare with national averages, rather than aggregated for use as a measure of social inclusion, meaning that Australian norms would need to be established.

Discussion

Interpreting the findings

Our review identified 10 candidate individual-level social inclusion measures. This number is fairly small, particularly considering the vast array of measures that exist for assessing clinical changes. The paucity of measures may reflect the fact that social inclusion has only come on to the agenda for mental health system reform relatively recently. The focus on social inclusion is now gaining momentum because there is acknowledgement that elements of social inclusion, like community participation, underpin good mental health. However, the absence of an agreed definition of social inclusion may still be curtailing the development of relevant measures to some extent, as may the lack of consensus about which life domains are essential to social inclusion (Morgan et al., 2007).

It is noticeable that the available measures have undergone fairly limited psychometric assessment. Some have not been scrutinised at all, and others have undergone testing with respect to one or two properties only. Again, this may in part be due to the fact that the measures are relatively new, so the window of opportunity for testing them is fairly restricted. The lack of clarity about the overarching construct(s) that each measure purports to assess may also have had an impact here (Priebe, 2007). Either way, further psychometric testing of all measures is required.

Our examination identified two measures that show the most potential for further testing in their current form: the APQ-6 and the SCOPE–short form. The APQ-6 has been trialled in the Australian environment. New South Wales is currently well advanced in the process of implementing it as a discretionary component of their local Mental Health Outcomes and Assessment Tool (MH-OAT) collection, and other jurisdictions have expressed interest in the possibility of the APQ-6 being used on a similar basis within their services. The limitation of the APQ-6 is that it does not measure housing, so it would require modification for the purpose under consideration here. The SCOPE–short form assesses the full gamut of social inclusion domains emphasised in the Fourth National Mental Health Plan (Australian Health Ministers, 2009), but might require modification for the Australian context and would certainly need to be tested here. As it is intended for comparison with national norms rather than for use independently as a measure of social inclusion, Australian norms would need to be established.

The current discussion of social inclusion measures acts as a starting point only in identifying a social inclusion measure for routine use. The two that have been identified are not perfect and could not be rolled out without further developmental work. The next steps in this process could involve head-to-head comparisons of the two measures. Further alternatives could be to examine sub-sections of other, longer measures that might be extracted for use for the current purpose, and/or to use some measures as self-report that have not yet been tested using this mode of administration. Such alterations to an existing measure would also require further testing to determine the psychometric properties and usability of the shorter measure.

Consideration should be given to nuances in the domains of social inclusion that the selected measure should assess. Consideration should also be given to how the data generated by a routine measure might be used. For example, if part of the assessment of performance against the indicators of the Fourth National Mental Health Plan (Australian Health Ministers, 2009) were to involve comparison of the degree of social inclusion of people with mental illness and the general population, then the APQ-6 might be given preference over the other measures on the grounds that its objective questions are designed to map directly to the Australian census. Depending on the outcomes of this evaluation process, one or more of the measures might be presented to stakeholders for consultation, perhaps in a modified form.

Limitations

This discussion of social inclusion measures is based only on published literature outlining the development and/or testing of a social inclusion measure. It was beyond the scope of this review to contact directly all corresponding authors responsible for development of the measures to determine if they have any further published work relating to that measure. This might be an important additional step in any future review.

This discussion also only considers the named measures in their existing format and with their recommended mode of administration. Some measures might be suitable for adaptation for the current purpose by using self-report in place of interviews and by selecting the most relevant components of longer measures. While these alterations would then require further testing, it might be an efficient way of developing a suitable standard measure for national use.

This is an initial scope of the current literature relating to social inclusion measures published in the literature and not a systematic review. For this reason, the review was largely conducted by a single researcher and there was no meta-analysis or critical review of the testing procedures used to establish the measures’ credentials. Any decisions regarding use of these measures resulting from this review should keep these limitations in mind.

Conclusion

Social inclusion is too important not to measure properly. As an initial scoping of available candidate measures, this discussion provides a springboard for selecting an appropriate measure for use in public sector mental health services. In their current format, the findings suggests two primary candidates, but neither of these is quite fit-for-purpose in its current form. Further exploration will reveal whether one of these is suitable, whether other measures might be adapted for the current purpose or whether a new, specifically designed measure needs to be developed.

Footnotes

Funding

This work was funded by the Australian Government Department of Health and Ageing.

Declaration of interest

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.

References

Anthony

(1993) Recovery from mental illness: The guiding vision of the mental health system in the 1990s. Psychosocial Rehabilitation Journal 16: 11–23.

Australian Health Ministers (2009) Fourth National Mental Health Plan: An agenda for collaborative government action in mental health 2009–2014. Canberra: Commonwealth of Australia.

Australian Mental Health Outcomes and Classification Network (2005) Reporting Framework for the National Outcomes and Casemix Collection (Version 1.0). AMHOCN: Brisbane.

Bates

Repper

(2001) Social inclusion: A framework for evaluation. A Life in the Day 5: 18–23.

Berry

Rodgers

Dear

(2007) Preliminary development and validation of an Australian community participation questionnaire: Types of participation and associations with distress in a coastal community. Social Science and Medicine 64: 1719–1737.

Bertram

Stickley

(2005) Mental health nurses, promoters of inclusion or perpetuators of exclusion? Journal of Psychiatric and Mental Health Nursing 12: 387–395.

Borba

DePadilla

Druss

. (2011) A day in the life of women with a serious mental illness: A qualitative investigation. Women’s Health Issues 21: 286–292.

Bowling

(2005) Mode of questionnaire administration can have serious effects on data quality. Journal of Public Health 27: 281–291.

Centre for Population Studies in Epidemiology (2002) The Kessler Psychological Distress Scale (K-10). Adelaide: Department of Human Services.

10.

Cohen

(1960) A coefficent of agreement for nominal scales. Education and Psychological Measurement 20: 37–46.

11.

Core System Group (1998) CORE System (Information Management) Handbook. Leeds: Core System Group, University of Leeds.

12.

Coste

Guillemin

Pouchot

. (1997) Methodological approaches to shortening composite measurement scales. Journal of Clinical Epidemiology 50: 247–252.

13.

Department of Health (2001) Safety First: Five-year report of the national confidential inquiry into suicide and homicide by people with mental illness. London: Department of Health.

14.

Dorer

Harries

Marston

(2009) Measuring social inclusion: A staff survey of mental health service users’ participation in community occupations. British Journal of Occupational Therapy 72: 520–530.

15.

Eisen

Grob

Klein

(1986) BASIS: The development of a self-report measure for psychiatric inpatient evaluation. Psychiatric Hospital 17: 165–171.

16.

Greenhalgh

Long

Brettle

. (1998) Reviewing and selecting outcome measures for use in routine practice. Journal of Evaluation in Clinical Practice 4: 339–350.

17.

Hacking

Bates

(2008) The Inclusion Web: A tool for person-centred planning and service evaluation. Mental Health Review Journal 13: 4–15.

18.

Harris

Gladman

Hennessy

. (2011a) Reliability and validity of a measure of role functioning among people with psychiatric disabilities. Australian Occupational Therapy Journal 58: 203–208.

19.

Harris

Gladman

Hennessy

. (2011b) The reliability of an education-related self-efficacy scale for people with psychiatric disabilities. Community Mental Health Journal 47: 136–142.

20.

Harrison

Sellers

(2008) Occupation for mental health and social inclusion. British Journal of Occupational Therapy 71: 216–219.

21.

Harvey

Evert

Herrman

. (2002) Disability, homelessness and social relationships among people living with psychosis in Australia. National Survey of Mental Health and Wellbeing Bulletin 5. Canberra: Commonwealth Department of Health and Ageing.

22.

Hooley

(2010) Social factors in schizophrenia. Current Directions in Psychological Science 19: 238–242.

23.

HM Government (2010). State of the nation report: poverty, worklessness and welfare dependency in the UK. London: Cabinet Office. Available at: http://mhfe.org.uk/sites/default/files/shared/State%20of%20the%20Nation%20May%202010.pdf

24.

Huxley

Evans

Madge

. (2006) Development of a ‘social inclusion index’ to capture subjective and objective domains (Phase 1). Birmingham, UK: National Co-ordinating Centre for Research and Methodology.

25.

Huxley

Evans

Madge

. (2012) Development of a social inclusion index to capture subjective and objective life domains (Phase II): Psychometric Development Study. Health Technology Assessment 16: 1–248.

26.

Kessler

Andrews

Colpe

. (2002) Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychological Medicine 32: 959–976.

27.

Landis

Koch

(1977) The measurement of observer agreement for categorical data. Biometrics 33: 159–174.

28.

Le Boutillier

Croucher

(2010) Social inclusion and mental health. British Journal of Occupational Therapy 73: 136–139.

29.

Lloyd

Waghorn

Best

. (2008) Reliability of a composite measure of social inclusion for people with psychiatric disabilities. Australian Occupational Therapy Journal 55: 47–56.

30.

Marino-Francis

Worrall-Davies

(2010) Development and validation of a social inclusion questionnaire to evaluate the impact of attending a modernised mental health day service. Mental Health Review Journal 15: 37–47.

31.

Mental Health Commission (2009) Mental health and social inclusion concepts and measurements. Wellington, New Zealand: Mental Health Commission.

32.

Morgan

Burns

Fitzpatrick

. (2007) Social exclusion and mental health: Conceptual and methodological review. British Journal of Psychiatry 191: 477–483.

33.

Pirkis

Callaly

(2010) Mental health outcome measurement in Australia. In: Trauer

(ed) Outcome Measurement in Mental Health: Theory and Practice. Cambridge: Cambridge University Press, pp. 15–25.

34.

Priebe

(2007) Social outcomes in schizophrenia. British Journal of Psychiatry 191(Suppl 50): s15–s20.

35.

Ramon

Griffiths

Nieminen

. (2009) Towards social inclusion through lifelong learning in mental health: Analysis of change in the lives of the EMILIA Project service users. International Journal of Social Psychiatry 57: 211–223.

36.

Sayce

(2001) Social exclusion and mental health. The Psychiatrist 25: 121–123.

37.

Schafer

(2000) Empowerment: Towards a participatory model for the evaluation of the empowering therapeutic environment. Mental Health Care 3: 233–237.

38.

Secker

Hacking

Kent

. (2009) Development of a measure of social inclusion for arts and mental health project participants. Journal of Mental Health 18: 65–72.

39.

Slade

(2009) Personal Recovery and Mental Illness: A Guide for Mental Health Professionals. London: Cambridge University Press.

40.

Social Exclusion Unit (2004) Mental health and social exclusion. Social Exclusion Unit Report. London: Office of the Deputy Prime Minister.

41.

Social Inclusion Unit (2011) What is social inclusion? Canberra: Department of the Prime Minister and Cabinet.

42.

Stewart

Sara

Harris

. (2010) A brief measure of vocational activity and community participation: Development and reliability of the Activity and Participation Questionnaire. Australian and New Zealand Journal of Psychiatry 44: 258–266.

43.

Stickley

Shaw

(2006) Evaluating social inclusion. Mental Health Practice 9: 14–20.

44.

Streiner

(2003) Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment 80: 99–103.

45.

Veit

Ware

(1983) The structure of psychological distress and well-being in general populations. Journal of Consulting and Clinical Psychology 51: 730–742.

46.

Waghorn

Chant

King

(2005) Work-related self-efficacy among community residents with psychiatric disabilities. Psychiatric Rehabilitation Journal 29: 105–113.

47.

Waghorn

Chant

King

(2007) Classifying socially-valued role functioning among community residents with psychiatric disorders. American Journal of Psychiatric Rehabilitation 10: 185–221.

48.

Webber

Huxley

(2007) Measuring access to social capital: The validity and reliability of the Resource Generator-UK and its association with common mental disorder. Social Science and Medicine 65: 481–492.

49.

Wing

Lelliott

Beevor

(2000) Progress on HoNOS. British Journal of Psychiatry 176: 392–393.