Assessing Cost-Effectiveness in Mental Health: Vocational Rehabilitation for Schizophrenia and Related Conditions

Abstract

Objective: Existing evidence suggests that vocational rehabilitation services, in particular individual placement and support (IPS), are effective in assisting people with schizophrenia and related conditions gain open employment. Despite this, such services are not available to all unemployed people with schizophrenia who wish to work. Existing evidence suggests that while IPS confers no clinical advantages over routine care, it does improve the proportion of people returning to employment. The objective of the current study is to investigate the net benefit of introducing IPS services into current mental health services in Australia.

Method: The net benefit of IPS is assessed from a health sector perspective using cost–benefit analysis. A two-stage approach is taken to the assessment of benefit. The first stage involves a quantitative analysis of the net benefit, defined as the benefits of IPS (comprising transfer payments averted, income tax accrued and individual income earned) minus the costs. The second stage involves application of ‘second-filter’ criteria (including equity, strength of evidence, feasibility and acceptability to stakeholders) to results. The robustness of results is tested using the multivariate probabilistic sensitivity analysis.

Results: The costs of IPS are $A10.3M (95% uncertainty interval $A7.4M–$A13.6M), the benefits are $A4.7M ($A3.1M–$A6.5M), resulting in a negative net benefit of $A5.6M ($A8.4M–$A3.4M).

Conclusions: The current analysis suggests that IPS costs are greater than themonetary benefits. However, the evidence-base of the current analysis is weak. Structural conditions surrounding welfare payments in Australia create disincentives to full-time employment for people with disabilities.

Employment rates among people with psychotic disorders are low. According to the Low Prevalence Disorder Study (LPDS), 28% of people with such disorders are employed. Conversely, over 80% are on government pensions or some form of social benefits [1].

Many approaches have been implemented to assist people with psychotic disorders to gain employment including skills training, vocational counselling, job club, sheltered workshops, workcrews, transitional employment and supported employment (SE). These are provided by various organizations such as psychosocial rehabilitation centres, community mental health centres (CMHC), vocational rehabilitation agencies or employment service agencies [2–4]. A recent Cochrane review [5] divided vocational rehabilitation approaches into two categories: prevocational training (PVT) and SE. Prevocational training assumes that people require a period of preparation before entering competitive employment including sheltered workshops, transitional employment, work crews, skills training and other preparatory activities. In contrast, SE approaches place clients in competitive employment without extended preparation and provide on the job support from ‘trained job coaches’ or employment specialists [6]. Supported employment contains three elements: competitive work, an integrated work setting and the provision of an ongoing support service [7]. Models of SE which have been developed and trialed include the Integrated Service Agency (ISA) model [8], Family-aided Assertive Community Treatment (FACT) [9] and individual placement and support (IPS) [10].

With respect to the effectiveness of these different models of vocational rehabilitation, the Cochrane review [5] found that there is no evidence to suggest that PVT is more effective than standard community care or hospital care in increasing competitive employment rates. There is evidence, however, that SE is superior to PVT in increasing competitive employment rates, but there appears to be no difference between SE and PVT in non-vocational outcomes, such as symptomatology, quality of life or social functioning.

A subanalysis of SE found that IPS was the most effective form of SE in helping people with severe mental illness gain competitive employment [5]. In addition, two recent reviews [11], [12] state that IPS is the most comprehensively documented approach. For these reasons, the current study focuses on the IPS model of SE.

This study forms part of a larger project, the Assessing Cost-Effectiveness in Mental Health (ACE-MH) study, which evaluated the cost-effectiveness of a number of interventions in mental health based on a standardized method [13]. The purpose of ACE-MH is to use existing evidence to determine the cost-effectiveness of various mental health interventions within an Australian context. For the measurement of outcomes, this involves a two-stage approach. The first stage entails translation of outcome information into a common metric of health benefit, the disability-adjusted life-year (DALY), while the second stage involves applying ‘second-filter criteria’ to the quantitative analysis. The filters include issues important to resource allocation decisions, but less amenable to quantitative analysis such as levels of evidence, equity, feasibility and acceptability. On the cost side, resource usage is determined using uniform assumptions tailored to Australian conditions. The key research question of the current study within the ACE-MH framework is: ‘What is the incremental cost-effectiveness of routinely introducing the IPS model of vocational rehabilitation to people with psychotic disorders who wish to work and are not accessing available vocational rehabilitation services from the economic perspective of the government and the client/family?’

Method

A comprehensive economic evaluation protocol specifically written for ACE-MH provided the basis for all evaluations [13], [14]. However, the current analysis required some digression from this protocol because of available studies failing to show significant clinical benefits of the intervention.

The intervention

Individual placement and support draws from components and philosophies of several other models; most heavily from the Program of Assertive Community Treatment (PACT) by Stein and Test, and the SE model developed by Wehman et al. [cited in [15]. The core IPS principles include:

Competitive employment as the goal;

Rapid job search;

Integration of rehabilitation and mental health;

Attention to consumer preferences;

Continuous and comprehensive assessment;

Time-unlimited support.

Individual placement and support comprises a vocational coordinator (usually with a postgraduate degree in rehabilitation) and two or more employment specialists (of varied backgrounds). Employment specialists work directly with clients in all phases, carry caseloads of 20–25 clients and are ideally integrated within the CMHC [10]. Direct contact time with clients is 62±37.1 hours [5], [16]. The main activities undertaken by the employment specialists include organization, referral, engagement, assessment, obtaining employment and job support.

This current analysis models IPS as an additional vocational rehabilitation service complementing current public mental health services. The time frame of the study is 1 year, in terms of tracking costs and outcomes, because there is little information describing outcomes beyond this period.

Current practice

Vocational rehabilitation services in Australia are mostly provided by the Commonwealth government via the Commonwealth Rehabilitation Service, known as ‘CRS Australia’ [17]. In contrast, the majority of employment services for people with psychiatric disabilities are provided by non-government agencies or non-government auspiced organizations, mostly funded by the Commonwealth government under the Disability Service Act 1986 and the Commonwealth/State Disability Agreement Act [18]. Such services tend to be termed ‘open employment services’. Most employment and vocational rehabilitation services for people with psychiatric disabilities in Australia are not integrated within routine community mental health services. They are usually organizationally separate and may liaise with community mental health services. In addition, most agencies provide services to people with different types of disabilities, not only psychiatric disabilities.

Unfortunately, information specific to people with psychotic disorders is limited because the term psychiatric disability includes many psychiatric conditions. We use information from the LPDS [1] to estimate the numbers of unemployed people with psychotic disorders who have received help from existing Australian vocational rehabilitation services. According to the LPDS, 37% of people with psychotic disorders aged less than 65 years who are unemployed received help from current practice. The remaining respondents who state that they are willing to work and have not received employment assistance (defined in the LPDS as received help from the Commonwealth Employment Service or received help toward improving ability to work) are considered eligible for the IPS program.

Estimation of population of people eligible to receive the intervention

The prevalent cases of schizophrenia and related conditions for the year 2000 are defined using operational criteria (OPCRIT) ICD-10 specific codes (11–17,19,20) from the LPDS. For ease of reference we refer to all these diagnostic categories as ‘schizophrenia’ but acknowledge that other diagnoses (such as schizoaffective disorder) are encompassed by this term.

We conservatively assume that only 70% of those stating they want to work in the LPDS and have not used current services will use the intervention. This proportion comes from a survey in the US cited in the Cochrane review [5]. We also assume 76% of the eligible population will actually use the IPS intervention (this proportion is based on the participation rate from Drake et al. [19].

Assessment of benefit

In contrast to the other ACE-MH interventions, a health benefit for IPS cannot be modelled. Crowther et al. [5] did not detect any significant difference between IPS group interventions and control groups on any clinical outcome measures, including self-esteem, mental state, overall functioning or quality of life at any time point [19], [20]. Unfortunately, a recent randomized study of IPS, not included in the Cochrane review, did not report clinical outcomes [21].

The available literature suggests that the main effect of IPS is vocational, defined as proportion of people returning to work. Crowther et al. [5] state that at 12 months, 31% of IPS participants are employed in contrast to 12% of control group subjects, resulting in a net gain of 19% for IPS.

Figure 1 represents an analytical pathway of the model constructed for the assessment of IPS based on all the proportions discussed so far.

Figure 1.

Analytical pathway.

The main economic benefits associated with improved employment rates include:

the change in government transfer/welfare payments (defined as sickness benefits, unemployment benefits etc.;

gains in tax revenue to the government; and

increased income to the individual.

Traditionally, transfer payments are not included in economic evaluations conducted from a societal or health sector perspective, because they reflect only a transfer of existing resources from one party to another, rather than creation or consumption of resources [22–24]. However, these payments are relevant when a government perspective to an evaluation is adopted, because the government bears the cost of these payments (though in reality the tax-payer bears the actual cost) and transfer payment savings may be used for other public programs or interventions (i.e. there is an opportunity cost to these transfer payments) [23], [25].

Quantification of economic benefits

Income accruing to the individual

The income threshold before welfare payments are affected (income-free area) is $2652 per year for a single person aged over 21 years without children (for the year 2000, which is the reference year for all ACE-MH analyses). This means that for every dollar earned more than this amount the benefit paid is reduced by 50 cents [26]. Therefore, the total net income earned by the individual is the income earned from employment minus any welfare payments received (taking into consideration reduced benefits).

There is considerable heterogeneity among trial participants in type of employment, total hours worked and income earned [19], [21]. We use Australian information on work patterns among people with psychiatric disability who receive assistance from open employment services [27] (Table 1). The inherent assumption is that IPS and open employment services achieve similar work-pattern outcomes.

Table 1.

Proportion of employed participants in each work category

No. of weeks worked		No. of hours worked per week			Total
	1–9 hours	10–19 hours	20–29 hours	30–39 hours
1–13	11%	12%	9%	9%	41%
14–26	8%	9%	6%	7%	30%
27–39	4%	5%	4%	4%	17%
40–48	3%	4%	3%	3%	12%
Total	26%	30%	21%	23%	100%

Furthermore, we assume the average hourly wage rate among employed IPS participants in the current study is $12.20. This is the average hourly wage rate earned by people with psychiatric disabilities who receive assistance from open employment services [28]. We consider this a reasonable assumption as most employed IPS participants gain employment in open labour markets.

Transfer payments

According to the LPDS, 88% of unemployed respondents are on disability support pensions (DSP). For employed participants who receive DSP, the amount of forgone pension due to earned income higher than the pension income-free area or income test are estimated by using the DSP rate and pension income test in the year 2000 [26], [29]. The income test rate for single people aged over 21 years without children is used in this analysis (only 12% of unemployed respondents in LPDS study are married or de facto, 2% are aged under 20% and 62% do not have children).

Tax income accruing to government

Although the DSP is exempt income from taxation, the earned income from work is taxable income [30]. We assume that participants do not have other taxable income. Tax offsets or other deductions are not included in this tax estimation, because it depends on individual circumstances and is difficult to estimate. Tax rates on taxable income from the Australian Taxation Office website are used to estimate income tax [31]. The individuals' forgone welfare payments and income tax are the main benefits to the government from implementing IPS in this analysis.

Costs

The costs of introducing IPS for the government as ‘third-party payer’ are estimated using pathway analysis and standardized methods described elsewhere [13]. Other health service costs are not included as there is little evidence the IPS significantly affects such costs [32], [33]. Costs to the individual and other sectors (e.g. housing, education) are not estimated because we have no reliable evidence describing what these may be. We assume that IPS intervention is in ‘steady state’ (i.e. working at full potential and trained employment specialists are available). Set-up costs are excluded.

Table 2 summarizes the cost assumptions.

Table 2.

Summary of unit cost information and assumptions

	Point estimate	Remark
Cost of IPS to government
Daily costs of employment specialist	$211	Assuming that employment specialist has a yearly salary rate of $34 400–$50 000 (equal to CRS rehabilitation consultant levels 1 and 2) [17] + 30% on-cost
Contact per day for employment specialist	5	Based on number of contacts with client per day for CRS rehabilitation consultants (3–5) and IPS employment specialist have a caseload of 20–25 clients
Cost per contact	$42	Estimate
No. of contacts among compliant participant per year	41	Estimate from trial in New Hampshire [16] and assuming 1hours per contact
No. of contacts among non-compliant participants	8	Range (1–15) is below average number of contacts minus its standard deviation
Cost per compliant participant per year	$1737	Estimate
Cost per non-compliant participant per year	$338	Estimate

CRS, Commonwealth Rehabilitation Service; IPS, individual placement and support.

Net benefit

We undertake a type of cost–benefit analysis (CBA) as the main outcomes of interest are monetarized (i.e. transfer payment savings, extra tax accrued and net income accruing to individuals). The goal of CBA is to identify interventions where the net social benefit (the difference between costs and benefits between two or more programs) is greater than zero. Theoretically, a CBA framework has a broader scope (societal perspective) and captures all effects of an intervention to society. We acknowledge that the perspective of this study is not societal and we may be missing important effects; however, we have no information available to judge what these effects may be. The formula used to calculate net benefit is

where ΔB refers to the incremental change in benefits (measured in $) and ΔC refers to the incremental change in costs (measured in $)

We model IPS as an add-on intervention for people with schizophrenia who have not received help from current practice (as detailed earlier). The net benefit is equal to the estimated benefits of IPS (extra tax revenue, reductions in transfer payments and incremental patient income) minus the estimated costs of IPS. We also calculate the benefit to cost ratio by simply dividing the benefits by the costs.

Uncertainty analysis

The robustness of the results is tested using probabilistic sensitivity analysis with the @RISK software. This is a type of multi-attribute sensitivity analysis whereby multiple parameters of the model are allowed to simultaneously vary. This type of analysis determines an empirical distribution around the net benefit. This is accomplished by applying distributions to the specified ranges with subsequent samples randomly drawn from these ranges to simulate uncertainty [22]. A set of common methods were derived for uncertainty testing across all interventions in ACE-MH [13]. However, each intervention also includes a unique set of uncertainty variables reflecting the applicable sources of uncertainty. Table 3 details which parameters of the current model are simultaneously varied.

Table 3.

Model parameters, uncertainty range and uncertainty distribution in the @RISK modelling simulations

Model parameter	Value (uncertainty range)	Distribution	Comments
DISMOD estimate prevalence cases	Prevalence in each sex and age group (±30% variation factor)	Variation factor = Risk Triang(0.7,1,1.3)	Estimate from DISMOD and it is assumed to vary together in the same direction between ±30%
Proportion of unemployed individual have received help from current practice in each age group	Proportion in each sex and age group (±10% variation factor)	Variation factor = Risk Triang(0.9,1,1.1)	Estimate from LPDS; assuming that it will vary together in the same direction between ±10%
IPS dropout rate	23% (23–36%)	Risk Uniform(23%,36%)	Minimum value is based on systemic review of two RCTs [5], maximum value is based on exit rate among psychiatric disability client of open employment service in Australia [28]
IPS employment rate	19% (SE = 4.67%)	Risk Triang(19% − SE, 19%,19% + SE)	Based on systemic review of two RCTs [5]
Cost per contact	$42 (34–50)	Risk Triang(34,42,50)	Estimate
No. of contacts among compliant participants	41 (SE = 3)	Risk Triang(38,41,44)	Estimate based on one RCT [16]
No. of contacts among non-compliant participant	8 (1–15)	Risk Triang(1,8,15)	Estimate
No. of hours worked per week
1–9 hours	5 (1–9)	Risk Triang(1,5,9)
10–19 hours	15 (10–19)	Risk Triang(10,15,19)
20–29 hours	25 (20–29)	Risk Triang(20,25,29)
30–39 hours	35 (30–39)	Risk Triang(30,35,39)
No. of weeks worked per year
1–13 weeks	7 (1–13)	Risk Triang(1,7,13)
14–26 weeks	20 (14–26)	Risk Triang(14,20,26)
27–39 weeks	33 (27–39)	Risk Triang(27,33,39)
40–48 weeks	44 (40–48)	Risk Triang(40,44,48)

DISMOD, Disease Modelling software; IPS, individual placement and support; LPDS, Low Prevalence Disorder Study; RCTs, randomized controlled trials; SE, supported employment; Risk Triang, triangular distribution; Risk Uniform, uniform distribution.

Second-stage filters

Readers are referred to Haby et al. [13] for details on the precise methods followed in the assessment of the second-filter criteria for ACE-MH. The second stage involves the assessment of issues that influence the degree of confidence placed in the results of the study (such as the quality of the available evidence), or broader issues that need to be considered in decision-making about resource allocation (such as equity and acceptability to stakeholders). The filters chosen for ACE-MH are:

Strength of evidence: We use the National Health and Medical Research Council (NHMRC) ‘Designation of levels of evidence’ but also emphasize threats to the quality of the evidence by identifying biases and issues that affect evidence generalizability. Based on the level and quality of evidence, strength of the evidence of effectiveness/ineffectiveness is characterized as: ‘sufficient’, ‘limited’ or ‘inconclusive’ [13]. This criterion is also addressed in the uncertainty modelling. For example, if the evidence is weak, large uncertainty is placed around the size of the impact measure.

Equity: This criterion addresses the capacity of the intervention to affect inequity in the distribution of the mental disorder, as well as emphasizing issues that may affect access to, or utilization of, the intervention. The special needs groups considered include lower socioeconomic status, non-English-speaking background, Aboriginal and Torres Strait Islanders or rural/remote residence.

Feasibility: This criterion addresses issues such as the availability of an appropriate workforce with appropriate training to conduct the intervention, whether the intervention can be implemented under current institutional arrangements and ease of implementation.

Acceptability to stakeholders: This criterion refers to the acceptability of the proposed interventions to the various stakeholders affected by the intervention. Stakeholders include patients and their family/carers, clinicians, policy makers, the general community and third-party funders.

Results

According to the analytical model, of the 7800 (6000–9700) people with psychotic disorders who join IPS, 1000 (700–1400) will gain employment. The IPS program costs $10.3M ($7.4M–$13.6M) to the government (the cost is approximately $8700 per participant). In return, the government can save $1.1M ($0.75M–$1.6M) in reduced income support pensions and gain $0.22M ($0.13M–$0.32M) from income tax. The net benefit to the government is −$8.9M (−$12M to −$6.3M). For individuals, net earnings total $3.3M ($2.2M–$4.6M). Thus, the total net benefit is −$5.6M (−$8.4M to −$3.4M). The benefit to cost ratio is 0.46 (0.33–0.61).

Second-stage filter considerations are:

Strength of evidence: Although there is some level 1 evidence used, this has limited generalizability outside the US context and there is certainly insufficient evidence on clinical outcomes (Brief Psychiatric Rating Scale (BPRS), self-esteem and broad measures). Longer-term trials set in an Australian context are therefore urgently required.

Equity: No major equity considerations.

Feasibility: The implementation of IPS is feasible but difficult. The most noteworthy difficulty being availability of trained providers. More significantly, problems surrounding the organization of the welfare system and related legislation create disincentives to ongoing significant employment for people with serious mental illnesses. Legislation covering eligibility criteria to benefit payments is the largest barrier which urgently needs to be re-assessed.

Acceptability to stakeholders: Acceptability to key stakeholders may be variable, particularly to the treatment team (where issues around treatment responsibility may arise) and policy or funding bodies (who require more evidence of benefit).

Discussion

This analysis has estimated costs and benefits associated with the addition of IPS to the current provision of vocational rehabilitation and employment services for people with schizophrenia in Australia. The main conclusion drawn from the preceding analysis is that IPS does not appear to be a cost-saving intervention. The major reason why the benefit to cost ratio is not favourable is that the employment rate associated with IPS is low and people do not usually return to full-time work. As mentioned in the second-filter criteria, one of the reasons why this may be is the unfortunate structure of the Australian welfare system, which creates disincentives for people with disabilities to undertake competitive employment. The disincentives are created through the difficulties encountered by people who need to return to the welfare system, after a period of employment, because of relapse or deterioration in their health. Such people often find themselves faced with the prospect of having to repeatedly verify and prove their need and eligibility for continued welfare payments, an experience many find cumbersome and humiliating.

The current results seem to agree with other limited economic evaluations of IPS. The New Hampshire trial also included a CBA of IPS undertaken from three different perspectives (government, individual and society). This study found that after 18 months of enrolment, IPS cost $US1.62 for each dollar earned, while Group Skills Training (GST; the control condition) cost $4.91 per dollar earned [20], [34]. Even though IPS may assist people to gain employment, most do so on a casual or part-time basis and do not earn enough income to cease receiving government benefits or to pay income tax. This is consistent with the finding from the uncertainty analysis that one of the main contributors to the results is the employment rate. Interestingly, Clark et al. also found that IPS did not reduce dependence on government support although IPS clients earned more than control (GST) clients [34].

However, the present results must be interpreted with some caution for a number of reasons, the first being the quality of evidence used in the modelling. The evidence of effectiveness of the IPS program is drawn from only two trials with relatively short follow-up time periods. In addition, both trials were conducted in the US where client characteristic, the mental health care system, welfare system, rehabilitation system, economic and employment environment all differ from Australia. These factors can affect the effectiveness of programs in practice. Furthermore, these trials may have been underpowered to capture clinical benefits and the short follow-up times may have been insufficient to capture the longer-term clinical and functional benefits associated with IPS. It is therefore very possible that this analysis has underestimated the benefit of IPS.

Second, it is also possible that we have underestimated the effect of IPS by assuming a 1 year time frame for the analysis. Even though costs incurring to IPS may span longer than 1 year (if continued support is required), benefits can likewise continue to accrue (in terms of reduced welfare payments, increased taxes and increased net income). This will be particularly so if IPS has the effect of maintaining people in long-term employment. Unfortunately, given the short time frame of IPS trials, this information is currently unknown.

Third, as acknowledged earlier, this analysis does not represent a full CBA, largely because of the limited study perspective and available information. The problem in undertaking limited CBA is that important consequences (and costs) may be missed thus biasing the final results. We have not included costs outside the government perspective or consequences outside those to the government and the individual. Significant positive effects of vocational rehabilitation, such as to families or carers, have not been included. Possible cost-offsets to other sectors, such as the criminal justice system, have also not been included. Such exclusions have the effect of underestimating the social benefit of vocational rehabilitation. However, there is very little, if any, published information addressing such issues.

In conclusion, large Australian-based IPS trials within CMHC need to be undertaken, including clinical vocational and quality of life outcomes, similar to a large multicentre trial of IPS currently being conducted in Europe. EQOLISE (enhancing the quality of life and independence of people disabled by severe mental illness through SE) will test the effectiveness and cost-effectiveness of IPS compared to standard rehabilitation and vocational services in six European countries [35]. However, EQOLISE will have limited external validity to the Australian context, thus necessitating local versions of this trial.

Caveat

The ACE-MH project was jointly funded by the Australian Department of Health and Ageing, Mental Health and Suicide Prevention Branch and the Department of Human Services, Mental Health Branch, Victoria, in recognition of the importance of research into the cost-effectiveness of interventions in mental health treatment and care. This work draws upon, but is also limited by the available research and the assumptions necessary to complete the work.

The results of the analyses provide valuable material, likely to contribute to future policy deliberations by all service providers. Conclusions drawn from the economic evaluations should be considered within the context of the second-stage filter process, which qualifies the results taking into account issues of equity, feasibility, strength of evidence and acceptability to stakeholders. This second-stage filter process addresses some of the practical considerations required for changes in actual service practice.