Abstract
Problematic substance use is the most common comorbid complication among people with a psychotic disorder [1, 2]. In 1995 in Central Sydney Area Health Service (CSAHS), substance dependence occurred in 58% of patients with schizophrenia aged 18–29 years [3]. Subsequent consultations with service providers identified a paucity of services for this patient group. In response to these findings CSAHS established the Psychosis and Substance Use Project in July 1997.
The Project's main aim was the development of an Area-wide strategy to improve services for patients with both psychotic illness and problematic substance use. It was funded for 2 years initially by the Mental Health Services and the Drug and Alcohol Services and was coordinated by a Project Officer (NS). The Project had five objectives:
1. To examine current services to identify ways of improving the identification and management of this patient group.
2. To develop a clinical services plan for the effective management of this patient group.
3. To develop an educational programme for staff to ensure a best-practice approach to providing effective care to this patient group.
4. To develop and maintain clinical and management information systems to support the clinical services plan.
5. To develop and implement a research agenda to improve understanding of comorbidity issues within CSAHS.
Before the first 2 years of the Project were complete, funding was approved for an additional 3 months for the Project Officer to conduct a formative evaluation. This paper describes the multifaceted approach adopted for this evaluation and summarizes the results. Our dual aims are (i) to encourage other health professionals to use similar evaluation methods; and (ii) to stimulate interest in improving services for people who have a psychotic disorder and problematic substance use.
Values underlying the evaluation
Broadly, evaluation is ‘any effort to increase human effectiveness through systematic data-based inquiry’ [4], p.11]. It can be formative (conducted during a project to refine it to make achievement of the goals more likely) or summative (conducted at the project's end to determine if the goals have been achieved).
Guba and Lincoln recommend that the issues included in an evaluation must be determined by the stakeholders, whom they define as groups put at risk by the evaluation [5]. Owen concurs, but warns that experience suggests that a given evaluation serves the needs of only one or two stakeholder groups [6]. It is these groups who must be identified and consulted when determining the design of an evaluation.
Hawe et al. describe evaluation as involving two processes: (i) observation and measurement; and (ii) comparison of what is observed with some standard considered an indicator of good performance [7]. As all standards are based on personal values, and evaluation is inevitably judgemental, Hawe et al. recommend that the process and values by which a programme is judged be clearly stated. Although this does not eliminate the values, it does permit their scrutiny.
The values which underpinned this formative evaluation were an emphasis on the Project participants' views about the progress of the Project, rather than an examination of client outcomes; involvement of the Project participants in the design of the evaluation; acceptance that the advantages of internal evaluation exceeded the disadvantages; use of external standards where possible; and the importance of making the results freely available.
Method
Two main stakeholder groups were identified as being primarily served by the evaluation: the Project Management Committee and the Project Special Interest Group (SIG). The Project Management Committee comprised nine senior managers and clinicians from both Mental Health Services and Drug and Alcohol Services and was accountable for ensuring the Project achieved its objectives. The SIG served as the think-tank of the Project and aimed to provide a forum where the views and needs of stakeholders could be voiced. Members included staff of the CSAHS Mental Health Services and Drug and Alcohol Services, the New South Wales Health Department and the non-government sector, two consumers with comorbid conditions (who were paid to attend) and the consumer coordinator for the Area Mental Health Service. The SIG met every 3 months on six occasions, with 16–34 (average 26) members attending each meeting.
The Project Officer negotiated the evaluation's design, key issues, timeline and budget with the Management Committee. Members of the SIG participated in a meeting to discuss the future of the Project before the evaluation began and issues raised were incorporated into the Project evaluation.
The Project Officer primarily conducted the evaluation. According to Owen, the validity of an evaluation focused on programme improvement relies on the availability of internal expertise [6]. Schalock identifies a number of advantages of internal evaluation: it reduces evaluation costs; focuses on self-monitoring and self-improvement; and attempts to be comprehensive, correct, and credible to all stakeholders [8]. However, internal evaluators need to be aware that their objectivity can be compromised and that such an evaluation can become a public relations tool of the organization rather than encouraging programme improvement [9].
The evaluation, which was approved by the CSAHS Ethics Committee, used four complementary research methods: (i) description and interpretation of the Project's documented processes and outcomes; (ii) a benchmark comparison of the Project processes and outcomes against the Australian National Standards for Mental Health Services; (iii) a survey of the Project's key stakeholders; and (iv) interviews with 12 purposefully sampled key informants. Triangulation, the combination of different research methods within one study, helps to reduce systematic bias in qualitative research. Patton describes four types of triangulation: methods, data sources, analysts, and theories [4]. Triangulation of methods and data sources were extensively used in this evaluation but resources did not permit triangulation of analysts and theories.
Description and interpretation
Evidence concerning outcome (the achievement, or not, of the aim and each of the five Project objectives) and process (activities designed to deliver those achievements) was sought in all the records and documents of the Project. These included bimonthly progress reports, minutes from meetings, published articles from journals and magazines, reports of quality improvement activities and Project correspondence. Identification was followed by interpretation during which the Project's records were explored in greater detail, that is, questions were asked regarding why outcomes were or were not achieved, the values underlying the processes were sought [10] and the implications of the processes and outcomes for future initiatives were identified.
Benchmark comparison
The National Standards for Mental Health Services were developed in 1996 as part of the National Mental Health Strategy [11]. The Standards provide a guide to good service delivery and quality improvement and were considered a useful benchmark for the Project. Three of the 11 National Standards were selected as benchmarks because they best illustrated the three key themes of the second National Mental Health Plan (1998–2003) [12]. Standard three focuses on consumer and carer participation and illustrates the theme of partnership development; standard six focuses on illness prevention and mental health promotion and illustrates the theme of prevention and promotion; standard nine focuses on service development (service evaluation, outcome measurement, research and quality improvement) and illustrates the theme of quality and effectiveness of service delivery. Locating the evaluation within the context of the National Plan ensured that the findings would be relevant to national policies.
Seventeen relevant criteria from the three standards were used to (i) rate the extent to which the Project's processes and outcomes attained each standard; and (ii) identify areas requiring improvement.
Special Interest Group survey
A 15-item questionnaire was developed to assess whether the SIG had allowed members to present their views satisfactorily and whether members wished the SIG to continue into the Project's second stage. The questionnaire was posted with a stamped, addressed envelope and a covering letter to the 50 members of the Group who had attended at least one meeting. To ensure anonymity, returned questionnaires could not be linked to individuals. Consequently, a reminder was not sent to non-respondents and respondents were not compared with non-respondents.
Questionnaire content included age, sex, professional designation and place of work and whether the respondent worked primarily in the mental health or drug and alcohol field. Respondents were also asked, with closed questions, how frequently they had attended the Group, how informative and useful they had found it (e.g. very informative, informative, slightly informative, not informative) and whether the Group should continue. Five open-ended questions allowed respondents to describe their experiences of the Group and whether it had enabled all stakeholders to voice their opinions. A final question invited other relevant comments.
Descriptive statistics only were computed for the quantitative questions due to the small sample size. Thematic content analysis was used to identify the general themes in respondents' answers to the openended questions. Content analysis involves generating as many categories as necessary to describe all aspects of the answers and then collapsing similar categories under higher-order headings [13].
Key informant interviews
We purposely selected key informants for the interviews using criterion and maximum variation sampling [4]. Criterion sampling involves choosing all cases that meet some criterion; in this study attendance at two or more Management Committee or SIG meetings. This yielded a sample of 35 individuals. As resources did not allow us to conduct 35 interviews, maximum variation sampling of the criterion sample was then used to yield a sample of 12. Maximum variation sampling maximizes heterogeneity in small sample sizes [4]. The informants were chosen by first identifying the shared and diverse characteristics of the 35 individuals in the criterion sample (e.g. professional designation, work setting) and then ensuring that each of these characteristics was represented in the smaller sample. The Project Officer's knowledge of individuals' involvement with the Project allowed the selection of participants with different experiences. This process ensured that the relatively small sample reflected the diversity of the individuals involved in the Project.
Each individual was invited to participate in an audiotaped interview about the processes and outcomes of the Project. The interview was semistructured and covered the following topics in depth: Project achievements and flaws; Project committee structures; staff perceptions of the Project; directions for the second stage of the Project; and other issues raised by the interviewee. The topics served to guide the interview process rather than to dictate it. Questions were primarily open-ended and ambiguous statements were clarified. Each audiotape recording was transcribed verbatim by the Project Officer. Thematic content analysis, as above, was used to analyse the transcripts [14].
Results
Description and interpretation
The documentation showed that the Project achieved its aim of developing a strategy to improve services for patients with both psychotic illness and problematic substance use. After an extensive literature review and participation in a Cochrane review [15], the Project Officer developed an evidence-based strategy for service delivery. This strategy was incorporated into the overarching strategic plan of the Mental Health Service and will serve as a foundation for planning systems of care.
Additionally, the records demonstrated that the Project achieved three of its five objectives: (i) the completed examination of current services; (ii) the development of a clinical services plan; and (iii) the development of an educational programme for staff. Objective four, the development and maintenance of clinical and management information systems, was only partially achieved because, during the Project, planning began for the replacement of the entire mental health information system. However, the Project's appraisal of the current system as it pertained to comorbidity was used in the development of the new system. Objective five, the development and implementation of a research agenda, was also partially achieved. No such agenda was developed, but the literature reviews and other work conducted will inform the development of a research agenda during the Project's second phase.
The records also demonstrated that the Project delivered several other important initiatives: (i) establishment of a Special Interest Group of key stakeholders; (ii) development, delivery and evaluation of a mandatory 1-day education and training programme regarding comorbidity for clinical mental health staff; (iii) completion of an extensive literature review regarding best-practice in comorbidity; (iv) a needs analysis, via a staff survey and patient focus groups, which identified changes required in current service delivery; (v) establishment and evaluation of a community-based support group for patients with comorbid issues; and (vi) development and strengthening of partnerships with the non-government sector and the New South Wales Health Department. In addition to these local achievements, the emphasis on evidence-based practice and the publications [2, 16] and conference presentations placed the Project in the forefront of Australian initiatives in this area.
Other benefits from the Project were also apparent from the records. For example, the relationships developed with the official consumer bodies operating within the Area were valuable for promoting organizational changes within the Mental Health and Drug and Alcohol Services, and provided a strong base for other collaborations.
Benchmark comparison
Benchmarking the Project processes and outcomes against the National Standards for Mental Health Services found that the Project had either attained or initiated the attainment of the 17 relevant criteria. However, there were three areas where improvements were necessary to achieve the criteria: greater involvement of carers and consumers in the Project; initiation of health promotion regarding comorbidity in the wider community; and use of baseline data regarding consumer outcomes (measured quantitatively and qualitatively) to monitor service delivery.
Special Interest Group survey
Thirty-three people (66%) responded to the SIG survey; 82% worked for government organizations and 18% for non-government organizations. Fifty-five per cent worked in the community setting, 33% in a hospital and 12% defined their work setting as ‘other’. Registered nurses represented 37% of respondents; allied health staff, 24%; medical staff, 15%; and the remaining 24% included patients, policy officers and counsellors. Two-thirds of the respondents worked in the mental health field and 31% in the drug and alcohol field. Sixty per cent of respondents had attended at least three SIG meetings.
Eighty-eight per cent rated the SIG ‘informative’ or ‘very informative’ and 12% rated it ‘slightly informative’. Eighty-one per cent rated the SIG ‘useful’ or ‘very useful’, 19% rating it ‘slightly useful’. Seventy per cent believed that the SIG was ‘effective’ in eliciting the views and needs of stakeholders, 30% believing it was ‘somewhat effective’. Sixty-six per cent believed the group should continue.
Thematic content analysis of the open-ended questions showed that a regular meeting of mental health and drug and alcohol stakeholders from the government and non-government sectors is both achievable and useful in engendering enthusiasm among participants. Good organization and structured chairing were widely recognized as major contributing factors to the success of the SIG. There was strong support for the SIG to continue, but several improvements were suggested: (i) regular use of small group discussions during meetings to facilitate participation by all stakeholders; (ii) managers' being more supportive of attendance by those groups that were poorly represented; and (iii) creation of mechanisms for members to provide feedback directly to the Project Management Committee.
Key informant interviews
Of the 12 key informants, 11 worked in a government and one in a non-government setting. Eight informants were currently employed within the mental health sector, three in the drug and alcohol sector and one worked across both sectors. Primary roles included clinician (medical, nursing and allied health), consumer, educator, policy developer and manager. Because the informants were members of the Project Management Committee or SIG, seven held (at least part-time) managerial positions. All but two of the informants were experienced in mental health work, with five having over 10 years' experience. Half of the informants had drug and alcohol work experience, with two having over 10 years' experience.
According to informants, the Project had many achievements. The SIG, the mandatory 1-day staff education and training programme, and the establishment of consumer support groups were the most frequently nominated achievements. However, the perceived lack of representation by medical staff within the SIG, its large size, and its apparent lack of autonomy were also noted by informants.
Although the education programme was nominated as an achievement, informants identified significant flaws within it. Most believed that a 1-day workshop was not sufficient time to address comorbidity adequately. The mandatory nature of the programme was also identified as requiring review. Four informants thought that shifts in individual and organizational attitudes and practices would be better achieved by targeting education to those already interested in comorbidity, rather than by compulsory training for all staff. Informants who identified flaws within the programme all recommended that these be addressed in the Project's second phase.
The establishment of consumer support groups was widely seen as an important and effective outcome of the Project and five informants recommended that the objectives of the second phase include the establishment of regular support groups in all wards and units. However, difficulties in ensuring the continuity of such groups were identified and included low staff numbers and the reluctance of staff to integrate support groups into current programmes.
The majority of informants thought that staff perceived the Project positively, with only three informants perceiving the opposite. However, most believed the Project had to prove its value before staff regarded it positively. Contributing factors to this perception were the longevity and momentum of the Project and the delivery of the education and training workshop.
Four informants identified as a flaw a perceived lack of active support for the Project by the senior management of Mental Health Services. All four believed that this had not significantly affected the Project, attributing this to the tenacity of the Project's processes. Further, three informants voiced their distrust of the Project Management Committee, and management in general, believing management to be secretive and lacking in transparency. (To protect confidentiality, details of these informants are not given.)
All informants expressed enthusiasm for a second phase of the Project; 10 recommended that the Project adopt a clinically oriented approach. Other frequently nominated objectives for the second phase were continuation of the education and training programme, development of a formal working policy between the Mental Health Services and Drug and Alcohol Services, establishment of regular consumer support groups, and entrenchment of comprehensive drug and alcohol assessments in the regular assessment of mental health consumers.
Discussion
The formative evaluation demonstrated that during its first 2 years the Project achieved considerable successes in implementing service developments. This confirms the value of the approach to quality improvement adopted by the Project; an approach based on staff and consumer involvement and similar in many ways to action research [17, 18]. The evaluation was conducted after funding for the Project had been approved for an additional 2 years. Thus, in the absence of pecuniary, managerial or staff pressure to portray the Project in a particular light, the authors were able to conduct their research in a genuine spirit of enquiry.
The Project's aim was to develop a strategy for future systems of care that would lead to improved consumer outcomes. During the formative evaluation we critically examined the process of developing that strategy. However, an improvement in consumer outcomes will be apparent only after the strategy has been implemented over the next 2 years. A summative evaluation of the Project, with consumer outcomes as the main focus, will then be appropriate.
The validity and credibility of a predominantly qualitative study are dependent on rigorous methods for gathering and analysing high quality data [4]. Triangulation of methods and data sources allowed us to check the consistency of information derived by different means. Documenting not only the consistency, but also the diversity, in the data contributes to the credibility of the study. Credibility would have been further improved by analyst and theory triangulation if additional resources had been available. Benchmarking the Project against the nationally agreed Mental Health Standards added to the study's credibility.
Bias may have been introduced by the sampling methods as all participants in the evaluation were either on the Management Committee or the SIG. However, as members of the SIG were key stakeholders in comorbidity rather than the Project itself, they were not necessarily positively predisposed towards the Project. During the purposeful sampling of key informants particular attention was paid to selecting participants with different views about the Project. Although criticisms are made of the small sample sizes employed by qualitative researchers, when the individual subjects are information-rich the insights obtained can be very valuable [4]. This is well illustrated by the detailed information derived from our interviews. Although our interview findings are specific to this project, we suggest that the rigour that can be applied to in-depth interviewing makes it an effective evaluation tool for other programmes.
All four evaluation methods confirm that staff education, consumer groups and key-stakeholder participation were the main strengths of the Project. During the second phase of the Project attention must be given to sustaining the current, and developing more advanced, staff education programmes. The opinions of key informants that education will be more effective if targeted to those who are already interested in comorbid issues contradicts some current evidence [19]. Providing basic education to all staff and advanced training to committed staff may be an appropriate response. However, an increase in knowledge does not necessarily lead to a change in practice [20]. We were encouraged that key informants felt that drug and alcohol assessments should become a routine part of mental health assessments, as assessment is the first and crucial step in the management of comorbidity.
The sustainability of consumer support groups relies on the enthusiasm of group facilitators and the provision of environments conducive to consumer participation. This can be difficult to achieve and it is imperative that managers are supportive of such initiatives and provide adequate staff time and resources. Experience from this project indicates that sharing the responsibility for such groups among clinicians from both the Mental Health and Drug and Alcohol Services promoted cooperation at a microlevel and had a positive impact on both organizations overall.
Most participants in the evaluation reported that the processes used to involve key stakeholders throughout the Project had been effective overall and helped them individually to reflect on the issues, their own practice and their involvement in the Project. Many believed that the evaluation itself kept comorbidity ‘on the agenda’.
Throughout the evaluation the Project Officer consulted the key stakeholders, including negotiating the final recommendations. In a responsive evaluation the evaluator and the stakeholders share responsibility so that, without compromising the evidence, the final recommendations are expected, even anticipated, by the main audience [21]. The final report of this evaluation, including its recommendations, was welcomed by management and was widely disseminated to all participants and staff. We believe this confirms that the evaluation was responsive.
In conclusion, formative evaluation demonstrated that the CSAHS Psychosis and Substance Use Project achieved its aim and most objectives and that the processes adopted during the Project were effective. With some improvements, the Project should be continued to implement the strategy developed. We believe that the project should be evaluated again in 2 years when improved consumer outcomes will be a key performance indicator. We encourage other services to use formative evaluation, qualitative research methods and benchmarking in the evaluation of health services.
Footnotes
Acknowledgements
The authors thank all those who participated in the Project and the evaluation; Marie Bashir, Victor Storm and Danny O'Connor for their support and advice; and Garth Alperstein and Lynne Madden for comments on an earlier draft. A special thanks to Michelle Cleary for her invaluable advice and support.