Abstract
This study identified and prioritized resources and outcomes that should be considered in more comprehensive and scientifically rigorous health and safety economic evaluations according to healthcare sector stakeholders. A literature review and stakeholder interviews identified candidate resources and outcomes and then a Delphi panel ranked them. According to the panel, the top five resources were (a) health and safety staff time; (b) training workers; (c) program planning, promotion, and evaluation costs; (d) equipment purchases and upgrades; and (e) administration costs. The top five outcomes were (a) number of injuries, illnesses, and general sickness absences; (b) safety climate; (c) days lost due to injuries, illnesses, and general sickness absences; (d) job satisfaction and engagement; and (e) quality of care and patient safety. These findings emphasize stakeholders’ stated priorities and are useful as a benchmark for assessing the quality of health and safety economic evaluations and the comprehensiveness of these findings.
In Canada, as elsewhere, the rates of occupational injuries and illnesses and their associated lost work days in healthcare are excessive. In 2010, the lost-time claim rate was three claims per 100 person-years of employment compared with a rate of 2.3 for all sectors in British Columbia (BC), a western province in Canada. Musculoskeletal injuries comprise approximately 60% of all time-loss claims in healthcare and are some of the most costly in terms of compensation payments (Koehoorn, Demers, Hertzman, Village, & Kennedy, 2006). In fact, musculoskeletal injury rates are the second highest across all sectors in British Columbia (Ochiobi, Lanphear, & Hansen, 2013). Healthcare workers are exposed to various health and safety risks, including patient handling; slips, trips, and falls; infectious diseases; allergens; and patient-related violence (Alamgir et al., 2009; K. G. Davis & Kotowski, 2015; Ngan et al., 2010; Yassi, Gilbert, & Cvitkovich, 2005). Given the high rate of injury and illness in this sector, evidence on effective health and safety interventions is critical to prevention. Cost-benefit and cost-effectiveness analyses are equally critical, given funding constraints in healthcare (van Dongen et al., 2013).
A systematic literature review of economic analyses of health and safety programs (Tompa, Dolinschi, de Oliveira, & Irvin, 2009) found that few evaluation studies included comprehensive economic evaluations. Of those that did, the quality of analyses was generally low. In particular, studies in the healthcare sector were found to be of modest quality such that no substantive summary statements could be made about the evidence generated. An earlier assessment of health and safety program evaluations in healthcare also noted that “well-designed and conducted evaluations of program costs and benefits were nearly impossible to find” (Niven, 2002, p. 294). A key shortcoming was the lack of consideration of program impacts on a broad range of resources and outcomes affecting diverse stakeholder groups. A recent systematic review confirmed this finding to still be the case (Rothmore, Aylward, & Karnon, 2014).
Published economic analyses have focused mostly on workers’ compensation claim expenses (e.g., Alamgir et al., 2008; Chhokar et al., 2005; Collins, Wolf, Bell, & Evanoff, 2004). A recent review of lifting interventions also found the focus of economic evaluations was claim expenses (Aslam, Davis, Feldman, & Martin, 2015). Yet, fluctuations in workers’ compensation expenses are a limited proxy for the value of health and other outcomes of a program introduced to avert injuries and illnesses because they capture only a small subset of these outcomes. Outcomes such as recruitment and training expenses incurred for replacement workers, expenses associated with accommodating workers, and impacts on quality of care and patient safety are not often included in these evaluations. Quality of care and patient safety have also been linked to the health and safety of healthcare professionals (Eklof, Torner, & Pousette, 2014; Jones, Stockwell, & Lake, 2015; Stone, Mooney-Kane, Larson, & Horan, 2007; Yassi & Hancock, 2005). Essentially, fatigue, stress, pain or injury, and illness can increase healthcare error, impede the ability of a worker to provide quality care and ensure patient safety. Some studies consider these outcomes (e.g., Hofmann & Mark, 2006; Singer et al., 2003; Zohar, 2014), but they have yet to be formally and widely reported or integrated in health and safety economic evaluations.
Given the narrow framing of existing economic evaluation studies in the literature, and the fact that decision making in not-for-profit healthcare may differ significantly from the for-profit perspective, a wider framing of economic evaluations might benefit stakeholders in this sector. A starting point is to identify the resources and outcomes that are most relevant for these stakeholders. This is the task undertaken in this research study.
This study had two specific objectives:
To identify, categorize, and define a comprehensive set of resources and outcomes associated with health and safety programs based on a review of the literature and key informant interviews.
To prioritize and rank these resources and outcomes based on a Delphi panel of stakeholder representatives.
The added value of this approach is that stakeholders are involved in the process of identifying and ranking the priorities, thus increased commitment should result if future studies consider the set of resources and outcomes proposed.
Materials and Methods
This study was undertaken in British Columbia, Canada, which has a not-for-profit, publicly financed healthcare system that provides hospital, long-term, and community care to all residents living in the province. The study time period was from 2009 to 2012 and was guided by an advisory group, which included policy makers from the provincial workers’ compensation board (WorkSafeBC), healthcare managers, union representatives, and representatives from other insurance providers.
Identification of Resources and Outcomes
Economic evaluations of health and safety programs identified in a systematic review of peer-reviewed literature (Tompa et al., 2009) were consulted to develop an initial list of resources and outcomes. Also reviewed were the methodological approaches used in those studies.
Key informant interviews with healthcare stakeholders focused on two objectives:
Determine the resources and outcomes that should be included in a comprehensive economic evaluation from key informants’ perspectives and
Determine how resources and outcomes should be measured and integrated in an economic evaluation, given data availability and preferences.
The interview guide is provided in the appendix.
The advisory group that guided the study provided an initial list of possible key informants. Candidate key informants included healthcare workers, employers, insurers, and staff from the British Columbia Ministry of Health Services. The snowball method was then used to identify additional key informants, with an effort made to have more than one representative from each stakeholder group. Twenty-five interviews were completed between April and August 2010 in person or over the phone. All interviews were recorded and transcribed. Most key informants were executives of healthcare organizations and healthcare unions; five informants were front-line workers.
Interview transcripts were analyzed by three research team members to identify resources and outcomes provided by the key informants. The list was combined with resources and outcomes identified from the literature review. Using a variation of a nominal group technique (Tague, 2004), the researchers analyzed possible definitions and overlapping concepts to derive a final list of reasonably unique categories of resources and outcomes for use in the Delphi panel prioritization process that followed.
Prioritization of Resources and Outcomes
An electronic Delphi panel method, modeled after Guzman, Hayden, et al. (2007) and Guzman, Jones, et al. (2007), was used. A set of cards listing the resources and outcomes was prepared for the Delphi panel exercise. Each card contained one resource or outcome with a definition and examples.
A list of Delphi panel participants was developed with assistance from the advisory group. A purposive snowball method was used to identify more than one representative from each stakeholder group. Delphi panel participants included mid-level managers of healthcare organizations and unions, the BC Ministry of Health Services and the provincial workers’ compensation board as well as patient representatives recruited through charities and non-governmental organizations.
Delphi panel members were asked to consider resources and outcomes separately. Specifically, they were asked to establish a ranking of each according to their opinion of which resource or outcome should be considered first, second, third, and so forth, when managers or directors are considering health and safety program investments. Panel participants were also asked to provide an explanation of why they ranked the resources and outcomes as they did. Delphi panel members completed three rounds of ranking.
After each of the first two rounds, the research team collected the rankings and explanations from the panel and prepared these data for the subsequent round. Specifically, rankings were tabulated and the lowest ranked items were removed from the list, so that in the subsequent round the Delphi panel members focused on those resources and outcomes ranked as most important by the panel in the previous round. Explanations for rankings provided by panel members from the previous round were summarized and shared with all panel members so they could consider these explanations when re-ranking the resources and outcomes. In the second and third rounds, Delphi panel members reconsidered personal rankings and resubmitted them to the research team, explaining any changes they made in their earlier ranking. The Delphi panel was conducted via electronic mail. Panel members did not meet and did not exchange opinions directly.
Results
Table 1 provides an overview of the three stages of the process used to identify and prioritize resources and outcomes in this study. Below the findings from each stage are described.
Three Stages Used to Identify and Prioritize Resources and Outcomes
Literature Review
Twenty-five healthcare studies that reported on a range of programs were identified by Tompa et al. (2009) and reviewed for this component of the study. The most common were ergonomic and other work-related musculoskeletal disorders (WMSD) prevention programs (Brophy, Achimore, & Moore-Dawson, 2001; Chhokar et al., 2005; Collins et al., 2004; Engst, Chhokar, Miller, Tate, & Yassi, 2005; Guthrie et al., 2004; Li, Wolf, & Evanoff, 2004; Ore, 2003), followed by disability management programs (Bernacki & Tsai, 2003; Koviack 2004) and occupational disease prevention programs (Korniewicz, Chookaew, El-Masri, Mudd, & Bollinger, 2005). Other studies included multi-faceted programs (P. M. Davis, Badii, & Yassi, 2004) and one that evaluated a program to reduce violence in the workplace (Martin, 1995). The literature on violence in the healthcare sector has grown substantially since the Tompa et al. (2009) review (Kynotch, Wu, & Chang, 2010; Wang, Hayes, & O’Brien-Pallas, 2008; Wassell, 2009).
The vast majority of studies in the Tompa et al. (2009) systematic literature review took employers’ perspectives, a few took a systems perspective, and some a societal perspective. As a result, most studies only considered resources and outcomes of significance to the employer. Many studies relied on workers’ compensation expenses as the only outcome measure. The resources identified included administrative time, worker time spent in training or education, specialist time (e.g., trainer, therapist, ergonomist, and attorney), and equipment costs (e.g., purchase, installation, modification, and maintenance). The outcomes identified in the systematic literature review included workers’ compensation (other insurance) expenditures, value of lost work time due to illness or injury, healthcare expenses, modified duty expenses, worker replacement expenses (e.g., overtime, recruitment, and training), and organizational performance (e.g., turnover, recruitment, productivity, quality, goodwill, client loss, motivation, and grievances). Some resources and outcomes identified in relevant studies were not always measured due to difficulty in retrieving associated data.
Key Informant Interviews
Twenty-five key informant interviews were held with executives of five healthcare unions, three healthcare insurers, nine healthcare employers, three Ministry of Health Services managers, and five healthcare workers. Three key messages emerged from interviews. First, many respondents put high priority on worker and patient outcomes regardless of the stakeholder group they represented, emphasizing the importance of including these outcomes, plus employer outcomes, in economic evaluations. Second, units other than monetary measures were important. They emphasized that this did not preclude translating resources and outcomes into monetary measures, as might be done in a cost-benefit framework, but rather, that more information should be provided than simply a summary measure such as the net-present-value or a benefit-to-cost ratio. Third, worker representatives commented on a range of labor-related issues that extend beyond the financial resources and outcomes of a program. Issues such as safety climate, job satisfaction, the pace of work, job security, and meaningful work were mentioned. Their concern was that health and safety decisions were sometimes driven by operational cost concerns rather than workers’ health and safety.
Based on the interviews, respondents preferred multiple ways to express the results of economic evaluations:
Monetary measures—return on investment, net cost impact, avoided costs, number of years to break even on an investment, impact on insurance premiums, cost per patient day;
Measurement in worker time lost—skilled resource work days saved, sustainable and durable return to work days;
Measurement in health and safety incidence—incidence rate, incidents prevented; and
Measurement in other units—productivity, length of patient stay, patient safety (e.g., avoided, near miss and patient incidents, healthcare errors), quality of care, worker satisfaction, worker retention.
A total of 61 resources and outcomes were identified in the first round of data extraction from published literature and key informant interviews. These resources and outcomes were distilled to 11 resources and 31 outcomes by the research team to minimize overlap. Definitions and examples were developed for each resource and outcome and used in the Delphi panel exercise.
Delphi Panel
The Delphi panel consisted of 46 participants, including 10 worker representatives, 12 employer representatives, 11 insurer representatives, three Ministry of Health Service staff, and 10 patient representatives. Of the 46 participants, 41 completed the three rounds.
Table 2 displays the resource ranking from the third (final) round. Heavier lines in the table identify items that cluster close together in their rankings. The resources are ranked according to the sum of ranks. A smaller sum of ranks means that those resources have a higher priority, and a higher sum means that they have lower priority. Some resource ranks are so close that they are virtually tied. Closely ranked resources should be viewed as a group having similar priority. Based on the sum of rankings, “health and safety staff time” and “training the worker” are clearly the two top resources, with “planning, promotion, and evaluation” a close third. At the bottom are “equipment removal and disposal” and “supply cleaning, recycling, and/or disposal.”
Final Ranking of Resources
Note. A heavy bar identifies a gap, and thus the “groupings” that resulted in the rankings.
The ranking in Round 1 varied across panel members, with the variance decreasing in Rounds 2 and 3. “Planning and promotion,” “administration costs,” and “consultant fees” were the resources with least consensus among panel members, and this lack of consensus persisted over the three rounds. Employers initially ranked “health and safety staff time” as the top priority, whereas it was ranked fourth by worker representatives. In contrast, “training the worker” was initially ranked fourth by employers and the top priority by worker representatives. By the third round, all groups concurred that these two resources were a top priority for health and safety decision making.
Table 3 displays outcome rankings from the third round ordered by their sum of ranks, from highest to lowest priority. Given the number of outcomes, the ranking process posed a greater challenge for the panel. To facilitate the process, outcomes clearly placed at the bottom were removed for ranking purposes in Round 2 and outcomes clearly placed at the top were removed in Round 3. Some outcomes had sums of ranks so close to be considered virtually tied. These outcomes should be considered as a group having a similar level of priority. A group of five outcomes are clearly top priority and were at the top in both the first and second round rankings. This group consisted of “number of injuries, illnesses, and general sickness absences,” “safety climate,” “job satisfaction and engagement,” “quality of care and patient safety,” and “days lost due to injuries, illnesses, and general sickness.” The second priority group consisted of four items: “accountability and reputation,” “accommodating injured or ill workers,” “attraction and retention,” and “meaningful return to work.” Noteworthy is the fact that “employer workers’ compensation insurance premiums,” which is often a key element considered in the published literature, ranked low in this study.
Final Outcome Rankings
Note. A heavy bar identifies a gap, and thus the “groupings” that resulted in the rankings.
Most outcomes had relatively similar variation in ranking across panel members, with the exception of “accountability and reputation” and “worker’s roles outside work,” which had substantial variation. Table 4 presents the top three outcome priorities ranked by stakeholder groups in Round 1 (before receiving any panel feedback). It is noteworthy that worker and patient representatives identified the effect of a health and safety program on “safety climate” and “job satisfaction” as high priorities, though these outcomes are difficult to quantify and seldom included in economic evaluations.
Top Three Outcome Priorities by Stakeholder Group
Discussion
The findings of this study are of interest to health and safety policy makers, researchers, and practitioners, including occupational health nurses. The study highlights stakeholders’ priorities and their expectations of health and safety economic evaluations. The identified resource and outcome priorities can inform the development of a framework for health and safety economic evaluation in healthcare that may better support and promote evidence-informed health and safety decision making.
Based on results from the Delphi panel, 11 resources were identified in four priority groupings for consideration in economic evaluations. The three top priority resources were “health and safety staff time,” “training the worker,” and “planning, promotion, and evaluation.” Given that this ranking was identified by stakeholders in the healthcare sector, it is critical for researchers to include these resources in economic evaluations.
Priority ranking for outcomes was less consistent across stakeholders within the Delphi panel, though remarkably, worker and patient outcomes were considered important by most study participants. Given this finding, some compromises should be made by healthcare stakeholders to improve support for economic evaluation. Nonetheless, five priority outcomes were identified by the panel. They were “number of injuries, illnesses, and general sickness absences,” “safety climate,” “job satisfaction and engagement,” “quality of care and patient safety,” and “days lost due to injuries, illnesses, and general sickness.” As with the priority resources, inclusion of these five outcomes is critical for economic evaluations to be useful to all healthcare stakeholders.
Worker representatives identified human resources issues that they felt were important to report (i.e., “safety climate,” “job satisfaction and engagement,” and “meaningful return to work”) even though these issues have not traditionally been part of economic evaluations. These outcomes highlight the fact that health and safety is ultimately about workers, and workers should be central to economic evaluations. In published studies, this principle is not always at the forefront of analyses. Rather, the focus is often on financial returns to the employer through reduced workers’ compensation expenses. A broader conceptualization of outcomes than typically considered in economic evaluations is particularly important when multiple stakeholder groups are affected by programs. It is also difficult to separate economic issues from program evaluation issues. As a result, it may be best practice to report on a range of relevant outcomes in an economic evaluation.
Surprisingly, injured worker outcomes such as “worker’s health-related quality of life,” “career impacts and employability,” “worker’s lost income,” “worker’s out-of-pocket expenses,” and “worker’s roles outside of work” were ranked 11th and lower. It appears that broad injury and human resources outcomes (i.e., leading indicators) were a priority for stakeholders, whereas injured worker-specific outcomes were less so (i.e., trailing indicators).
Stakeholders identified “quality of care and patient safety” as a top priority. In contrast, “impacts on productivity” and “number of healthcare services provided” were in the second tier of priority outcomes. Clearly, quality of care cannot be substituted by volume of services or other measures of output and productivity.
Workers’ compensation expenses were low on the priority list in contrast to studies in the published literature where these expenses had been the key outcome of interest (e.g., Alamgir et al., 2009; Chhokar et al., 2005; Collins et al., 2004). As noted, previous studies have focused on the employer perspective (e.g., monetary summary measures; Rothmore et al., 2014; Tompa et al., 2009). Yet the outcome priority list identified by the Delphi panel included many items not traditionally captured in monetary terms. This broader scope suggests that the framing of economic evaluations in this sector should go beyond traditional monetary measures if they are to be relevant to all stakeholders in the healthcare sector.
The key strength of this study is that it is grounded in stakeholder engagement throughout the research process. As noted, an advisory group consisting of stakeholders guided the study. The identification and ranking of priorities included representation from several stakeholder groups. The Delphi panel approach provided a means for reaching consensus on priority rankings that was acceptable to all groups.
Several limitations were identified in this study. First, the Delphi panel had no face-to-face discussions. This lack of contact ensured no opportunity for pressure or persuasion, though this also negated the possibility of discussion to illuminate new insights. Second, stakeholders who participated in the advisory group, in-depth interviews and the Delphi panel had little knowledge of economic evaluation. The research team, which included two economists, was conscious of this issue, and efforts were made throughout to ensure that all materials were understandable by non-economists. Piloting ensured that all materials appealed to a broad stakeholder audience. The advisory group and key informant interviewees were provided a background document on economic evaluation. The Delphi panel received a one-page summary on economic evaluation. A third limitation is that Delphi panel employer representatives were at a lower level of decision making than employer representatives who were key informant interviewees. Including higher level employer representatives on the Delphi panel might have introduced new or different viewpoints to the process and results.
The findings are likely relevant for not-for-profit, publicly financed healthcare in British Columbia, Canada. It remains to be seen whether the findings are readily transferable to other jurisdictions in Canada or elsewhere. Similar studies should be undertaken in multiple jurisdictions to confirm the generalizability of the findings.
Conclusion
This study identified top tier resources and outcomes that should be considered in economic evaluations of health and safety programs in the not-for-profit, publicly financed healthcare sector, according to key sector stakeholders. Given the findings, researchers undertaking economic evaluations could improve their study findings by including as many of the priority resources and outcomes as possible in their health and safety healthcare economic evaluations.
The proposed list of priority resources and outcomes may be a starting point for a new, more comprehensive framework for the economic evaluation of health and safety interventions in many not-for-profit sectors. Incremental methods research is needed to develop a complete framework within which these resources and outcomes can be meaningfully integrated to ensure no overlap or double counting. In particular, research is needed to further develop and integrate constructs such as safety climate, job satisfaction and engagement, quality of care, and patient safety more formally into economic evaluations.
Applying Research to Practice
This study identified top tier resources and outcomes that should be included in economic evaluations of health and safety programs in healthcare and other not-for-profit, publicly financed sectors. Given the findings, researchers and policy makers should consider a broader set of resources and outcomes than have traditionally been used in economic evaluation of health and safety programs in healthcare. The findings provide a reference for occupational health nurses and other health and safety practitioners to use when assessing the merits of interventions or planning such evaluations. Occupational health nurses can use study findings to support the need for health and safety resources and demonstrate, more comprehensively, the outcomes that flow to diverse stakeholder groups.
Footnotes
Appendix
Authors’ Note
Any research materials related to this article can be obtained from the first author upon request.
Conflict of Interest
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research study was funded by the WorkSafeBC Research Secretariat. M. Koehoorn was supported in-part by a Michael Smith Foundation for Health Research Senior Scholar Award (WorkSafeBC 10.13039/501100000182 RS2009-OG12).
Author Biographies
Emile Tompa is a health and labor economist and a senior scientist at the Institute for Work & Health. He holds appointments as associate professor in the Department of Economics at McMaster University and assistant professor at the Dalla Lana School of Public Health at the University of Toronto. He is co-director of the Centre for Research on Work Disability Policy (CRWDP), a seven-year initiative funded by the Canadian Social Sciences and Humanities Research Council.
Henriette de Boer was a research associate at British Columbia Injury Research and Prevention Unit, University of British Columbia, at the time of this study. She is currently applying for her master’s in adult education at a First Nations vocational college.
Sara Macdonald is a knowledge transfer and exchange associate at the Institute for Work & Health. She engages stakeholders in research development and dissemination, manages knowledge exchange networks and events, and creates tools from research findings. She received an occupational health and safety certificate from Ryerson University and a diploma of business systems management at Gothenburg, Sweden.
Hasanat Alamgir is an associate professor of epidemiology, human genetics, and environmental sciences at The University of Texas School of Public Health. His degrees include MPharm from Dhaka University in Bangladesh, MBA from West Texas A&M University, and PhD from the University of British Columbia. His research interests include occupational injury epidemiology, economic consequences of injury, evaluation of interventions, and global occupational health.
Mieke Koehoorn is a professor and the head of Occupational and Environmental Health Division at the School of Population and Public Health, Faculty of Medicine at the University of British Columbia. She holds CIHR chair in Gender, Work, and Health (2013-2018), with a focus on investigating differences in occupational health and workers’ compensation outcomes and experiences by gender/sex.
Jaime Guzman is a clinical associate professor at the Faculty of Medicine, University of British Columbia. He has received a master of science degree in clinical epidemiology from the University of Toronto and completed specialty in physical medicine and rehabilitation at The University of Manitoba.