Abstract
Objectives:
This article describes a meta-analysis of The Working Mind, a program that was developed to address workplace mental health. The basic program addresses issues related to stigma in the workplace, the use of a mental health continuum model to evaluate signs and indicators of mental illness, and the development of coping skills. A manager version further addresses issues such as how to work with an employee who struggles with mental health issues, workplace accommodations, and overall management issues.
Methods:
A total of 8 replications evaluated program effects on stigma, self-reported resilience, and coping abilities.
Results:
The implementation of the program was associated with moderate reductions in stigma and increased self-reported resilience and coping abilities. These results were generally consistent across settings and showed nonsignificant differences when various potential moderators of the program were evaluated (e.g., employees versus managers, public versus private sector, gender, age). Qualitative comments collected at the end of the program suggested that many program participants found the program to be helpful and that the skills were being employed.
Conclusions:
Directions for future research, including the need for a randomized trial of The Working Mind, are discussed. Overall, the results suggest that the program is successful in its aims, but further inquiry is encouraged.
Recent data demonstrate that the mental health of employees significantly affects their ability to work productively and to maximize both their own satisfaction in the workplace, as well as the economic activity of the employment setting itself. 1,2 Furthermore, it is recognized that many workplaces have inherent stressors built into them, including normal stresses associated with workplace performance, interpersonal relationships, conflicts, and systemic pressures, which can all contribute to potential mental health problems. 3,4 It is also clear that significant mental health challenges in the workplace create additional costs to disability programs and that mental health problems themselves are one of the greater contributors to overhaul costs of health care employee benefits. 5 In light of these facts, the promotion and maintenance of workplace mental health have become national imperatives in many countries and are supported by a variety of agencies and policies, including the National Standard for Psychological Health and Safety in the Workplace developed by the Mental Health Commission of Canada, 6 the Equity Act in the United Kingdom, 7 and the 2001 Americans with Disabilities Act 8 in the United States.
There is little doubt that an effective mental health strategy is needed for the contemporary workplace. This overall strategy is likely to include mental health promotion, policies, and practices that foster a healthy workplace environment; stigma reduction to improve employee help seeking and to foster the creation of a supportive workplace culture; early intervention for appropriate care; policies to support employees who are struggling; and employee and family wellness benefits. 9 These elements of a workplace mental health strategy also require support from a general health care system that can effectively treat mental health problems and assist employees to return to work as efficiently as possible. A review of workplace antistigma and mental health promotion programs 10 unfortunately revealed that there are few evidence-based mental health workplace programs. Indeed, many programs that purport to reduce stigma and enhance mental health in the workplace are not evaluated, are evaluated using acceptability ratings (which is only weak evidence), or are proprietary and so the evidence is unavailable for public use.
The Opening Minds initiative of the Mental Health Commission of Canada 11 was established to promote stigma reduction. Within its scope of work, Opening Minds targeted the workplace, as it was recognized that this is a critical location where stigma can impede access to timely and appropriate care, 12,13 reduce employee satisfaction and performance, and ultimately increase losses and thereby reduce the profitability of employments settings. As part of its work, Opening Minds became aware of the work of the Canadian Department of National Defence, which had developed a program for mental health resiliency training in the military, entitled the Road to Mental Readiness (R2MR). As described elsewhere in this volume, 14 the R2MR program was adapted for use in first responder groups. This adaptation included a demilitarization of the content, enhanced efforts to reduce stigma through videotaped contact, and explicit discussion about stigma and its negative effects within first responder groups. As a further adaptation of the R2MR program, The Working Mind was created as a general workplace program. The current study is a meta-analysis of The Working Mind program, as applied in a variety of workplace settings across Canada.
The core elements of The Working Mind reflect its genesis, and both its structure and content resemble those seen in the R2MR program. The program is intended to reduce stigma, improve awareness of various signs and indicators of mental health using the mental health continuum model, promote coping skills, and provide information about policies and practices related to the workplace that could promote mental health and assist employees who experience mental health challenges. Details of the program are available elsewhere (see Szeto et al. 15 ).
There are 2 versions of the program. One is intended for frontline workers and is a 4-hour group program. The other is an 8-hour program intended for managers, which not only examines their own mental health and coping resources but also highlights their obligations in the workplace to those who report to them. For example, there is an extended discussion of how to talk with employees about their mental health challenges, how to manage mental health accommodations in the workplace, and how to facilitate the successful return to work for employees who have been off work on medical leave. Both versions of the program use trained facilitators, workshop manuals, contact-based videos that present actual employees and managers dealing with issues related to the program’s content, discussion exercises, and personal goal setting to begin to enact the coping skills within the program.
The Working Mind program is delivered in 1 of 2 ways, as selected by the workplace that wishes to engage the program. The preferred method for delivery is a “train-the-trainer” model, wherein employers send their prospective workshop leaders to an intensive training week. These leaders experience the program, learn the manuals and other materials, and then practice delivery, before being evaluated and (hopefully) approved to deliver the program. This method is preferred primarily as this training results in group facilitators from within the same organization that is going to receive the program and so builds internal experts and the capacity to discuss and train mental health in the workplace. This said, if an employer prefers, trained group facilitators can be sent to directly deliver The Working Mind program.
The Working Mind went into development in late 2012. This article reports the results from 8 replications of the program in numerous Canadian jurisdictions between December 2013 and May 2015. Ethics approval for these evaluation studies was granted by the University of Calgary Conjoint Faculties Research Ethics Board (ID: REB14-1611).
Methods
All of the replications reported in this study used a comparable design, which was an open trial methodology, in which The Working Mind program was delivered, and outcomes were assessed before the program, immediately at its conclusion, and at a 3-month follow-up period. While participation in the evaluation of the program was not compelled by most organizations, their employees were encouraged to complete all 3 assessments. Each employee generated an anonymous but unique identifier, so that his or her data could be matched across time periods. A pooled analysis was used to examine the outcomes of the program, and study-level meta-analysis with moderators was used to interpret the data.
Primary Outcomes
The 2 primary outcomes identified for the program were stigma reduction and improvement in resiliency skills. Stigma was expected to reduce as a result of the program, whereas coping skills were expected to increase. Stigma was measured using the Opening Minds Scale for Workplace Attitudes 16 (OMS-WA). The OMS-WA is a 22-item scale designed specifically for workplace environments to assess attitudes, stereotypes, and behavioural intentions toward persons with mental illnesses. Examples of scale items include “Most employees with a mental illness are too disabled to work,” “Employees with a mental illness often become violent if not treated,” “I would help a co-worker who got behind in their work because of a mental illness,” “You can’t rely on an employee with a mental illness,” and “I would try to avoid an employee with a mental illness.” Five main dimensions of stigma are captured in the scale, including the desire to avoid, perceptions of dangerousness and unpredictability, negative attitudes about mental illness in the workplace, negative attitudes toward helping people with a mental illness, and beliefs about responsibility for having a mental illness. All items are scored on an agreement scale from 1 to 5, where lower scores indicate less stigma. Mean scores were used for the full scale as well for each of the 5 factors.
Improvement in resiliency skills was assessed with a 5-item scale that was developed for the evaluations. The scale captures participants’ perceptions of their level of skill and ability to recover from adverse or traumatic situations. Scale items include “I have the skills to cope with traumatic events or adverse situations,” and “I believe I can recover quickly if I am negatively affected by traumatic events or adverse situations.” Responses are scored on an agreement scale from 1 to 5. Higher scores indicate greater perceived resiliency.
Analytic Strategy
The analysis approach was 2-fold. First, the “metan” command was used to show change in scores from pre- to posttest by study, using a forest plot. 17 The Q test was used to assess the homogeneity of study results in this meta-analysis, but a random-effect meta-analysis was chosen a priori since this approximates the fixed-effect model when heterogeneity is low. Then, to explore determinants of the 2 primary program outcomes, a pooled data set was produced. A random intercept linear mixed-model approach was used to conduct a study-level meta-analysis for changes in stigma and resiliency skills before and after the intervention. Site was modeled as a random effect, and to examine possible moderators of treatment outcomes, participant characteristics were examined as independent variables using this approach. Individual tests included pre- to postchange by participant type (frontline staff or supervisor) and by work sector, as well as by gender, age, education, marital status, and self-rated mental health at baseline. These participant factors were captured as part of the pretest questionnaire. Analyses were completed using version 12 of Stata. 18
Additional Outcomes
Three additional outcomes were of interest to this study. These outcomes were all related to effects of the program over time. This included the retention of stigma and resiliency skill improvements over time; the extent to which participants improved their understanding, willingness to discuss, and willingness to seek help or provide support to colleagues regarding mental health in the workplace; and the extent to which participants were using the program’s skills and knowledge. While the primary outcomes may be viewed as a direct assessment of program impacts, the additional outcomes were identified to assess changes over the longer term, which may be one way to glimpse whether broader organizational or cultural shifts regarding workplace mental health might be taking place.
Assessment of change from post to follow-up for the OMS-WA and the resiliency skills scale was undertaken using the same method as for the primary pre- to postoutcome described above, that of a random intercept linear mixed-model analysis. This same method was also employed for the 4 measures related to understanding and intentions regarding mental health in the workplace. These measures each contain a 5-item agreement scale and were asked at pretest and 3-month follow-up. Usability of the skills and knowledge enhancement were assessed by asking participants at follow-up if they had used any of what they learned in the program at home or at work (a yes/no response) and asking them also to describe their response in more detail. Open-ended responses were then coded for themes and analyzed with frequency tables. Other measures were used to evaluate the program but are not discussed here. 19
Data Sources
Details about the various program implementations is provided in Table 1, including number of participants and survey completions across the various program replications. All implementations were evaluated using a nonrandomized pre-post design, and all studies included a 3-month follow-up survey. Surveys were linked across time points through a process whereby participants provided the last digit of their year of birth, the last digit of their day of the month in which they were born, the last 2 digits of their home phone number, and the last letter of their last name on their survey forms. Due to participant errors in the generation of these unique codes and the nonresponse of some participants at some time points, the sample sizes vary across the various replications and time intervals.
The Working Mind Individual Evaluation Details: Setting, Industry Type, Total Participants, and Pre-Post Completed Surveys.
Results
The data set for the pooled analysis included 1292 participants across the 8 replications and a total of 1155 of completed and matched pre- and postsurveys. Participant characteristics for the pooled sample can be seen in Table 2. The number of matched surveys at the follow-up assessment period was 415. Internal reliabilities of the scales, as assessed by Cronbach’s alphas, were .91 at pretest, .92 at posttest, and .90 at follow-up for the OMS-WA total score. For the OMS-WA subscales, Cronbach’s alphas were acceptable at all time points with the exception of the “helping behaviour” subscale, which was slightly under the generally acceptable minimum of .70 (social avoidance: .87, .90, .85; danger/unpredictability: .75, .80, .79; work-related beliefs: .81, .80, .78; helping behaviour: .58, .68, .63; responsibility for one’s illness: .72, .77, .74). The subscale for helping behaviour contained only 4 items, which may contribute to lower Cronbach’s alphas at 1 or more time points.
Summary of Participant Characteristics (n = 1155).
Marital status, n = 340; self-rated mental health, n = 885.
aNot collected in all evaluations/sites.
Cronbach’s alphas for the resiliency skills scale were .81 at pretest, .86 at posttest, and .84 at follow-up, indicating a high level of internal consistency at all 3 time points for both primary outcome measures. An examination of a histogram and QQ plot showed normal distribution of change scores for both primary outcome measures.
Primary Outcomes
Figure 1 shows the forest plot of the individual program effects for the OMS-WA measure (I2 ∼ 0.0; heterogeneity χ2 = 1.20, df = 7, P = 0.99). Despite this strong degree of homogeneity, a random-effects model was used in the meta-analysis. Program effect sizes ranged from .15 to .52 (standard mean difference [SMD]), with an overall combined effect size of .38. The test of SMD = 0 revealed a z score of 9.13 and was significant at the 95% confidence interval (P < 0.001). Figure 2 shows the shows the forest plot of the individual program effects for the resiliency skills outcome (heterogeneity χ2 = 2.34, df = 7, P = 0.94). Program effects ranged from .41 to .65 (SMD), with an overall combined effect size of .50. The pooled effect was statistically significant (z = 11.66, P < 0.001).
Forest plot of The Working Mind program effects by study/site: change in the Opening Minds Scale for Workplace Attitudes. Study ID numbers match those from Table 1. CI, confidence interval; SMD, standard mean difference.
Forest plot of The Working Mind program effects by study/site: change in resiliency skills. Study ID numbers match those from Table 1. CI, confidence interval; SMD, standard mean difference.
For the pooled sample data set, OMS-WA mean (SD) score at pretest was 1.77 (.47). At posttest, the mean (SD) score was 1.61 (.45), representing an overall mean improvement of .17 scale points. The results of the mixed-model analysis for the pre- to postchange on the OMS-WA revealed statistically significant reductions in stigma for the total scale, coefficient = .167, SE = .08, z = 20.72, P < 0.001, and all subscales (all Ps < 0.001). Analysis of participant factors found no differences in outcomes by participant type (frontline staff vs. supervisor), work sector (public vs. private), gender, age, education, marital status, or self-rated mental health.
For the outcome of improvement in resiliency skills, the mean (SD) score was 3.50 (.64) at pretest and 3.81 (.61) at posttest, representing an overall mean improvement of .31 scale points. The mixed-model analysis for the pre- to postchange on the resiliency skills scale revealed statistically significant improvement at the 95% level of confidence (P < 0.001). As with the above analyses, the total scale was a significant predictor of this effect, coefficient = –.308, SE = .015, z = –.20.39, P < 0.001, but none of the participant factors was associated with significant prediction.
Additional Outcomes
A total of 564 follow-up surveys were completed, 414 of which could be matched to corresponding pre- and postsurveys. An analysis of sample characteristics found that participants who completed all 3 surveys had lower (i.e., more positive) baseline stigma scores than did participants who did not complete all 3 surveys (1.72 and 1.81 respectively, P = 0.002). Women were also more likely than men to complete all 3 surveys (noncompleters: male [42.0%], female [58.0%]; completers: male [28.7%], female [71.3%], P < 0.001), as were managers/supervisors (noncompleters: frontline staff [52.1%], supervisors/managers [47.9%]; completers: frontline staff [42.9%], supervisors/managers [57.1%], P = 0.002). No significant differences were observed in age, marital status, education level, self-rated mental health at baseline, or baseline resiliency skills score between those who completed all 3 surveys and those who did not.
Table 3 shows the results of the post to follow-up analysis for the OMS-WA total scale and subscales, the resiliency skills scale, and the pre to follow-up change for the 4 statements pertaining to mental health knowledge and intentions in the workplace. As shown, reductions in stigma were maintained to the time of follow-up for the subscales of avoidance/social distance, work-related beliefs/competency, and responsibility for illness. A significant loss in gains was observed for the subscales of danger/unpredictability and helping behaviour. A repeat of the analysis from pretest to follow-up on these 2 subscales showed that scores were still significantly improved from baseline for the danger/unpredictability subscale (coefficient = .201; SE = .025; z = 7.90; P < 0.001) but not for the helping subscale (coefficient = –.023; SE = .035; z = –0.66; P = 0.510). Mean (SD) scores for the total OMS-WA scale at the 3 time points were as follows: baseline = 1.72 (.43), posttest = 1.53 (.41), and follow-up = 1.62 (.42) (n = 415 matched). No differences were observed across any of the participant factors measured.
Random Intercept Mixed-Model Regression: OMS-WA Post to Follow-up Change, Post to Follow-up Resiliency Skills Change, and Pre to Follow-up Change for Mental Health Knowledge/Intentions Statements.
OMS-WA, Opening Minds Scale for Workplace Attitudes.
For the resiliency skills scale, a loss in gain was also observed from post to follow-up (Table 3). Again, a repeat of the analysis from pretest to follow-up showed that scores were still significantly improved over those at baseline (coefficient = –.186; SE = .026; z = –7.21; P < 0.001). Mean (SD) scores for the resiliency skills scale at the 3 time points were as follows: baseline = 3.52 (.65), posttest = 3.84 (.64), and follow-up = 3.70 (.63) (n = 398 matched). An analysis of participant characteristics showed private-sector participants had better retention of resiliency skills scores compared to public-sector employees (coefficient = –.154; SE = .063; z = –2.43; P = 0.015; constant = .165). No other differences from post to follow-up were observed.
For the 4 measures pertaining to participants’ understanding of mental health in the workplace, intentions toward seeking help, and behaviours related to openness and supporting fellow colleagues, significant improvements from pretest to follow-up were observed on 3 of the 4 statements: “I understand how mental health problems present in the workplace,” “I plan to seek help for my mental health problems, when needed,” and “When I am concerned, I ask my colleagues how they are doing” (Table 3). Mean (SD) scores were as follows: question 1: baseline = 3.32 (1.71), follow-up = 4.07 (.91); question 2: baseline = 3.15 (1.65), follow-up = 3.75 (1.08); question 3: baseline = 3.09 (1.64), follow-up = 3.79 (1.16); and question 4: baseline = 3.23 (1.57), follow-up = 3.46 (1.20).
Most (69.4%, n = 268) respondents responded positively to the follow-up question, “Have you used any of what you learned in TWM [The Working Mind] at work or at home?” (see Table 4). Many of these participants mentioned actively using the tools taught in the program, particularly the mental health continuum model (35.4%), and, to a lesser extent, the “Big 4” skills (19.3%). Many also indicated they used the empathy skills learned in the program to provide support and/or reach out to colleagues regarding their mental health (29.2%). Talking more about mental health in the workplace and being more open with colleagues was also commonly mentioned as a way program learnings were being used at follow-up (20.5%). Additionally, 5.0% indicated that the program led them to seek professional help for their own mental health and/or enabled them to get friends or colleagues to seek professional help for their mental health.
Have You Used Any of What You Learned at The Working Mind at Work or at Home—Specify: Respondents Who Answered “Yes” (n = 268 Cases; n = 339 Responses).
Most respondents who replied “no” (n = 101) did not provide further details. Those who did typically indicated that they had not used the skills because they had not had the need/opportunity (5.1%), because they had forgotten what they had learned (2.6%), or because the course had not taught them anything new (4.3%).
Discussion
This study provided a meta-analysis of the effects of a workplace mental health awareness and stigma reduction program, entitled The Working Mind. The outcomes of this program were evaluated in 8 diverse workplace settings across Canada, with a focus on the primary outcomes of stigma in the workplace and perceived resiliency. The results of the program revealed improvement in both primary measures. The observed effect sizes were comparable to, if not a bit higher, than successful stigma reduction interventions with other target groups. 10,19,20 We also note that the effect sizes observed in the current study were somewhat stronger than for the results of the R2MR program also recently conducted in Canada and reported in this issue 11 despite the fact that The Working Mind program is in fact derived from the R2MR program. We also note that these results were obtained, even though the baseline scores on the stigma scale were already quite low at the outset of the program, which could have limited the results that might have been seen otherwise.
The low variability seen in the results across the various sites (see Figure 1) suggests that consistent program results can be obtained from a program that is delivered in a fairly standardized manner and, as such, likely has a high level of fidelity. The current results also demonstrated minimal outcome variability across different participant factors, which further suggest that the program is applicable to diverse workplace audiences.
Follow-up scores revealed that positive outcomes were maintained on many dimensions of stigma reduction. One exception to this general pattern was for the helping subscale of the stigma scale. This factor includes items that ask to what extent the respondent is willing to help others with mental health problems. It is at present unclear why these subscale scores were not maintained. It is possible that dimensions of stigma may not all respond equivalently to such a program or that the program itself did not address this dimension sufficiently. It is also worth reiterating that less than acceptable Cronbach’s alphas were observed on the helping subscale of the OMS-WA at pre, post, and follow-up (see above). Supplementary materials or booster sessions in this domain may be indicated. 21 Further research that distinguishes among the various dimensions of stigma will also help to discern if cultural change is not a linear process. 22,23
Many of the respondents who provided comments at the follow-up assessment indicated that the program had benefits and that they were using the skills that had been learned. Many also indicated that they had either personally sought professional help or encouraged others to do so. It has been suggested that in the short term, these incremental assessment and treatment costs may be associated with higher costs, but in the long term, this process likely yields more psychological healthy, effective, and productive workers and workplace cultures. 2,24 It was also observed that the more formally evaluated self-reported resiliency skills were not fully maintained between the end of the program and the 3-month follow-up assessment, but they were still significantly improved relative to the observed baseline skills. It was also observed that the self-reported use of the program’s skills was modest, which suggests that perhaps other elements were in play (e.g., general culture change), that participants were functioning well and simply did not need to use the program’s skills, or that some form of refresher may be necessary to remind participants about these skills. This pattern is not unusual in many intervention programs, 10 and it also supports the potential value of booster sessions or refresher modules.
Study Strengths and Limitations
This study benefitted from a number of strengths. It employed a well-conceived program with strong attention to professionally developed materials and concern about training and fidelity of the delivery of the program. The program was delivered in multiple sites, which yielded a large sample, with diverse characteristics that enabled a preliminary assessment of potential modifiers of the program’s outcomes. The evaluation captured both immediate and longer term changes on a range of primary and secondary outcomes, which included both quantitative and qualitative measures.
Despite the above set of study strengths, it also had several limitations. None of the assessments included a control group, and so although it seems unlikely that the observed pattern of results was the result of regression toward the mean, natural change, or some common third variable that influenced change across the set of workplaces included in the current study, such possibilities cannot be definitively ruled out. Further research employing a control group, or potentially with a randomized trial methodology, will help to discern the program’s true efficacy. A second study limitation is that, while all of the sites included a follow-up assessment period, the participants in this study were neither incentivized nor compelled to complete these assessments. As a consequence, there was a relatively large attrition at the naturalistic follow-up period, and an examination of participant characteristics did reveal some differences between participants who completed all 3 surveys compared to those who completed the survey at only pre and post. Future studies should consider strategies to retain a higher proportion of respondents at all assessment periods and might also wish to study longer follow-up intervals, to establish the program’s effects more definitively.
Summary and Directions for Future Research
In summary, the current study provides a compelling set of results for The Working Mind in a diverse set of workplace settings in Canada. The replicability of these results in other comparable workplace settings and their generalizability to other cultures, workplaces, and populations remain topics for future study. Such research should ideally employ a comprehensive set of evaluation outcomes, as was the case in the current study, to discern both the dimensions wherein the program had benefit and might be enhanced. We also strongly recommend alternative methods to track participants over time and, if possible, some method to incentivize follow-up assessments over a longer time interval, to minimize the issue of attrition observed in the above analyses. For example, the current results suggest the need to enhance or reinforce the helping dimension of stigma reduction, so it will be important to evaluate such considerations in future research. At a broader level, we also encourage the assessment of program and organizational outcomes that may be more indirectly affected by a program such as The Working Mind. Such outcomes could include the use of employee health benefits but also improved corporate culture and overall respect for employees in the workplace.
Footnotes
Acknowledgements
We thank all the organizations that have partnered with us on our projects, past and present. This work would not be possible without your collaboration.
Data Access
Request for access to the data can be addressed to the first author at
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: All authors have received funding, including grants and/or consulting fees, for the work described above.
Funding
The author(s) disclosed receipt of the following financial support for the research and publication of this article: The work described in this article was supported by the Mental Health Commission of Canada, which is supported by Health Canada.