Sage Journals: Discover world-class research

Abstract

Background:

Acceptance and Commitment Therapy is a form of Cognitive Behavioural Therapy which uses behavioural psychology, values, acceptance and mindfulness techniques to improve mental health and wellbeing. Acceptance and Commitment Therapy is efficacious in treating stress, anxiety and depression in a broad range of settings including occupational contexts where emotional labour is high. This approach could help palliative care staff to manage work-related stress and promote wellbeing.

Aim:

To develop, and feasibility test, an online Acceptance and Commitment Therapy intervention to improve wellbeing of palliative care staff.

Design:

A single-arm feasibility trial of an 8-week Acceptance and Commitment Therapy based intervention for staff, consisting of three online facilitated group workshops and five online individual self-directed learning modules. Data was collected via online questionnaire at four time-points and online focus groups at follow-up.

Setting/participants:

Participants were recruited from Marie Curie hospice and nursing services in Scotland.

Results:

Twenty five staff commenced and 23 completed the intervention (93%). Fifteen participated in focus groups. Twelve (48%) completed questionnaires at follow-up. Participants found the intervention enjoyable, informative and beneficial. There was preliminary evidence for improvements in psychological flexibility (Cohen’s d = 0.7) and mental wellbeing (Cohen’s d = 0.49) between baseline and follow-up, but minimal change in perceived stress, burnout or compassion satisfaction.

Conclusion:

Online Acceptance and Commitment Therapy for wellbeing is acceptable to palliative care staff and feasible to implement using Microsoft Teams in a palliative care setting. Incorporating ways to promote long-term maintenance of behaviour changes, and strategies to optimise data collection at follow-up are key considerations for future intervention refinement and evaluation.

Keywords

Psychosocial intervention occupational stress psychological resilience self-compassion professional burnout hospices palliative care

What is already known about the topic?

Working in palliative care involves a high degree of emotional labour and potential for stress.

Acceptance and Commitment Therapy is effective in workplace stress management interventions.

What this paper adds

Online Acceptance and Commitment Therapy consisting of self-directed learning resources and facilitated online sessions is feasible to deliver and acceptable to palliative care staff.

Challenges to retention of participants post-intervention delivery to evaluate longer terms impacts are identified.

The RESTORE intervention needs to be refined and then evaluated with a larger sample to assess effectiveness and cost-effectiveness.

Implications for practice

Training palliative care staff to apply individualised Acceptance and Commitment Therapy based strategies in their own lives may help them to manage the emotional impact of their work.

Communities of practice may be needed to help sustain this work.

Background

Palliative care work is generally viewed as rewarding, bringing meaning, and purpose to professional care staff.^1–3 However, alongside positive experiences, stress, distress and burnout are common.^4–6 Some stressors are typical of those identified across healthcare settings such as unmanageable workloads, shift work, staff shortages and lack of training on specific issues.⁶ Additional stressors specific to palliative care include regular exposure to death, loss and grief. This can include caring for a dying person of a similar age or background; caring for a dying child or person with dependent children; caring for a person with a complex physical condition; managing uncontrolled symptoms and breaking bad news.^6–8 Balancing the rewards associated with palliative care work with the emotional challenges is an ongoing task.

The COVID-19 pandemic led to increased stress, distress, and burnout across the health and social care workforce,^9–12 including palliative care^13–16 and exacerbated the need for healthcare staff support internationally.^13–17 A considerable proportion of the healthcare workforce experienced mood and sleep disturbances, raising concerns about risks to mental health.⁹ In palliative care, staff encountered rapidly changing clinical roles; new modes of service delivery, reduced staffing and increased patient numbers; coupled with risk of infection and limitations in personal protective equipment.^13,14 Many palliative care staff felt overwhelmed by the need to support families through telephone or video platforms due to social distancing measures; and remote consultations concerning sensitive and emotive end-of-life care concerns were challenging.^15,16 Supports needs of staff increased, and the 2020 Nursing Standard-Marie Curie survey revealed that 45% of respondents reported insufficient support at work to manage grief and emotional stress, up from one-third in 2019.¹⁷

Interventions to support wellbeing in palliative care staff are needed now more than ever. A systematic review prior to the COVID pandemic identified nine intervention studies to improve palliative care staff wellbeing including music therapy, a stress reduction programme, art therapy, a sleep intervention and a psycho-existential intervention.¹⁸ Evidence for the effectiveness of any particular intervention was inconclusive, and the authors concluded that more theory-driven development of psychosocial interventions to improve staff wellbeing were needed.¹⁸ Acceptance and Commitment Therapy is used in hospice settings,¹⁹ but has not previously been evaluated as an approach for improving wellbeing in palliative care staff. Outside of palliative care, there is a growing evidence base that ACT is effective in improving general and work-related distress compared to a control condition, though overall quality of evidence is low.²⁰ High quality research that seeks to design and further evaluate interventions, including ACT, to promote wellbeing amongst palliative care staff are warranted.

We propose an Acceptance and Commitment Therapy based psychological intervention to help palliative care staff to manage work-related stress and distress.^21,22 Acceptance and Commitment Therapy is an empirically supported form of Cognitive Behavioural Therapy which uses behavioural psychology, values, acceptance, and mindfulness techniques to improve mental health and wellbeing by increasing psychological flexibility.²¹ Psychological flexibility refers to the ability to act mindfully, guided by our values.²³ Acceptance and Commitment Therapy differs from traditional behavioural and cognitive interventions in that, rather than teaching methods of suppressing or challenging thoughts, it encourages greater awareness and acceptance of both positive and negative internal events. This promotes ‘opening up’ to unwanted inner experiences, when doing so is helpful in pursuing what is important to the person. Previous research identified common responses to workplace stressors in palliative care staff including emotional suppression, over-identification with patient suffering, and neglect of self-care behaviours.^2,6 These kinds of responses are typically targetted by Acceptance and Commitment Therapy. Acceptance and Commitment Therapy is efficacious in treating stress, anxiety and depression in a broad range of populations, including occupational contexts where emotional labour is high.^22,24–26 It may be especially suited to promoting palliative care staff wellbeing because it supports being present in the face of distress, being open to initiating difficult conversations, and acting in a way that is underpinned by values.^2,27 It may also reduce unhelpful responses such as self-doubt, rumination, and over-identifying with patients and families.²

Our aim was to develop, and feasibility test an Acceptance and Commitment Therapy intervention to improve wellbeing of staff working in palliative care settings. Given the rapid emergence of digital interventions in palliative care,²⁸ combined with the flexibility and scalability of an online intervention format, and the need to replace in-person with remote interventions during the COVID-19 pandemic, an online intervention was designed. We chose RESTORE (Research Evaluating Staff Training for Resilience) as the study acronym, as it reflected the aim of maintaining and restoring workplace mental health and wellbeing. The research questions were:

What is the experience of palliative care staff undertaking online Acceptance and Commitment Therapy and is it acceptable and feasible?

Is there preliminary evidence that online Acceptance and Commitment Therapy leads to improvements in psychological flexibility, wellbeing and stress?

What are the barriers and facilitators to implementing online Acceptance and Commitment Therapy for palliative care staff and what are the implications for future evaluation?

Methods

Design

A single-arm feasibility trial of an online Acceptance and Commitment Therapy -based intervention for staff providing palliative care for terminally ill adults. We used convergent mixed methods to address the research questions.²⁹ A convergent design allowed us to combine the results of both qualitative and quantitative data analysis in order to obtain a more complete understanding of the impact of the acceptability, feasibility and potential impact of the intervention. The study protocol was registered (https://doi.org/10.1186/ISRCTN14313559) and published.³⁰

Setting

The study was hosted by Marie Curie Scotland. Marie Curie is the largest independent provider of end-of-life care and the largest charitable provider of hospice-based care in Scotland (https://www.mariecurie.org.uk). Two Marie Curie hospices, located in Edinburgh and Glasgow, provide short-term inpatient hospice care and outpatient services for terminally ill people and families in surrounding areas. Across Scotland, the Marie Curie Nursing Service (MCNS) provides palliative care to people in their own homes. The intervention was delivered to staff participants online via Microsoft Teams. Participants could choose to take part in their personal time or during working hours, or both.

The original study was designed as an in-person intervention, to be delivered over 2.5 days (two full days and a follow up) at the Edinburgh hospice. However, the study was put on hold due to the COVID pandemic. Subsequently, in consultation with the hospice management team, it was redesigned for online delivery and also opened to staff at the Glasgow hospice and Marie Curie Nursing Service.

Participants

Participants were recruited from two Marie Curie hospices and from the Marie Curie Nursing Service in Scotland. As this was a feasibility study, sample size was not formally calculated.³¹ Based on prior experience, resources available and the format of the intervention, we judged that a sample of 30 would be sufficient to enable a decision about feasibility to be made. Eligible participants were health and social care professionals providing direct support to terminally ill patients and their families. This included nursing and medical staff, allied health professionals, social workers and healthcare assistants. Staff who had previously undertaken Acceptance and Commitment Therapy were excluded.

Participant recruitment

The research team held an online information session with service managers across Marie Curie. The study was promoted via hospice newsletters, posters and internal chat forums. Invitation letters were shared with staff by their service managers. Interested staff completed an online consent form prior to participating.

Intervention design and content

Content

The RESTORE intervention was developed by the research team. The content was informed by previous interventions delivered by the team for health professionals in other settings, as well as research on occupational stress, wellbeing, self-compassion and resilience in palliative care.^2,22,32,33

Format

Intervention format was guided by the format of previous interventions developed for related populations including patients,^34,35 and dementia care staff³⁶ and by advice from the host organisation in terms of what format might work best given clinical demands for staff (i.e. flexible format with some self-directed content). The intervention consisted of eight modules delivered via Microsoft Teams over an 8-week period, including three facilitated online group workshops and five self-directed, e-learning modules (Table 1). Participants received a workbook outlining the content for each week. The RESTORE manual and workbook is available on the ISRCTN registry: https://doi.org/10.1186/ISRCTN14313559.

Table 1.

Overview of the RESTORE online Acceptance and Commitment Therapy intervention for palliative care staff wellbeing.

Week	Module	Time required	Content	Delivery mode
1	Introduction to Acceptance and Commitment Therapy	90 minutes	Introduction	Synchronous online group workshop led by a peer-reviewed Acceptance and Commitment Therapy trainer, workbook exercises. Group chat via Teams
2	Values	30 minutes	Identifying and acting in line with your values	Self-directed, asynchronous materials, three videos, and workbook exercises, Group chat via Teams
3	Awareness	30 minutes	Present moment awareness; mindfulness; grounding	Self-directed, asynchronous materials, 2 videos and workbook exercises Group chat via Teams
4	Review of materials	90 minutes	Review, troubleshooting, clarifying materials. Discussion.	Synchronous online group workshop led by a certified Acceptance and Commitment Therapy trainer, workbook exercise. Group chat via Teams
5	Openness	30 minutes	Developing self-awareness, becoming more open, making room.	Self-directed, asynchronous materials, 2 videos and workbook exercises, Group chat via Teams
6	Defusion	30 minutes	Unhooking from difficult thoughts and feelings	Self-directed, asynchronous materials, three videos and workbook exercises, Group chat via Teams
7	Compassion	30 minutes	Kindness to self and others	Self-directed, asynchronous materials, two videos and workbook exercises. Group chat via Teams
8	Review and trouble shooting	90 minutes	Review, troubleshooting, clarifying materials. Discussion.	Synchronous online group workshop led by a certified Acceptance and Commitment Therapy trainer. Workbook exercises, Group chat via Teams

Facilitation

Online workshops were delivered by a Senior Academic Clinical Psychologist with extensive training and experience in Acceptance and Commitment Therapy (DG). DG is also a Peer Reviewed Acceptance and Commitment Therapy Trainer and Fellow of the Association for Contextual Behavioural Science (a mark of quality and fidelity of Acceptance and Commitment Therapy knowledge and skill). Workshops were also facilitated by a research psychologist (AF).

Homework

Participants were asked to complete ‘homework’ each week. For example, to practice a brief mindfulness exercise or try out different strategies to ‘unhook’ from unhelpful thoughts.

Adherence

To facilitate adherence, participants were sent weekly reminders to alert them that the new module content for that week was available.

Intervention fidelity

Intervention fidelity was not formally assessed. However, in the live sessions, the second facilitator (AF) took a monitoring role, ensuring that all elements of the sessions were covered as planned in an Acceptance and Commitment Therapy consistent way.

Research data collection

Feasibility data criteria were informed by a review of previous feasibility studies in palliative care research,³¹ and the research teams prior experiences of what might be expected in an intervention study involving palliative care staff. These data were assessed retrospectively and included:

number of participants recruited (target: 30).

number of participants completing intervention (target: two-thirds of those who commenced the intervention), this measure also reflected intervention acceptability.

number who completed the outcome measures (target: two-thirds of those who commenced the intervention).

number of participants who took part in post-intervention focus groups/interviews (target: 50%–75% of participants commencing the intervention).

Quantitative data collection

Five key outcomes were assessed prospectively at four time-points via online survey (https://www.onlinesurveys.ac.uk): before the intervention, mid-way (week 4), on completion (week 8) and 1-month post-intervention (follow-up). Further detail is available in the protocol.³⁰ The questionnaire was designed such that all questionnaire items needed to be completed before moving on the next page, thus minimising missing data.

Psychological flexibility was assessed by the CompACT.³⁷ It consists of 23 items over three sub-scales – openness to experience, behavioural awareness and valued action. Participants are asked to rate the degree to which each statement is true for them using a 7-point scale. CompACT total scores range from 0 to 138 with higher scores indicating greater psychological flexibility. Previous studies have found that general population samples have a mean total score of between 81 and 86, with a standard deviation of between 13 and 21.^38,39

Self-reported, perceived stress was assessed using the 10-item Perceived Stress Scale (PSS).⁴⁰ This assesses participants’ appraisals of stressful situations, including perceptions of how unpredictable, uncontrollable, and overloaded their lives have been over a defined period. Participants are asked how often they have felt a certain way on 5-point scale from ‘never’ to ‘very often’. Scores range from 0 to 40 with higher scores indicating higher perceived stress. The PSS is not a diagnostic instrument and does not provide cut-offs for high or low stress; though mean scores of 16.1 have been recorded for large samples of US women, and 15.5 for US men.⁴¹

Workplace quality of life was assessed by the Professional Quality of Life scale (ProQOL) Version 5.⁴² The ProQOL consists of 30 items across three subscales measuring compassion satisfaction, burnout and secondary traumatic stress. Each item rates the frequency of an experience on a scale from 0 (never) to 5 (very often). Compassion satisfaction scores of 22 or less, 23–41 and 42 or more reflect low, moderate and high compassion satisfaction respectively. Burnout scale scores of 22 or less, between 23 and 41 or 42 and over reflect low, moderate and high burnout respectively. Secondary traumatic stress scores of 22 or less, 23–41 or 42 and over reflect low, moderate and high secondary traumatic stress respectively.

Wellbeing was assessed using the Warwick Edinburgh Mental Wellbeing Scale.^43,44 This is a 14-item scale, covering both hedonic and eudaemonic aspects of mental health, including positive affect, satisfying interpersonal relationships and positive functioning. Scores range from 14 to 70 with higher scores reflecting higher levels of mental wellbeing. At an individual level, a change of three or more units is interpreted as representing ‘meaningful change’.⁴⁵

Healthcare Psychological Flexibility was assessed by the Mindful Healthcare Scale, a 13-item scale that assesses engagement, awareness and defusion in healthcare professionals.⁴⁶ Total scores ranges from 13 to 78 with higher scores reflecting greater psychological flexibility. This measure is currently in development.

Qualitative data collection

Online focus groups (MS Teams) were held approximately 1-month post-intervention (November 2021) and explored participant’s perspectives regarding acceptability, feasibility, and potential effectiveness (See Supplemental Information). All participants were invited to take part. Focus groups were facilitated by an experienced qualitative researcher (BS), not personally known to participants. Participants who dropped out were invited to share their reasons in a brief interview.

Quantitative data analysis

The main outcomes were feasibility related (recruitment, attrition, retention and evaluation completion); as such, this study was underpowered to detect statistically significant improvements in the non-feasibility outcomes (e.g. stress, quality of life, wellbeing). However, we did conduct paired sample t-tests to explore any changes in outcomes between baseline and follow-up for participants who completed both measures. We were most interested in evidence in change in outcomes from baseline (pre-intervention) to follow-up (1-month post-intervention). We calculated effect size (Cohen’s d) to illustrate the size and direction of any change that might be anticipated in future studies.

Qualitative data analysis

Focus groups were transcribed verbatim prior to analysis. Qualitative data analysis was undertaken by AF and BS guided by the framework approach using NVivo 12.³² AF undertook the initial coding, developed and applied the analytical framework and summarised the data. BS, who had facilitated the focus groups, provided additional support at the data interpretation stage.

Ethics and governance

The intervention was originally designed for in-person delivery and sponsorship approval was obtained from the University of Edinburgh on 18 February 2020. However, due to the COVID pandemic, the study was suspended in March 2020. In consultation with the hospice management team, the intervention was redesigned for online delivery in 2021. Sponsorship approval for the redesigned study was obtained from the University of Edinburgh Research Governance Office on 7/4/21; ethical approval was obtained from the University of Edinburgh Clinical Psychology Research Ethics Committee on 20/5/2021 (Ref: CLPS021s); and approval from the Marie Curie Research Governance Committee in Scotland was obtained on 15/6/2021. Full details on the management of personal data, data transfer, data storage and confidentiality are described in the study protocol.³⁰

Results

Sample characteristics and feasibility outcomes

Twenty-eight participants were recruited (Table 2), just under our recruitment target (Table 3). The sample was predominantly white female, aged between 25 and 54 years. Professionally they were mostly doctors and community or inpatient nurses, with over 5 years’ experience in palliative care. Prior to the intervention, three dropped out, and did not complete baseline measures, with the following reasons given: changed circumstances (n = 1); sick leave (n = 1) and no information (n = 1). Nearly all of those who commenced the intervention completed it: only two dropped out, citing sick leave (n = 1) and personal reasons/lack of time (n = 1). Questionnaire completion fell over the course of the study with just 48% completing questionnaires 1-month post-intervention. Focus group participation was affected by annual leave, sick leave and work patterns, but was in line with target. Overall, 15 participants took part in a focus group at follow-up (13 female and 2 male). This included nine doctors, four nurses and two allied health professionals or social workers.

Table 2.

Participant characteristics.

Participant characteristics*	Number	%
Gender
Female	26	93
Male	2	7
Age band (years)
25–34	5	18
35–44	6	21
45–54	11	39
55–64	4	14
65+	2	7
Role
Community nurse	6	21
Inpatient nurse	4	14
Doctor	10	36
Healthcare assistant	5	18
Allied health professional or social worker	3	11
Ethnicity
White	28	100
Years’ experience in palliative care
Less than 1 year	2	7
1–5 years	5	18
6–10 years	7	25
More than 10 years	14	50

Of those recruited for the full study, 15 took part in focus groups – 13 female, 2 male; 9 doctors, 4 nurses and 2 allied health professionals or social workers.

Table 3.

Feasibility outcomes compared with targets.

Feasibility outcome	Target	Actual
Number of participants recruited	30	28
Number of participants who commenced the intervention	30	25 (drop-out of three prior to commencement)
Number of participants who dropped out during the 8 weeks intervention	<33% (i.e. less than 8)	7.1% (two participants)
Number of participants who completed the intervention	16–17 (i.e. 66% of those who commenced the intervention)	23 (92%)
Number and % of participants who took part in online workshops (of those commencing the intervention)	Not pre-specified	Workshop 1: 24/25 (96%)
		Workshop 2: 20/25 (80%)
		Workshop 3: 17/25 (68%)
		Overall: 81%
Number of participants who completed the questionnaires	16–17 (i.e. 66% of those who commenced the intervention)	25/25: 100% at baseline
		23/25: 92% at week 4 (mid)
		18/25: 72% at week 8 (post)
		12/25: 48% at follow-up
Number of participants taking part in post-intervention focus groups or interviews	12–18 participants (50%–75% of those who commenced the intervention)	15 (60%)
		Doctors (9); Nurses (4); AHPs/social workers (2); Female (13); Male (2)
Number of participants reporting that they used Acceptance and Commitment Therapy strategies on some or most days since the last questionnaire was completed (based on questionnaire responses)	Not pre-specified	17/23: 74% at week 4 (mid)
		18/18: 100% at week 8 (post)
		11/12: 91.6% at follow-up

Staff perspectives on acceptability of the intervention

Participants reported that RESTORE provided a positive experience overall. They found it to be enjoyable, informative and beneficial. Participants valued the opportunity to step back and reflect on how they approach challenges in everyday life.

“I really enjoyed it and you know, quite a lot of the techniques that they were showing us and asking us to do I found really helpful in allowing yourself some space and time to think, or to clear your mind. . .”. (FG 5, P3)

The structure, format and mode of delivery was acceptable to participants. The blended approach, consisting of a mix of online self-guided and facilitated sessions, was acceptable and highly valued. The online group sessions allowed participants share experiences with colleagues, seek clarification and engage with the materials on a deeper level.

“I definitely wouldn’t have got as much just going through the material without that ability to hear other people’s reflections and stories, and the challenges. . .. “(FG4, P1)

The self-directed modules allowed participants to engage flexibly with the content and created opportunities to reflect personally on the material:

“. . .good balance with the self-work that you needed to do. . . I think if it had all been face-to-face sessions I wouldn’t have got so much out of it because you really do need to work through it” (FG4, P2)

The workbook helped participants stay on track each week and provided dedicated place for note-keeping that could be referred to even after the intervention had been completed.

“It was really helpful to have the workbook as well. I think you know it wasn’t just about you doing your own notes, the workbook running alongside it was really good” (FG5, P3)

Participants noted the high quality of facilitation of group sessions. The online aspect did not impede the ability of participants to feel they were in a safe space and open-up about their experiences.

“It was really well facilitated, you know like there’s all these problems where everyone talks at the same time or who knows when to speak, but the facilitation of it was really good and putting your hand up when people wanted to speak like yeah it was much better on Teams than I thought it was going to be. (FG2, P1)”

Preliminary evidence for intervention effectiveness

There was no difference in baseline outcome measure scores for the participants who completed outcome measures at follow-up (n = 12) compared to those who did not (n = 13). (Supplemental information file). For the 12 participants completing questionnaires at follow-up, we found preliminary evidence for improvements in psychological flexibility (Cohen’s d = 0.70), particularly valued action (Cohen’s d = 0.83) and openness (Cohen’s d = 0.66) (Table 4). There were increases in defusion between baseline and follow-up (Cohen’s d = 0.64), but only small or minimal changes for engagement and awareness. There were improvements in overall wellbeing (Cohen’s d = 0.49), but only a small reduction in stress (Cohen’s d = −0.34). There was little change in burnout or compassion satisfaction, though secondary trauma scores fell between baseline and follow-up (Cohens d = −1.03). At baseline, participant scores were high on compassion satisfaction, and low on burnout suggesting floor and ceiling effects.⁴² No participant met the threshold for burnout at baseline, thus improvements on these outcomes were unlikely.

Table 4.

Outcome data from baseline to follow-up.

Outcomes	Baseline N = 25		Mid-way N = 23		Endpoint N = 18		Follow-up N = 12		Questionnaire completers only at baseline (n = 12)		Completers baseline v follow-up	Effect size for completers only (n = 12)
Outcomes	Mean	SD	Mean	SD	Mean	SD	Mean	SD	Mean	SD	Sig. two-tailed	Cohen’s d (pooled)	Interpretation
Pychological flexibility
CompACT total	86.5	18.6	87.9	19.7	96.4	21.0	95.4	25.0	87.1	21.4	0.11	0.70	Medium increase
Valued action	35.6	6.5	36.9	7.7	37.3	6.5	37.3	8.4	35.5	7.8	0.17	0.83	Large increase
Awareness	16.9	7.0	15.9	7.3	19.2	8.1	17.4	8.8	16.0	7.6	0.46	0.29	Small increase
Openness	34.0	9.4	35.0	10.1	40.0	10.0	40.8	9.9	35.6	9.2	0.06	0.66	Medium increase
Wellbeing
Warwick Edinburgh Mental Wellbeing Scale	47.3	7.5	47.8	7.2	50.8	7.5	50.4	7.8	48.5	7.1	0.25	0.49	Medium increase
Stress
Perceived stress scale	17.3	6.2	16.7	6.1	14.9	5.7	14.1	7.6	15.8	5.6	0.33	−0.34	Small reduction
Workplace quality of life
Compassion satisfaction	38.8	5.1	38.7	4.7	40.2	4.7	38.9	4.9	38.8	5.2	0.86	0.08	Minimal change
Secondary trauma	23.1	5.6	22.9	5.5	22.5	5.0	20.3	5.4	21.5	5.0	0.12	−1.03	Large reduction
Burnout	23.6	4.2	24.3	4.3	23.2	5.1	22.3	5.7	22.6	4.2	0.78	−0.15	Minimal change
Mindful Healthcare scale
Total	57.2	5.1	58.6	4.5	58.8	6.5	58.9	6.3	56.7	5.1	0.09	0.75	Medium increase
Engagement	25.6	2.9	25.6	3.1	25.4	3.0	25.7	3.5	25.3	2.9	0.68	0.15	Minimal change
Awareness	21.7	3.8	21.5	3.0	21.7	3.8	20.9	3.7	20.8	3.7	0.75	0.20	Small increase
Defusion	9.9	3.7	11.5	3.1	11.7	2.7	12.3	3.4	10.6	4.3	0.09	0.64	Medium increase

Effect size calculated by comparing baseline with follow-up (comparison of groups with different sample sizes) https://www.psychometrica.de/effect_size.html. Completers only: All those who provided follow-up data (n = 12) compared to data for the same participants at baseline (n = 12). Medium to large effect size changes are highlighted in bold.

Preliminary evidence for effectiveness could also be interpreted in the qualitative data. No negative impacts were reported. Participants reported that they became more aware of their thoughts, emotions and feelings:

“I suppose one of the themes that ran through it was you’re not necessarily your thoughts and you don’t have to buy into them or to live by them, you can just notice them. So, all really helpful concepts. . .(FG5, P1)

Participants developed strategies to ‘unhook’ from unwanted thoughts and feelings and valued the practical nature of the exercises.

“I think probably the most helpful things [were] the kind of practical tips . . . ways to break those anxious thoughts or negative thoughts on a day-to-day basis” (FG3, P3)

The present moment awareness exercises helped participants to slow down, connect with the present moment, relax and unwind.

“I remember one day in work it was just chaotic . . .I sat down and my head was still buzzing, like I just couldn’t concentrate on anything. But then actually when I started sitting. . . eating, and reflecting on that, it definitely made a difference”(FG2, P2)

The self-care component highlighted the value of reducing self-criticism and creating opportunities to look after oneself.

“. . .I’ve started to be kinder to myself in those situations than I was before. . .”.(FG4, P2)

Barriers and facilitators to intervention engagement

Common barriers to engagement with the intervention were (i) time constraints and (ii) staff wellbeing not seen as a priority. For instance:

I have struggled to find the time to engage as much as I would have liked. I think setting aside dedicated time when I am able to devote my full attention to it in a quiet space has been difficult. . .. (QT1, P17).

Engagement was more difficult for staff who work shift patterns, or who needed colleagues to cover their regular duties:

“. . .because I was rostered on shift quite a lot of the time when we had the meetings and occasionally it would be too busy, there would be no staff to cover me, so I found that a bit of a barrier”. . . (FG5, P2)

Some participants felt guilty about taking time out for a wellbeing intervention given their current workloads and did not view their own wellbeing as a priority.

“ I think for me, what was a big barrier is the typical. . .or what I view as the typical. . . personality that goes into palliative care of caring for others, and not necessarily yourself and actually re-prioritising”. . . (FG5, P4)

Clinical demands resulted in prioritising patient care over staff wellbeing.

“Ultimately the top-down pressure is because of staffing so they can’t get time off the ward because there’s not enough of them to cover. . .certainly though, it [staff wellbeing] would never be seen as a priority. (FG2, P1)

Intervention engagement was facilitated by: (i) senior management support; (ii) protected time; (iii) flexible format and (iv) private space to engage in the intervention:

“We were very much supported by the senior (profession) team to take part in this. . . on the day making sure that . . .the service is covered. . . allowing us to get to sessions. . .” (FG6, 2)

Protected time, or the ability to plan your own time also made participation easier:

“I’m not on the ward all the time, my office is downstairs so I can ensure that time is protected. Like I can work my diary around it. . .” (FG2, P2)

The flexible format allowed participants to engage with the resources in a time that suited them – either in the workplace or at home. Having a private office or home space facilitated full participation in the online sessions. For those working in open offices, this was more difficult due to interruptions, and lack of privacy:

“I actually was working from home so I was removed from the office, no phone calls, no nothing, so that was actually really, really helpful” (FG5, P6)

Future implementation and evaluation

Most participants would have liked more opportunities to consolidate their learning, and allow more time to practice new strategies between workshops:

“It was great but to me actually it felt quite short and quite rushed and not giving me enough time to really consolidate some of the [techniques]”. . .(FG5, P5)

Many reported that they would like a top-up or follow-up session in the future:

‘It might be quite helpful to check in again at 3 months and say ok you know how is everybody and how have you got on?’ (FG1, P1)

Participants viewed the online questionnaires as onerous, with two participants referring to the process as ‘tortuous’!

“I would agree, they felt really long. . .it was a wee bit onerous, definitely”. (FG5, P6)

Although nearly all participants remained in the intervention, just under half (48%) completed the questionnaires at follow up. Despite reminders, questionnaire completion at follow-up was not a priority for staff and was often forgotten with time pressures and clinical priorities:

“it’s not a lack of enthusiasm for wanting to complete the questionnaire itself. . ., it’s just it being lost in a ream of other things. And forgetting, like when it comes through you may be getting emails of other things you need to prioritise in that moment so you see it and you think ‘I’ll go back and do that’ and then you forget to go back and do it” (FG4, P2)

Discussion

An online Acceptance and Commitment Therapy intervention to improve wellbeing was perceived as feasible and acceptable to palliative care staff. Once commenced, retention was good, and participants found RESTORE enjoyable, informative and beneficial. Data collection was feasible during the intervention but challenging at follow-up. Between baseline and follow-up there was preliminary evidence for improvements in psychological flexibility and mental wellbeing. Overall, these findings suggest that online Acceptance and Commitment Therapy shows promise as an accessible and flexible intervention that could support staff wellbeing in palliative care settings. Further intervention refinement and pilot-testing is warranted.

Improvements in psychological flexibility and overall mental wellbeing were not statistically significant. This was to be expected given the low sample size and emphasis on acceptability and feasibility over effectiveness. Many feasibility studies are not powered to detect significant differences in outcomes, rather the focus is on whether the intervention looks promising and can be evaluated.³¹ However, the general direction of change was aligned with improved wellbeing, and when combined with qualitative findings we judge that these findings support further intervention refinement and evaluation.

The intervention was acceptable to staff. Despite workload constraints, and intervention delivery during the COVID-19 pandemic (September–December 2021), recruitment was close to target and there was high staff retention, with staff reporting positive experiences and impacts relating to participation. Participants reported examples of increased contact with the present moment, greater likelihood to prioritise self-care and use of techniques to help them step back from overwhelming thoughts and feelings. Similar impacts were identified by dementia care workers, who reported greater present moment awareness, clearer focus on values, and changes in self-care behaviour following an Acceptance and Commitment Therapy intervention.³⁶

None of our participants were experiencing burnout at baseline pointing to the value of offering online Acceptance and Commitment Therapy as a protective measure, allowing them to develop skills that they could draw on in future challenging situations. Proactively preparing for challenging future situations, both at an individual and organisational level, may help mitigate experiences of psychological distress when a future crisis occurs.⁴⁷ Healthcare staff value the opportunity to access psychological support and learn techniques that support wellbeing. Those who have access to such supports feel valued and supported by their employer,^48,49 thus organisational-level benefits may accrue when wellbeing interventions are provided.

Our findings provide further evidence that online psychological support is acceptable to palliative care staff. Prior to the COVID-19 pandemic, there was already evidence for the utility of digital health interventions for education and training, with online learning and video-conferencing playing a critical role.²⁸ Cost-savings associated with reduced travel, alongside greater access to support were identified as benefits of online formats previously⁵⁰ and were found to facilitate participation in this study. Previously there was mixed views on the delivery of psychological support online, with some suggesting that face-to-face interventions are better for psychological concerns.⁵¹ However, staff in our study were comfortable discussing psychological issues and felt safe to share their views and experiences in online group session. A recent review showed that online Acceptance and Commitment Therapy improves mental health outcomes including anxiety, depression and psychological flexibility in a range of contexts, especially with therapist guidance.⁵² Given the scalability of online formats, continued work to adapt in-person formats for online delivery, so they can be accessed by more people, are essential.

Sustaining the effects of any intervention beyond the course of a research project is challenging.⁵³ Consideration of longer terms plans for intervention implementation is needed from the early stages of development.⁵⁴ Sustainable interventions should have perceived benefits or value to participants; they need to be embedded into the habitual practices of individuals, organisations and systems; and the intervention needs to evolve to fit with individual and organisational needs over time.⁵³ For successful implementation, clearly communicating the potential benefits of Acceptance and Commitment Therapy who to staff; identifying protected time to participate; and routinely collecting feedback to refine the intervention, will be essential. Leadership and contextual factors also influence sustainability.⁵³ We had strong senior manager and clinician support, yet there were still context-related barriers to engagement such as shift patterns, time constraints, and prioritisation of clinical priorities over staff wellbeing. Recognition of these tensions, and planning to prioritise staff participation early on, will likely optimise engagement and sustainability over time.

Implications for future intervention development and evaluation

In a follow-up study, the Warwick-Edinburgh Mental Wellbeing Scale^43,44 would be most useful to include as a main outcome, given that it showed greater sensitivity to change than the other measures examined. Participants would have welcomed more time to embed the new techniques in practice, so follow-up sessions or peer-support groups to sustain practice in the longer term will need to be built into intervention design. Furthermore, our intervention was delivered by an experienced Acceptance and Commitment Therapy trainer who led the online group sessions; future interventions need to be designed to allow delivery by organisational staff such as a Palliative Care Nurse Specialists, Doctors, Social Workers or Allied Health Professionals. To support upscaling, training for professionals to deliver the intervention will need to be developed and evaluated. Longer-term, there is potential for RESTORE to be adapted to support palliative care staff wellbeing in other countries: even though care systems may differ, the personal coping skills trained in RESTORE are likely to have commonalities across culture. As with any validation to other health care settings, this would require consideration of the intervention-context fit to inform judgements on transferability and decisions on the nature and extent of adaptations required.

Strategies to improve data collection at follow-up will be needed for future studies. In the present study, quantitative data collection fell at the 1-month post-intervention point. The fact that this study was conducted in the midst of the COVID pandemic, with intense pressures on staffing alongside clinical challenges, is likely to have influenced participants ability to prioritise completion of quantitative data at follow-up. Some suggestions to improve data collection at follow-up in the future include: (i) availability of a research nurse or assistant to prompt participants to complete follow-up data; (ii) encourage participants to add prompts to their diaries indicating when data needs to be returned; (iii) reducing the number of data collection points (e.g. removing the mid-study data collection point); (iv) consider incentivising questionnaire completion, for example providing a voucher or entry to a prize draw for those who complete outcomes at the follow-up. (v) Clearly communicate, and reiterate, the importance of the evaluation aspect of the study to participants, which may seen as of less value to participants than the act of taking part in the intervention. While much previous research has examined recruitment issues in palliative care settings, further research on effective strategies to support retention and follow-up beyond the course of an intervention are recommended.

Further research is needed to refine the intervention and evaluate its effectiveness compared with other forms of wellbeing support or no support. The effect of the intervention on outcomes such as stress-management self-efficacy, need also to be explored.⁵⁵ If effectiveness is established, there will be a need for an implementation study, examining how to implement and sustain the intervention when delivered in practice, including when it is delivered by non-psychologists. Finally, evaluating this intervention with health and social care workers in other emotionally charged work contexts may be fruitful.

Strengths and limitations

As far as we are aware, this is the first study to examine the use of Acceptance and Commitment Therapy to promote palliative care staff wellbeing. As a feasibility study, it was underpowered to detect statistically significant differences in quantitative outcomes. Yet, the use of mixed methods was a strength, and qualitative data provided strong evidence for acceptability and preliminary evidence of potential impact. As this was a pre-post study, we did not have a control condition, so we do not yet know how feasible it will be to recruit and evaluate participants for an RCT. As this study was funded by a small research grant, intervention fidelity was not independently assessed, and this is recommended in future studies. We collected data at four time-points, but under half completed quantitative measures at follow-up. This may have been due to participant burden as they had been asked to complete questionnaires and participate in a focus group, at a time when COVID-19 was still heavily impacting staffing. In future studies, identifying strategies to ensure data collection at follow-up will be essential.

Conclusion

An online Acceptance and Commitment Therapy intervention to improve palliative care staff wellbeing was acceptable, with preliminary evidence that participation may improve psychological flexibility and wellbeing. Mixed method evaluation was useful in identifying areas for improvement in the structure of the intervention and considerations for future efficacy studies. The data presented here suggest that such studies are worth pursuing.

Supplemental Material

sj-pdf-1-pmj-10.1177_02692163221143817 – Supplemental material for Feasibility of RESTORE: An online Acceptance and Commitment Therapy intervention to improve palliative care staff wellbeing

Supplemental material, sj-pdf-1-pmj-10.1177_02692163221143817 for Feasibility of RESTORE: An online Acceptance and Commitment Therapy intervention to improve palliative care staff wellbeing by Anne M Finucane, Nicholas J Hulbert-Williams, Brooke Swash, Juliet A Spiller, Brigid Wright, Libby Milton and David Gillanders in Palliative Medicine

Footnotes

Acknowledgements

We are very grateful to Julie Pearce, Chief Nurse Executive Director of Quality & Caring at Marie Curie for support and advice on intervention design and participant recruitment at the early stages of this project. We also thank Jean Lugton, research volunteer at Marie Curie Hospice Edinburgh, for proof-reading and commenting on the final manuscript.

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: AFs research fellowship is funded by Marie Curie. JS, LM and BL are employed by Marie Curie.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and publication of this article: This work was supported by a Marie Curie small grant (£9804) to AF and DG (7.02.20), and a small grant from the University of Edinburgh (£2177) to support the design and production of the RESTORE workbook (22.03.2).

Study registration

This study has been registered on the ISRCTN registry. No: SRCTN14313559

Study protocol

The full study protocol has been published on AMRC Open Research.

Data available statement

Given the potential for identification of study participants, full transcripts from qualitative interviews are not available – though further detail can be provided on request. The original protocol for this study had been referenced in this article and is available online. Requests for quantitative data access should be addressed to the corresponding author (AF) and will be reviewed and responded to.

ORCID iD

Anne M Finucane

Supplementary materials

Additional materials including the RESTORE workbook and manual can be located here:

References

Ablett

Jones

RSP

. Resilience and well-being in palliative care staff: a qualitative study of hospice nurses’ experience of work. Psychooncology 2007; 16: 733–740.

Fisher

Gillanders

Ferreira

The experiences of palliative care professionals and their responses to work-related stress: a qualitative study. Br J Health Psychol 2022; 27: 605–622.

Ingebretsen

Sagbakken

Hospice nurses’ emotional challenges in their encounters with the dying. Int J Qual Stud Health Well Being 2016; 11: 31170–31113.

Dijxhoorn

Brom

van der Linden

, et al. Prevalence of burnout in healthcare professionals providing palliative care and the effect of interventions to reduce symptoms: a systematic literature review. Palliat Med 2021; 35: 6–26.

Smith

Vasileiou

Kagee

Experiences of work-related stress and coping among palliative care staff in South Africa: a qualitative study. S Afr J Psychol 2020; 50: 425–437.

Powell

Froggatt

Giga

Resilience in inpatient palliative care nursing: a qualitative systematic review. BMJ Support Palliat Care 2020; 10: 79–90.

Heynemann

Philip

McLachlan

S-A.

An exploration of the perceptions, experience and practice of cancer clinicians in caring for patients with cancer who are also parents of dependent-age children. Support Care Cancer 2021; 29: 3895–3902.

McCloskey

Taggart

How much compassion have I left? An exploration of occupational stress among children’s palliative care nurses. Int J Palliat Nurs 2010; 16: 233–240.

Pappa

Ntella

Giannakas

, et al. Prevalence of depression, anxiety, and insomnia among healthcare workers during the COVID-19 pandemic: a systematic review and meta-analysis. Brain Behav Immun 2020; 88: 901–907.

10.

Ruiz-Fernández

Ramos-Pichardo

Ibáñez-Masero

, et al. Compassion fatigue, burnout, compassion satisfaction and perceived stress in healthcare professionals during the COVID-19 health crisis in Spain. J Clin Nurs 2020; 29: 4321–4330.

11.

Sarma

Vig

Rathore

, et al. Concerns of health care professionals managing non-COVID patients during the COVID-19 pandemic: a descriptive cross-sectional study. Indian J Palliat Care 2020; 26: S21–S26.

12.

Shahrour

Dardas

LA.

Acute stress disorder, coping self-efficacy and subsequent psychological distress among nurses amid COVID-19. J Nurs Manag 2020; 28: 1686–1695.

13.

Mills

Ramachenderan

Chapman

, et al. Prioritising workforce wellbeing and resilience: what COVID-19 is reminding us about self-care and staff support. Palliat Med 2020; 34: 1137–1139.

14.

Pastrana

De Lima

Pettus

, et al. The impact of COVID-19 on palliative care workers across the world: a qualitative analysis of responses to open-ended questions. Palliat Support Care 2021; 19: 187–192.

15.

Kates

Gerolamo

Pogorzelska-Maziarz

The impact of COVID-19 on the hospice and palliative care workforce. Public Health Nurs 2021; 38: 459–463.

16.

Hanna

Rapa

Dalton

, et al. Health and social care professionals’ experiences of providing end of life care during the COVID-19 pandemic: a qualitative study. Palliat Med 2021; 35: 1249–1257.

17.

Dean

Care at the end of life: the lessons of the first wave: Marie Curie-Nursing Standard survey highlights the emotional toll of the pandemic and the need for more staff support. Nurs Stand 2020; 35: 51–54.

18.

Hill

Dempster

Donnelly

, et al. Improving the wellbeing of staff who work in palliative care settings: a systematic review of psychosocial interventions. Palliat Med 2016; 30: 825–833.

19.

McInnerney

Candy

Stone

, et al. Access to and adequacy of psychological services for adult patients in UK hospices: a national, cross-sectional survey. BMC Palliat Care 2021; 20: 31.

20.

Prudenzi

Graham

Clancy

, et al. Group-based acceptance and commitment therapy interventions for improving general distress and work-related distress in healthcare professionals: a systematic review and meta-analysis. J Affect Disord 2021; 295: 192–202.

21.

Hayes

SC.

Acceptance and commitment therapy, relational frame theory, and the third wave of behavioral and Cognitive Therapies - republished article. Behav Ther 2016; 47: 869–885.

22.

Kent

Hochard

Hulbert-Williams

NJ.

Perceived stress and professional quality of life in nursing staff: how important is psychological flexibility?

J Contextual Behav Sci 2019; 14: 11–19.

23.

Harris

ACT made simple. 2nd ed. Oakland, CA: New Harbinger Publications, Inc, 2019.

24.

Gloster

Walder

Levin

, et al. The empirical status of acceptance and commitment therapy: a review of meta-analyses. J Contextual Behav Sci 2020; 18: 181–192.

25.

Han

Yuen

Jenkins

Acceptance and commitment therapy for family caregivers: a systematic review and meta-analysis. J Health Psychol 2021; 26: 82–102.

26.

Fashler

Weinrib

Azam

, et al. The use of acceptance and commitment therapy in oncology settings: a narrative review. Psychol Rep 2018; 121: 229–252.

27.

Hayes

Lillis

Acceptance and commitment therapy. Washington, D.C: American Psychological Association, 2012.

28.

Finucane

O’Donnell

Lugton

, et al. Digital health interventions in palliative care: a systematic meta-review. NPJ Digit Med 2021; 4: 64.

29.

Creswell

Plano Clark

VL.

Designing and conducting mixed methods research. London: SAGE Publications, 2018.

30.

Finucane

Hulbert-Williams

Swash

, et al. Research Evaluating Staff Training Online for Resilience (RESTORE): Protocol for a single-arm feasibility study of an online acceptance and commitment therapy intervention to improve staff wellbeing in palliative care settings. AMRC Open Res 2022; 3: 26.

31.

Jones

Olds

Currow

, et al. Feasibility and pilot studies in Palliative Care Research: a systematic review. J Pain Symptom Manag 2017; 54: 139–151.e4.

32.

Wardley

MNJ

Flaxman

Willig

, et al. ‘Feel the feeling’: psychological practitioners’ experience of acceptance and commitment therapy well-being training in the workplace. J Health Psychol 2016; 21: 1536–1547.

33.

Mesquita Garcia

Domingues Silva

Oliveira da Silva

, et al. Self-compassion in hospice and Palliative Care: a systematic integrative review. J Hosp Palliat Nurs 2021; 23: 145–154.

34.

Arch

Fishbein

Ferris

, et al. Acceptability, feasibility, and efficacy potential of a multimodal acceptance and commitment therapy intervention to address psychosocial and advance care planning needs among anxious and depressed adults with metastatic cancer. J Palliat Med 2020; 23: 1380–1385.

35.

Muscara

McCarthy

Rayner

, et al. Effect of a videoconference-based online group intervention for traumatic stress in parents of children with life-threatening illness: a randomized clinical trial. JAMA Netw Open 2020; 3: e208507.

36.

Montaner

Tárrega

Pulgarin

, et al. Effectiveness of acceptance and Commitment Therapy (ACT) in Professional dementia caregivers burnout. Clin Gerontol 2022; 45: 915–926.

37.

Francis

Dawson

Golijani-Moghaddam

The development and validation of the comprehensive assessment of acceptance and Commitment Therapy processes (CompACT). J Contextual Behav Sci 2016; 5: 134–145.

38.

Muñoz-Martínez

Otto-Scheiber

Zuluaga-Jassir

, et al. Scalability of an ACT-Based strategy for improving well-being in health care providers: a mix-method and preliminary evaluation of efficacy. J Contextual Behav Sci 2022; 25: 136–144.

39.

Trindade

Ferreira

Mendes

, et al. Comprehensive assessment of acceptance and Commitment Therapy processes (CompACT): measure refinement and study of measurement invariance across Portuguese and UK samples. J Contextual Behav Sci 2021; 21: 30–36.

40.

Cohen

Kamarck

Mermelstein

A global measure of perceived stress. J Health Soc Behav 1983; 24: 385–396.

41.

Cohen

Janicki-Deverts

Who’s stressed? Distributions of psychological stress in the United States in probability samples from 1983, 2006, and 2009. J Appl Soc Psychol 2012; 42: 1320–1334.

42.

Stamm

BH.

The concise proqol Manual. 2nd ed. Pocatello: https://ProQol.org, 2010.

43.

Stewart-Brown

Tennant

, et al. Internal construct validity of the Warwick-Edinburgh mental well-being scale (WEMWBS): a Rasch analysis using data from the Scottish Health Education Population Survey. Health Qual Life Outcomes 2009; 7: 15–15.

44.

Tennant

Hiller

Fishwick

, et al. The Warwick-Edinburgh mental well-being scale (WEMWBS): development and UK validation. Health Qual Life Outcomes 2007; 5: 63–63.

45.

Maheswaran

Weich

Powell

, et al. Evaluating the responsiveness of the Warwick Edinburgh mental well-being scale (WEMWBS): group and individual level analysis. Health Qual Life Outcomes 2012; 10: 156.

46.

Kidney

Acceptance and commitment therapy training and psychological flexibility for helping professionals. Doctoral Thesis, The University of Edinburgh, Edinburgh, 2018. https://era.ed.ac.uk/handle/1842/33107

47.

Porter

Zile

Peryer

, et al. The impact of providing end-of-life care during a pandemic on the mental health and wellbeing of health and social care staff: Systematic review and meta-synthesis. Soc Sci Med 2021; 287: 114397–114397.

48.

Blake

Gupta

Javed

, et al. COVID-well study: qualitative evaluation of supported Wellbeing Centres and psychological first aid for healthcare workers during the COVID-19 pandemic. Int J Environ Res Public Health 2021; 18: 3626.

49.

Blake

Yildirim

Wood

, et al. COVID-well: evaluation of the implementation of supported Wellbeing Centres for hospital employees during the COVID-19 pandemic. Int J Environ Res Public Health 2020; 17: 9401.

50.

Jess

Timm

Dieperink

KB.

Video consultations in palliative care: A systematic integrative review. Palliative Medicine 2019; 33: 942–958.

51.

Kidd

Cayless

Johnston

, et al. Telehealth in palliative care in the UK: a review of the evidence. J Telemed Telecare 2010; 16: 394–402.

52.

Thompson

Destree

Albertella

, et al. Internet-based acceptance and Commitment Therapy: a transdiagnostic systematic review and meta-analysis for mental health outcomes. Behav Ther 2021; 52: 492–507.

53.

Fleiszer

Semenic

Ritchie

, et al. The sustainability of healthcare innovations: a concept analysis. J Adv Nurs 2015; 71: 1484–1498

54.

Skivington

Matthews

Simpson

, et al. Framework for the development and evaluation of complex interventions: gap analysis, workshop and consultation-informed update. Health Technol Assess 2021; 25: 1–132.

55.

Jin

SAA

. The effects of incorporating a virtual agent in a computer-aided test designed for stress management education: the mediating role of enjoyment. Comput Human Behav 2010; 26: 443–451.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.70 MB