Which outcome domains are important in palliative care and when? An international expert consensus workshop,using the nominal group technique

Abstract

Background:

When capturing patient-level outcomes in palliative care, it is essential to identify which outcome domains are most important and focus efforts to capture these, in order to improve quality of care and minimise collection burden.

Aim:

To determine which domains of palliative care are most important for measurement of outcomes, and the optimal time period over which these should be measured.

Design:

An international expert consensus workshop using nominal group technique. Data were analysed descriptively, and weighted according to ranking (1–5, lowest to highest priority) of domains. Participants’ rationales for their choices were analysed thematically.

Setting/participants:

In all, 33 clinicians and researchers working globally in palliative care outcome measurement participated. Two groups (n = 16; n = 17) answered one question each (either on domains or optimal timing). This workshop was conducted at the 9th World Research Congress of the European Association for Palliative Care in 2016.

Results:

Participants’ years of experience in palliative care and in outcome measurement ranged from 10.9 to 14.7 years and 5.8 to 6.4 years, respectively. The mean scores (weighted by rank) for the top-ranked domains were ‘overall wellbeing/quality of life’ (2.75), ‘pain’ (2.06), and ‘information needs/preferences’ (2.06), respectively. The palliative measure ‘Phase of Illness’ was recommended as the preferred measure of time period over which the domains were measured.

Conclusion:

The domains of ‘overall wellbeing/quality of life’, ‘pain’, and ‘information needs/preferences’ are recommended for regular measurement, assessed using ‘Phase of Illness’. International adoption of these recommendations will help standardise approaches to improving the quality of palliative care.

Keywords

Palliative care quality indicators patient outcome assessment patient reported outcome measures consensus

What is already known about the topic?

Measures of outcomes can help determine the difference that palliative care interventions can make. However, they are challenging to capture due to the deteriorating health of patients receiving palliative care.

The outcomes important to patients cover a range of person-centred domains that can be hard to quantify (e.g. social and cultural), and no consensus on the most important domains has been reached.

A more uniform approach to outcome measures is needed to improve the quality of care across palliative care services. It is necessary to identify which domains are most important in capturing patient-level outcomes, while minimising the burden of collection for patients and staff.

What this paper adds?

This paper provides insight into the outcome-based domains that palliative care clinicians and researchers recommend for regular measurement.

‘Overall wellbeing/quality of life’, ‘pain’, and ‘information needs and preferences’ are recommended for regular measurement, with ‘Phase of Illness’ proposed as most useful to measure the time period.

Collated overview of important outcome-domains from the experts’ point of view.

Implications for practice, theory or policy

International adoption of these recommendations will help standardise approaches to improving the quality of palliative care.

The key domains to measure are ‘overall wellbeing/quality of life’, ‘pain’, and ‘information needs/preferences’. Each domain needs to be measured over each palliative ‘Phase of Illness’ to allow for national and international comparability.

It is important to provide training at all levels to ensure reliable application of palliative ‘Phase of Illness’.

Background

Measuring how a person with advanced illness is affected by symptoms over time (i.e. at more than one time point) can demonstrate the difference that palliative care interventions make,¹ especially when contrasted with change over time without the intervention. Donabedian² defined an outcome as a ‘change in current or future health status attributable to a preceding healthcare intervention’. A change in health status in the palliative care population is, for instance, an improvement or a worsening of a symptom (e.g., pain or breathlessness).³ Outcomes in palliative care are not easy to capture, as patients living with advanced illness and receiving palliative care are steadily declining in health, often too ill to self-report outcomes, and the outcomes most important to them cover a range of person-centred domains. This requires interpretation of the extent a person is ‘concerned by’ a symptom or issue, which may be hard to quantify, particularly for individuals no longer able to self-report.^4,5

The recent white paper on outcome measures produced by the European Association for Palliative Care (EAPC) taskforce recommends adoption and implementation of uniform outcome measures to improve the quality of care across palliative care services.^4,6–8 In the Outcomes Assessment and Complexity Collaborative (OACC), we have successfully developed and implemented a core set of outcome measures for palliative care in the United Kingdom. The project (conducted in collaboration with Hospice UK and the Collaboration for Leadership in Applied Health Research and Care South London (CLAHRC)) enabled implementation of standardised person-level outcomes collected at point of care,⁹ both in core OACC sites and the extended OACC network (200 + providers across the United Kingdom). A similar, but more established initiative is the Palliative Care Outcome Collaboration (PCOC) in Australia,⁸ which utilises standardised clinical assessment tools to measure and benchmark patient reported outcomes.¹⁰ The PCOC team is measuring and benchmarking patient-reported outcome measures (PROMS) at individual patient-level within 127 services across Australia, identifying the need to improve aspects of services and achieving quality improvement.^7,11 Both the OACC and PCOC teams have worked together, benefitting from each other’s experiences.¹²

Specific outcomes are important for palliative care for a variety of reasons: to improve quality of care,¹³ to demonstrate whether services are achieving their intended goals,⁸ to establish the effectiveness and cost-effectiveness of care¹⁴; and to evaluate new services or interventions.¹⁵ Over the last few decades, there has been extensive research into patient perspectives on the important domains in advanced disease,^16–19 but relatively little consensus-building about how to prioritise domains among those working to introduce outcome measurement into routine palliative care practice. A further challenge in routine measurement is how to define the period of time over which to measure outcomes, in order to compare patient-centred outcomes across services.⁴ The Organisation for Economic Co-operation and Develop-ment (OECD)²⁰ defines a time period as ‘the actual unit in which the associated values are measured’. This consensus workshop therefore aimed to determine–from experts working to implement outcomes measurement–which domains of palliative care are most important for measurement of outcomes, and the optimal time period over which these outcomes should be measured. This workshop was a joint endeavour between the OACC and PCOC.

Methods

Design

The workshop adopted a nominal group technique (NGT), a highly structured method used for producing ideas and identifying solutions within groups, with the intention of generating recommendations for best practice.^21,22 This method has been used recently, and successfully, to ask a range of critical questions to experts (patients, families, people from the public, and stakeholders) within a short period of time.^21–23 It is considered feasible for similar palliative care research questions.^22–25

Identification of experts and eligibility for participation

Participants were identified through OACC and PCOC networks, through authorship of the European White Paper on Outcome Measurement,⁴ and through screening of oral presentation titles for the 9th World Research Congress of the EAPC 2016, where this workshop was conducted. All participants had to be (1) working in palliative care (research and/or clinical) and (2) actively using outcome measures or scientific publications about outcome measures.

Setting and participants

Potential participants were invited via email. Another four participants received information through word of mouth or expressed interest in outcome measurement to the study team, and fulfilled eligibility criteria. The workshop was a closed session lasting approximately 90 min. Participants were in groups of 6–10 in line with recommended sizes of focus groups, optimising facilitated discussion flow.^21,26

Workshop preparation and conduct

The participants received and provided specific information before the NGT rating exercise to inform and develop the considered questions. In advance of the workshop, they were asked to provide information about outcome measures and any tools currently used in their practice to measure time periods. This information, plus literature scoping, informed the potential domains presented and considered in the workshop. Second, they received information about the process of NGT (Box 2) as well as the critical questions to be addressed in the workshop (Box 1), a week in advance via email. Two short 5-min presentations were held on the day to re-visit the questions under consideration and reiterate the workshop aim to participants. The first presentation defined outcomes in the context of palliative care and reviewed possible domains to be considered, including a collated list from participants of all the outcome measures, plus details of all time periods for these domains currently in use internationally and retrieved from the literature. The second presentation covered practical examples of outcome measures, in order to help define and clarify outcomes, and domains for the purpose of this workshop. Participants were divided by geographical area and experience in palliative care and outcome measures. Four groups were consequently formed (2 groups to answer question 1, and 2 groups to answer question 2).

Box 1.

Two critical questions, each answered by two groups:

Question 1:
Do the outcomes presented to you cover the right domains of palliative care; where are the gaps?
Question 2:
Are the outcomes presented to you measured in the right time period or not?

Box 2.

Steps during the workshop following the nominal group technique:

1. Without discussion or conferring, the groups on each table were asked to individually write down their top five (rating from 1–5; 1 equals top) outcome domains or preferred time period (according to which question they were addressing).
2. Each participant then verbally shared their priorities and explained their rationale in turn around the table.
3. The facilitator combined similar statements and removed duplicates to facilitate discussion, presenting these back to the group in refined form.
4. The group then discussed together (30 min) the rationale and reasoning behind each individual’s choice, and all domains/units were compared and contrasted in discussion.
5. Without discussion or conferring further, every person in the group was then asked to individually re-consider and re-rank their top five outcome domains or defined time periods again, writing this down on paper.
6. The facilitators then shared a brief summary from each table with the floor.

Each group had a facilitator experienced in NGT, and two scribes to capture the rationale for choices and narratives of the discussions. The steps taken by the groups in answering question 1 or question 2 are outlined in Box 2.

The workshop concluded with a vote from each participant for their top outcome domain, indicating simply a first and most important recommendation.

Analysis

Individual rankings of question 1, as described in Box 2 step 5, were transferred into an Excel spreadsheet for analysis. Any outcome domains, ranked by participants as one of their top five, was included in analysis. The weighted mean score was calculated by adding the scores for each rank (rank 1 = 5 points, rank 2 = 4 points, etc.) divided by the number of participants in a group, so higher mean scores represent higher ranking and choice among the expert group. The findings from the second question (time period) were analysed descriptively. Rationales and narratives for question 2, collected by two scribes, were analysed thematically on a semantic level. SdW collated and ordered the data according to themes, in order to enable comparison of comments from participants. The reporting of the qualitative aspects follows the Consolidated Criteria for Reporting Qualitative Studies (see Appendix 1).²⁷

Ethical considerations

As the workshop involved participation from professionals, ethical approval was sought in accordance with King’s College London guidelines for research with professionals. Ethical approval was received prior to the workshop (LRS-15/16-2954). Written informed consent was gained from all participants before the start of the workshop.

Results

Demographics

The study team approached 50 experts, and received 29 positive replies confirming attendance, 11 apologies, 9 non-responders, and 1 ‘undeliverable’. Four additional participants were included after they approached us through ‘word of mouth’ or expressed interest. Altogether, 33 clinicians and researchers (22 female, 11 male) working in palliative care globally took part in the workshop. They all came from different professional backgrounds with medical doctors most prominent (n = 20) but also nurses (n = 6) and others (statisticians and researchers from the public health sector; n = 6) were represented. In all, 30 participants were from European countries (GER = 7, SWE = 2, UK = 8, BEL = 3, IRL = 4, ITA = 1, FRA = 1, NOR = 1, DNK = 3), two participants were from Australia, and one participant was from South Africa. A total of 16 participants answered question 1 and 17 participants answered question 2. The experts answering question 1 had more years of experience in palliative care (16.5 years’ experience (median)) than the experts answering question 2, but were less experienced in using outcome measures (4.25 years of experience using outcome measures (median)) (Table 1).

Table 1.

Characteristics of participants addressing question 1 and 2.

Characteristics of participants addressing question 1 (n = 16)
	Years of experience in palliative care	Years of experience in outcome-measures
Mean	14.7 years	5.8 years
Median	16.5 years	4.25 years
Range	1–26 years	1–15 years
Missing	0	2
Characteristics of participants addressing question 2 (n = 17)
	Years of experience in palliative care	Years of experience in outcome measures
Mean	10.9 years	6.4 years
Median	10 years	4.5 years
Range	2–20 years	0–20 years
Missing	2	1

Question 1: ‘do the outcomes presented to you cover the right domains of palliative care? where are the gaps?’

Step 3 of the NGT: discussion in groups following individual ranking exercise

‘Overall wellbeing and quality of life’ was identified as the most important domain. Participants recommended that the patient and family are defined as the unit of care, and participants stressed preferences from both patients and families need to be listened to and acted upon as a priority.

The care quality of older patients was critical as participants discussed this as the biggest, emerging group in need and therefore bespoke outcomes such as pain, overall emotions (including loneliness etc.) would need to be chosen wisely. In particular, participants were concerned that patient reported outcomes would not be measured on a continuous basis because many older people would not have a support network who could help them complete measures at follow-up.

Step 4 of the NGT: individual re-ranking after discussion

Participants felt that the outcomes presented covered the right domains in palliative care, with an added outcome of ‘staff distress’ proposed by one participant. Table 2 details the proposed outcome domains. The highest domains–using weighted mean scores–comprised overall wellbeing and quality of life (2.75) and pain and information needs had equal mean weighted score (2.06).

Table 2.

Illustration of individual re-ranking after discussion: Preferred outcome domains of participants in question 1 (n = 16).

Category / Items	Participants 1–16																Weighted mean score ^a
Overall wellbeing	P1	P2	P3	P4	P5	P6	P7	P8	P9	P10	P11	P12	P13	P14	P15	P16
Overall wellbeing and quality of life	3		3	5		5	5		5		5	5		3		5	2.75
Personal perception of wellbeing		5															0.31
Cognitive dysfunction											4						0.25
Physical wellbeing	P1	P2	P3	P4	P5	P6	P7	P8	P9	P10	P11	P12	P13	P14	P15	P16
Pain and pain reduction of two points on a VAS/NRS 0–10		5	5	5				5				4	5	4			2.06
Breathlessness and Breathlessness reduction of two points on a VAS/NRS 0–10		5	5									4	5				1.18
Fatigue		4		5				5							4		1.13
Nausea and vomiting										3							0.18
Physical symptoms in general					5	4	5		4		5					4	1.68
Emotional Wellbeing	P1	P2	P3	P4	P5	P6	P7	P8	P9	P10	P11	P12	P13	P14	P15	P16
Overall emotions including loneliness	5				5	4	5		4		5						1.75
Feeling safe in institution										4							0.25
Depression/psychological care															3	4	0.43
Social and Family Wellbeing	P1	P2	P3	P4	P5	P6	P7	P8	P9	P10	P11	P12	P13	P14	P15	P16
Family anxiety and wellbeing		5				2							5				0.75
Family carer burden			4				5		4		4						1.06
Social care									2								0.13
Relationship with family including sharing feelings												3			1		0.25
Spiritual wellbeing	P1	P2	P3	P4	P5	P6	P7	P8	P9	P10	P11	P12	P13	P14	P15	P16
Feeling at peace												2			2		0.25
Information and preferences	P1	P2	P3	P4	P5	P6	P7	P8	P9	P10	P11	P12	P13	P14	P15	P16
Communication (feeling listened to, shared decision making)	4									5	5					4	1.13
Place of care – choice of place of care – treatment preferences		5	3														0.5
Information needs and preferences (patient and family)		5		5	5	4	5	5						2		2	2.06
Adverse Events and Staff distress	P1	P2	P3	P4	P5	P6	P7	P8	P9	P10	P11	P12	P13	P14	P15	P16
Adverse events including medication adherence and pressure ulcer	1				4			4									0.56
Length of unstable phase			4					4			4		5	5	5		1.69
Timing and duration of intervention									4								0.25
Staff distress													1				0.06

VAS: visual analog scale; NRS: numerical rating scale.

Ranks are weighted (rank 1 = 5 points, rank 2 = 4 points, rank 3 = 3 points etc.). The mean score is calculated by dividing the weighted sum with the total number of participants (n = 16).

Question 2: ‘are the outcomes presented to you measured in the right time period or not?’

Step 3 of the NGT: thematic analysis of group discussion

The recommendation to use the palliative measure of ‘Phase of Illness’²⁸ to capture and report the time period over which change in health status occurred was discussed. Participants discussed continued education and teaching for staff about the measure, to support consistency in its use. This was considered paramount in order to overcome the cultural change within an organisation when beginning to measure ‘Phase of Illness’. One participant felt education around ‘Phase of Illness’ needed to be consistent internationally as well:

Phase of Illness’ is good, if we have international agreement, thorough and repeated training, and education on ‘Phase of illness’. (Participant 1)

A lot of thinking surrounded the clinical context, as well as the resources available to implement and use ‘Phase of Illness’ successfully. The meaning of case-mix adjustment was debated and then recommended to include as the first step, in order to move towards the decision that ‘good quality care’ was being reflected in any outcome measure findings. One participant related ‘Phase of Illness’ together with the patients’ complexity:

‘Phase of Illness’ meets the individual patient situation on complexity. Length of particular phases of illness would need to be included (i.e. length of unstable phase). (Participant 8)

Participants illustrated using ‘Phase of Illness’ as the anchor point, providing context about the acuity and urgency of the patient’s needs, which then triggered ideas about how to support clinical practice and compare the data between services. One participant highlighted the ease of use of ‘Phase of Illness’ and the importance of the validation of such a measure in the palliative care population:

To avoid positive and negative measures and to focus on good assessments to help clinical practice with having ‘anchor points’ at start. (Participant 17)

Measuring the time period with the ‘Phase of Illness’ measure can be stated as a recommendation. However, the discussion around education, context, and consistency of its use needs to be continued as shown in Figure 1.

Figure 1.

Synthesis of comments in relation to question 2.

Step 4 of the NGT: individual re-ranking after discussion

Participants rated ‘Phase of Illness’ as the preferred time period measure for the measurement of outcomes. Most participants felt that ‘Phase of Illness’ was a good universal measure but some items under the domain ‘adverse events’ (i.e. falls) were proposed to be used with a different measure; as ‘falls’ do not necessarily relate to the palliative care phase. An important suggestion was that patient outcomes may be measured with different time period measures, for instance, ‘last week of life’ instead of ‘Phase of Illness’, according to which domain is considered (Table 3). One challenge raised for international benchmarking was to ensure comparability in the time period measure.

Table 3.

Frequency table referring to preferred ‘time period’ reported by participant addressing question 2 (n = 17).

Time period	After 24 hours	After 48 hours	After 72 hours	Over ‘Phase of Illness’	Over End of Life Care admission	Over Last week of life	Over Episode of symptom occurrence	Over a period of high symptom severity (VAS 0–10)	Comments
Symptom
Physical
Pain (Average pain < 3)	2	2		7	1	2			Needs clear definition of starting and finish point of measurement.Pain needs to be controlled as soon as possible.
Breathless-ness		2		6		1			Distinction between palliative care and end-of-life care.
Fatigue				7		1			Realistic and lower benchmark for fatigue (i.e. 70%)
Death rattle			1			5			Only applicable to end-of-life care. No benchmarking feasible.
Agitation				5		2
Psychological
Anxiety			1	3		2			Difference in scores best visible in ‘Phase of Illness’ length.
Depression				7		1
Spiritual
Feeling at peace				6		1
Social
Family anxiety				5		1			Outcome measure is dependent on staff to complete as patient not always able to report.
Sharing feelings with family				5		1
Family/carer problems				5		1		1
Practical problems				3
Quality of life (overall), information needs, treatment preferences, adverse events and length of phase
Quality of life				6		1		1
Information needs				1
Treatment preferences				1			1		Requires clear guidance on time point of measurement
Most important symptom for patients				1					‘Time period measure may change in line with the symptom/problem.
Adverse events (e.g. falls)				2			1		Requires very clear guidelines regarding starting and ending the time period.
Length of time in unstable phase				3					Should be analysed together with symptom pain.
Palliative care			1						Experience measure to be included
Percentage of patients becoming unstable once in receipt of palliative care				1

VAS: visual analog scale; NRS: numerical rating scale.

Recommendation on outcome domains by all participants

Ranking of the top three outcome domains

After the floor discussion, participants were invited to vote for their top outcome domain by a show of hands. This vote showed consensus that, in order to attribute outcomes to the care provided, ‘quality of life’ and ‘overall wellbeing’ need to be measured continuously over each period of care.

Discussion

This workshop reached expert consensus from international and multi-disciplinary perspectives, on some of the most important outcome domains in palliative care, with the highest scores for patients’ and families’ quality of life and overall wellbeing, pain, and information needs/preferences. This compliments and adds to existing evidence from patients themselves about what is important to them.¹⁹ With regard to the optimal timing of measurement, palliative ‘Phase of Illness’ was discussed as the preferred measure for a time period–reflecting acuity and urgency of care needs–but the appropriate period of time for outcome measurement may vary according to the outcome domain being assessed.

Towards a uniform outcome measure

There was strong agreement among experts on which outcome measures should be used; implying the profession and specialty is moving towards uniform outcome measurement. The expert group reported that domains of quality of life, overall wellbeing, cognitive dysfunction, physical symptoms, emotional and psychological wellbeing, family health, spiritual issues, and autonomy, information and preferences were all important. This concurs with findings from a systematic review of patient perspectives by McCaffrey et al.,¹⁹ which found similarly that physical symptoms and function, emotional wellbeing, social domains, spirituality, cognition, and preparation for death are all prioritised by those with advanced illness themselves.

Awareness by professionals of outcome measures has changed since Dawson et al.²⁹ found that PROMS are rarely used in clinical healthcare settings. For instance, the use of the Palliative care Outcome Scale (POS) and Support Team Assessment Schedule (STAS) in different countries and settings was evident shortly after Dawson’s publication,³⁰ and increased over time as the result of translations to other languages, as well as the introduction of use in non-cancer palliative care patients according to a recent systematic review.³¹ Experts stated clearly the importance of a multi-dimensional outcome measure (such as POS or Integrated Palliative Care Outcome Scale (IPOS), European Organisation for Research and Treatment of Cancer (EORTC), or Edmonton Symptom Assessment System (ESAS)) to elicit the individual needs of patients and their families, and which follow recommendations from the EAPC taskforce on outcome measures with regard to psychometric properties.^4,32 Furthermore, it would help to reduce reliance on process-based measures, as these measures address patient-centred outcomes. Clark et al.³³ stated that by embedding objective measurements of quality into routine practice before implementing outcome measures, palliative care services risked relying on process measures rather than PROMS and therefore were not considering patients’ needs and experiences. Interestingly, this was congruent with our information sent to participants prior to the workshop, to gather information about outcome measures in use. Many replied with a list of process-based measures, which were clarified with the participants at the beginning of the workshop.

Phase of illness is an emerging measure about period of time

PCOC and OACC agreed on the same period of time measure throughout their projects, which is palliative ‘Phase of Illness’³⁴ reflecting palliative care phases of stable, deteriorating, unstable, and dying for patients and their families.^8,28 A change in ‘Phase’ represents a change in the person’s clinical condition and/or a change in the patient’s/family well-being.^28,34 Both changes lead to a change in the patient’s care plan.

Prominent consistency and agreement among participants occurred, with the majority selecting ‘Phase of Illness’ as the time period measure of choice. This result may have been biased due to the founder of ‘Phase of Illness’ and its development participating in the workshop.³⁵ However, the palliative care ‘Phase of Illness’ embraces clinical needs without acknowledging the patient’s diagnosis and prognosis. This makes it a very useful and simple way to state the patient’s current situation, and therefore so appealing to the participants.⁷ The unstable phase, was recommended as a potential quality indicator by correlating the length of the unstable phase, including the patient’s complexity (a shorter unstable phase particularly reflects an improved outcome for patients, as this enables better quality time, when time may be limited). We recognise that ‘Phase of Illness’ needs further research, particularly into validity and reliability in a wider range of settings.³⁶ However, our finding from the expert group does imply some consensus on using a measure – like ‘Phase of Illness’ – which reflects the acuity and urgency of a patient’s needs.³⁴

It was important to get as many experts as possible to answer these questions by consensus. With the consensus method, challenges can be identified and best practice is discussed, with new directions to identify patients’ needs within a very short period of time.²⁵ Hence, questions like these – which need answering from a clinical and research context collaboratively – can be more usefully addressed using a consensus method.³⁷

Limitations

This piece of research has several limitations. The major limitation is this study did not have any patient and public involvement advising on the most important outcome domain despite its importance to improve the quality of research.³⁸ However, as discussed, there is some congruence of findings with priorities identified in a recent priority setting partnership³⁹ and with prior evidence. Patients, their families, and people from the public will be invited when taking these findings to individual services.²¹ One bias is the study sample, as we only invited experts in the field of palliative care and outcome measurement, and only those attending the 9th World Research Congress of the European Association for Palliative Care 2016 could attend. Importantly, a number had been involved in the EAPC taskforce on outcome measures and initiatives such as OACC and PCOC, which may have inhibited expression of a broad spectrum of views or divergent and more critical views in the workshop. Finally, no clinical frontline staff such as nurses, physiotherapists or chaplaincy, attended this workshop because of the research focus on that year’s EAPC congress.

Conclusion

The domains of ‘overall wellbeing/quality of life’, ‘pain’, and ‘information needs/ preferences’ are recommended for regular measurement by palliative care clinicians and researchers, with change in urgency and acuity of palliative care needs – as measured by palliative ‘Phase of Illness’ – proposed as the optimal time period over which to measure change in the domains. These experts felt that timing should be determined by clinical presentation and acuity of needs, not by fixed periods of time. Training in use of ‘Phase of Illness’ and international adoption of these recommendations will help standardise approaches to using outcome measures and improving the quality of palliative care.

Footnotes

Appendix

Appendix 1.

Consolidated criteria for reporting qualitative studies (COREQ): 32-Item Checklist (Tong et al.²⁷).

Domain 1: Research team and reflexivity
1. Interviewer	SdW, MD, CJE, FM facilitated the discussion on the tables (1 table each).
2. Credentials	MSc Palliative Care, BScN Hons, PhD – Senior lecturer, Reader – Senior lecturer, Professors.
3. Occupation	Researcher in palliative care, part time clinicians
4. Gender	All female
5. Experience and training	CJE, FM, KE and IJH have substantial research experience. They are all senior lecturers teaching research methods, while conducting their own cutting edge research in palliative care. SdW, MD, SP, and FW are students of the senior authors above, closely supervised throughout the conduct of this study. All authors have academic credentials.
6. Relationship with participants	A relationship has been established via email prior to conducting this workshop.
7. Participants knowledge	All participants have been informed about the seniority and goal of the researchers in a formal letter sent by email.
8. Interviewer characteristics	Assumption and potential bias with regards to phase of illness measurement tool (founder was part of the research team and present in the room) is addressed in the paper.
Domain 2: study design
9. Methodological orientation and Theory	The method of conduct of this workshop was an adapted nominal group technique. Documentation from scribes were analysed thematically.
10. Sampling	It was a mixture of purposive and snowball sampling. We invited experts in the field and those we were unable to reach via email, for example, heard from others and asked us if they could participate. Sampling is well described in the paper.
11. Method of approach	Participants were approached via email.
12. Sample size	33 Palliative Care clinicians and researchers
13. Non-participation	11 apologised, 9 non-responders, one email was not delivered, and four additional participants were included as they approached us and fulfilled the inclusion criteria.
14. Setting of data collection	The workshop was conducted at the 9th World Research Congress of the EAPC in 2016.
15. Presence of non- participants	We had help from administrative members of staff of which 2 were present, looking after the well-being of participants.
16. Description of sample	Important characteristics of the sample are years of experience in palliative care and outcome measures, as the research question asked for an opinion based on experience.
17. Interview guide	The four facilitators were following the steps to conduct the workshop on each table as outlined in Box 2 in the paper.
18. Repeat interviews	No repeat interviews were carried out as they were not appropriate.
19. Audio/visual recording	No audio or visual recordings were used for data collection.
20. Field notes	No field notes were taken, however scribes were assigned to document the conversations at each table in detail.
21. Duration	The duration of the workshop was 90 min including two presentations at the beginning.
22. Data saturation	Participants were given time to discuss their point of views in detail. The facilitator made sure that all the inputs were discussed and the participants had no others to propose (see outline of discussion in Box 2).
23. Transcripts returned	Transcripts from facilitators, participants, and scribes were collected at the end of the workshop for thematic analysis.
Domain 3: analysis and findings
24. Number of data coders	SdW coded the data – there was no second coder but the codes and analysis were circulated among the authors for accuracy checking.
25. Description of the coding tree	Not appropriate as the codes supported one question as described in the analysis section.
26. Derivation of themes	Themes were derived from the data originating from the discussion.
27. Software	Not applicable
28. Participant checking	No participant checking occurred, see also point number 24.
29. Quotations presented	Quotations are presented in the manuscript to illustrate the theme including participant number.
30. Data findings consistent	Yes, we tried to present the data consistent with the findings.
31. Clarity of major themes	Major themes are presented in Figure 1 as well as in words and repeated in the conclusion.
32. Clarity of minor themes	Yes, minor themes are presented as well, particular with regards to the time period of when to measure outcome domains.

Acknowledgements

Our sincere thanks goes to all the participants of the workshop. The OACC project is led by the Cicely Saunders Institute, King’s College London. It is funded by the Guy’s and St Thomas’ Charity and supported by project BuildCARE. OACC is supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care: South London. The Collaboration for Leadership in Applied Health Research and Care (CLAHRC) South London is part of the National Institute for Health Research (NIHR), and is a partnership between King’s Health Partners, St. George’s, University London, and St George’s Healthcare NHS Trust.

Author’s Note

The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the National Health Service, the National Institute of Health Research, Medical Research Council, Central Commissioning Facility, NIHR Evaluation, Trials and Studies Coordinating Centre, the National Institute of Health Research Programme Grants for Applied Research, or the Department of Health and Social Care.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The C-CHANGE project is funded by the National Institute for Health Research under the Programme Grants for Applied Research (project number RP-PG-1210-12015 – C-CHANGE: Delivering high-quality and cost-effective care across the range of complexity for those with advanced conditions in the last year of life).

ORCID iD’s

de Wolf-Linder, S.

Murtagh, F.E.M.

References

Harding

How and what do you measure to ensure quality palliative and end-of-life care? EAPC blog, Vol. 2017, https://eapcnet.wordpress.com/2017/07/11/how-and-what-do-you-measure-to-ensure-quality-palliative-and-end-of-life-care/ (accessed 29 May 2019).

Donabedian

Exploratings in quality assessment and monitoring definition of quality and approaches to its assessment. Ann Arbor, MI: Health Administration Press, 1980.

Payne

Leget

Peruselli

, et al. Quality indicators for palliative care: debates and dilemmas. Palliat Med 2012; 26(5): 679–680.

Bausewein

Daveson

Currow

, et al. EAPC White Paper on outcome measurement in palliative care: improving practice, attaining outcomes and delivering quality services–Recommendations from the European Association for Palliative Care (EAPC) Task Force on Outcome Measurement. Palliat Med 2016; 30(1): 6–22.

Murtagh

Ramsenthaler

Firth

, et al. A brief, patient-and proxy-reported outcome measure for the adult palliative care population: validity and reliability of the integrated palliative outcome scale (IPOS). Palliat Med 2016; 30(6): NP11.

Bausewein

Simon

Benalia

, et al. Implementing patient reported outcome measures (PROMs) in palliative care–users’ cry for help. Health Qual Life Outcomes 2011; 9: 27.

Currow

Allingham

Yates

, et al. Improving national hospice/palliative care service symptom outcomes systematically through point-of-care data collection, structured feedback and benchmarking. Support Care Cancer 2015; 23(2): 307–315.

Eagar

Watters

Currow

, et al. The Australian palliative care outcomes collaboration (PCOC)–measuring the quality and outcomes of palliative care on a routine basis. Aust Health Rev 2010; 34(2): 186–192.

Pinto

Bristowe

Witt

, et al. Perspectives of patients, family caregivers and health professionals on the use of outcome measures in palliative care and lessons for implementation: a multi-method qualitative study. Ann Palliat Med 2018; 7: S137–S150.

10.

Health AGDo. PCOC – Palliative care outcome collaboration, 2016, http://ahsri.uow.edu.au/pcoc/index.html (accessed 9 August 2016).

11.

Allingham

Clapham

Quinsey

, et al. Patient outcomes in palliative care: national results for January-June 2017 (Detailed report), Palliative Care Outcomes Collaboration, Australian Health Services Research Institute, University of Wollongong, Wollongong, NSW, Australia, 2017.

12.

Witt

Murtagh

Barbara

, et al. International advances in outcome measurement in palliative care: one step closer to cross-national comparisons of routinely collected outcome data in palliative care, 2015. http://ro.uow.edu.au/ahsri/689/ (accessed 1 July 2017).

13.

Kiley

Ast

, et al. Measuring what matters: top-ranked quality indicators for hospice and palliative care from the American Academy of Hospice and Palliative Medicine and Hospice and Palliative Nurses Association. J Pain Symptom Manage 2015; 49(4): 773–781.

14.

Bainbridge

Seow

Sussman

Common components of efficacious in-home end-of-life care programs: a review of systematic reviews. J Am Geriatr Soc 2016; 64(3): 632–639.

15.

Brereton

Clark

Ingleton

, et al. What do we know about different models of providing palliative care? findings from a systematic review of reviews. Palliat Med 2017; 31(9): 781–797.

16.

Heyland

Dodek

Rocker

, et al. What matters most in end-of-life care: perceptions of seriously ill patients and their family members. CMAJ 2006; 174(5): 627–633.

17.

Henry

Findlay

Leech

, et al. What’s important to me. A review of choice in end of life care. London: Choice in End of Life Care Programme Board, 2015.

18.

Emanuel

Alpert

Baldwin

, et al. What terminally ill patients care about: toward a validated construct of patients’ perspectives. J Palliat Med 2000; 3(4): 419–431.

19.

McCaffrey

Bradley

Ratcliffe

, et al. What aspects of quality of life are important from palliative care patients’ perspectives? a systematic review of qualitative research. J Pain Symptom Manage 2016; 52(2): 18–328. e5.

20.

Organisation for Economic Co-operation and Development (OECD). Glossary of statistical terms ‘unit of measure’, 2005, https://stats.oecd.org/glossary/detail.asp?ID=2806 (accessed 22 July 2017).

21.

Daveson

de Wolf-Linder

Witt

, et al. Results of a transparent expert consultation on patient and public involvement in palliative care research. Palliat Med 2015; 29(10): 939–949.

22.

Wainwright

Boichat

McCracken

LM.

Using the nominal group technique to engage people with chronic pain in health service development. Int J Health Plann Manage 2014; 29(1): 52–69.

23.

Evans

Benalia

Preston

, et al. The selection and use of outcome measures in palliative and end-of-life care research: the MORECare international consensus workshop. J Pain Symptom Manage 2013; 46(6): 925–937.

24.

Gysels

Evans

Lewis

, et al. MORECare research methods guidance development: recommendations for ethical issues in palliative and end-of-life care research. Palliat Med 2013; 27(10): 908–917.

25.

Higginson

Evans

Grande

, et al. Evaluating complex interventions in end of life care: the MORECare statement on good practice generated by a synthesis of transparent expert consultations and systematic reviews. BMC Med 2013; 11: 111.

26.

Krueger

RA.

Focus groups: a practical guide for applied research. Thousand Oaks, CA: SAGE, 2014.

27.

Tong

Sainsbury

Craig

Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007; 19(6): 349–357.

28.

Witt

de Wolf-Linder

Dawkins

, et al. Introduction to the Outcome Assessment and Complexity Collaborative (OACC) suite of measures: a brief introduction (Version 2). London, 2014, https://pdfs.semanticscholar.org/3174/1cef800aed424f1b9421997441de35453aa7.pdf?_ga=2.48382309.230135650.1558523101-1605112808.1545128257 (accessed 22 May 2019).

29.

Dawson

Doll

Fitzpatrick

, et al. The routine use of patient reported outcome measures in healthcare settings. BMJ 2010; 340: c186.

30.

Bausewein

Le Grice

Simon

, et al. The use of two common palliative outcome measures in clinical care and research: a systematic review of POS and STAS. Palliat Med 2011; 25(4): 304–313.

31.

Collins

Witt

Bausewein

, et al. A systematic review of the use of the palliative care outcome scale and the support team assessment schedule in palliative care. J Pain Symptom Manage 2015; 50(6): 842–853. e19.

32.

Addington-Hall

JM.

Research methods in palliative care. Oxford: Oxford University Press, 2007.

33.

Clark

Eagar

Currow

DC.

Embedding objective measurements of quality into routine practice in hospice/palliative care. J Pain Symptom Manage 2016; 52: e5–e7

34.

Masso

Allingham

Banfield

, et al. Palliative care phase: inter-rater reliability and acceptability in a national study. Palliat Med 2015; 29(1): 22–30.

35.

Eagar

Gordon

Green

, et al. An Australian casemix classification for palliative care: lessons and policy implications of a national study. Palliat Med 2004; 18(3): 227–233.

36.

Mather

Guo

Firth

, et al. Phase of illness in palliative care: cross-sectional analysis of clinical data from community, hospital and hospice patients. Palliat Med 2018; 32(2): 404–412.

37.

Mokkink

Terwee

Patrick

, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol 2010; 63(7): 737–745.

38.

Brighton

Pask

Benalia

, et al. Taking patient and public involvement online: qualitative evaluation of an online forum for palliative care and rehabilitation research. Res Involv Engagem 2018; 4: 14.

39.

NIHR. Palliative and end of life care Top 10. Southampton: James Lind Alliance Priority Setting Partnerships, 2015.