Developing key performance indicators for adult critical care transfer services: Scoping review and Delphi technique

Abstract

In 2021 NHS England commissioned regional Adult Critical Care Transfer Services. These services will replace a historically predominant ad hoc approach to adult critical care transfers nationally. It is anticipated that these new formal services will provide a system of robust regional & national governance previously acknowledged to be deficient. As part of this process, it is important that an agreed set of transfer service quality indicators are developed to drive equitable improvement in patient care. We used a Delphi technique to develop a set of key performance indicators through consensus for a recently established London critical care transfer service. We believe this may be the first-time key performance indicators have been developed for adult critical care transfer services using a consensus method. We hope services will consider tracking similar measures to enable benchmarking and drive improvements in patient care.

Introduction

In 2021 NHS England commissioned dedicated, nationwide, Adult Critical Care Transfer Services (ACCTS). A significant development, in part triggered by the surge in inter-hospital critical care transfers experienced during the COVID-19 pandemic. ¹ The provision of formal ACCTS will enable improvements in the safety and efficiency of adult critical care transfer which have long been recognised as deficient in England’s historical ad-hoc approach to adult critical care transfers.^2–6 Quality metrics, developed through consensus methods, have been published for paediatric and neonatal critical care transfer services in both Europe and the USA.^7,8 In adult services, however, whilst many likely use quality metrics, or have specified contractual obligations, a set of quality metrics specifically designed for adult services through consensus methods have not been published. Agreement about these metrics or key performance indicators are essential as part of any continuous quality and safety monitoring system. The North East London Critical Care Transfer And Retrieval (NECCTAR) service, a recently established ACCTS, sought to develop KPIs through consensus using a Delphi process.

Methods

Potential KPIs were identified from a scoping review (Supplementary Material) alongside consultation within the service, referring hospitals, and a commissioner. The authors reviewed the final list by removing duplicates, and those identified from the literature not relevant to an ACCTS (e.g. ‘bystander CPR time’). These were evaluated in an e-Delphi process with three rounds of questionnaires before being discussed at a consensus meeting among NECCTAR’s working group. Online questionnaires were created and distributed using Google Forms (Google LLC, Mountain View, CA, USA). The first questionnaire asked respondents to suggest ‘the most optimal’ KPIs. The following rounds asked panel members to rate each KPI using a seven-point Likert scale. Anonymity was maintained throughout, and the composition of the panel unknown to its members. Consensus was defined a priori as a minimum of 70% agreement associated with a Likert scale score above four. The Delphi process was stopped when at least 50 KPIs reached consensus. An a priori agreed threshold.

Results

Figure 1 summaries the Delphi process. The Delphi panel consisted of a wide range of individuals including NECCTAR clinicians, managers & service users; the London Ambulance Service; a patient representative; and representation from NHS England & NHS Improvement London (see Supplemental Table for detailed description of the Delphi panel). Following independent screening by the authors (NH & SF) for duplicates or similar KPIs a total of 76 KPIs were formulated from the initial Delphi Round. 26/37 (70%) responded to the second-round invitation and 24/37 (65%) responded to the third-round invitation. The Delphi process was completed at this stage as over 50 KPIs had reached consensus.

Figure 1.

Flowchart detailing the Delphi process and selection of KPIs for adult critical care transfer services.

The final 54 KPIs that reached positive consensus were reviewed by the authors and NECCTAR working group to produce a balanced scorecard (Table 1). For practical application full definitions and specifications of each KPI from the final scorecard are presented (Supplemental Material).

Table 1.

Final scorecard.

KPI number	KPI name	Short definition	NHS Outcome Framework Domain
1	Transfers completed	Total number of completed transfers sub-categorised into capacity and time critical.	3
2	Transfers declined	Percentage of requested transfers declined by the duty consultant.	3
3	Transfers cancelled/abandoned transfers	Percentage of transfers not completed after transfer team has been activated.	5
4	Shifts operational by target green time in mornings	The time the transport service is ready to dispatch in the morning.	5
5	Stabilisation time at referring centre <60 minutes	Time from ambulance arrival at referring hospital to departure of ambulance from referring hospital.	3
6	Delays in returning to operational readiness after a transfer (> 30 minutes)	Time from arrival at receiving hospital to time team are ready for further tasking.	5
7	Shifts running overtime	Completion of sign out checklist performed after the end of the shift.	4
8	Ambulance arriving fully equipped	Ambulance arriving fully kitted at beginning of shift as per sign in checklist.	5
9	Vehicle failure	Any vehicle fault or malfunction that requires exchange of vehicle or repair.	5
10	Medical equipment failure	Rate of medical equipment failure per attempted transfer.	5
11	Adverse events that occur during transfer	Harmful or potentially harmful event to patient or staff whilst tasked.	5
12	Significant physiological deterioration during transfer	Percentage of transfers in which significant physiological deterioration* occurs during transfer.	5
13	Death in transit	Death at any point during transit on transfer trolley.	1
14	Unexpected mortality within 48 hours	Number of deaths 48 hours after departure time.	1
15	Adherence to key operational checklists	Rate of 100% compliance with all operational checklists.	5
16	Cases undertaken receive a complete patient report form	All mandatory sections of patient report form completed. If information unavailable this must be annotated as such.	5
17	Patient report forms reviewed by a consultant	Documented review of patient report form by a consultant.	5
18	Transfers debriefed by a consultant and recorded as a formal learning activity for all team members	Percentage of transfers debriefed with the team by a consultant.	4
19	Next of kin informed of the transfer by referring clinicians prior to departure	Next of kin informed of transfers prior to departure from referral hospital as a proportion of all transfers.	4
20	Feedback from patients, relatives & staff	Qualitative feedback collected from patients, relatives and non-NECCTAR staff via formal feedback form.	4

KPI: key performance indicator; NECCTAR: North East Critical Care Transfer and Retrieval service.

*

See Supplemental Material for full definition of physiological deterioration.

Discussion

We have developed a balanced set of KPIs for a newly established ACCTS in England following a Delphi process. To the authors’ knowledge this is the first published set of KPIs developed through consensus for an ACCTS. The final scorecard emphasises a number of important areas in critical care transfer that have previously been identified in the literature as requiring improvement. ⁹ Namely – safety; efficiency and timeliness; and senior leadership presence. As well as embracing broader aspects of a quality service including education; focus on communication with relatives; and feedback from patients.

Concerns regarding the safety of adult critical care transfer have been consistently voiced^2–5 and re-emphasised in recent guidelines. ⁶ A recurring theme, however, has been a lack of adherence to published national guidelines despite their existence. ³ Continual monitoring of the proposed KPIs will provide some assurance that these aspects are being monitored; drive quality improvement; and afford the opportunity for services to benchmark themselves against one another.

Delays in transfers are also common.^4,9 A national target of less than 4 hours from acceptance of referral to delivery of the patient to the destination hospital for time-critical transfers has been suggested. ¹⁰ This timeline, however, will be dictated by numerous factors outside the control of an ACCTS. To mitigate this our KPIs focus on time periods specifically influenced by the performance of the transfer team. We feel these are fair across services with different geography and regional setup.

NHS England has specified that all ACCTS should be consultant led. ¹⁰ Our scorecard includes metrics monitoring consultant involvement in clinical and non-clinical roles. Including consultant led multidisciplinary learning, debriefing and case review.

Our final scorecard has some similarities to previously published quality indicators developed for paediatric and neonatal transport^7,8 including emphasis on process safety (e.g. adverse event rate); efficiency/timeliness (e.g. time to mobilisation); and physiological changes (e.g. cardiovascular, temperature). There are, however, important differences as our scorecard aims to capture the quality of a transfer service rather than just the transport process itself. For example, our scorecard, incorporates feedback from relatives, patients and staff; documented communication with the next of kin; and formal consultant led debrief and learning. This reflects our broader stakeholder representation and is a strength of our study.

The development of formal ACCTS throughout England represents a paradigm shift in the approach to adult critical care transfer nationally. We will use this scorecard and monitor the KPIs over time to ascertain their ability to highlight areas for improvement and to assure our users about quality within the service. In the future, we hope other services will use the scorecard to facilitate benchmarking across transport providers.

Supplemental Material

sj-docx-2-inc-10.1177_17511437231153049 – Supplemental material for Developing key performance indicators for adult critical care transfer services: Scoping review and Delphi technique

Supplemental material, sj-docx-2-inc-10.1177_17511437231153049 for Developing key performance indicators for adult critical care transfer services: Scoping review and Delphi technique by Nick Haslam, Aurelien Giouse, Jonathon Dean, Mamoun Abu-Habsa and Simon J Finney in Journal of the Intensive Care Society

Supplemental Material

sj-docx-3-inc-10.1177_17511437231153049 – Supplemental material for Developing key performance indicators for adult critical care transfer services: Scoping review and Delphi technique

Supplemental material, sj-docx-3-inc-10.1177_17511437231153049 for Developing key performance indicators for adult critical care transfer services: Scoping review and Delphi technique by Nick Haslam, Aurelien Giouse, Jonathon Dean, Mamoun Abu-Habsa and Simon J Finney in Journal of the Intensive Care Society

Supplemental Material

sj-docx-4-inc-10.1177_17511437231153049 – Supplemental material for Developing key performance indicators for adult critical care transfer services: Scoping review and Delphi technique

Supplemental material, sj-docx-4-inc-10.1177_17511437231153049 for Developing key performance indicators for adult critical care transfer services: Scoping review and Delphi technique by Nick Haslam, Aurelien Giouse, Jonathon Dean, Mamoun Abu-Habsa and Simon J Finney in Journal of the Intensive Care Society

Supplemental Material

sj-docx-5-inc-10.1177_17511437231153049 – Supplemental material for Developing key performance indicators for adult critical care transfer services: Scoping review and Delphi technique

Supplemental material, sj-docx-5-inc-10.1177_17511437231153049 for Developing key performance indicators for adult critical care transfer services: Scoping review and Delphi technique by Nick Haslam, Aurelien Giouse, Jonathon Dean, Mamoun Abu-Habsa and Simon J Finney in Journal of the Intensive Care Society

Supplemental Material

sj-docx-6-inc-10.1177_17511437231153049 – Supplemental material for Developing key performance indicators for adult critical care transfer services: Scoping review and Delphi technique

Supplemental material, sj-docx-6-inc-10.1177_17511437231153049 for Developing key performance indicators for adult critical care transfer services: Scoping review and Delphi technique by Nick Haslam, Aurelien Giouse, Jonathon Dean, Mamoun Abu-Habsa and Simon J Finney in Journal of the Intensive Care Society

Supplemental Material

sj-jpg-1-inc-10.1177_17511437231153049 – Supplemental material for Developing key performance indicators for adult critical care transfer services: Scoping review and Delphi technique

Supplemental material, sj-jpg-1-inc-10.1177_17511437231153049 for Developing key performance indicators for adult critical care transfer services: Scoping review and Delphi technique by Nick Haslam, Aurelien Giouse, Jonathon Dean, Mamoun Abu-Habsa and Simon J Finney in Journal of the Intensive Care Society