Abstract
Background
Anaesthesia Quality Improvement New Zealand developed a set of five quality improvement indicators pertaining to postoperative nausea and vomiting, pain, respiratory distress, hypothermia and a prolonged post-anaesthesia care unit stay. This study sought to assess the proportion of eligible institutions that were able to measure and provide data on these indicators, produce an initial national estimate of these, and a measure of variability in the quality improvement indicators across hospitals in New Zealand.
Methods
All public hospitals that provide a representative to Anaesthesia Quality Improvement New Zealand were eligible for inclusion. Participating institutions were required to provide the number and proportion of patients with each of the five quality improvement indicators over a continuous 2-week period between 1 June 2019 and 25 October 2019. The overall percentage of patients and the median percentage with each outcome were calculated.
Results
A total of 79.2% of eligible hospitals participated. The median incidence of the indicators ranged from 1.67% for respiratory distress to 6.31% for prolonged post-anaesthesia care unit stay. The indicator with the largest interquartile range was hypothermia and the smallest was respiratory distress (13.48 and 2.29, respectively). A large variation was seen for prolonged post-anaesthesia care unit stay, hypothermia, pain and postoperative nausea and vomiting.
Conclusion
The majority of eligible institutions were able to measure and provide data on the quality improvement indicators. There was a low rate of respiratory distress with low variability. A large amount of variability was observed in the other indicators. Future studies are needed to explore the nature of this variability.
Introduction
The measurement and analysis of clinical indicators is one component of ensuring the delivery of quality care. 1 Clinical indicators are commonly used to assess structures, processes and outcomes of a service. 2 They aid standardisation of care, reduction of unwanted variability and may allow for benchmarking. 3
Many clinical indicators exist in the perioperative setting; 4 however, the utility of these in quality improvement requires choosing appropriate indicators and reliably collecting accurate data. 3 In a recent systematic review, 1282 clinical indicators were identified over 43 relevant indicator programme publications. 5 Most focused on the effectiveness, safety and efficiency of care, but the majority were not evidenced based. 5 Accurate and reliable measurement can be problematic when using clinical indicators, and underestimation of the true incidence of problems can lead to false reassurance of maintained quality care. 6 , 7
There is currently no uniform set of postoperative quality improvement indices (QIIs) consistently measured in New Zealand. The Australian and New Zealand College of Anaesthetists (ANZCA) endorses the clinical indicator programme (CIP) produced by the Australian Council of Healthcare Standards (ACHS). 8 , 9 These are, however, not universally accepted or measured within New Zealand due to the cost of subscription, the perceived differences between healthcare in Australia and New Zealand, and the wide range of clinical indicator programmes already in existence from which an institution can select. 5
Anaesthesia Quality Improvement New Zealand (AQINZ) is a volunteer group, supported by ANZCA, consisting of the anaesthesia quality assurance coordinators from each public hospital in New Zealand where anaesthetic services are provided. It meets biannually with the aim of providing a national forum for anaesthetic quality improvement and quality assurance. In an attempt to address the lack of universally accepted and utilised quality improvement indicators AQINZ developed a draft memorandum of understanding for five key QIIs (based on those from the CIP) measured in the post-anaesthesia care unit (PACU). Use of these indicators would be subscription free, New Zealand specific, and each institution would be able to participate in the indicators’ definitions and revisions. These indices were defined using available evidence-based literature and consensus from AQINZ. They pertain to postoperative nausea and vomiting (PONV), postoperative pain, postoperative respiratory distress, prolonged stay in the PACU and hypothermia in the PACU. The initial focus on indicators relating to the PACU was pragmatic. They are temporally and spatially easier to collate and measure as opposed to other outcomes such as mortality and quality of life. Although often viewed as surrogate measures, there is increasing opinion that a number of these outcomes are significant to patients (and therefore) anaesthetists in their own right. 10 , 11
To optimise quality improvement initiatives large datasets of standardised outcome measures are imperative in order to understand the reasons for variance in outcome; to ascertain where best to channel efforts for improvement; and to be certain that the initiatives have achieved the desired effect. Standardised, national anaesthetic QIIs would greatly benefit the generation of such datasets and allow national, rather than just siloed, institutional learning to occur.
Currently, in New Zealand there are no standardised perioperative QIIs collected, and no analysis or comparisons are made at a national level. This initial descriptive study aimed to address this by assessing the proportion of eligible hospitals that would be able to collect and provide data on the five QIIs and to compare the observed proportions of each, to enable a national estimate of their incidence and assess their variability across hospitals.
Methods
Ethical approval
National ethics approval was obtained from the health and disability committee (reference 19/STH/185) with locality approval obtained by each participating hospital.
Inclusion/exclusion criteria
All public hospitals in New Zealand that provide a representative to AQINZ and have contact details on record were eligible for inclusion.
Participants
All patients receiving post-anaesthesia care who were admitted to the PACU of a participating institution were eligible to be participants.
Quality improvement indicators
The quality improvement indicators were based on those from the CIP, but adapted to improve their utility for quality improvement work. The definitions for the indictors used and the rationale for any changes from the CIP are detailed below.
Respiratory distress: Any patient with any airway or respiratory concern that requires an anaesthetist to manage.
The original definition was limited to reporting of airway concerns only if a laryngeal mask airway (LMA) or reintubation was required. It was thought that the numerator would be extremely small and would miss other significant issues such as severe bronchospasm or laryngospasm which often do not require LMA or reintubation but require anaesthesia personnel to attend. Therefore, the definition was expanded.
PONV: Any patient undergoing treatment for symptomatic PONV in the PACU.
The addition of ‘symptomatic’ was added as in the authors’ experience a significant proportion of antiemetic used in the PACU was for prophylaxis against PONV rather than active treatment.
Hypothermia: Any patient admitted to the PACU with a temperature recorded in the recovery period of less than 36°C. Severe postoperative pain: Any patient who is reviewed by an anaesthetist to manage severe pain in the recovery period.
This was adapted to include either a physical or phone review by an anaesthetist for pain. This was thought to be a more sensitive indicator as it would capture both acute postoperative pain crises, in addition to those patients failing to be adequately treated by pain protocols.
A prolonged PACU stay was defined as any patient who required ongoing medical review and unplanned stay in the PACU for more than 2 hours.
‘Ongoing medical review’ was added to this QII as a substantial number of patients (over 20% in one institution) had prolonged PACU stays due to non-clinical reasons (e.g. lack of ward beds, nursing staff, porters, etc.). Therefore, the focus was shifted to the clinical reasons for the prolonged PACU stay.
Data collection
Representative members from each institution were emailed with information on the project and invited to participate. A total of three emails was sent to each member at weekly intervals. If there had been no reply after three emails it was assumed that institution did not wish to participate.
Participating institutions were then asked to collect the proportion of patients who fulfilled the criteria of each of the five indicators over a continuous 2-week period of their choice between 1 June 2019 and 25 October 2019. The exact 2-week period and method of collection were determined by the institution. The decision to allow flexibility in the 2-week audit period was a pragmatic one: it would enable the maximum number of hospitals to participate (as those institutions collecting data manually would be able to select a period with optimal workforce availability) while was unlikely to change the underlying baseline indices significantly.
Institutions were asked to submit their methods for data collection which are included in Appendix 1. Institutions were anonymised by being assigned a numerical value.
Statistics
Ninety-five per cent confidence intervals were calculated for proportions using the Clopper–Pearson method. The overall percentage of patients and median percentage with each outcome were calculated.
Institutions that collected data using different definitions were excluded from the relevant indicator. No attempt was made to correct for missing data.
All analyses and graphs were performed using RStudio (version 1.1.453). Tables were constructed using Microsoft Excel (version 16.30).
Results
Participating institutions
A total of 24 hospitals was eligible for inclusion, representing all 20 New Zealand district health boards (DHBs). The hospitals varied in size, catchment demographics and surgical complexity. 12 Two hospitals did not reply to invitations to participate. Three hospitals were unable to participate due to staffing constraints. This left 19 participating hospitals, representing 14 DHBs (79.2% of eligible hospitals and 70.0% of eligible DHBs).
Descriptive data
The total number of patients included ranged from 5161 to 6171 depending on the QIIs. Hospitals 11 and 17 used definitions that differed from those of AQINZ for PONV, respiratory distress and pain, respectively, due to their use of pre-existing data collection systems, therefore their results for these QIIs have been excluded from the analysis. In addition, hospital 11 did not collect data on respiratory distress. Hospitals 2 and 4 only included patients during daylight hours (09:00–17:00) and hospital 4 was only able to provide one week’s worth of data.
Outcome data
The number of participants with each outcome and number of eligible patients for each indicator for both individual institutions and overall are shown in Table 1. The median and interquartile range (IQR) for each QII is shown in Figure 1. The median incidence of the QIIs ranged from 1.67% for respiratory distress to 6.31% for prolonged PACU stay. The percentage and 95% confidence interval for each indicator by hospital is displayed in Figure 2.
Five performance indicators by hospital.
n: numerator; d: denominator. Percentage = ((n/d) × 100); 95% CI: 95% confidence interval calculated using the Clopper–Pearson method; PONV: postoperative nausea and vomiting; PACU: post-anaesthesia care unit; Median is the median percentage; IQR: interquartile range.
*Only collected during daylight hours (09:00–17:00).
†Only collected for one week.
‡Used different definitions.
Box and whisker plot for five indicators. The upper whisker extends from the upper quartile to the largest value no further than 1.5 times the interquartile range (IQR). The lower whisker extends from the lower quartile to the smallest value no further than 1.5 times the IQR. Data beyond the end of the whiskers are plotted individually as outlying points. PACU: post-anaesthesia care unit; PONV: postoperative nausea and vomiting.
Percentage and 95% confidence interval of patients with outcome for each of the five indicators by hospital. (a) Respiratory distress. (b) Postoperative nausea and vomiting. (c) Hypothermia. (d) Severe postoperative pain. (e) Prolonged post-anaesthesia care unit stay. *Only collected during daylight hours (09:00–17:00). †Only collected for one week. ‡Used different definitions.
Individual indicators
The indicator with the greatest variability in our cohort was hypothermia (range 0.00–41.33%; IQR 1.73–15.21%). This was followed by prolonged PACU stay (range 0.00–29.34%; IQR 3.25–9.28%). Both severe pain and PONV showed substantial variation, while respiratory distress showed the lowest variation (range 0.00–4.32%; IQR 0.49–2.78%).
Discussion
Key results
This project represents the first attempt to collect and apply standardised definitions for anaesthetic quality improvement indicators in New Zealand, and the first estimates of the national proportions of these outcomes and their variation. We observed high rates of variation in the clinical indicators of PONV, severe pain, hypothermia and prolonged PACU stay and low rates of variation were observed in respiratory distress. It is not possible to make comparisons between hospitals from these data due to the degree of heterogeneity in the case mix at each, and the lack of underlying patient demographics that might affect the QIIs outcomes. This is in part the reason why the term ‘quality improvement indices’ was used for the markers rather than ‘key performance indicators’, as we felt it important to be explicit that performance cannot be measured by the reported rates currently.
Strengths
The participation rate from the hospitals within New Zealand was high, demonstrating cooperation and collaboration and a commitment by anaesthetic departments and individuals to quality improvement. This has also stimulated a number of national level discussions, individualised hospital quality improvement initiatives and focused audits, and plans for future nationwide projects. The ultimate goal is to have a nationally agreed, evidence-based set of QIIs that are continuously collected in all institutions and would allow subanalyses into specific operations or patient groups (thereby reducing heterogeneity) and would be linked to clinically meaningful outcomes.
One of the strengths of our chosen indicators is that they were evidence-based, commonly used clinical indicators, adapted to our clinical practice in New Zealand, without the need for a paid subscription. Modifications were made to help identify problematic PACU problems. Our severe pain clinical indicator acted both as a process (as it requires the necessity to have a severe pain protocol), and a patient-centred indicator. The threshold for respiratory distress was to be recorded such that it was significant enough to require anaesthetic assistance, and the PONV indicator was to be triggered when antiemetics were to be given for symptomatic PONV.
Our rates of severe pain, PONV and respiratory distress were acceptable; however, we did observe variability between hospitals. Although this study was not designed to look at causality, we cannot underestimate the contribution of hospital heterogeneity in these observed differences.
The clinical indicators which returned the greatest variability were hypothermia and prolonged PACU stay. These are both amenable to reduction from process and system improvements, which can lead to improvements in patient outcomes. In New Zealand, thermal care bundles are not mandated as part of the surgical site infection programme but are assumed as part of good clinical care. 13 Anecdotally, there has been acknowledgement in the past that there is heterogeneity of practice throughout the country. These data may enable a move towards better prevention of inadvertent hypothermia. It has been reported that one reason for delays in PACU discharge may be attributable to the need to complete necessary treatments. 14 Contributors include addressing hypotension due to a body of evidence that demonstrates an increase in organ dysfunction relating to the duration of low mean arterial pressure.15–17 Whether there is a reduced tolerance for hypotension and the flow-on effect of prolonged stay needs further investigation.
Limitations
Despite its strengths, there were a number of limitations. Firstly: data quality. Hospitals were able to choose how and when to collect data. This introduced bias but was an accepted limitation to increase participation. There was variation in how missing data were handled by different hospitals, some required positive or negative responses to be actively entered, while others assumed no positive responses meant no outcome occurred. This would have a large impact on the observed proportions. Few hospitals were able to estimate the amount of missing data with their collection methods. Secondly: confounders such as age, sex, physiological status and type of surgery were not accounted for. These are likely to contribute to the observed variation but to what degree and in what direction was unable to be explored. Thirdly: the definitions are broad and open to interpretation. For example, a patient who requires the anaesthetist to be called to provide a simple jaw thrust would be classified as respiratory distress according to our definition, as would the anaesthetist being called to manage severe laryngospasm in the PACU despite being different clinical scenarios. We would, however, argue that our definitions represent pragmatic outcome measures and are useful for quality improvement. A PACU where the staff is not comfortable performing basic airway manoeuvres represents a significant opportunity for improvement, as does a high incidence of laryngospasm. In addition, measures with modest sensitivity and specificity that are easily applied across institutions in a standardised fashion produce a better estimate of the true incidence and its variation than a more accurate measure that is impractical, poorly applied and difficult to use in a standardised fashion.
Implications and future research
We have identified several areas of potentially unwarranted variation. Dissemination of this information will inform hospitals of their own practice with the hope that they will evaluate their own data, build this into their individual quality improvement programmes, improve their own processes and systems, and in turn their patient outcomes. Further clarification in the clinical indicators will assist in this.
This project opens a number of areas of further research. Firstly, repetition over time would enable the analysis of trends which would be more informative than point estimates. Secondly, collecting data over a longer time period and with an increase in participating institutions would enable more accurate estimates of the nationwide proportions of these indicators and their variation. Thirdly, assessing and improving data quality would enable more useful comparisons, result in more accurate estimates and result in improved data collection systems. Establishing a standard for data quality would probably address this and enable comparison of actual to ideal data collection and reporting. It also provides evidence to providers that collecting this data is difficult but important and requires support from information technology units throughout the country. Finally, attempting to address confounding : future iterations of this project should include measures of likely confounders such as age, sex, physiological status and type of surgery so the impact of these on variation can be explored.
Conclusion
In conclusion, this study demonstrated that the majority of eligible hospitals were able to collect data on AQINZ QIIs. It also provided the first national estimates of the national proportions of these indicators and their variation. Future research that capitalises on the high levels of participation and collaboration seen in this project, and addresses the limitations identified, would lead to improved estimates and identification of areas for improvement in care in anaesthesia.
Supplemental Material
sj-pdf-1-aic-10.1177_0310057X211027884 - Supplemental material for National quality improvement indicators project: an initial descriptive study
Supplemental material, sj-pdf-1-aic-10.1177_0310057X211027884 for National quality improvement indicators project: an initial descriptive study by Daniel P Ramsay in Anaesthesia and Intensive Care
Footnotes
Author Contribution(s)
Acknowledgements
Daniel Ramsay would like to acknowledge and thank Davina McAllister for assistance with obtaining ethical approval; James Ramsay for feedback on figures; and Dr Lisa Woods for statistical assistance and advice.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
