Utility of the National Emergency Laparotomy Audit prognostic model in predicting outcomes in an Australian health system

Abstract

The Royal College of Anaesthetists was commissioned by the United Kingdom Health Quality Partnership to conduct the National Emergency Laparotomy Audit of England and Wales (NELA), to compare outcomes of patients undergoing emergency laparotomy in order to promote quality improvement. Prior to 2016 there were minimal data for emergency laparotomy patients in Australia. The aim of this cohort study was to assess the utility and applicability of the NELA model in a tertiary centre in Western Australia. NELA-related data of patients who underwent emergency laparotomy, between June 2018 and May 2020, were merged with other administrative databases and clinical records. The discriminative ability and calibration of the model were assessed by the area under the receiver operating characteristic (AUROC) curve and calibration plot, respectively. Cox proportional hazards regression was used to assess whether the NELA-predicted risks were an independent predictor of hospital mortality. Of the 502 patients included, 168 (33.5%) patients had a NELA-predicted risk >10%, and of these, 93 (55.4%) were admitted to a critical care unit in a planned fashion immediately after surgery. The NELA model had a good ability to discriminate between survivors and non-survivors (AUROC 0.892, 95% confidence intervals 0.854 to 0.93, P <0.001). However, the model was not perfectly calibrated, with the predicted risks tending to overestimate the observed risks of mortality, especially when the predicted risks were >50%. A high NELA-predicted risk remained significantly associated with mortality after adjusting for other covariates, including sepsis and plasma lactate concentration, suggesting that it is a reliable screening tool for identifying high-risk patients requiring emergency laparotomy.

Keywords

ANZELA-QI emergency laparotomy NELA mortality perioperative care

Introduction

Emergency laparotomy is increasingly offered to an ageing population with multiple comorbidities. Improving surgical outcomes of these patients is challenging. The United Kingdom Health Quality Improvement Partnership in conjunction with the Royal College of Anaesthetists initiated the National Emergency Laparotomy Audit of England and Wales (NELA) in December 2012 and developed the NELA prognostic model.¹ After implementing the NELA prognostic model into clinical practice, data from the UK showed that there was a significant increase in the number of planned admissions to a critical care environment for patients undergoing emergency laparotomy and, at the same time, also a significant reduction in unexpected escalation in level of care in the postoperative period.²

A review of the care of patients undergoing emergency laparotomy in tertiary Australian hospitals showed that there was wide inter-hospital variability in how these patients were managed and also a lack of accurate patient-centred outcome data.^3,4 A binational audit in Australia and New Zealand similar to the one in the UK (the Australian and New Zealand Emergency Laparotomy Audit – Quality Improvement (ANZELA-QI)) was launched in 2018.⁵ The ANZELA-QI includes recommendations similar to the NELA key standards on the best perioperative practices, such as the timing of computed tomography abdomen, operating theatre logistics, involvement of consultant staff and postoperative care.¹ Specifically, both the UK and the local binational audits have recommended that patients with a high predicted risk of death should be admitted to and monitored in either an intensive care or a high-dependency care unit after surgery in a planned fashion.^1,5

A recent Australian study showed that the NELA prognostic model was highly sensitive in predicting 30-day mortality, and its performance was comparable to two other perioperative prognostic models, the Portsmouth Physiological and Operative Severity Score (P-POSSUM) for the enUmeration of mortality and Morbidity, and the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) score, that were not specifically designed for patients requiring emergency laparotomy.⁶ Whether the findings of this study are generalisable to other Australian centres remains uncertain. As one of the pioneer sites in Australia to adopt the NELA prognostic model in clinical practice, we aimed to validate the accuracy of the NELA prognostic model in an Australian context, and assess whether elective intensive care unit (ICU) or high dependency unit (HDU) admission for those who were deemed to be at high risk of dying would improve their chance of survival to hospital discharge.

Methods

Approval was obtained from the Sir Charles Gairdner Hospital Clinical Safety and Quality Unit (approval number: 34861). Data of all patients who had an emergency laparotomy between June 2018 and May 2020 were retrieved from the ANZELA-QI Research Electronic Data Capture tool database and merged with the data from patient records and other administrative databases, including the operative theatre, laboratory and ICU databases. Inclusion and exclusion criteria for patients to be included in the ANZELA-QI, and hence also in this audit, can be found in more detail on the ANZELA-QI Resources page.⁷

The primary outcomes of this study were (a) the proportion of patients who had a greater than 10% NELA-predicted risk of mortality who had a planned ICU/HDU admission immediately after emergency laparotomy or an unplanned ICU/HDU admission within seven days of laparotomy, and (b) the prognostic performance of the NELA model in predicting hospital mortality. The secondary outcome was whether a planned ICU/HDU admission was associated with a reduced risk of mortality among the high-risk patients, defined by a NELA-predicted mortality risk of >10%. For patients who did not have a preoperative NELA risk recorded (n = 69), the NELA risk was calculated using data from patient records and the NELA online risk calculator (http://data.nela.org.uk/riskcalculator/). In this study, admissions to either ICU or HDU were pooled together as a single entity.

Statistical methods

Categorical and continuous variables were analysed by chi-square and Mann–Whitney tests respectively. The discriminative ability and calibration of the NELA model were assessed by the area under the receiver operating characteristic (AUROC) curve and calibration plot, respectively. The slope and intercept of the calibration curve were used to reflect the predictive accuracy of the NELA model across the full range of predicted risks and systematic bias, respectively.^8,9 We used the Spearman (non-parametric) correlation coefficient and a scatter plot to assess whether the NELA-predicted risks were correlated with the Acute Physiology and Chronic Health Evaluation (APACHE) II score for those who were admitted to the ICU.

We used the Cox proportional hazards regression to assess whether the NELA-predicted risks were an independent predictor of hospital mortality, with and without adjusting for other important confounders or covariates that could influence mortality, including suspected sepsis, Goals of Care discussion, plasma lactate concentration before surgery and planned admission to ICU/HDU. To determine the potential benefit of planned ICU/HDU admission among those who had a high predicted risk of mortality (>10%), a separate survival analysis was conducted for this group of patients. All statistical analyses were two-tailed and conducted using the MedCalc® Statistical Software (version 19.6.4, MedCalc Software Ltd, Ostend, Belgium; 2021) and SPSS for Windows (version 24, IBM, Armonk, NY,USA). A P-value <0.05 was taken as significant in this study.

Results

Difference in patient characteristics between those with a high and those with a low National Emergency Laparotomy Audit predicted risk of death

Of the 502 patients included in the study, 433 patients (86.3%) had their NELA-predicted risk of mortality recorded prospectively. The baseline patient characteristics were similar between those with and those without NELA data recorded prospectively (Supplemental material Table 1 online). Of the 502 patients, 168 (33.5%) patients were considered to have a high predicted risk (>10%) of death. These patients were older, more likely to have sepsis, had a higher plasma lactate concentration, and had higher rates of Goals of Care discussion and planned admission to ICU/HDU immediately after emergency laparotomy compared with those who had a lower NELA-predicted risk of mortality (≤10%) (Table 1).

Table 1.

Differences in characteristics and outcomes between those with a National Emergency Laparotomy Audit predicted risk of mortality >10% and those with a predicted risk of mortality ≤10% (N = 502).

Variable	NELA ≤ 10% n = 334, 66.5%	NELA > 10% n = 168, 33.5%	P-value^a
Age, years	63 (46–73)	76 (66–85)	0.001
Male, no. (%)	154 (46.1%)	84 (50.0%)	0.449
Indigenous ethnicity, no. (%)	30 (9.0%)	18 (10.7%)	0.524
Sepsis prior to laparotomy, n = 473, no. (%)	77 (23.6%)	93 (61.6%)	0.001
Plasma lactate prior to laparotomy, n = 324, mmol/l	1.4 (1.0–2.1)	2.0 (1.3–3.1)	0.001
Goals of Care discussion with treatment limitations documented prior to surgery, n = 471, no. (%)	94 (30.2%)	98 (61.3%)	0.001
Postoperative ICU/HDU admission planned prior to surgery no. (%)	77 (23.1%)	93 (55.4%)	0.001
Unplanned ICU or HDU admission within seven days of emergency laparotomy among those who were not admitted to ICU initially, n = 303, no. (%)	10 (4.0%)	5 (9.4%)	0.152
Length of ICU stay, n = 67, days	3 (2–3)	3 (1–5)	0.878
APACHE II score, n = 67	13 (5–16)	17 (12–22)	0.126
Length of hospital stay, days	10 (7–18)	13 (7–25)	0.130
Length of hospital stay excluding non-survivors, days	11 (7–18)	15 (9–27)	0.001
Hospital mortality, no. (%)	6 (1.8%)	40 (23.8%)	0.001

Data are median (interquartile range) unless stated otherwise.In the Variable column, n refers to the number of patients for which information regarding the particular predictor was available.

^aDerived from Mann–Whitney or chi-square test.

NELA: National Emergency Laparotomy Audit of England and Wales; ICU: intensive care unit; HDU: high dependency unit; APACHE: Acute Physiology and Chronic Health Evaluation.

Compliance with admitting high-risk patients to the intensive care or high dependency unit electively

Only 93 (55.4%) of those with a high predicted mortality risk were admitted to ICU/HDU electively immediately after surgery, suggesting that compliance with the recommendation to admit higher-risk patients electively to ICU/HDU after emergency laparotomy was relatively low. Conversely, 77 patients (23.1%) with a low (≤10%) NELA-predicted risk were admitted to ICU/HDU after surgery electively. Unplanned ICU/HDU admission within seven days of laparotomy occurred more frequently among those with a NELA >10% compared with those with a lower NELA-predicted risk, but this difference was not statistically significant (9.4% (n = 5) versus 4.0% (n = 10) respectively, P = 0.152).

National Emergency Laparotomy Audit predicted risk of mortality as a predictor of severity of illness and hospital mortality

As a dichotomised predictor, >10% NELA-predicted risks were significantly associated with a much higher risk of hospital mortality (23.8% versus 1.8%) than those with a lower NELA risk (≤10%) (Table 1). As a continuous predictor, the NELA-predicted risks had a good ability to discriminate between hospital survivors and non-survivors (AUROC 0.892, 95% confidence intervals (CI) 0.854 to 0.93, P < 0.001) (Figure 1). The excellent discriminative ability of the NELA model was not different between those who had their NELA-predicted mortality data collected prospectively and those whose were collected retrospectively (Supplemental Table 1).

Figure 1.

Area under the receiver operating characteristic (AUROC) curve of the National Emergency Laparotomy Audit of England and Wales predicted risks of mortality in differentiating between hospital survivors and non-survivors. The dotted lines define the 95% confidence intervals (CI) of the AUROC curve and the diagonal line defines the AUROC = 0.5.

Nonetheless, the calibration of the NELA model was not perfect (Hosmer–Lemeshow chi-square statistic of the model was 15.3, P = 0.053). The slope of the model’s calibration curve (6.11, standard error 0.74) was substantially >1, suggestive of overestimating the observed mortality risks for high-risk patients — especially when the NELA-predicted risks were greater than 50% (Figure 2) — and, at the same time, underestimating the observed mortality risks for lower risk patients. The intercept of the calibration curve was <0 (–3.51, standard error 0.26), suggestive of a systematic bias in overestimating the observed risks of mortality.

Figure 2.

The calibration plot of the National Emergency Laparotomy Audit of England and Wales (NELA) risk model. Error bars indicate the standard error of mean of the observed risks of mortality.

As a dichotomised predictor, a NELA-predicted mortality risk >10% was associated with a longer length of hospital stay after excluding non-survivors (median 15 versus 11 days respectively, P = 0.001) (Table 1), and remained highly predictive of hospital mortality both in univariable analysis without adjusting for covariates (hazard ratio 12.8, 95% CI 5.4 to 30.2; P = 0.001) and multivariable analysis after adjusting for sepsis, plasma lactate concentration, Goals of Care decision and planned ICU/HDU admission immediately after laparotomy (hazard ratio 17.9, 95% CI 2.2 to 148; P = 0.007) (Figure 3 and Table 2). Among those who were admitted to the ICU who also had the APACHE II scores available, the NELA-predicted risks were significantly correlated with the APACHE II scores (Spearman correlation coefficient 0.395, P = 0.001).

Figure 3.

Difference in survival time between those with a National Emergency Laparotomy Audit of England and Wales (NELA)-predicted risk of mortality >10% and those with a NELA-predicted risk of mortality ≤10%, after adjusting for sepsis, plasma lactate level, Goals of Care decision and planned intensive care or high dependency admission prior to laparotomy. The longest patient hospital stay was 373 days.

Table 2.

Univariable and multivariable Cox proportional hazards regression showing the association between the National Emergency Laparotomy Audit predicted risks of mortality and survival to hospital discharge.

	HR (95% confidence intervals)	P-value
Univariable models:
NELA as a continuous predictor (HR per % predicted risk increment) only	1.05 (1.04 to 1.06)	0.001
NELA >10% as a categorical predictor only	12.8 (5.4 to 30.2)	0.001
Multivariable models:
Model 1
NELA as a continuous predictor (HR per % predicted risk increment)	1.03 (1.01 to 1.05)	0.011
Postoperative ICU/HDU admission planned prior to surgery	1.48 (0.42 to 5.25)	0.545
Goals of Care decision with treatment limitations before surgery	4.80 (1.55 to 14.89)	0.007
Plasma lactate level (HR per mmol/l increment)	1.21 (1.01 to 1.46)	0.037
Sepsis prior to surgery	2.39 (0.85 to 6.75)	0.099
Model 2
NELA >10% as a categorical predictor	17.9 (2.16 to 148.2)	0.007
Postoperative ICU/HDU admission planned prior to surgery	1.05 (0.31 to 3.64)	0.935
Goals of care decision with treatment limitations before surgery	2.71 (0.91 to 8.04)	0.073
Plasma lactate level (HR per mmol/l increment)	1.22 (0.99 to 1.48)	0.056
Sepsis prior to surgery	2.97 (1.08 to 8.20)	0.036

NELA: National Emergency Laparotomy Audit of England and Wales; HR: hazard ratio; ICU: intensive care unit; HDU: high dependency unit.

Planned ICU/HDU admission as a predictor of mortality among high-risk patients

Planned ICU/HDU admission was significantly associated with mortality (N = 502: 14.2% versus 6.1%, P = 0.004).

Among those who had a >10% NELA-predicted risk of mortality (n = 146, after exclusion of those patients in the original 168 for whom relevant predictors were unavailable), only the Goals of Care treatment limitations (hazard ratio 3.59, 95% CI 1.03 to 12.49; P = 0.044) and sepsis prior to surgery (hazard ratio 2.84, 95% CI 1.01 to 8.03; P = 0.049) were significantly associated with mortality. Planned admission to ICU/HDU (n = 93) was not significantly associated with a reduced risk of mortality in the multivariable analysis (hazard ratio 1.06, 95% CI 0.28 to 4.01; P = 0.931) (Table 3).

Table 3.

Multivariable Cox proportional hazards regression showing a lack of association between planned intensive care unit/high dependency unit admission (n = 93) immediately after surgery among those who had a National Emergency Laparotomy Audit of England and Wales predicted risk of mortality >10% (n = 146) and risk of hospital mortality.

Predictors	HR (95% confidence intervals)	P-value
Planned ICU/HDU admission immediately after surgery	1.06 (0.28 to 4.01)	0.931
Goals of Care decision with treatment limitations before surgery	3.59 (1.03 to 12.49)	0.044
Sepsis prior to surgery	2.84 (1.01 to 8.03)	0.049
Plasma lactate level (HR per mmol/l increment)	1.21 (0.99 to 1.48)	0.067

The total cohort included 168 patients with a >10% NELA-predicted risk of mortality, however after patients for whom the predictors in this table were excluded, the number reduced to 146.HR: hazard ratio; ICU: intensive care unit; HDU: high dependency unit.

Discussion

The current cohort study showed that the NELA model had a good ability to differentiate between hospital survivors and non-survivors for patients requiring emergency laparotomy. Patients who were considered at high risk of dying (>10%) by the NELA model were — as expected and recommended — more likely to be admitted to ICU/HDU in a planned fashion immediately after surgery, but the compliance to this national recommendation was relatively low (55%). In addition, our study did not confirm the clinical intuition that admitting high-risk patients to ICU/HDU immediately after emergency laparotomy in a planned fashion was associated with a higher chance of survival. These results are clinically relevant and require careful consideration.

First, stratifying patients who are expected to deteriorate after emergency laparotomy and admitting them to the ICU/HDU electively after emergency surgery in a planned fashion makes sense. In this context, our results are useful in confirming that the NELA model has a good ability to predict mortality. These results are, indeed, in good agreement with the national recommendation, and are certainly supportive of the utility of the NELA model as a risk score for patients requiring emergency laparotomy. Nonetheless, we could not confirm the hypothesis that for higher risk patients (who had >10% of predicted risk of mortality), elective admission to the ICU/HDU (regardless of their clinical status) after emergency laparotomy would reduce their risks of unplanned ICU/HDU admission within seven days after surgery nor their mortality. These unexpected results may, at least in part, be related to the limited statistical power of this study. We had only 168 patients who were judged to be at high risk of dying (>10%), and an unexpected proportion of them were not admitted to the ICU/HDU in the first instance after surgery (n = 75).

Second, our results showed that although the NELA model had a good discriminative power in differentiating between survivors and non-survivors, its calibration was imperfect with a tendency to overestimate the actual risks of death, especially when the predicted risks were greater than 50%. Clinicians should therefore be prudent not to use the NELA-predicted risks of mortality as an absolute ‘gold standard’ to guide the Goals of Care discussions with their patients, because the predicted risks had a tendency to mislead clinicians to be overly pessimistic in predicting their patients’ risks of death. This issue is important and has substantial clinical implications because our results showed that Goals of Care discussion with treatment limitations was an independent predictor of mortality among the high-risk patients (hazard ratio 3.59, 95% CI 1.03 to 12.49; P = 0.044) (Table 3). Limiting life support based on overinflated predicted risks of death could result in poor outcomes by committing a self-fulfilling prophecy.

Third, we found that our centre had a lower ICU/HDU admission rate than the national standard (55.4% versus 69.6% respectively). On the other hand, there were also a fair number of patients who were deemed to be at low risk of dying (according to the NELA model) but were admitted to ICU/HDU electively after emergency surgery. Of the patients who had a high NELA-predicted risk (>10%) who did not get admitted to ICU/HDU immediately after surgery, detailed examination of the clinical records showed that the most common reasons for not admitting these patients were related to a change in patient treatment plan, including palliation (for irreversible intra-abdominal pathology such as non-resectable ischaemic bowel disease), or the patients had been assessed by the critical care team after surgery and were deemed to be not requiring close monitoring or intensive care therapy. Of the patients who were admitted to the ICU/HDU despite a low NELA-predicted risk (≤10%), the commonest reasons were related to hospital logistics and specific medical issues (e.g. ongoing anaesthetic medications, those with a difficult airway, or an anaphylactoid reaction intraoperatively). As such, our results did not refute the utility of the NELA model or suggest that clinicians did not adhere to the national recommendation in using the NELA model to risk-stratify patients requiring emergency laparotomy. The low compliance rate to the national recommendation was more related to the fact that clinicians would consider a number of clinical factors in addition to the NELA-predicted risk to decide whether they would admit the patients to the ICU/HDU electively after emergency laparotomy.

We acknowledge that this was a single-centre study; however, the sample size was comparatively large and the NELA data were collected prospectively in a significant proportion (86.3%) of our patients. As we can see in the calibration plot, the number of patients who were judged to be at very high risk (>30%–40%) by the NELA model was small and, hence, the results of these patients were imprecise. Notably since the inception of the ANZELA-QI project, the definition of a high-risk laparotomy has been internationally redefined by the NELA group as patients with a NELA-predicted risk of >5%. Clearly this would have altered the results of this dataset, which is based on the definition of NELA-predicted risk >10%. Further validation of the NELA model, especially in confirming the survival benefits of admitting high-risk cases to ICU/HDU electively after emergency surgery, will be useful to justify the use of scarce critical care resources. Until then, the NELA-predicted risks should be used judiciously after careful consideration of each individual patient’s background and clinical status.

In summary, this study validated the utility of the NELA model for patients who require emergency laparotomy in a tertiary centre in Western Australia. Clinicians appeared to use the NELA-predicted risk of death as well as other information to determine whether such patients were admitted to ICU/HDU electively after surgery. A multicentre study is required to confirm whether admitting all high NELA-predicted risk patients (taking into consideration the new NELA defined high-risk laparotomy of >5%) to an ICU/HDU electively after emergency laparotomy, regardless of their clinical status, is associated with survival benefit.

Supplemental Material

sj-pdf-1-aic-10.1177_0310057X221105291 - Supplemental material for Utility of the National Emergency Laparotomy Audit prognostic model in predicting outcomes in an Australian health system

Supplemental material, sj-pdf-1-aic-10.1177_0310057X221105291 for Utility of the National Emergency Laparotomy Audit prognostic model in predicting outcomes in an Australian health system by Elizabeth T Tran, Kwok M Ho in Anaesthesia and Intensive Care

Footnotes

Author Contribution(s)

Elizabeth T Tran: Conceptualization; Data curation; Methodology; Writing – original draft.

Kwok M Ho: Formal analysis; Investigation; Methodology; Supervision; Writing – review & editing.

Acknowledgements

Thanks to Mr James Aitken, Dr Katherine Broughton, the General Surgery Department, and the Anaesthetic Department at Sir Charles Gairdner Hospital for their collection of data and generous permissions to use the NELA data for this study. Special thanks to Rashmi Rauniyar, Research Coordinator of the Intensive Care Unit and General High Dependency Unit at Sir Charles Gairdner Hospital, for help with APACHE II databases and admission data.

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: KMH is an intensive care editor for Anaesthesia & Intensive Care.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Elizabeth T Tran

Kwok M Ho

Supplemental material

Supplemental material for this article is available online.

References

NELA Project Team. Sixth patient report of the National Emergency Laparotomy Audit. London: RCoA, 2020.

Mak

Hakeem

Chitre

Pre-NELA versus NELA – has anything changed, or is it just an audit exercise?

Ann R Coll Surg Engl 2016; 98: 554–559. doi:10.1308/rcsann.2016.0248

Burmas

Aitken

Broughton

Outcome following emergency laparotomy in Australian public hospitals. ANZ J Surg 2018: 88: 998–1002.

Broughton

Aldridge

Pradhan

, et al. Perth Emergency Laparotomy Audit (PELA). ANZ J Surg 2017; 87; 893–897.

Australian and New Zealand Emergency Laparotomy Audit – Quality Improvement Group. First ANZELA-QI program summary report 1 June 2018 to 30 June 2020. RACS Australia 2020, https://www.surgeons.org/-/media/Project/RACS/surgeons-org/files/morbidity-audits/ANZELA-QI-2020-Report.pdf?rev=3e444531bc894ae2945a3682982ab778&hash=1ED5EE581C5F364CFDC29DE36C663842 (accessed 12 December 2020).

Eliezer

Holmes

Sullivan

, et al. ; forming the Hunter Emergency Laparotomy Collaborator Group. High-risk emergency laparotomy in Australia: Comparing NELA, P-POSSUM, and ACS-NSQIP calculators. J Surg Res 2020; 246: 300–304. doi:10.1016/j.jss.2019.09.024

Royal Australasian College of Surgeons. Australian Emergency Laparotomy Audit – Quality Improvement Project, https://www.surgeons.org/-/media/Project/RACS/surgeons-org/files/morbidity-audits/ANZELA-QI-Inclusion-Exclusion-Criteria.pdf?rev=896e02ca23284bf2b651bab70921b463&hash=ACAB5E71DBF554A8970D2B074871E3B1 (accessed 18 November 2021).

KM.

Forest and funnel plots illustrated the calibration of a prognostic model: A descriptive study. J Clin Epidemiol 2007; 60: 746–751. doi:10.1016/j.jclinepi.2006.10.017

Vergouwe

Steyerberg

Eijkemans

, et al. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol 2005; 58: 475–483.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB