External Validation of the Cardiac Surgery Score in a Quaternary Hospital in the United States of America

Abstract

Background

The Cardiac Surgery Score (CASUS) was developed to assist in predicting post-cardiac surgery mortality using parameters measured in the intensive care unit. It is calculated by assigning points to ten physiologic variables and adding them to obtain a score (additive CASUS), or by logistic regression to weight the variables and estimate the probability of mortality (logistic CASUS). Both additive and logistic CASUS have been externally validated elsewhere, but not yet in the United States of America (USA). This study aims to validate CASUS in a quaternary hospital in the USA and compare the predictive performance of additive to logistic CASUS in this setting.

Methods

Additive and logistic CASUS (postoperative days 1-5) were calculated for 7098 patients at Cleveland Clinic from January 2015 to February 2017. 30-day mortality data were abstracted from institutional records and the Death Registries for Ohio State and the Centers for Disease Control. Given a low event rate, model discrimination was assessed by area under the curve (AUROC), partial AUROC (pAUC), and average precision (AP). Calibration was assessed by curves and quantified using Harrell's Emax, and Integrated Calibration Index (ICI).

Results

30-day mortality rate was 1.37%. For additive CASUS, odds ratio for mortality was 1.41 (1.35-1.46, P <0.001). Additive and logistic CASUS had comparable pAUC and AUROC (all >0.83). However, additive CASUS had greater AP, especially on postoperative day 1 (0.22 vs. 0.11). Additive CASUS had better calibration curves, and lower Emax, and ICI on all days.

Conclusions

Additive and logistic CASUS discriminated well for postoperative 30-day mortality in our quaternary center in the USA, however logistic CASUS under-predicted mortality in our cohort. Given its ease of calculation, and better predictive accuracy, additive CASUS may be the preferred model for postoperative use. Validation in more typical cardiac surgery centers in the USA is recommended.

Keywords

cardiac surgery score CASUS cardiac surgery outcomes 30-day mortality additive CASUS logistic CASUS

Get full access to this article

View all access options for this article.

References

Jin

Furnary

Fine

Blackstone

Grunkemeier

. Using society of thoracic surgeons risk models for risk-adjusting cardiac surgery results. Ann Thorac Surg. Mar 2010;89(3):677-682. doi:10.1016/j.athoracsur.2009.10.078

Nashef

Roques

Sharples

, et al. EuroSCORE II. Eur J Cardiothorac Surg. Apr 2012;41(4):734-744; discussion 744–5. doi:10.1093/ejcts/ezs043

Holmes

Patel

Pritchard

Shuman

Halpin

. Comparison of EuroSCORE II, original EuroSCORE, and The society of thoracic surgeons risk score in cardiac surgery patients. Ann Thorac Surg. Aug 2016;102(2):573-579. doi:10.1016/j.athoracsur.2016.01.105

Hekmat

Kroener

Stuetzer

, et al. Daily assessment of organ dysfunction and survival in intensive care unit cardiac surgical patients. Ann Thorac Surg. May 2005;79(5):1555-1562. doi:10.1016/j.athoracsur.2004.10.017

Marshall

Cook

Christou

Bernard

Sprung

Sibbald

. Multiple organ dysfunction score: a reliable descriptor of a complex clinical outcome. Crit Care Med. Oct 1995;23(10):1638-1652. doi:10.1097/00003246-199510000-00007

Badreldin

Kroener

Heldwein

, et al. Prognostic value of daily cardiac surgery score (CASUS) and its derivatives in cardiac surgery patients. Thorac Cardiovasc Surg. Oct 2010;58(7):392-397. doi:10.1055/s-0030-1250080

Hekmat

Doerr

Kroener

, et al. Prediction of mortality in intensive care unit cardiac surgical patients. Eur J Cardiothorac Surg. Jul 2010;38(1):104-109. doi:10.1016/j.ejcts.2010.01.053

Doerr

Badreldin

Bender

, et al. Outcome prediction in cardiac surgery: the first logistic scoring model for cardiac surgical intensive care patients. Minerva Anestesiol. Aug 2012;78(8):879-886.

Doerr

Badreldin

Heldwein

, et al. A comparative study of four intensive care outcome prediction models in cardiac surgery patients. J Cardiothorac Surg. Mar. 2011;1(6):21. doi:10.1186/1749-8090-6-21

10.

Badreldin

Doerr

Ismail

, et al. Comparison between sequential organ failure assessment score (SOFA) and cardiac surgery score (CASUS) for mortality prediction after cardiac surgery. Thorac Cardiovasc Surg. Feb 2012;60(1):35-42. doi:10.1055/s-0030-1270943

11.

Wilson

Tran

DTT

Dupuis

McDonald

. External validation and updating of the cardiac surgery score for prediction of mortality in a cardiac surgery intensive care unit. J Cardiothorac Vasc Anesth. Nov 2019;33(11):3028-3034. doi:10.1053/j.jvca.2019.03.066

12.

Justice

Covinsky

Berlin

. Assessing the generalizability of prognostic information. Ann Intern Med. Mar. 16 1999;130(6):515-524.

13.

Exarchopoulos

Charitidou

Dedeilias

Charitos

Routsi

. Scoring systems for outcome prediction in a cardiac surgical intensive care unit: a comparative study. Am J Crit Care. Jul 2015;24(4):327-334; quiz 335. doi:10.4037/ajcc2015500

14.

Howitt

Caiado

McCollum

, et al. Validation of three postoperative risk prediction models for intensive care unit mortality after cardiac surgery. Thorac Cardiovasc Surg. Nov 2018;66(8):651-660. doi:10.1055/s-0037-1608897

15.

Austin

Steyerberg

. The integrated calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models. Stat Med. Sep. 20 2019;38(21):4051-4065. doi:10.1002/sim.8281

16.

Saito

Rehmsmeier

. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PloS one. 2015;10(3):e0118432-e0118432. doi:10.1371/journal.pone.0118432

17.

Carrington

Fieguth

Qazi

, et al. A new concordant partial AUC and partial c statistic for imbalanced data in the evaluation of machine learning algorithms. BMC Med Inform Decis Mak. Jan 6 2020;20(1):4. doi:10.1186/s12911-019-1014-6

18.

DeLong

Clarke-Pearson

. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. Sep 1988;44(3):837-845.

19.

Hanley

McNeil

. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. Apr 1982;143(1):29-36. doi:10.1148/radiology.143.1.7063747

20.

Vergouwe

Steyerberg

Eijkemans

Habbema

. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol. May 2005;58(5):475-483. doi:10.1016/j.jclinepi.2004.06.017

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.17 MB