Abstract
Background
Out-of-hospital cardiac arrest (OHCA) represents a major burden for society and health care, with an average incidence in adults of 67 to 170 cases per 100,000 person-years in Europe and in-hospital survival rates of less than 10%. Patients and practitioners would benefit from a prognostication tool for long-term good neurological outcomes.
Objective
We aim to develop a machine learning (ML) pipeline on a local database to classify patients according to their neurological outcomes and identify prognostic features.
Methods
We collected clinical and biological data consecutively from 595 patients who presented OHCA and were routed to a single regional cardiac arrest centre in the south of France. We applied recursive feature elimination and ML analyses to identify the main features associated with a good neurological outcome, defined as a Cerebral Performance Category score less than or equal to 2 at six months post-OHCA.
Results
We identified 12 variables 24 h after admission, capable of predicting a six-month good neurological outcome. The best model (extreme gradient boosting) achieved an AUC of 0.96 and an accuracy of 0.92 in the test cohort.
Conclusion
We demonstrated that it is possible to build accurate, locally optimised prediction and prognostication scores using datasets of limited size and breadth. We proposed and shared a generic machine-learning pipeline which allows external teams to replicate the approach locally.
Introduction
Out-of-hospital cardiac arrest (OHCA) represents a major healthcare burden and one of the leading causes of death. 1 In Europe, the incidence of OHCA with attempted resuscitation is estimated between 67 and 170 per 100,000 person-years. 2 Nowadays, thanks to communication campaigns, improved early recognition and ‘chain of survival’, more and more patients benefit from specialised care.3–5 Over the years, specialised care improved outcomes thanks to intensive resuscitation and invasive organ support, from mechanical ventilation to extracorporeal membrane oxygenation (ECMO).6,7 However, the current survival rate at hospital discharge is estimated at only 8.8%, 8 and many survivors suffer from severe neurological damage secondary to cerebral anoxia. 5 Return to normal cerebral function (cerebral performance category (CPC) score = 1) or moderate cerebral disability (CPC score = 2) varies between 2.8% and 18.2%. 9
Prognostication post-OHCA is central to clinical decision-making, since it can inform family discussions, level of care and the initiation or not of invasive therapies, in the complex medical, ethical and economic landscape of post-cardiac arrest care. While many intensive care unit (ICU) prediction scores exist (OHCA, CAST, MIRACLE2, etc.), 10 their performance fluctuates and invariably degrades when applied to populations different from those used to derive the scores. Machine learning (ML) holds the promise of generating new insight into a vast number of medical topics,11–13 including for the prediction of sepsis or renal failure.14,15 Data-driven analyses with built-in explainability might become the new go-to approach for building prediction scores. The purpose of this study was to build an ML model to predict good neurological outcomes after OHCA and identify key clinical predictors.
Methods
Study design
Data was collected retrospectively from patients admitted between the 21st of January 2014 and the 22nd of December 2021 to Toulouse University Hospital, a tertiary centre located in the south of France. Toulouse University Hospital provides 24/7 access to a large technical platform including a cardiac arrest centre, and cardiovascular surgery. We included patients older than 18 years old, admitted for OHCA regardless of the aetiology. We excluded patients who died before ICU admission, and patients who did not remain comatose after return to spontaneous circulation (ROSC), who were directly routed to another specialised cardiology care centre. Upon arrival, after the patient's assessment and initial blood tests, the decision was made by clinicians to perform coronary angiography and/or to start ECMO, based on available local guidelines. According to the French ethics and regulatory law (public health code), this study was registered in the register of retrospective studies of the Toulouse University Hospital (number RnIPH 2023-02) and covered by reference methodology of the French National Commission for Informatics and Liberties (CNIL). Due to the retrospective nature of the study, passing through an ethics committee was not mandatory.
Outcome
The primary outcome was good neurological outcome, defined as a CPC score of 1 or 2 at six months post-OHCA. Non-survivors were classified as CPC 5.
Data collection
Data were collected from patients’ electronic medical records. The physiological variables, specifically arterial pressure, heart rate, body temperature, and blood results were collected at the time of ICU admission. Pre-hospital data including time of no-flow and low-flow, location of the cardiac arrest (home, street, or workplace) and bystander CPR, were extracted from the emergency department records. Simplified Acute Physiology Score II (SAPS II) was collected at 24 h after ICU admission. Patient medical history and outcomes of interest were also extracted. In total, we included 30 features for each patient.
Data preprocessing
Two features were removed due to high missingness: Neuron-specific enolase (NSE 16 ) at day 3, and duration before ECMO. Categorical variables were transformed into binary variables using the one hot encoding method. We split the dataset into training and test sets, with respectively 80% and 20% of the cohort. Next, we used a feature selection algorithm in the training dataset called recursive feature elimination (RFE) to select only the relevant features for predicting good neurological outcomes. RFE consists of evaluating a model over the recursive deletion of the least contributive input feature, in order to keep the minimal set of features that preserve a high model performance. To perform RFE, we follow the following steps. First, we randomly split the training dataset into training and validation sets (respectively 80% and 20% of the training data), and we fit a model (we selected XGBoost) on the data using grid search cross-validation. It is crucial to use grid search cross-validation at this step because the best hyperparameters for the model may change depending on the number of features. 17 Then, we assess the feature's contributions using SHapeley Additive exPlanations (SHAP), and we remove the least contributive feature from the dataset. We repeat these two steps (fitting the model and removing the least contributive feature) until there is no feature left and report the model's performances in train and validation each time. We repeated the whole RFE procedure 100 times to ensure the robustness of our method.
Model training and performance
Once the final set of features were selected, we trained a model to predict good neurological outcome using the training data (80% of the entire cohort). We built and compared four different models: random forest (RF), gradient boosting model (XGBoost, XGB), logistic regression, and neural network (multilayer perceptron). We then tested each model on the test dataset (representing 20% of the cohort) and reported train and test set confusion matrices, accuracy (ratio of number of correct predictions divided by the total number of predictions), and the area under the receiver operating characteristic (ROC) curves. We selected our best final model based on model accuracy.
Statistical analysis
Analyses were performed using GraphPad Prism 9.0 (GraphPad Software, Inc., San Diego, CA). Student's t-test or Mann–Whitney tests were used for continuous variables, while the Chi-squared test or Fisher's exact test was used for binary features. Variables are presented as mean and standard derivation or percentage. A two-sided p-value of less than .05 was considered to be statistically significant. The code of the analysis is available for reference. 18
Explainability
We used SHAP, which is a post-hoc explainability method where the contribution of each variable to the prediction is estimated for each individual using game theory Shapeley values.19,20
Results
Figure 1 presents a flowchart of the patients included in the study. All patients admitted for OHCA included in the study received CPR and presented ROSC upon arrival. They were all attended to by pre-hospital specialised medical teams which included a medical doctor (the French ‘SAMU’). All the patients could receive advanced medical care during transportation including intubation and mechanical ventilation, mechanical CPR (LUCAS® chest compression system), epinephrine, and other advanced cardiac arrest treatment (amiodarone, lignocaine) as indicated in recommendations at the time of the event.21,22 Patients’ characteristics, in particular severity index, the suspected cause of cardiac arrest, and treatments administered are detailed in Table 1. The median age in the cohort was 56 years old (±17), men were overrepresented (68.2%) and the median SAPS II score was 68.9 (±19.3). Most of the patients received bystander CPR (76.5%), and 62.5% had a no-flow time of less than 5 min. The aetiology remained unknown in 31.6% of cases, and almost a quarter of patients presented ST-elevated myocardial infarction (24.5%).
Flow chart of the patient included in the study.a
Development cohort: descriptive data of patients included in the study.
All values are reported as mean and standard derivation (SD) for continuous variables, and percentage for categorical variables.
BMI: body mass index.
SAPS II: simplified acute physiology score version II.
CPR: cardio-pulmonary resuscitation.
TTM: targeted temperature management.
ECMO: extracorporeal membrane oxygenation.
Coronarography within 2 h after admission.
PCI: percutaneous coronary intervention.
HS: high-sensitive.
STEMI: ST-elevated myocardial Infarction
Predictive performance comparison of the four machine learning models in the test cohort.
XGB: XGBoost.
RF: Random forest.
LR: Logistic regression.
MLP: Multilayer perceptron.
Variables of interest
The results of the RFE to predict good neurological outcome is presented in Figure 2. We identified that a minimum of 12 features were necessary before model accuracy started to deteriorate. We showed that removing features was not associated with any accuracy improvement as it could have reduced background noise.
Receiver operating curves and precision-recall curves for the four models in the test cohorta,b; confusion matrix for neurological outcome prediction in (b) training cohort and (c) test cohort for the best model (xGBoost).
Mean SHAP value for the top 12 features of the XGBoost model.
Discrimination performance was determined using ROC curves and precision-recall curves analysis (Figure 2), while calibration was assessed with confusion matrices (Figure 2(b) and (c)). In the train cohort, the different models achieved AUC between 0.91 and 0.95 and average precision between 0.68 and 0.82. The results were higher in the test cohort with AUC between 0.95 and 0.96 and average precision between 0.77 and 0.85. The best F1-score (0.76) was obtained by the XGBoost model (Table 2). The confusion matrix in the test cohort reveals that the XGBoost model misclassified 20 patients out of 119. Out of the misclassified patients, two were false negatives. The false negative rate was 8% in that model. The O:E ratio was 0.75 in the validation cohort and 0.89 in the development cohort, showing an overestimation of good neurological outcomes.
Explainability
Mean SHAP value extracted from the XGBoost model identified 12 features associated with a good neurological outcome (Figure 3 and Figure 4): emergency coronary angiography, shockable initial rhythm, blood lactate, pH, no-flow duration, SAPS II, low-flow duration, age, serum troponin, epinephrine administration, admission temperature, patient height. The aetiology of OHCA was not identified by the model as a significant contributing feature.
SHAP dependence contribution plots, illustrating the link between individual feature values and SHAP values for the prediction of a good neurological outcome. For each feature, we identified the thresholds which differentiate between positive and negative SHAP values (vertical red lines). The grey histograms represent the distribution of patients for each feature value.
The SHAP analysis also allows us to build dependence contribution plots for each feature, which illustrates further the link between individual feature values and SHAP values. We identified the thresholds which differentiate between positive and negative SHAP values. Figure 4 shows that the SAPS-II score higher than 70.5 at 24 h after admission, no-flow longer than 1.5 min, low-flow longer than 39 min, lactate level higher than 6.35 mmol/l, age under 54.5 years and an admission temperature under 34.7°C were associated with a worse outcome.
Discussion
We deployed ML methods on a cohort of patients following OHCA and constructed a sparse model that accurately predicts a six-months neurological outcome. The main contribution of this study is related to the establishment of a locally tailored model development pipeline, identifying recognised predictive variables, and capable of achieving satisfactory performance. XGBoost-based predictive model achieved the highest prediction score in terms of Accuracy, AUC, F1 score and average precision (respectively 0.92, 0.96, 0.76, 0.84). The aim of this study was to demonstrate the value of modern prediction processes with a new level of dynamism and adaptability. While recent publications focus on utilising vast cohorts 23 that are inaccessible to many clinicians, we aimed to highlight the significant extrapolation capabilities provided by ML-based methods today.
The analyses highlighted 12 features that were considered critical to determine prognosis. Blood lactate was the main predictive feature identified. A low value has the most important impact on the prognostication as the SHAP value might reach a maximum of 0.10. The role of blood lactate as a mortality prognostication marker has been known for a long time. Its metabolism in circulation failure or ischaemia is also well known. This ML approach emphasised its role respecting the known physiological model concerning cardiac arrest. On the contrary, some other features have a more unclear impact. Serum troponin seemed to have either a positive or negative impact. The model interpreted missing serum troponin value as an unfavourable prognostication marker. In fact, some patients were routed to our ICU with an unclear history of the disease and without return on spontaneous circulation. Resuscitation was then stopped at arrival for obvious ethical reasons (such as excessive no-flow duration, known palliative disease, advances directives, etc.). These data, however, were not collected in the study dataset. Height was the least contributive feature in the model. There is no literature comforting its role in OHCA outcomes prognostication. As the serum troponin, it's the missing values that are predictive. We think that it emphasises that our ML model extrapolates from data regardless of the known prevailing physiological model and extracts data even from less obvious features. It emphases also the main limitation of the model prediction as it is strongly affected by initial features selection. This limitation concerns all the prognostication models with or without ML. 24
The thresholds identified by the algorithm were close to those already described in the previous study. 25 Some of the variables we identified were already used in previous prognostication models for neurological outcomes post cardiac arrest at six months, such as SALTED and MIRACLE2 score.26,27 Both these scores were built with the intent of being as parsimonious as possible to simplify prognostication at the time of admission and included only a limited set of features. As a consequence, they likely suffer from low temporal (across time) and external (in new settings) validity. 28 We obtained higher AUC values in the development and validation cohort than SALTED (respectively 0.90 and 0.82), and MIRACLE2 (respectively 0.90 and 0.92). C-GRApH and CAST scores predict neurological outcomes at discharge and reach AUC values of respectively 0.81 and 0.86 (in validation cohorts).29,30
We created an adaptive tool allowing optimised prediction in a dataset of relatively limited size (number of patients) and breadth (number of features). As our study shows, a 600-patient dataset with only 30 features appears large enough to generate a locally robust and highly accurate ML prognostication system. The danger of building prediction models from small datasets is the risk of over-fitting, when the model gives (highly) accurate predictions for the training data but not for new data. We showed that this obstacle could be overcome by using cross-validation, and RFE. 31 Generalisability (the ability of an ML model to perform well across time and/or settings) has been put into question. 32 Instead, ML experts have called for “locally optimised” models that are more likely to find clinical utility at the bedside. The pipeline we propose (data pre-processing, RFE, then model selection) could be replicated by external teams in other institutions to enable them to build local prediction models and prognostication tools. We argue that external validation of the current model in a different centre would be of limited value. Instead, we call for the assessment of the performance of a new model, built implementing our strategy (RFE, model selection and then algorithm testing) with the features and data available at that centre.
The main limitation of our study is the absence of prospective validation in the clinical setting, which represents the next step in the assessment of its clinical utility. However, the model's performance showed great stability between the test set and the development set, as well as during cross-validation. These are positive indicators that mitigate the risk of overfitting. Another point of discussion is that we did not estimate how active withdrawal of care changed the results of neurological prognostication. Neurological outcome evaluation in OHCA patients was based at the beginning of this study (2014), on the review from Sandroni and al. 33 This evaluation considers mainly day-3 features (as blood NSE, EEG results and cerebral scan results) to determine end-of-life support. More recent studies did not bring substantial changes.33,34 We believe that based on that strategy, the patients angled towards end-of-life support had worse feature values and would mainly have been classified in the poor neurological outcome prediction group (CPC > 2).
A central concern with the use of predictive scores such as this one is the rate of false negatives, where patients are erroneously predicted to have a poor neurological outcome, when in reality they evolved favourably. This could have serious consequences, as it might lead to withdrawing assistance from a patient who could have benefited from it. The rate of false negatives in our study in the test cohort was 8%, which is higher than the recommended value of 5%. 34 Modulating the hyperparameters of the models did not allow us to achieve better performance on this element. This limitation probably stems from deficiencies in the dataset.
The ability of models to extrapolate data, to account for feature correlation, and to reduce noise enable its use in many diverse settings. Adaptive prediction models could help evaluate the specificity of predictive features for example: population base, cohort's derivation with time, and calibration drift. 35 As computational power grows and analytical skills in healthcare institutions become more ubiquitous, developing algorithm strategies for outcome prognostication based on ML will become the new standard. This study shows that the future of ICU patient prognostication may include evolving ML models, and should be ICU-specific (regarding the level of care and the population recruitment).
Conclusion
ML models managed to accurately identify the predictive factors of good neurological outcomes following OHCA in a single centre cohort of 595 patients. We proposed and shared a pipeline that can be replicated by external teams in different centres to generate locally optimised prediction and prognostication tools.
Footnotes
Acknowledgements
The authors thank Anais Chekroun and Louise Richez for their participation in collecting data and revising the manuscript.
Contributorship
Vincent Pey and Emmanuel Doumard wrote the article, and take responsibility for the content of the manuscript. Matthieu Komorowski, Antoine Rouget, Clément Delmas, Fanny Vardon-Bounes, Michaël Poette, Valentin Ratineau, Cédric Dray and Isabelle Ader helped revise the manuscript and Vincent Minville helped draft and revise the manuscript.
Consent statement
According to French law on ethics, patients were informed that their codified data will be used for the study. According to the French ethics and regulatory law (public health code) retrospective studies based on the exploitation of usual care data should be submitted at an ethic committee but they have to be declared or covered by reference methodology of the French CNIL. A collection and computer processing of personal and medical data was implemented to analyze the results of the research. Toulouse University Hospital signed a commitment of compliance to the reference methodology MR-004 of the French CNIL. After evaluation and validation by the data protection officer and according to the General Data Protection Regulation*, this study completes all the criteria, and it is registered in the register of a retrospective study of the Toulouse University Hospital (number's register: RnIPH 2023-02) and cover by the MR-004 (CNIL number: 2206723 v 0). This study was approved by Toulouse University Hospital and confirmed that ethic requirements were totally respected in the above report.
*Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article
Guarantor
Vincent Pey: This author wrote the article and takes responsibility for the content of the manuscript.
