Predicting Outcomes Following Carotid Artery Stenting Using Machine Learning

Abstract

Background:

Carotid artery stenting (CAS) carries important perioperative risks. Outcome prediction tools may help guide clinical decision-making but remain limited. We developed machine learning (ML) algorithms that predict 30-day outcomes following transfemoral CAS.

Methods:

The National Surgical Quality Improvement Program (NSQIP) targeted vascular database was used to identify patients who underwent transfemoral CAS between 2011 and 2021. Input features included 36 preoperative demographic/clinical variables. The primary outcome was a 30-day major adverse cardiovascular event (MACE; composite of stroke, myocardial infarction [MI], or death). The secondary outcomes were 30-day stroke, MI, death, carotid-related morbidity, other morbidity, non-home discharge, and unplanned readmission. Our data were split into training (70%) and test (30%) sets. Using 10-fold cross-validation, we trained six ML models using preoperative features with logistic regression as the baseline comparator. The primary model evaluation metric was area under the receiver operating characteristic curve (AUROC). Model robustness was evaluated with calibration plot and Brier score. Variable importance scores were calculated to determine the top 10 predictive features. Performance was assessed on subgroups based on age, sex, race, ethnicity, symptom status, stent type, and urgency.

Results:

Overall, 2093 patients underwent CAS during the study period. Thirty-day MACE occurred in 130 (6.2%) patients. The best-performing prediction model for 30-day MACE was XGBoost, achieving an AUROC (95% CI) of 0.93 (0.92–0.94). In comparison, logistic regression had an AUROC (95% CI) of 0.67 (0.65–0.68), and existing tools in the literature demonstrate AUROCs ranging from 0.58 to 0.74. For secondary outcomes, XGBoost achieved AUROCs between 0.86 and 0.97. The calibration plot showed good agreement between predicted and observed event probabilities with a Brier score of 0.02. The top three predictive features in our algorithm were (1) symptomatic carotid stenosis, (2) age, and (3) American Society of Anesthesiologists classification. Model performance remained robust on all subgroup analyses of specific demographic and clinical populations.

Conclusions:

Our ML models accurately predict 30-day outcomes following transfemoral CAS using preoperative data. They have the potential for important utility in guiding risk-mitigation strategies for patients being considered for CAS to improve outcomes.

Clinical Impact

Transfemoral carotid artery stenting (CAS) carries important perioperative risks. Outcome prediction tools may help guide clinical decision-making but remain limited. Using data from the National Surgical Quality Improvement Program (NSQIP) targeted vascular database, we developed machine learning (ML) models that accurately predict 30-day outcomes following transfemoral CAS using preoperative data, outperforming logistic regression and existing tools in the literature. The models were well-calibrated and remained robust across demographic and clinical subpopulations. These ML algorithms have the potential for important utility in guiding risk-mitigation strategies for patients being considered for transfemoral CAS to improve outcomes.

Keywords

prediction major adverse cardiovascular event MACE carotid artery stenting machine learning

Introduction

Carotid artery stenosis accounts for approximately one-third of global ischemic strokes and significantly impacts morbidity and mortality.¹ Traditionally, moderate-severe carotid artery stenosis has been treated with surgical carotid endarterectomy (CEA).² In recent decades, transfemoral carotid artery stenting (CAS) has emerged as a less invasive alternative.³ The Carotid Revascularization Endarterectomy versus Stenting Trial (CREST) found no significant difference in the primary outcome of stroke, myocardial infarction, or death between CAS and CEA.⁴ However, CAS was associated with a higher risk of stroke, while CEA was associated with a higher risk of myocardial infarction.⁴ Despite this, CAS procedures have increased by 72% over the past decade based on population-level data.⁵ Although considered minimally invasive, transfemoral CAS carries significant perioperative risks, including a 30-day major adverse cardiovascular event (MACE) rate exceeding 9% in high-risk patients, as defined by the Centers for Medicare and Medicaid Services (CMS), including age ≥80 years, New York Heart Association congestive heart failure (CHF) class III/IV, left ventricular ejection fraction <30%, unstable angina within 30 days prior to intervention, myocardial infarction (MI) within 30 days prior to intervention, restenosis, previous radical neck dissection, contralateral carotid occlusion, prior neck radiation, contralateral laryngeal nerve injury/palsy, or high anatomic lesion.⁶ The Society for Vascular Surgery (SVS) recommends CEA for low-risk patients⁷ and reserves CAS for those with high-risk anatomical or physiological features.⁸ Therefore, accurate risk assessment is crucial for guiding clinical decisions.

Currently, there are no widely adopted tools available to predict adverse events following CAS. A systematic review analyzing 37 studies that assessed outcome prediction models for carotid revascularization highlighted significant methodological shortcomings, incomplete reporting, and insufficient predictive accuracy.⁹ For instance, many existing models do not report how missing data were managed and/or demonstrate suboptimal discriminatory ability, with area under the receiver operating characteristic curve (AUROC) values ranging from 0.58 to 0.74.⁹ Moreover, these tools rely on traditional modeling techniques that necessitate manual input of clinical variables, which limits their practical use in busy clinical environments.¹⁰ Notably, the SVS Vascular Quality Initiative (VQI) Cardiac Risk Index (CRI) offers risk assessments for CEA but does not include CAS.¹¹ Consequently, there is a pressing need to develop more effective risk prediction tools specifically tailored for patients undergoing CAS.

Machine learning (ML) is advancing rapidly, enabling computers to learn from data and accurately predict outcomes.¹² Through sophisticated analytics, ML can model complex relationships between inputs (e.g., patient characteristics) and outputs (e.g., clinical outcomes), driven by the vast availability of electronic data and enhanced computational capabilities.¹² ML techniques excel over traditional statistical methods in capturing intricate, multicollinear relationships among variables and outcomes in health care data.^13,14 Previously, ML has successfully leveraged the American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) database to develop algorithms predicting peri-operative complications.¹⁵ Using data from more than 2900 procedures, the authors achieved AUROC values ranging from 0.85 to 0.88.¹⁵ Given the heterogeneity of this cohort, there is potential to enhance predictive accuracy by tailoring ML algorithms specifically for patients undergoing CAS. In this study, we applied ML to the ACS NSQIP database to predict 30-day MACE following transfemoral CAS using preoperative variables.

Materials and Methods

Design

We conducted a multicenter retrospective ML-based prognostic study reported according to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis + Artificial Intelligence (TRIPOD + AI) statement.¹⁶

Dataset

The ACS NSQIP database contains demographic, clinical, and 30-day outcomes data on surgical patients across over 700 hospitals in approximately 15 countries worldwide.¹⁷ The information is prospectively collected from electronic health records by trained clinical reviewers and regularly audited by ACS for accuracy.¹⁸ Targeted NSQIP registries for vascular operations contain additional procedure-specific variables and outcomes.¹⁹ This study was exempt from institutional ethics board review and informed consent was not required as the data came from a large, deidentified registry.

Cohort

All patients who underwent transfemoral CAS from 2011-2021 in the ACS NSQIP targeted CAS database were included. This information was merged with the main ACS NSQIP database using unique case identification numbers for a complete set of generic and procedure-specific variables and outcomes.²⁰ Patients with unreported presenting symptom status or stent type or treated for carotid aneurysm or dissection or malignancy were excluded.

Features

Thirty-six preoperative variables were used as input features for the ML models, as determined based on the availability of the variables in the NSQIP database and their important impact on CAS outcomes based on the literature.^21
–23 Given the unique advantage of ML techniques in handling a large number of input features, all available preoperative variables in the NSQIP database were used to maximize predictive performance. There were no significant confounding effects between variables. Demographic variables included age, sex, body mass index, race, ethnicity, and origin status. Comorbidities included hypertension, diabetes, smoking status, CHF, chronic obstructive pulmonary disease (COPD), end-stage renal disease requiring dialysis, functional status, and physiologic high-risk factor [defined as at least one of (1) CHF class III/IV, (2) left ventricular ejection fraction <30%, (3) unstable angina within 30 days prior to intervention, or (4) MI within 30 days prior to intervention]. Medications included antiplatelets, statins, and beta-blockers. Preoperative laboratory investigations included serum sodium, blood urea nitrogen (BUN), serum creatinine, albumin, white blood cell count, hematocrit, platelet count, international normalized unit (INR), and partial thromboplastin time (PTT). Anatomic characteristics included ipsilateral/contralateral carotid stenosis percentage and anatomic high-risk factor [defined as at least one of (1) previous ipsilateral carotid endarterectomy or stent, (2) previous radical neck dissection, (3) contralateral carotid occlusion, (4) prior neck radiation, (5) contralateral laryngeal nerve injury/palsy, or (6) high anatomic lesion (cervical vertebrae 2 or higher)]. Other pre-procedural characteristics recorded were symptom status [asymptomatic or symptomatic (history of stroke, transient ischemic attack [TIA], or amaurosis fugax within 180 days prior to CAS)], stent type (single straight, single straight with cerebral protection device [CPD], single tapered, single tapered with CPD, multiple stents, or multiple stents with CPD), urgency [elective, urgent, or emergent], American Society of Anesthesiologists (ASA) classification, and specialty of the primary physician performing the procedure. A complete list of features and definitions can be found in Supplemental Table 1.

Outcomes

The primary outcome was a 30-day MACE, defined as a composite of stroke, MI, or death. Stroke was defined as motor, sensory, or cognitive dysfunction that persists for 24 hours in the setting of a suspected ischemic or hemorrhagic stroke in the ipsilateral or contralateral cerebral hemisphere. MI was defined as electrocardiogram changes indicative of acute MI (ST elevation >1 mm in two or more contiguous leads, new left bundle branch block, or new q-wave in two or more contiguous leads), new elevation in troponin greater than 3 times the regular upper level of the reference range in the setting of suspected myocardial ischemia, or physician/advanced provider diagnosis of MI. Death was defined as all-cause mortality. This composite outcome was chosen because it is frequently reported as a primary outcome in landmark clinical trials including CREST.^4,24

Secondary outcomes were 30-day stroke, MI, death, carotid-related morbidity, other morbidity, non-home discharge, and unplanned readmission. Carotid-related morbidity was defined as a composite of distal embolization causing ipsilateral cerebral infarcts demonstrated on Doppler ultrasound, computed tomography angiography (CTA), magnetic resonance angiography (MRA), or angiogram, acute occlusion or thrombosis of the ipsilateral carotid artery demonstrated on Doppler ultrasound, CTA, MRA, or angiogram, TIA (neurologic dysfunction lasting <24 hours without evidence of cerebral infarction), puncture site bleeding/pseudoaneurysm or embolization of arterial closure device, re-stenosis >50% on postoperative Doppler ultrasound, CTA, MRA, or angiogram, or repeat carotid revascularization (endarterectomy or stent). Other morbidity was defined as a composite of surgical site infection (SSI), pneumonia, unplanned reintubation, pulmonary embolism (PE), failure to wean from ventilator (cumulative time of ventilator-assisted respirations >48 hours), acute kidney injury (AKI; a rise in creatinine of >2 mg/dL from preoperative value or requirement of dialysis in a patient who did not require dialysis preoperatively), urinary tract infection (UTI), cardiac arrest, bleeding requiring blood transfusion within 72 hours of intervention, deep vein thrombosis (DVT) requiring therapy, Clostridium difficile infection, sepsis, or septic shock. Non-home discharge was defined as discharge to rehabilitation, skilled care, or other facility.

Model Development

Six ML models were trained to predict 30-day primary and secondary outcomes following CEA: Extreme Gradient Boosting (XGBoost), random forest, Naïve Bayes classifier, radial basis function (RBF) support vector machine (SVM), single-layer perceptron artificial neural network (ANN) with a single hidden layer, sigmoid activation function, and cross-entropy loss function, and logistic regression. These were selected because they demonstrate the best performance for predicting surgical outcomes.^25
–27 Logistic regression was the baseline comparator to assess relative model performance because it is the most common modeling technique used in traditional risk predictors.²⁸

Our data were randomly split into training (70%) and test (30%) sets.²⁹ Unique patient identification numbers were used to ensure that the training and testing populations were separated. Ten-fold cross-validation and grid search were performed on the training set to find optimal model hyperparameters.^30,31 Preliminary analysis of our data demonstrated that the primary outcome was uncommon, occurring in 130/2093 (6.2%) patients. To improve class balance, Random Over-Sample Examples (ROSE) was applied to training data.³² ROSE employs smoothed bootstrapping to draw new samples from the feature space around the minority class and is a commonly used method to support predictive modeling of rare events.³² The models were then evaluated on test set data and ranked based on the primary discriminatory metric of AUROC. The best-performing model was XGBoost, which had the following optimized hyperparameters: number of rounds = 200, maximum tree depth = 3, learning rate = 0.3, gamma = 0, column sample by tree = 0.6, minimum child weight = 1, subsample = 1. The process for selecting these hyperparameters through grid search and cross-validation is detailed in Supplementary Table 2. Once the best-performing model for the primary outcome was identified, we trained the same model to predict secondary outcomes. Parameter space was optimized for XGBoost given previous literature demonstrating the superiority of this model for using structured data to predict binary outcomes.^33
–35

Statistical Analysis

Baseline demographic and clinical characteristics for patients with versus without 30-day MACE were summarized as mean (standard deviation) or number (proportion). Differences between groups were assessed using independent t-test (continuous variables) or chi-square test (categorical variables). Statistical significance was adjusted using Bonferroni correction to account for multiple comparisons. The p-values for categorical variables with multiple categories, such as race, were determined based on multi-cell chi-square tests.

The primary metric for assessing model performance was AUROC (95% CI), a validated discriminatory metric considers both sensitivity and specificity.³⁶ Secondary performance metrics were accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Confidence intervals for AUROC, accuracy, sensitivity, specificity, PPV, and NPV were calculated using Newcombe’s Wald method.^37,38 To further assess model performance, we plotted a calibration curve and calculated the Brier score, a measurement of the agreement between predicted and observed event probabilities.³⁹ In the final model, feature importance was determined by ranking the top 10 predictors based on the variable importance score (gain), a measure of the relative impact of individual covariates in contributing to an overall prediction.⁴⁰ Feature importance was determined separately for the overall cohort, symptomatic patients, and asymptomatic patients. This was done to understand the distinction between features that may be more pronounced in one group versus the other, as factors that influence CAS outcomes may be different in symptomatic versus asymptomatic patients. To assess model robustness across demographic/clinical subpopulations, we performed a subgroup analysis of predictive performance based on age, sex, race, ethnicity, symptom status, stent type with or without CPD, and urgency.

Based on a validated sample size calculator for clinical prediction models, to achieve a minimum AUROC of 0.9 with an outcome rate of ~6% and 36 input features, the minimum sample size required is 2,006 patients with 121 events.^41,42 Our cohort of 2093 patients with 130 primary events satisfied this sample size requirement. The variables of interest displayed <5% missing data; hence, complete-case analysis was applied whereby only non-missing covariates for each patient were considered.⁴³ This is a valid analytical method for datasets with minimal missing data (<5%) and reflects predictive modeling of real-world data, which inherently includes missing information.^44,45 For ML algorithms that require an input value for missing data (e.g., neural network and logistic regression), the mean value of the variable was used, which has no significant impact on model performance for small amounts of missing data.⁴⁶ All analyses were performed in R version 4.3.0⁴⁷ with the following packages: caret,⁴⁸ xgboost,⁴⁹ ranger,⁵⁰ naivebayes,⁵¹ e1071,⁵² nnet,⁵³ and pROC.⁵⁴

Results

Patients and Events

From an initial cohort of 2212 patients who underwent transfemoral CAS in the NSQIP targeted vascular database between 2011 and 2021, we excluded 119 patients for the following reasons: undocumented symptom status (n=59) or stent type (n=43) and treatment for carotid aneurysm (n=5), carotid dissection (n=2), or malignancy (n=10). Overall, we included 2093 patients. The primary outcome of 30-day MACE occurred in 130 (6.2%) patients. The 30-day secondary outcomes occurred in the following distribution: 65 (3.1%) patients had a stroke, 49 (2.3%) patients had a MI, 36 (1.7%) patients died, 171 (8.2%) patients had a carotid-related morbidity (distal embolization [n=24], acute occlusion or thrombosis [n=22], TIA [n=23], puncture site bleeding/pseudoaneurysm or embolization of arterial closure device [n=31], restenosis [n=18], repeat carotid revascularization [n=83]), 145 (6.9%) patients had other morbidity (SSI [n=5], pneumonia [n=30], unplanned reintubation [n=37], PE [n=3], failure to wean from ventilator [n=20], AKI [n=14], UTI [n=20], cardiac arrest [n=6], bleeding requiring transfusion [n=61], DVT [n=6], Clostridium difficile infection [n=4], sepsis [n=6], septic shock [n=9]), 222 (10.6%) patients had a non-home discharge, and 162 (7.7%) patients had an unplanned readmission.

Preoperative Demographic and Clinical Characteristics

Compared to patients without a primary outcome, those who developed MACE at 30 days were older (72.9 [SD 10.1] vs. 69.8 [SD 9.8] years, p<0.001) and more likely to reside in nursing homes (2.3% vs. 0.9%, p<0.001) or be transferred from another hospital (22.3% vs. 10.5%, p<0.001). There were no significant differences in comorbidities between groups, although patients with 30-day MACE were less likely to receive antiplatelets (89.2% vs. 93.7%, p=0.004). Notable differences in laboratory investigations included higher mean creatinine (118.48 [SD 41.7] vs. 100.8 [SD 22.9] umol/L, p=0.002) and BUN (22.7 [SD 13.7] vs. 20.0 [SD 9.4] mmol/L, p=0.002) in patients with an event. A greater proportion of patients with a primary outcome had symptomatic carotid stenosis (59.2% vs. 48.2%, p=0.005), required urgent/emergent intervention (40.8% vs. 24.2%, p<0.001), and had an ASA class above 3 (39.2% vs. 23.3%, p=0.006) (Table 1).

Table 1.

Preoperative Demographic and Clinical Characteristics of Patients Undergoing Carotid Artery Stenting With and Without Major Adverse Cardiovascular Events at 30 Days.

	Absence of MACE at 30 days (n=1963)	Presence of MACE at 30 days (n = 130)	P
Demographics
Age, years, mean (SD)	69.8 (9.8)	72.9 (10.1)	< 0.001
Female	699 (35.6)	49 (37.7)	0.70
BMI, kg/m², mean (SD)	29.0 (6.3)	28.7 (7.0)	0.59
Race
White	1,682 (85.7)	106 (81.5)	0.24
Black or African American	168 (8.6)	16 (12.3)
American Indian or Alaskan Native	1 (0.05)	0
Native Hawaiian or Other Pacific Islander	2 (0.1)	1 (0.8)
Asian	18 (0.9)	2 (1.5)
Other	5 (0.3)	0
Unknown or not reported	87 (4.4)	5 (3.9)
Hispanic ethnicity^a	56 (2.9)	5 (3.9)	0.43
Origin status
Transferred from another hospital	206 (10.5)	29 (22.3)	< 0.001
Home	1,727 (88.0)	98 (75.4)
Nursing home	17 (0.9)	3 (2.3)
Other facility	13 (0.7)	0
Comorbidities
Hypertension	1,612 (82.1)	110 (84.6)	0.55
Diabetes
Non-insulin-dependent	350 (17.8)	28 (21.5)	0.27
Insulin-dependent	300 (15.3)	24 (18.5)	0.27
Current smoker	545 (27.8)	32 (24.6)	0.44
Congestive heart failure	90 (4.6)	10 (7.7)	0.11
Chronic obstructive pulmonary disease	260 (13.2)	19 (14.6)	0.66
Dialysis	28 (1.4)	4 (3.1)	0.13
Functional status
Independent	1,886 (96.1)	121 (93.1)	0.26
Partially/totally dependent	75 (3.8)	9 (6.9)
Unknown	2 (0.1)	0
Physiologic high-risk factor^b	191 (9.7)	13 (10.0)	0.92
Medications
Antiplatelet	1,839 (93.7)	116 (89.2)	0.004
Statin	1,661 (84.6)	106 (81.5)	0.35
Beta-blocker	1,119 (57.0)	73 (56.2)	0.85
Laboratory investigations
Sodium, mmol/L, mean (SD)	139.0 (3.0)	138.0 (3.5)	0.08
BUN, mmol/L, mean (SD)	20.0 (9.4)	22.7 (13.7)	0.002
Creatinine, umol/L, mean (SD)	100.8 (22.9)	118.48 (41.7)	0.002
Albumin, g/L, mean (SD)	38.6 (3.7)	37.4 (5.0)	< 0.001
White blood cell count, cells/mm³, mean (SD)	7.8 (2.5)	7.8 (2.5)	0.97
Hematocrit, L/L (%), mean (SD)	39.3 (5.2)	37.7 (5.8)	< 0.001
Platelet count, 10⁹/L, mean (SD)	231.0 (72.2)	228.0 (89.1)	0.68
INR, mean (SD)	1.1 (0.3)	1.1 (0.3)	0.23
PTT, sec, mean (SD)	34.4 (10.1)	38.8 (17.2)	< 0.001
Anatomy
Ipsilateral carotid stenosis percentage
<50%	28 (1.4)	4 (3.1)	0.33
50–79%	518 (26.4)	31 (23.8)
80–99%	1,308 (66.6)	84 (64.6)
Occluded	67 (3.4)	6 (4.6)
Unknown	42 (2.1)	5 (3.9)
Contralateral carotid stenosis percentage
<50%	861 (43.9)	49 (37.7)	0.09
50–79%	470 (23.9)	28 (21.5)
80–99%	149 (7.6)	14 (10.8)
Occluded	152 (7.7)	7 (5.4)
Unknown	331 (16.9)	32 (24.6)
Anatomic high-risk factor^c	737 (37.5)	46 (35.4)	0.62
Other pre-procedural characteristics
Symptom status
Asymptomatic	1,017 (51.8)	53 (40.8)	0.005
Symptomatic	946 (48.2)	77 (59.2)
Previous stroke	793 (40.4)	71 (54.6)
Previous transient ischemic attack or amaurosis fugax	153 (7.8)	6 (4.6)
Stent type
Single straight	298 (15.2)	26 (20.0)	0.09
Single straight with cerebral protection device	1,127 (57.4)	62 (47.7)
Single tapered	108 (5.5)	10 (7.7)
Single tapered with cerebral protection device	339 (17.3)	21 (16.2)
Multiple stents	37 (1.9)	4 (3.1)
Multiple stents with cerebral protection device	54 (2.8)	7 (5.4)
Urgency
Elective	1,490 (75.9)	77 (59.2)	< 0.001
Urgent	372 (19.0)	37 (28.5)
Emergent	101 (5.2)	16 (12.3)
ASA class
1	4 (0.2)	0	0.006
2	85 (4.3)	4 (3.1)
3	1,358 (69.2)	72 (55.4)
4	453 (23.1)	51 (39.2)
5	3 (0.2)	0
Not reported	60 (3.1)	3 (2.3)
Primary specialty
Interventional radiology	83 (4.2)	7 (5.4)	0.04
Neurosurgery	359 (18.3)	36 (27.7)
Vascular surgery	1,503 (76.6)	86 (66.2)
Other^d	18 (0.9)	1 (0.8)

Values are reported as number (%) unless otherwise indicated.

Abbreviations: MACE, major adverse cardiovascular event; BMI, body mass index; BUN, blood urea nitrogen; INR, international normalized ratio; PTT, partial thromboplastin time; ASA, American Society of Anesthesiologists; SD, standard deviation.

Hispanic ethnicity is reported independently from race because the US Census Bureau classifies Hispanic status as an ethnicity rather than a race, and they note that Hispanic people can be of any race (https://www.census.gov/topics/population/race/about.html).

At least one of the following: (1) New York Heart Association congestive heart failure class III/IV, (2) left ventricular ejection fraction <30%, (3) unstable angina within 30 days prior to intervention, or 4) myocardial infarction within 30 days prior to intervention.

At least 1 of the following: (1) previous ipsilateral carotid endarterectomy or stent, (2) previous radical neck dissection, (3) contralateral carotid occlusion, (4) prior neck radiation, (5) contralateral laryngeal nerve injury/palsy, or (6) high anatomic lesion (cervical vertebrae 2 or higher).

Includes cardiac surgery and general surgery.

Model Performance

Of the six ML models evaluated on test set data for predicting 30-day MACE following CAS, XGBoost had the best performance with an AUROC (95% CI) of 0.93 (0.92–0.94) compared to random forest [0.92 (0.91–0.93)], Naïve Bayes [0.84 (0.83–0.86)], RBF SVM [0.83 (0.82–0.84)], MLP ANN [0.73 (0.71–0.74)], and logistic regression [0.67 (0.65–0.68)]. The other performance metrics of XGBoost were the following: accuracy 0.86 (95% CI 0.84–0.87), sensitivity 0.84, specificity 0.88, PPV 0.88, and NPV 0.83 (Table 2).

Table 2.

Model Performance on Test Set Data for Predicting 30-Day Major Adverse Cardiovascular Events Following Carotid Artery Stenting Using Preoperative Features.

	AUROC (95% CI)	Accuracy (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)
XGBoost	0.93 (0.92–0.94)	0.86 (0.84–0.87)	0.84 (0.83–0.85)	0.88 (0.86–0.89)	0.88 (0.87–0.89)	0.83 (0.82–0.85)
Random forest	0.92 (0.91–0.93)	0.85 (0.84–0.86)	0.85 (0.84–0.86)	0.85 (0.84–0.86)	0.86 (0.84–0.87)	0.84 (0.82–0.86)
Naïve Bayes	0.84 (0.83–0.86)	0.84 (0.83–0.86)	0.83 (0.82–0.84)	0.86 (0.85–0.87)	0.87 (0.86–0.89)	0.81 (0.80–0.83)
RBF SVM	0.83 (0.82–0.84)	0.75 (0.73–0.76)	0.72 (0.70–0.74)	0.78 (0.77–0.79)	0.79 (0.77–0.80)	0.70 (0.69–0.71)
ANN	0.73 (0.71–0.74)	0.67 (0.66–0.69)	0.65 (0.63–0.67)	0.71 (0.70–0.73)	0.78 (0.77–0.80)	0.57 (0.55–0.59)
Logistic regression	0.67 (0.65–0.68)	0.61 (0.59–0.62)	0.57 (0.56–0.58)	0.73 (0.72–0.74)	0.68 (0.67–0.70)	0.53 (0.51–0.54)

Abbreviations: XGBoost, Extreme Gradient Boosting; AUROC, area under the receiver operating characteristic curve; CI, confidence interval; PPV, positive predictive value; NPV, negative predictive value; RBF SVM, radial basis function support vector machine; ANN, artificial neural network.

For 30-day secondary outcomes, XGBoost attained the following AUROCs (95% CI): stroke [0.89 (0.88–0.90)], MI [0.93 (0.92–0.94)], death [0.94 (0.93–0.95)], carotid-related morbidity [0.86 (0.85–0.87)], other morbidity [0.92 (0.91–0.93)], non-home discharge [0.97 (0.96–0.98)], and unplanned readmission [0.86 (0.85–0.88)] (Table 3).

Table 3.

XGBoost Performance on Test Set Data for Predicting 30-day Secondary Outcomes Following Carotid Artery Stenting Using Preoperative Features.

	AUROC (95% CI)	Accuracy (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)
Stroke	0.89 (0.88–0.90)	0.81 (0.79–0.82)	0.80 (0.79–0.81)	0.82 (0.80–0.83)	0.84 (0.83–0.85)	0.77 (0.75–0.78)
Myocardial infarction	0.93 (0.92–0.94)	0.85 (0.84–0.86)	0.84 (0.83–0.85)	0.86 (0.85–0.87)	0.86 (0.85–0.87)	0.83 (0.82–0.85)
Death	0.94 (0.93–0.95)	0.89 (0.88–0.90)	0.89 (0.88–0.91)	0.90 (0.89–0.91)	0.90 (0.89–0.91)	0.88 (0.87–0.89)
Carotid-related morbidity	0.86 (0.85–0.87)	0.78 (0.76–0.79)	0.77 (0.76–0.78)	0.79 (0.77–0.80)	0.80 (0.79–0.81)	0.75 (0.74–0.76)
Other morbidity	0.92 (0.91–0.93)	0.84 (0.83–0.86)	0.84 (0.83–0.85)	0.85 (0.84–0.86)	0.86 (0.85–0.88)	0.83 (0.82–0.85)
Non-home discharge	0.97 (0.96–0.98)	0.91 (0.90–0.92)	0.91 (0.90–0.92)	0.91 (0.90–0.92)	0.91 (0.90–0.92)	0.91 (0.90–0.92)
Unplanned readmission	0.86 (0.85–0.88)	0.77 (0.76–0.79)	0.77 (0.76–0.78)	0.78 (0.77–0.80)	0.79 (0.78–0.80)	0.76 (0.75–0.78)

Abbreviations: XGBoost, Extreme Gradient Boosting, AUROC, area under the receiver operating characteristic curve; CI, confidence interval; PPV, positive predictive value; NPV, negative predictive value.

The ROC curve of the XGBoost model is reported in Figure 1. The calibration plot demonstrated good agreement between predicted and observed event probabilities with a Brier score of 0.02 (Figure 2). The top 10 predictors of increased risk of 30-day MACE following CAS in the XGBoost model were the following: (1) symptomatic carotid stenosis, (2) older age, (3) higher ASA class, (4) higher preoperative creatinine, (5) transferred from another hospital, (6) urgency (urgent/emergent procedure), (7) stent type (single straight/tapered stent without a cerebral protection device or multiple stents), (8) preoperative CHF, (9) lack of preoperative antiplatelet, and (10) primary specialty of the proceduralist (interventional radiology, neurosurgery, or other non-vascular surgery specialty) (Figure 3). On subgroup analysis based on symptom status, 9/10 of the most important features were the same for symptomatic and asymptomatic patients, with the top 3 predictors being age, ASA class, and preoperative creatinine for both groups. Transfer from another hospital was a top 10 predictor for 30-day MACE in patients with symptomatic carotid stenosis, but not in patients with asymptomatic carotid stenosis. Preoperative BUN was a top 10 predictor for 30-day MACE in patients with asymptomatic carotid stenosis, but not in patients with symptomatic carotid stenosis (Supplemental Figure 1).

Figure 1.

Receiver operating characteristic curve for predicting 30-day major adverse cardiovascular events following carotid artery stenting using Extreme Gradient Boosting (XGBoost) model. AUROC, area under the receiver operating characteristic curve; CI, confidence interval.

Figure 2.

Calibration plot with Brier score for predicting 30-day major adverse cardiovascular events following carotid artery stenting using Extreme Gradient Boosting (XGBoost) model.

Figure 3.

Variable importance scores (gain) for the top 10 predictors of 30-day major adverse cardiovascular events following carotid artery stenting in the Extreme Gradient Boosting (XGBoost) model. ASA, American Society of Anesthesiologists; CHF, congestive heart failure.

Subgroup Analysis

The XGBoost model performance for predicting 30-day MACE remained excellent on all subgroup analyses of demographic and clinical subpopulations, with AUROCs ranging from 0.92 to 0.94 and no significant differences between majority and minority groups (Supplemental Figures 2–8).

Discussion

Summary of Findings

In this study, we leveraged data from the ACS NSQIP targeted vascular files between 2011 and 2021 consisting of 2,093 patients who underwent transfemoral CAS to develop ML models that accurately predict 30-day MACE with an AUROC of 0.93. Additionally, our algorithms predicted 30-day stroke, MI, death, carotid-related morbidity, other morbidity, non-home discharge, and unplanned readmission with AUROCs ranging from 0.86 to 0.97. Several significant findings emerged from our analysis. First, patients who develop 30-day MACE following CAS constitute a high-risk population with predictive factors at the preoperative stage, including older age and higher creatinine with a greater proportion having symptomatic carotid stenosis and requiring urgent/emergent intervention. Second, we trained six ML models to predict 30-day MACE using preoperative features and showed that XGBoost achieved the best performance. Our model was well-calibrated and remained robust on subgroup analyses based on age, sex, race, ethnicity, symptom status, stent type, and urgency. Finally, we identified the top 10 predictors of 30-day MACE in our ML models. These features offer clinicians valuable insights into the factors influencing risk predictions, thereby guiding patient selection and preoperative optimization. Overall, we have developed a robust ML-based risk assessment tool that can help guide clinical decision-making to improve outcomes and reduce costs from complications, reinterventions, and readmissions associated with CAS.

Comparison to Existing Literature

Volkers et al (2018) conducted a systematic review encompassing 37 studies that developed 46 prediction models for patients undergoing carotid revascularization.⁹ The majority of these models were for CEA (74%), with a minority focused on CAS (26%).⁹ Most studies utilized traditional statistical methods like logistic regression or Cox proportional hazards analysis, achieving AUROC values ranging from 0.58 to 0.74,⁹ while we achieved AUROCs >0.90 using ML methods. Most (54%) models did not discuss how missing data was handled,⁹ while we used complete-case analysis due to a small amount of missing data. Furthermore, none of the existing models predicted readmission,⁹ whereas our models included secondary outcomes such as non-home discharge and unplanned readmission, which impact patient outcomes and healthcare costs.⁵⁵ Compared to current CAS risk prediction tools, our ML algorithms exhibit methodological strength and better performance on more clinically relevant outcomes. Additionally, our models fill a gap by providing accurate risk predictions for a procedure that has often not been included in existing tools such as the SVS VQI CRI.¹¹

Bonde et al (2021) trained ML algorithms using data from over 2900 procedures in the ACS NSQIP database to predict peri-operative complications, achieving AUROC values between 0.85 and 0.88.¹⁵ Considering the distinct characteristics and vascular comorbidities of CAS patients, generic surgical risk prediction tools may have limitations.⁵⁶ By developing tailored ML algorithms for CAS, we surpassed an AUROC of 0.90 and included specific outcomes such as distal embolization, acute occlusion/thrombosis, restenosis, and repeat revascularization, crucial for vascular surgeons and interventionalists. Our study underscores the importance of procedure-specific ML models in enhancing performance and clinical relevance. This effort complements previous work on predicting outcomes for CEA using ML.⁵⁷

Explanation of Findings

There are several explanations for our findings. First, patients who develop adverse events following CAS represent a high-risk group, which is corroborated by previous literature.⁵⁸ Aggressive medical management including antiplatelet therapy is a Grade 1A recommendation by SVS guidelines,⁵⁹ yet patients who developed MACE in our cohort were less likely to receive antiplatelets. This underscores a critical opportunity to improve patient care by understanding their surgical risk and medically optimizing them prior to CAS. According to SVS guidelines, patients undergoing CAS should receive dual antiplatelet therapy perioperatively (Grade 1C), an embolic protection device should be used during the procedure to reduce the risk of cerebral embolization (Grade 1B), and no specific recommendation has been made regarding the type of anesthesia for CAS.⁵⁹ Given the increased risk of perioperative strokes without the use of cerebral protection devices, it is critical to employ these devices intraoperatively when feasible.⁵⁹ Second, our ML models demonstrated performance superior to existing tools for several reasons. Compared to traditional logistic regression, advanced ML techniques can better model complex, non-linear relationships.⁶⁰ This is particularly important in health care data, where patient outcomes can be influenced by many factors.⁶¹ Our top-performing algorithm was XGBoost, which has unique advantages over other ML approaches including relatively fewer issues with overfitting and faster computing while maintaining precision.^35,62,63 Furthermore, XGBoost is well-suited to structured data, likely explaining its better performance compared to more complex algorithms such as neural networks on our dataset.⁶⁴ It is important to note that in some cases, carefully constructed logistic regression models can achieve equivalent or superior performance compared to ML models.⁶⁵ Third, our XGBoost model performance remained robust across demographic/clinical subpopulations. This is an important finding given that algorithm bias against underrepresented populations is a significant issue in ML studies.⁶⁶ We were likely able to avoid these biases due to the excellent capture of sociodemographic data by ACS NSQIP, a multi-national database that includes diverse patient populations.^67,68 Fourth, a small proportion of CAS (<5%) was performed for ipsilateral carotid stenosis percentage <50% or occlusions. The reasons for these interventions are unclear from our dataset but may be related to patient preference, poor adherence to guideline-directed therapy, or coding errors.^7,8

Implications

Our ML models can guide clinical decision-making in several ways. Preoperatively, a patient predicted to be at heightened risk of adverse events should be further assessed in terms of modifiable and non-modifiable factors. Patients with significant non-modifiable risks may benefit from medical management alone.⁶⁹ Conversely, individuals with low predicted risk may be considered for CEA based on SVS guidelines.⁷ Those with modifiable risks, such as cardiovascular comorbidities, should be further evaluated and optimized with consideration of referral to cardiologists or internal medicine specialists.^70,71 At the postoperative stage, patients at high risk of 30-day MACE may benefit from close monitoring in the intensive care unit.⁷² Additionally, patients at high risk of non-home discharge or readmission should receive early support from allied health professionals to facilitate safe discharge planning.⁷³ These peri-operative decisions guided by our ML models have the potential to improve outcomes and reduce costs related to adverse events.

The programming code for our ML models is publicly available on GitHub (https://shorturl.at/AEiV2). These tools can be used by clinicians involved in the peri-operative management of patients being considered for CAS. On a broader scale, our models can be implemented by the >700 centers worldwide that participate in ACS NSQIP. Their utility may also extend beyond NSQIP sites, as the input features are commonly captured variables for routine vascular care.⁷⁴ A distinct advantage of our ML models lies in their capacity to provide automated risk predictions, thereby enhancing practicality in busy clinical environments compared to traditional risk predictors that generally require manual input of variables.⁷⁵ Specifically, our ML algorithms can autonomously extract a patient’s NSQIP information to provide predictions of procedural risk. Predictive performance declined significantly for all models with the reduction of the number of features. We advocate for dedicated health care data analytics teams at the institutional level, as their benefits have been previously demonstrated and these experts can facilitate model implementation using our code.⁷⁶

Limitations

Our study has several limitations. First, our models were developed using ACS NSQIP data. Hospitals that participate in ACS NSQIP tend to be larger with more resources, which may limit the generalizability of our models.⁷⁷ Notably, the top 10 predictive features in our models are generally accessible across hospital settings in the work-up of patients with carotid stenosis, and therefore, future models that limit their input features to those that are common to obtain while maintaining predictive performance may increase generalizability. Future investigations are needed to evaluate whether model performance remains robust at institutions not enrolled in ACS NSQIP. Additionally, prospective validation whereby our ML models are tested for predictive performance and/or impact on outcomes in a prospectively recruited cohort of patients, rather than a clinical registry, would further demonstrate the clinical utility of the models. Second, the sample size was lower than expected over a 10-year period likely because ACS NSQIP is primarily a surgical database, and procedures performed by interventional radiologists or other non-surgical specialists may be under-captured. Additional investigation of CAS performed by non-surgical specialists may increase the sample size for analysis. Third, the ACS NSQIP database captures 30-day outcomes. Evaluation of ML algorithms on other data sources with longer follow-up may improve our understanding of long-term risk. Fourth, although preoperative medications were captured, postoperative antiplatelet therapy use was not available in our dataset. Additionally, patients who were not taking antiplatelets may be on anticoagulants. However, information on anticoagulants was not available within our dataset. Future model training on datasets that capture more detailed medication information would be prudent. Furthermore, some anatomic variables including the location, thickness, and distribution of the stenosis were not available in our dataset. Future models that incorporate these input features may improve model predictive performance. Moreover, the NSQIP dataset does not clarify the specific imaging modality by which the anatomic characteristics were determined. Given that there may be greater diagnostic accuracy achieved with some imaging modalities, this information would be helpful for future studies. Fifth, our models are limited to patients undergoing transfemoral CAS. A ML model for predicting CEA outcomes has been previously described,⁵⁷ and work is ongoing to develop predictive algorithms for transcarotid artery revascularization (TCAR). Particularly, a combined ML model for transfemoral CAS, CEA, and TCAR that predicts risk for the various treatments may provide additional information to clinicians regarding the optimal treatment approach based on short, middle, and long-term outcomes.

Conclusions

In this study, we leveraged the ACS NSQIP targeted vascular database to develop robust ML models that preoperatively predict 30-day MACE following CAS with excellent performance (AUROC 0.93). Our models also predicted stroke, MI, death, carotid-related morbidity, other morbidity, non-home discharge, and readmission with AUROCs of 0.86–0.97. Given that our ML algorithms perform better than existing tools and logistic regression, they have the potential for important utility in the peri-operative management of patients being considered for transfemoral CAS to mitigate adverse outcomes. Prospective validation of our prediction models is warranted.

Supplemental Material

sj-docx-1-jet-10.1177_15266028251333670 – Supplemental material for Predicting Outcomes Following Carotid Artery Stenting Using Machine Learning

Supplemental material, sj-docx-1-jet-10.1177_15266028251333670 for Predicting Outcomes Following Carotid Artery Stenting Using Machine Learning by Ben Li, Badr Aljabri, Derek Beaton, Mohamad A. Hussain, Douglas S. Lee, Duminda N. Wijeysundera, Ori D. Rotstein, Charles de Mestral, Muhammad Mamdani, Graham Roche-Nagle and Mohammed Al-Omran in Journal of Endovascular Therapy

Footnotes

Acknowledgements

The American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) and the hospitals participating in the ACS NSQIP are the source of the data used herein; they have not verified, and are not responsible for, the statistical validity of the data analysis or the conclusions derived by the authors.

Code Availability

The complete code used for model development and evaluation in this project is publicly available on GitHub: .

Data Availability

The data used for this study come from ACS NSQIP. Access to and use of the data requires approval through an application process available at .

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded partially by the Canadian Institutes of Health Research, Ontario Ministry of Health, PSI Foundation, and University of Toronto Schwartz Reisman Institute for Technology and Society (Dr. Li). Dr. Hussain is funded by a Brigham and Women’s Hospital Heart and Vascular Center Faculty Award. The funding sources did not play a role in the design or conduct of the research.

Ethical Approval and Informed Consent

This study was exempt from institutional ethics board review and informed consent was not required as the data came from a large, deidentified registry (ACS NSQIP).

ORCID iD

Ben Li

Supplemental Material

Supplemental material for this article is available online.

References

Donkor

ES.

Stroke in the 21st century: a snapshot of the burden, epidemiology, and quality of life. Stroke Res Treat. 2018;2018:3238165. doi:10.1155/2018/3238165

DaCosta

Tadi

Surowiec

SM.

Carotid endarterectomy. In: StatPearls. StatPearls Publishing;2023. Accessed August 23, 2023. http://www.ncbi.nlm.nih.gov/books/NBK470582/

Saleem

Baril

DT.

Carotid artery stenting. In: StatPearls. StatPearls Publishing; 2023. Accessed May 17, 2023. http://www.ncbi.nlm.nih.gov/books/NBK470541/

Brott

Hobson

Howard

, et al Stenting versus endarterectomy for treatment of carotid-artery stenosis. N Engl J Med. 2010;363(1):11–23. doi:10.1056/NEJMoa0912321

Hussain

Mamdani

, et al Impact of clinical trial results on the temporal trends of carotid endarterectomy and stenting from 2002 to 2014. Stroke. 2016;47:2923–2930. doi:10.1161/STROKEAHA.116.014856

Schermerhorn

Fokkema

Goodney

, et al The impact of Centers for Medicare and Medicaid Services high-risk criteria on outcome after carotid endarterectomy and carotid artery stenting in the SVS Vascular Registry. J Vasc Surg. 2013;57(5):1318–1324. doi:10.1016/j.jvs.2012.10.107

AbuRahma

Avgerinos

Chang

, et al Society for Vascular Surgery clinical practice guidelines for management of extracranial cerebrovascular disease. J Vasc Surg. 2022;75(1S):4S–22S. doi:10.1016/j.jvs.2021.04.073

Ricotta

AbuRahma

Ascher

, et al Updated Society for Vascular Surgery guidelines for management of extracranial carotid disease: Executive summary. J Vasc Surg. 2011;54:832–836. doi:10.1016/j.jvs.2011.07.004

Volkers

Algra

Kappelle

, et al Prediction models for clinical outcome after a carotid revascularisation procedure: a systematic review. Eur Stroke J. 2018;3:57–65. doi:10.1177/2396987317739122

10.

Sharma

Ali

van der Veer

, et al Adoption of clinical risk prediction tools is limited by a lack of integration with electronic health records. BMJ Health Care Inform. 2021;28:e100253. doi:10.1136/bmjhci-2020-100253

11.

Bertges

Neal

Schanzer

, et al The Vascular Quality Initiative Cardiac Risk Index for prediction of myocardial infarction after vascular surgery. J Vasc Surg. 2016;64:1411–1421.e4. doi:10.1016/j.jvs.2016.04.045

12.

Baştanlar

Özuysal

Introduction to machine learning. Methods Mol Biol. 2014;1107:105–128. doi:10.1007/978-1-62703-748-8_7

13.

Liew

BXW

Kovacs

, et al Machine learning versus logistic regression for prognostic modelling in individuals with non-specific neck pain. Eur Spine J Off Publ Eur Spine Soc Eur Spinal Deform Soc Eur Sect Cerv Spine Res Soc. 2022;31:2082–2091. doi:10.1007/s00586-022-07188-w

14.

Ngiam

Khor

IW.

Big data and machine learning algorithms for health-care delivery. Lancet Oncol. 2019;20(5):e262–e273. doi:10.1016/S1470-2045(19)30149-4

15.

Bonde

Varadarajan

Bonde

, et al Assessing the utility of deep neural networks in predicting postoperative surgical complications: a retrospective study. Lancet Digit Health. 2021;3(8):e471–e485. doi:10.1016/S2589-7500(21)00084-4

16.

Collins

Moons

KGM

Dhiman

, et al TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. doi:10.1136/bmj-2023-078378

17.

ACS NSQIP. ACS. Accessed August 1, 2023. https://www.facs.org/quality-programs/data-and-registries/acs-nsqip/

18.

Shiloach

Frencher

Steeger

, et al Toward robust information: data quality and inter-rater reliability in the American College of Surgeons National Surgical Quality Improvement Program. J Am Coll Surg. 2010;210(1):6–16. doi:10.1016/j.jamcollsurg.2009.09.031

19.

Cohen

Bilimoria

, et al Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus. J Am Coll Surg. 2013;217:336–346.e1. doi:10.1016/j.jamcollsurg.2013.02.027

20.

Dasenbrock

Smith

Gormley

, et al Predictive score of adverse events after carotid endarterectomy: the NSQIP Registry Carotid Endarterectomy Scale. J Am Heart Assoc. 2019;8:e013412. doi:10.1161/JAHA.119.013412

21.

AbuRahma

DerDerian

Hariri

, et al Anatomical and technical predictors of perioperative clinical outcomes after carotid artery stenting. J Vasc Surg. 2017;66:423–432. doi:10.1016/j.jvs.2017.02.057

22.

Dua

Romanelli

Upchurch

, et al Predictors of poor outcome after carotid intervention. J Vasc Surg. 2016;64:663–670. doi:10.1016/j.jvs.2016.03.428

23.

Lal

Roubin

Jones

, et al Influence of multiple stents on periprocedural stroke after carotid artery stenting in the Carotid Revascularization Endarterectomy versus Stent Trial (CREST). J Vasc Surg. 2019;69:800–806. doi:10.1016/j.jvs.2018.06.221

24.

Anand

Bosch

Eikelboom

, et al Rivaroxaban with or without aspirin in patients with stable peripheral or carotid artery disease: an international, randomised, double-blind, placebo-controlled trial. Lancet Lond Engl. 2018;391:219–229. doi:10.1016/S0140-6736(17)32409-1

25.

Elfanagely

Toyoda

Othman

, et al Machine learning and surgical outcomes prediction: a systematic review. J Surg Res. 2021;264:346–361. doi:10.1016/j.jss.2021.02.045

26.

Bektaş

Tuynman

Costa Pereira

, et al Machine learning algorithms for predicting surgical outcomes after colorectal surgery: a systematic review. World J Surg. Published online September 15, 2022. doi:10.1007/s00268-022-06728-1

27.

Senders

Staples

Karhade

, et al Machine learning and neurosurgical outcome prediction: a systematic review. World Neurosurg. 2018;109:476–486.e1. doi:10.1016/j.wneu.2017.09.149

28.

Shipe

Deppen

Farjah

, et al Developing prediction models for clinical use using logistic regression: an overview. J Thorac Dis. 2019;11(Suppl 4):S574–S584. doi:10.21037/jtd.2019.01.25

29.

Dobbin

Simon

RM.

Optimally splitting cases for training and testing high dimensional classifiers. BMC Med Genomics. 2011;4:31. doi:10.1186/1755-8794-4-31

30.

Jung

A K-fold Averaging Cross-validation procedure. J Nonparametric Stat. 2015;27:167–179. doi:10.1080/10485252.2015.1010532

31.

Adnan

Alarood

AAS

Uddin

, et al Utilizing grid search cross-validation with adaptive boosting for augmenting performance of machine learning models. PeerJ Comput Sci. 2022;8:e803. doi:10.7717/peerj-cs.803

32.

Wibowo

Fatichah

Pruning-based oversampling technique with smoothed bootstrap resampling for imbalanced clinical dataset of Covid-19. J King Saud Univ - Comput Inf Sci. 2022;34:7830-7839. doi:10.1016/j.jksuci.2021.09.021

33.

, et al Prediction of preoperative in-hospital mortality rate in patients with acute aortic dissection by machine learning: a two-centre, retrospective cohort study. BMJ Open. 2023;13:e066782. doi:10.1136/bmjopen-2022-066782

34.

Inoue

Ichikawa

Ueno

, et al XGBoost, a machine learning method, predicts neurological recovery in patients with cervical spinal cord injury. Neurotrauma Rep. 2020;1:8–16. doi:10.1089/neur.2020.0009

35.

Wang

Zhang

Shan

, et al XGBoost machine learning algorithm for prediction of outcome in aneurysmal subarachnoid hemorrhage. Neuropsychiatr Dis Treat. 2022;18:659–667. doi:10.2147/NDT.S349956

36.

Hajian-Tilaki

Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation. Casp J Intern Med. 2013;4:627–635.

37.

Cho

Matthews

Harel

Confidence intervals for the area under the receiver operating characteristic curve in the presence of ignorable missing data. Int Stat Rev Rev Int Stat. 2019;87:152–177. doi:10.1111/insr.12277

38.

Newcombe

RG.

Confidence intervals for an effect size measure based on the Mann-Whitney statistic. Part 2: asymptotic methods and evaluation. Stat Med. 2006;25:559–573. doi:10.1002/sim.2324

39.

Redelmeier

Bloch

Hickam

DH.

Assessing predictive accuracy: how to compare Brier scores. J Clin Epidemiol. 1991;44:1141–1146. doi:10.1016/0895-4356(91)90146-z

40.

Loh

Zhou

Variable importance scores. J Data Sci. 2021;19:569–592. doi:10.6339/21-JDS1023

41.

Riley

Ensor

Snell

KIE

, et al Calculating the sample size required for developing a clinical prediction model. BMJ. Published online March 18, 2020:m441. doi:10.1136/bmj.m441

42.

Ensor

Martin

Riley

RD.

pmsampsize: calculates the minimum sample size required for developing a multivariable prediction model. Published online February 12, 2022. Accessed December 11, 2022. https://CRAN.R-project.org/package=pmsampsize

43.

Schafer

JL.

Multiple imputation: a primer. Stat Methods Med Res. 1999;8:3–15. doi:10.1177/096228029900800102

44.

Ross

Breskin

Westreich

When is a complete-case approach to missing data valid? The importance of effect-measure modification. Am J Epidemiol. 2020;189:1583–1589. doi:10.1093/aje/kwaa124

45.

Hughes

Heron

Sterne

JAC

, et al Accounting for missing data in statistical analyses: multiple imputation is not always the answer. Int J Epidemiol. 2019;48:1294–1304. doi:10.1093/ije/dyz032

46.

Markey

Tourassi

Margolis

, et al Impact of missing data in evaluating artificial neural networks trained on complete data. Comput Biol Med. 2006;36:516–525. doi:10.1016/j.compbiomed.2005.02.001

47.

R Core Team (2022). R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Accessed December 1, 2022. https://www.R-project.org/

48.

Kuhn

Wing

Weston

, et al caret: classification and regression training. Published online August 9, 2022. Accessed October 6, 2022. https://CRAN.R-project.org/package=caret

49.

Chen

Guestrin

XGBoost: a scalable tree boosting system. Proc 22nd ACM SIGKDD Int Conf Knowl Discov Data Min - KDD ’16. Published online August 13, 2016. doi: 10.1145/2939672.293978.

50.

Wright

Wager

Probst

ranger: a fast implementation of random forests. Published online June 18, 2022. Accessed October 6, 2022. https://CRAN.R-project.org/package=ranger

51.

naivebayes: High Performance Implementation of the Naive Bayes Algorithm version 0.9.7 from CRAN. Accessed October 6, 2022. https://rdrr.io/cran/naivebayes/

52.

svm function - RDocumentation. Accessed October 6, 2022. https://www.rdocumentation.org/packages/e1071/versions/1.7-11/topics/svm

53.

Ripley

Venables

nnet: feed-forward neural networks and multinomial log-linear models. Published online September 28, 2022. Accessed October 6, 2022. https://CRAN.R-project.org/package=nnet

54.

Robin

Turck

Hainard

, et al pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011;12:77. doi:10.1186/1471-2105-12-77

55.

Friedman

Basu

The rate and cost of hospital readmissions for preventable conditions. Med Care Res Rev MCRR. 2004;61(2):225-240. doi:10.1177/1077558704263799

56.

Hers

Van Schaik

Keekstra

, et al Inaccurate Risk Assessment by the ACS NSQIP Risk Calculator in Aortic Surgery. J Clin Med. 2021;10(22):5426. doi:10.3390/jcm10225426

57.

Beaton

Eisenberg

, et al Using machine learning to predict outcomes following carotid endarterectomy. J Vasc Surg. Published online May 19, 2023:S0741-5214(23)01252-1. doi:10.1016/j.jvs.2023.05.024

58.

Gates

Botta

Schlosser

, et al Characteristics that define high risk in carotid endarterectomy from the Vascular Study Group of New England. J Vasc Surg. 2015;62(4):929-936. doi:10.1016/j.jvs.2015.04.398

59.

Ricotta

AbuRahma

Ascher

, et al Updated Society for Vascular Surgery guidelines for management of extracranial carotid disease. J Vasc Surg. 2011;54(3):e1–e31. doi:10.1016/j.jvs.2011.07.031

60.

Stoltzfus

JC.

Logistic regression: a brief primer. Acad Emerg Med Off J Soc Acad Emerg Med. 2011;18:1099–1104. doi:10.1111/j.1553-2712.2011.01185.x

61.

Higgins

JP.

Nonlinear systems in medicine. Yale J Biol Med. 2002;75:247–260.

62.

Ravaut

Sadeghi

Leung

, et al Predicting adverse outcomes due to diabetes complications with machine learning using administrative health data. Npj Digit Med. 2021;4:1–12. doi:10.1038/s41746-021-00394-8

63.

Fang

Yang

, et al Application of a data-driven XGBoost model for the prediction of COVID-19 in the USA: a time-series study. BMJ Open. 2022;12:e056685. doi:10.1136/bmjopen-2021-056685

64.

Viljanen

Meijerink

Zwakhals

, et al A machine learning approach to small area estimation: predicting the health, housing and well-being of the population of Netherlands. Int J Health Geogr. 2022;21:4. doi:10.1186/s12942-022-00304-5

65.

Austin

Lee

Wang

, et al Comparison of machine learning and the regression-based EHMRG model for predicting early mortality in acute heart failure. Int J Cardiol. 2022;365:78-84. doi:10.1016/j.ijcard.2022.07.035

66.

Gianfrancesco

Tamang

Yazdany

, et al Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med. 2018;178:1544–1547. doi:10.1001/jamainternmed.2018.3763

67.

Mazmudar

Vitello

Chapman

, et al Gender as a risk factor for adverse intraoperative and postoperative outcomes of elective pancreatectomy. J Surg Oncol. 2017;115:131–136. doi:10.1002/jso.24488

68.

Halsey

Asti

Kirschner

RE.

The impact of race and ethnicity on surgical risk and outcomes following palatoplasty: an analysis of the NSQIP pediatric database. Cleft Palate-Craniofacial J Off Publ Am Cleft Palate-Craniofacial Assoc. Published online February 9, 2022:10556656221078154. doi:10.1177/10556656221078154

69.

Hackam

DG.

Optimal medical management of asymptomatic carotid stenosis. Stroke. 2021;52(6):2191-2198. doi:10.1161/STROKEAHA.120.033994

70.

Davis

Park

Grey

, et al The clinical impact of cardiology consultation prior to major vascular surgery. Ann Surg. 2018;267:189–195. doi:10.1097/SLA.0000000000002014

71.

Rivera

Nguyen

Martinez-Osorio

, et al Preoperative medical consultation: maximizing its benefits. Am J Surg. 2012;204:787–797. doi:10.1016/j.amjsurg.2012.02.018

72.

Gillies

Harrison

Pearse

, et al Intensive care utilization and outcomes after high-risk surgery in Scotland: a population-based cohort study. Br J Anaesth. 2017;118(1):123–131. doi:10.1093/bja/aew396

73.

Henke

Karaca

Jackson

, et al Discharge planning and hospital readmissions. Med Care Res Rev MCRR. 2017;74:345–368. doi:10.1177/1077558716647652

74.

Nguyen

Barshes

NR.

Analysis of large databases in vascular surgery. J Vasc Surg. 2010;52:768–774. doi:10.1016/j.jvs.2010.03.027

75.

Bilimoria

Liu

Paruch

, et al Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg. 2013;217:833–842. doi:10.1016/j.jamcollsurg.2013.07.385

76.

Batko

Ślęzak

The use of Big Data Analytics in healthcare. J Big Data. 2022;9:3. doi:10.1186/s40537-021-00553-4

77.

Osborne

Nicholas

Ryan

, et al Association of hospital participation in a quality reporting program with surgical outcomes and expenditures for medicare beneficiaries. JAMA. 2015;313:496–504. doi:10.1001/jama.2015.25

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.55 MB