Abstract
Background:
Prediction of bleeding is critical for acute myocardial infarction (AMI) patients after percutaneous coronary intervention (PCI). Machine learning methods can automatically select the combination of the important features and learn their underlying relationship with the outcome.
Objectives:
We aimed to evaluate the predictive value of machine learning methods to predict in-hospital bleeding for AMI patients.
Design:
We used data from the multicenter China Acute Myocardial Infarction (CAMI) registry. The cohort was randomly partitioned into derivation set (50%) and validation set (50%). We applied a state-of-art machine learning algorithm, eXtreme Gradient Boosting (XGBoost), to automatically select features from 98 candidate variables and developed a risk prediction model to predict in-hospital bleeding (Bleeding Academic Research Consortium [BARC] 3 or 5 definition).
Results:
A total of 16,736 AMI patients who underwent PCI were finally enrolled. 45 features were automatically selected and were used to construct the prediction model. The developed XGBoost model showed ideal prediction results. The area under the receiver-operating characteristic curve (AUROC) on the derivation data set was 0.941 (95% CI = 0.909–0.973, p < 0.001); the AUROC on the validation set was 0.837 (95% CI = 0.772–0.903, p < 0.001), which was better than the CRUSADE score (AUROC: 0.741; 95% CI = 0.654–0.828, p < 0.001) and ACUITY-HORIZONS score (AUROC: 0.731; 95% CI = 0.641–0.820, p < 0.001). We also developed an online calculator with 12 most important variables (http://101.89.95.81:8260/), and AUROC still reached 0.809 on the validation set.
Conclusion:
For the first time, we developed the CAMI bleeding model using machine learning methods for AMI patients after PCI.
Trial registration:
NCT01874691. Registered 11 Jun 2013.
Introduction
Dual antiplatelet therapy is critical to reduce thrombus events for acute myocardial infarction (AMI) patients after percutaneous coronary intervention (PCI). However, it is a two-edged sword; once patients occurred severe bleeding, the risk of death will increase significantly.1,2 Therefore, it is very important to identify high-risk bleeding patients with AMI. The traditional bleeding scores for acute coronary syndrome patients includes Can Rapid risk stratification of Unstable angina patients Suppress ADverse outcomes with Early implementation of the ACC/AHA Guidelines (CRUSADE) score, 3 Acute Catheterization and Urgent Intervention Triage Strategy and Harmonizing Outcomes with Revascularization and Stents in Acute Myocardial Infarction (ACUITY-HORIZONS) score, 4 etc. However, these scores are derived from limited clinical parameters using traditional statistical methods, so their predictive values are relatively limited. The recommendation level of CRUSADE score in the ESC guidelines for patients with non–ST-segment elevation acute coronary syndromes, downgraded from Ib class in 2011 5 to IIb class in 2015. 6
Machine learning is based on computer technology and big data developed from the artificial intelligence science. In recent years, with the emergence of big medical data, machine learning methods have improved the performance of the predictive models rapidly. In previous studies, machine learning shows better predictive values than traditional prediction models.7–12 However, so far, we have not seen any research report on the establishment of in-hospital bleeding risk prediction model for AMI patients by machine learning method. Therefore, we aimed to establish a new bleeding score by machine learning, in order to reduce the risk of bleeding in patients with AMI.
Methods
Data description
Our study was a prospective cohort analysis of data from the China Acute Myocardial Infarction (CAMI) registry, a prospective, nationwide, multicenter observational study for AMI patients, which was organized and conducted by the Fuwai Hospital, National Center for Cardiovascular Diseases, China. The methodology of the CAMI registry (NCT01874691) was described previously. 13 The patients fulfilled the inclusion/exclusion criteria were consecutively selected into current analysis. The registry includes three levels of hospitals (at the provincial, prefecture, and county level) which reflect typical Chinese governmental and administrative model and covers all the provinces and municipalities with broad coverage of geographic region across mainland China (except for Hong Kong and Macau). Patients with AMI were consecutively enrolled and data were collected upon their arrival and throughout their hospital stay until discharge. Data were collected, validated, and submitted by trained clinical cardiologists or cardiovascular fellows to ensure the accuracy and reliability of data at each participating site. The study was conducted in accordance with the Declaration of Helsinki and was approved by the Ethics Committee of Fuwai Hospital (Approval No. 2012-431), and all of the patients provided written informed consent.
Study population
Among 41,348 consecutive AMI patients in the registry study between January 1, 2013 and June 30, 2016, 24,612 patients were excluded according to our cohort exclusion criteria, including patients not received PCI (n = 22,607), only received bare metal stent (n = 892) or not received dual antiplatelet therapy (n = 1113). The study population consisted of 16,736 patients, among which 70 (0.42%) patients experienced in-hospital bleeding (Figure 1).
Patient flow chart for the study cohort.
Endpoint and follow-up
We used the Third Universal Definition of Myocardial Infarction as a diagnostic criterion for AMI. 14 Major bleeding was defined by the Bleeding Academic Research Consortium (BARC), 15 which included BARC type 3 or 5 of non-coronary artery bypass grafting–related bleeding. Major bleeding events occurred in hospital were recorded in detail.
Statistical analysis
Baseline characteristics
Continuous variables were presented as mean ± standard deviation and compared using student’s t test. Categorical variables were expressed as number and proportion and compared by means of the chi-square test or Fisher exact test. All tests were two-tailed and a value of p < 0.05 was considered to represent statistical significance. All analyses were performed with R statistical software version 3.4.3 (R Foundation for Statistical Computing).
Model construction and validation
We randomly split the study population into derivation set (n = 8368, 50%) and validation set (n = 8368, 50%). The 50/50 scheme was used to guarantee the stability of the validation results given the relatively low in-hospital bleeding rate. Based on the derivation data set with 98 candidate variables (Table 1), we developed a risk prediction model to predict in-hospital bleeding using eXtreme Gradient Boosting (XGBoost) algorithm. 16 XGBoost is a recently developed state-of-the-art machine learning algorithm, which ensembles a series of decision trees into a stronger classifier. And XGBoost integrated the sparsity-aware algorithm devoted to accurately handle missing values. It can automatically rank the most important variables that have the largest contribution to the prediction of the clinical outcome. The model parameters were tuned using 10-fold cross-validation.
A total of 98 variables were included for analysis.
ACEI, angiotensin-converting enzyme inhibitors; ARB, angiotensin receptor antagonist; BMI, body mass index; BNP, B type natriuretic peptide; CABG, coronary artery bypass grafting; CK-MB, creatine kinase isoenzyme-MB; COPD, chronic obstructive pulmonary diseases; ECG, electrocardiogram; FMC, first medical contact; HDL-C, high-density lipoprotein cholesterol; Hs-CRP, high-sensitivity C-reactive protein; LDL-C, low density lipoprotein cholesterol; NT-proBNP, N-terminal pro brain natriuretic peptide; PCI, percutaneous coronary intervention; STEMI, ST-segment elevation myocardial infarction; TIMI, thrombolysis in myocardial infarction.
The predictive performance of the constructed XGBoost model in predicting BARC 3 or 5 bleeding was validated in the validation data set. The discriminatory capacity of the XGBoost model was assessed by AUROC (area under the receiver-operating characteristic curve), and the calibration was estimated by the Hosmer–Lemeshow goodness-of-fit test. To gain insights into the relative contribution of each predictor to the prediction of in-hospital bleeding, we also computed the importance scores of the variables in the XGBoost model. And in order to test the practicability and robustness of the model, we evaluated the impact on the prediction performance of the model when reducing the number of variables. The least important variable was eliminated for each evaluation. We stratified the cohort into low/medium/high risk groups according to the patient distribution.
Results
Baseline characteristics
Among 16,736 patients, the mean age of the patients was 60.08 ± 12.64 years, and 13,311 (79.5%) were male. 16,736 patients (100%) took aspirin, 16,552 (98.90%) took P2Y12 receptor inhibitors, and 103 (0.62%) took bivalirudin. A total of 70 patients (0.42%) had in-hospital bleeding (BARC 3 or 5 bleeding). As showed in Table 2, patients who had in-hospital bleeding were older (p = 0.010), more often female (p < 0.001), less often current smoker (p = 0.019) compared with patients without in-hospital bleeding. More in-hospital bleeding patients had sudden persistence (p = 0.007) and obvious clinical symptoms (p = 0.002) when the disease occurred. They also had higher heart rate (p = 0.013), lower systolic blood pressure (p = 0.001), lower diastolic blood pressure (p = 0.040) and higher Killip class (p < 0.001). In addition, more in-hospital bleeding patients had triple-vessel lesion (p = 0.004) and femoral artery puncture (p < 0.001). And, creatine kinase isoenzyme-MB (CK-MB) (p < 0.001), white Blood Count (p = 0.011), potassium (p = 0.001), N-terminal pro brain natriuretic peptide (NT-proBNP) (p = 0.001) at baseline were also much higher, while much lower of hemoglobin in the in-hospital bleeding group (p < 0.001) (Table 2).
Baseline characteristics of the study population.
Values are mean ± SD or n (%).
BMI, body mass index; CK-MB, creatine kinase isoenzyme-MB; ECG, electrocardiogram; HDL-C, high-density lipoprotein cholesterol; Hs-CRP, high-sensitivity C-reactive protein; LDL-C, low-density lipoprotein cholesterol; NT-proBNP, N-terminal pro brain natriuretic peptide; STEMI, ST-segment elevation myocardial infarction; TIMI, thrombolysis in myocardial infarction.
Prediction of in-hospital bleeding
In total, 45 features were automatically selected from the 98 candidate features (Table 1) and were used to construct the CAMI bleeding score. The discrimination ability of different models, on derivation set and validation set, as represented by, was shown in Figure 2. The XGBoost model showed significant better discrimination ability compared with the CRUSADE risk score and ACUITY-HORIZONS score. On the derivation data set, the AUROC of the XGBoost model was 0.941 (95% CI = 0.909–0.973, p < 0.001), the AUROC of CRUSADE score was 0.765 (95% CI = 0.686–0.845, p < 0.001), and the AUROC of ACUITY-HORIZONS score was 0.639 (95% CI = 0.541–0.738, p = 0.003); similarly, the AUROC of the XGBoost model on the validation data set was 0.837 (95% CI = 0.772–0.903, p < 0.001), which was also higher than that of the CRUSADE score [AUROC (0.741, 95% CI = 0.654–0.828), p < 0.001] and the ACUITY-HORIZONS score [AUROC (0.731, 95% CI = 0.641–0.820), p < 0.001] (Table 3). Calibration of the XGBoost model was evaluated using the quintile plot of observed versus predicted risk and Hosmer–Lemeshow goodness-of-fit test (Figure 3). The Hosmer–Lemeshow test statistic was 11.507, with p = 0.201, which indicated that the XGBoost model on the validation data set suggested no evidence of lack of fit (p > 0.05).
AUROC of the XGBoost model, the CRUSADE risk score, and the ACUITY-HORIZONS score on training data (a) and validation data (b).
AUROCs of the XGBoost model, CRUSADE score, and ACUITY-HORIZONS score in the derivation and validation data sets.
ACC, American College of Cardiology; ACUITY-HORIZONS, Acute Catheterization and Urgent Intervention Triage Strategy and Harmonizing Outcomes with Revascularization and Stents in Acute Myocardial Infarction; AHA, American Heart Association; AUROC, Area Under the Receiver-Operating Characteristic curve; CRUSADE, Can Rapid risk stratification of Unstable angina patients Suppress ADverse outcomes with Early implementation of the ACC/AHA Guidelines; XGBoost, eXtreme Gradient Boosting.
Calibration plot of the XGBoost model on the validation data set.
We found that the model performance was basically stable as the number of variables decreased. When 17 variables remained in the model, the AUROC of the model was 0.830 (95% CI = 0.764–0.895, p < 0.001). Even if only 12 variables were included, the model still showed good prediction performance with AUROC of 0.809 (95% CI = 0.745–0.873, p < 0.001) on the validation data set (Figure 4).
AUROC of XGBoost corresponding to different number of features on the validation data set. The model performance was basically stable as the number of variables decreased. When 12 variables were included, the AUROC was of 0.809 (95% CI = 0.745–0.873, p < 0.001).
Variable importance
Figure 5 demonstrated the importance scores of the 12 relatively important variables in the XGBoost model for the bleeding outcome. Clinical characteristics (age, systolic blood pressure, diastolic blood pressure, and heart rate) and blood test indicators (creatinine clearance rate, direct bilirubin, hematocrit, CK-MB, potassium, NT-proBNP, hemoglobin, and total bilirubin) were important predictors. We established the calculator from these 12 clinical variables online (http://101.89.95.81:8260/), and estimated the low, medium, and high risk of bleeding.
Relative contribution of the 12 most important features.
Validity analysis of scores in STEMI and NSTEMI subgroups
In subgroup analysis, validation data set (n = 8368) was divided into STEMI group (n = 4792) and NSTEMI group (n = 3576). The XGBoost model showed similar predictive value in the STEMI group (AUROC: 0.826) and NSTEMI group (AUROC: 0.821), and outperformed the CRUSADE risk score and ACUITY-HORIZONS score, respectively (Table 4).
Validity analysis of scores in STEMI and NSTEMI subgroups.
ACUITY-HORIZONS, Acute Catheterization and Urgent Intervention Triage Strategy and Harmonizing Outcomes with Revascularization and Stents in Acute Myocardial Infarction; AUROC, area under the receiver-operating characteristic curve; CI, confidence interval; CRUSADE, Can Rapid risk stratification of Unstable angina patients Suppress ADverse outcomes with Early implementation of the ACC/AHA Guidelines; NSTEMI, Non-ST-segment elevation myocardial infarction; STEMI, ST-segment elevation myocardial infarction; XGBoost, eXtreme Gradient Boosting.
Discussion
With the rapid development of medical technology, clinical prediction model and risk assessment have entered a new era of large sample and artificial intelligence. This study analyzed a multicenter, large-sample (16,736 patients) CAMI study database, including the patients’ clinical basic information, coronary angiography data, treatment, NT-proBNP, and so on, a total of 98 baseline indicators. For the first time, the advanced XGBoost method was used to establish a bleeding risk prediction model for patients with AMI. The results are as follows: (1) we have established a good predictive model by using the method of machine learning to identify the risk of in-hospital bleeding in AMI patients. The model shows good predictive effect in both training set and verification set; (2) the model prediction effect established by the XGBoost risk prediction model is better than the traditional CRUDADE score and ACUITY-HORIZONS score in both the training set and the verification set; and (3) we set up a calculator for the 12 most important clinical parameters, which can be applied to the clinical practice.
It is generally believed that acute coronary syndrome patients have higher risk of ischemic events during in-hospital period, and previously D’Ascenzo et al. 17 also demonstrated a significant excess of average daily ischemic risk in the first 2 weeks. As a result, early antithrombotic therapy is important to prevent ischemic events during in-hospital period. However, it is a two-edged sword, once patients occurred severe bleeding, the risk of death will increase significantly. Therefore, it is very important to identify high-risk bleeding patients with AMI. However, the predictive value of the bleeding score which recommended by the guideline is relatively limited, 18 and the related research is obviously lagging behind. This study established a prediction model of in-hospital bleeding in AMI patients by machine learning method. The study showed that whether the CAMI bleeding score is in the derivation data set (AUROC: 0.941), or the validation data set (AUROC: 0.837) showed good prediction results. And, the good fit of the XGBoost model on the validation data set was proved by goodness-of-fit test (p = 0.201). Machine learning method is an artificial intelligence science based on computer technology and big data. In recent years, it has shown good clinical application value in many medical fields, including tumor, 7 neurology, 19 diabetes, 20 and pathological diagnosis. 21 It has also come to the fore recently in the cardiovascular field, including cardiovascular images (such as cardiac ultrasound 12 and cardiac CT 11 ); predicting poor cardiovascular outcomes, such as death after PCI, 9 rehospitalization of heart failure, 9 and in-hospital death after transcatheter aortic valve replacement. 8 For the first time, we assessed and reported the risk of in-hospital bleeding in patients with AMI by machine learning.
It is worth noting that this study included more risk prediction factors and found that in addition to traditional bleeding risk factors, such as creatinine clearance rate, age, systolic blood pressure, and other bleeding prediction indicators, some nontraditional factors were also found to be related to bleeding, such as serum potassium, NT-proBNP, bilirubin, and so on. This often happens in the machine learning methods results. For example, Al’Aref et al. 10 predicted the in-hospital death of PCI by machine learning method, the results showed that there were both traditional indicators such as body mass index and serum creatinine, and nontraditional indicators were also incorporated in risk scores such as day of the week. This is because the previous Logistic regression model based on the presence of a significant bivariate relationship with the primary outcome, tends to ignore the potential prognostic value of the interaction between weaker risk factors and preliminary results. Second, continuous variables are often divided into different intervals, which may also lead to the loss of prediction information.
To the best of our knowledge, no study has reported the relationship between the serum bilirubin, potassium, and in-hospital bleeding in patients with AMI. However, some studies have shown that serum bilirubin and potassium are associated with poor in-hospital prognosis of AMI.22,23 Recently, a meta-analysis (323,891 patients) showed 24 that total bilirubin was significantly positively related to in-hospital cardiovascular mortality and major adverse cardiac events. Celik et al. 25 reported that serum bilirubin levels were independently associated with no-reflow and in-hospital major adverse cardiac events in patients with STEMI undergoing primary PCI. Similarly, previous studies have shown that serum potassium is associated with poor prognosis in patients with AMI.26,27 Goyal et al. 26 showed that a U-shaped relationship between serum potassium levels and in-hospital mortality. The advantage of machine learning score is that it can find out the correlation between variables and prognosis, regardless of whether the correlation is linear or not. Our results suggest that bilirubin and serum potassium are closely related to AMI bleeding. In addition to the fact that bilirubin and serum potassium are related to the severity of patients with AMI, it may also be related to the reflection of hepatic or renal insufficiency. At present, it is consistent to consider that the risk of bleeding is significantly increased in patients with hepatic or renal insufficiency.
Our study showed that the machine learning model had a higher predictive value for identifying in-hospital bleeding than the traditional CRUSADE score and the ACUITY-HORIZONS score. Machine learning has a clear advantage over traditional statistical analysis. In 2016, Obermeyer and Emanuel 28 mentioned that traditional prognostic models are limited to a small number of variables, while machine learning greatly improves the outcome. In our CAMI bleeding score, 45 valuable indicators from 98 candidate indicators including clinical baseline data, coronary angiography data, drug treatment, and the biomarkers. In comparison, the traditional CRUSADE score published in 2009 included eight variables, and the predictive value of bleeding was moderate (C statistic 0.72) 3 in original study. As well as the ACUITY score 4 published in 2010 included seven variables for bleeding, and the C statistic was 0.74. In short, with the increasing number of variables in the current big clinical electronic database and the integration of new biomarkers, machine learning model has the ability to better sort out and analyze the massive data of patients. It breaks through the limitations of the accuracy and scope of application of the traditional statistical model, and brings us a better prediction effect than the traditional score in the real world.
At the end of the study, we performed a weight analysis on the valuable variables. It was found that when the variables were reduced from 45 to 12, the model’s AUROC in the verification set only decreased from 0.837 to 0.809, which still maintained a good predictive value. Although the size of 12 is greater than the traditional bleeding score in the number of parameters, they are all clinically easy to obtain indicators. In addition, we established the calculator from these 12 clinical variables which is expected to be easily applied to clinical practice. This is in line with the general trend of the development of artificial intelligence in cardiovascular precision medicine, and improves the prediction effect of AMI in-hospital bleeding. In the future, we look forward to further evaluating the predictive value of this model through external verification.
Limitations
First, our research has not been externally validated, and external validation of large samples should be conducted in the future to further evaluate the predictive value of the model; second, although the XGBoost model has been validated by the internal validation data set, there may still be overfitting; third, this study removed patients who did not receive PCI, so whether it is suitable for thrombolytic therapy and conservative treatment patients, further study is needed; fourth, because the inclusion parameters of different clinical center databases are different, and the number of parameters included in the machine learning model is higher than the traditional score, these reasons may cause the model to be inconvenient to use. Therefore, we selected 12 clinical parameters from the included 45 parameters according to the relative importance, and established a calculator to facilitate useful exploration for clinical use; fifth, since most of the patients included in this study were taking clopidogrel, future evaluation of the predictive value of ticagrelor or prasugrel remains to be studied.
Conclusion
This is the first study to use a large-sample, multicenter CAMI database, and the results showed that the XGBoost method can establish a risk model for accurately predicting in-hospital hemorrhage in patients with AMI, and its predictive value is better than the traditional bleeding scores. At the same time, we optimize the model parameters and set up a calculator for the most important 12 clinical parameters, which is expected to be used in clinical practice.
