Abstract
Background
This is a validation study comparing the European System for Cardiac Operative Risk Evaluation (EuroSCORE) II with the previous additive (AES) and logistic EuroSCORE (LES) and the Society of Thoracic Surgeons' (STS) risk prediction algorithm, for patients undergoing valve replacement with or without bypass in Pakistan.
Patients and Methods
Clinical data of 576 patients undergoing valve replacement surgery between 2006 and 2013 were retrospectively collected and individual expected risks of death were calculated by all four risk prediction algorithms. Performance of these risk algorithms was evaluated in terms of discrimination and calibration.
Results
There were 28 deaths (4.8%) among 576 patients, which was lower than the predicted mortality of 5.16%, 6.96% and 4.94% by AES, LES and EuroSCORE II but was higher than 2.13% predicted by STS scoring system. For single and double valve replacement procedures, EuroSCORE II was the best predictor of mortality with highest Hosmer and Lemmeshow test (H-L) p value (0.346 to 0.689) and area under the receiver operating characteristic (ROC) curve (0.637 to 0.898). For valve plus concomitant coronary artery bypass grafting (CABG) patients actual mortality was 1.88%. STS calculator came out to be the best predictor of mortality for this subgroup with H-L p value (0.480 to 0.884) and ROC (0.657 to 0.775).
Conclusions
For Pakistani population EuroSCORE II is an accurate predictor for individual operative risk in patients undergoing isolated valve surgery, whereas STS performs better in the valve plus CABG group.
Introduction
Risk prediction models play an important role in current cardiac surgical practice. Such models are useful in surgical decision-making, preoperative patient education and consent and quality assurance measures (1). Several tools have been developed in recent years, and among them the Society of Thoracic Surgeons' (STS) mortality risk score and the European System for Cardiac Operative Risk Evaluation (EuroSCORE) scoring system have gained widest popularity (2, 3).
These models have been developed and validated in a surgical population composed mainly of those undergoing coronary artery bypass grafting (CABG). However, valvular heart disease (VHD) comprises a unique subset, with etiologies that are different in developed and developing countries. Most valvular pathology in industrialized countries is now degenerative and is a disease of advancing age (4). On the other hand, rheumatic heart disease and infective endocarditis remain endemic in developing countries, inflicting VHD in young patients (5). This difference in clinical profile of patients could affect their postoperative outcomes as well. The objective of this study was to test and compare the predictive performance of additive EuroSCORE (AES), logistic EuroSCORE (LES), EuroSCORE II and STS scoring system in Pakistani patients undergoing valvular surgery.
Patients and Methods
Retrospective data were extracted from the prospectively collected cardiothoracic database of the Aga Khan University Hospital. A total of 576 patients who underwent valvular heart surgery between January 2006 and July 2013 were included. We included mitral valve replacement (MVR), aortic valve replacement (AVR), double valve replacement (DVR), CABG + MVR and CABG + AVR patients. STS calculator does not code for DVR, therefore 490 patients were analyzed for this subgroup. Thirty-day mortality was taken as the primary outcome (hospital mortality or death within 30 days postoperatively). The data set was eventually imported and analyzed using SPSS version 19 software (SPSS, Inc., Chicago, IL, USA).
LES was calculated for each patient based on the original EuroSCORE criteria by running a syntax code incorporating the regression coefficients and intercept developed by EuroSCORE. An individual score for the additive model was also calculated by a simple additive SPSS syntax. The EuroSCORE II and STS risk score were calculated with free online calculators available at http://www.euroscore.org and http://209.220.160.181/STSWebRiskCalc261/de.aspx, respectively. The calculated score corresponds to the individual risk of death in the 30-day postoperative period (predicted mortality).
Patient demographics are presented as percentages for discrete variables and mean (_standard deviation) for continuous variables. Thirty-day mortality was taken as the primary outcome (actual mortality) and determined for the overall patient population, and trends in actual mortality were analyzed across the entire risk spectrum. Performance of the models was also assessed by comparing the observed and expected mortality in quintiles of risk.
The discriminatory power of the models was assessed using the area under the receiver operating characteristic (ROC) curve with 95% confidence intervals; an area of 0.5 indicates no predictive ability, whereas an area of 1.0 represents perfect discrimination (6). Model calibration (the degree to which observed outcomes are similar to the predicted outcomes from the model across patients) of AES, LES, EuroSCORE II and STS was examined by comparing average observed and predicted values within each of 10 equal-sized subgroups arranged in increasing order of patient risk. To evaluate model calibration, the Hosmer-Lemeshow test for the lack of “goodness of fit” was applied and graphically represented by a calibration plot. The smooth curve in a calibration plot reflects the nonparametric relationship between observed and predicted risk mortality. The straight dotted line through the origin of a calibration plot represents perfect calibration. Hosmer-Lemeshow p values above 0.05 indicate a well-calibrated model for the study population in question (7).
Results
The study population included 576 patients eligible for calculation of EuroSCORE and 490 patients for STS risk score as it does not stand valid for DVRs. Mean age was 47.36 ± 15.47 years with female population being 46.53%. Table I presents the distribution of preoperative risk variables identified in our population. Majority of the patients (66.67%) underwent single valve replacement.
Prevalence of risk factors in study population
AVR = aortic valve replacement; CABG = coronary artery bypass grafting; DVR = double valve replacement; MVR = mitral valve replacement; NYHA = New York Heart Association; CC = creatinine clearance; CCS = Canadian Cardiovascular Society Score
There were a total of 28 deaths among 576 operated patients. The actual mortality rate of 4.86% was lower than 5.16%, 6.96% and 4.94% predicted by AES, LES, EuroSCORE II, respectively, but was higher than 2.13% predicted by the STS scoring system. Table II shows the predictive ability, calibration and discriminatory power for all the risk stratifying algorithms. As shown in Figure 1, all the models had good discriminatory power (ROC >0.8) in the global patient sample ranging from 0.812 to 0.819. However, EuroSCORE II was better calibrated with highest Hosmer and Lemmeshow p value (0.552). Table III shows the performance of risk models in quintiles of risk. It shows EuroSCORE II overestimated mortality up to the 5th decile and thereafter under predicted mortality. However, the difference in actual and predicted mortalities was less as compared to differences with AES and LES estimates.
Performance of risk models in patient subsets
AES = additive EuroSCORE; AVR = aortic valve replacement; CABG = coronary artery bypass grafting; DVR = double valve replacement; LES = logistic EuroSCORE; MVR = mitral valve replacement; ROC = receiver operating characteristic; STS = Society of Thoracic Surgeons (STS)
Receiver operating characteristic curves.
Performance of risk models in quintiles of risk
AES = additive EuroSCORE; AVR = aortic valve replacement; CABG = coronary artery bypass grafting; DVR = double valve replacement; LES = logistic EuroSCORE; MVR = mitral valve replacement; ROC = receiver operating characteristic; STS = Society of Thoracic Surgeons (STS)
For patients undergoing single valve replacement, there was a general trend of overestimation by AES, LES and EuroSCORE II, while STS predicted mortality was lower than the actual value. A total of 14.9% patients had DVR. STS risk score does not code for DVR procedures and was henceforth not used in this comparison. In this subgroup, EuroSCORE II estimates were closest to actual mortality with AES and LES both overestimating mortality. Interestingly the actual mortality paradoxically decreased in patients undergoing DVR as compared to single valve replacement, but the predicted mortality increased as expected. A total of 106 patients underwent mitral valve replacement or AVR along with concomitant CABG surgery, with observed mortality being 1.88% and 1.84%, respectively. Although ROC and H-L p values were lower than those for single valve replacement procedures, when compared with the other three models, STS came out to be the best risk stratifying algorithm for valve replacement surgeries along with bypass.
Discussion
External validation of a risk-predicting model seeks to verify whether it can be generalized outside the boundaries of the population it was built upon, as a preliminary to using it as a reference for quality assessment (8). Differences in the clinical profile of reference population and the testing sample are generally considered as a serious hindrance to such a validation process. In addition, currently used risk score systems have been developed for quite some time and therefore require periodic recalibration and validation to reflect improved surgical techniques and postoperative patient management advances which occurred in recent times (9, 10).
Clinical profile of patients is quite different in Asia as opposed to the Western world. The difference lies in the differences of patient demographics; delayed clinical presentation due to socioeconomic, cultural and geographical reasons; inequitable distribution of medical facilities and different treatment patterns (11). Similar differences combined with altered risk-adjusted mortality were also found in China and India when compared with that of STS and EuroSCORE II sample population (10, 12). The prevalence of rheumatic heart disease (RHD) has fallen drastically in developed countries, with VHD being a disease of the elderly only. On the contrary, prevalence of RHD in Pakistan has remained unchanged from 6 in 1000 in the 1970s to 5.7 in 1000 in 2004 (13). Moreover, Rizvi et al (13) showed that less than 20% of those found to have RHD were aware of their diagnosis before participation in the study and only 8% of affected people (8%) were taking rheumatic prophylaxis. Secondly, we also had a higher incidence of active endocarditis (5.4%) compared to 2.2% in EuroSCORE II population. This may have led to the need for valve replacement at a younger age in our patients. Mean age was 47.4 years as compared to 64.6 years in EuroSCORE II patients (14). About 67.4% of patients in our study were <55 years of age and only 3.8% were above 75 years, compared to 25.6% <55 years and 27.5% above 75 years in the STS database.
Recently the role of EuroSCORE and STS has been questioned. In fact, several papers have suggested that these algorithms may be no more adequate for risk estimation due to a two to threefold overestimation of the adult cardiac surgical risk. The lack of performance has been especially observed in valvular surgical subgroups because these models were developed and validated in a surgical population mainly composed of CABG. EuroSCORE II has been recently published to improve on the older EuroSCORE. In the EuroSCORE II more attention was given on the weight of surgical procedures in order to ameliorate the risk estimation even in non-CABG procedures although the core of risk factors is almost the same (14).
Since publication, the EuroSCORE II has been validated in large, European, multicenter cohorts. Grant et al (15) found that the performance of the tool was “more than satisfactory,” and well calibrated for a large proportion of patients. Barili et al (16) demonstrated that the EuroSCORE II is a good predictor of perioperative mortality in isolated aortic valve surgery, with lower discrimination when compared with STS and a better calibration when compared with LES, Age, creatinine and left ventricular ejection fraction model (ACEF) and STS scores. Its performance is optimal in the lowest tertile of patients, whereas it underpredicts mortality afterward. Overall, EuroSCORE II has better performance although it does not seem to significantly improve the performance of older EuroSCORE versions in the higher tertiles of risk. Although high-risk cases constitute only a small proportion of databases from which risk scores are elaborated and validated, they also express a significant proportion of the overall mortality, for which poor approximation in stratification cannot be easily tolerated. In this study, by applying both models on the surgical operation types (Tab. II), no statistically significant differences were observed comparing AUC-AES, AUC-LES with AUC-EuroSCORE II (AUC = Area under the curve). However, EuroSCORE II was better calibrated than the older version. The worst discrimination of all three models was observed in the patients with combined valve and coronary surgery. Similar to our study, Carnero-Alcázar et al also found that EuroSCORE showed excellent discriminative capacity that has been slightly improved with the new EuroSCORE II version (17).
Few studies have drawn head to head comparison of STS and EuroSCORE II (12, 18, 19). Consistent with our findings, Wang et al (19) found that compared with the EuroSCORE II, the STS score gives an accurate prediction for individual operative risk in patients undergoing heart valve surgery with concomitant CABG. Contrary to this, Kirmani et al (20) concluded that EuroSCORE II and STS both provide equivalent discrimination (area under ROC EuroSCORE II vs. STS, 0.816 vs. 0.810, p = 0.714) in predicting mortality in all cardiac surgical procedures. However, STS risk values can be calculated only for selected procedures. In a recent study from India, large proportion of patients undergoing cardiac surgery had to be excluded because STS does not code for DVR, concomitant tricuspid valve surgery or MAZE procedure (12). In our study, we had to exclude 14.9% patients from analysis who underwent surgeries for which STS risk score was not available. Our study showed STS to be well calibrated and showing high discrimination power for CABG and valve replacement patients.
There are several limitations to our study. First, our study is based on a relatively small sample size of patients from a single center, and so the truth of the performance of four risk models for predicting operative mortality in heart valve surgery in Pakistan could be biased, thus the results need to be further examined using a large number of patients across the multicenter database. Second, this is a retrospective investigation and deviation, such as ignoring a strong predictor (pulmonary hypertension), may weaken the results of the validation procedure. Third, we examined all-cause mortality and were unable to determine the cause of death (cardiac or noncardiac). However, prediction of all-cause mortality is probably more practical than cardiac death alone.
The main results of our study are that (i) EuroSCORE II is the best predictor of mortality in isolated valve surgery patients (including both single valve replacements and DVRs). (ii) STS risk prediction algorithm is better calibrated for CABG + valve replacement, (iii) Clinical profile of Pakistani patients undergoing valvular surgery is uniquely different from European or American population where these risk models were developed and validated.
Conclusion
In conclusion, the EuroSCORE II gives an accurate prediction for individual operative risk in patients undergoing isolated valve surgery, whereas STS performs better in the valve plus CABG group.
Footnotes
Financial support: None of the authors has financial interest related to this study to disclose.
Conflict of interest: The authors declare that they have no conflicting interests.