Abstract
Background
This study aimed to develop and validate predictive models based on machine learning (ML) algorithms for patients with bone metastases (BM) from clear cell renal cell carcinoma (ccRCC) and to identify appropriate models for clinical decision-making.
Methods
In this retrospective study, we obtained information on ccRCC patients diagnosed with bone metastasis (ccRCC-BM), from the Surveillance, Epidemiology, and End Results (SEER) database from 2010 to 2015 (n = 1490), and collected clinicopathological information on ccRCC-BM patients at our hospital (n = 42). We then applied four ML algorithms: extreme gradient boosting (XGB), logistic regression (LR), random forest (RF), and Naive Bayes model (NB), to develop models for predicting the overall survival (OS) of patients with bone metastasis from ccRCC. In the SEER dataset, 70% of the patients were randomly divided into training cohorts and the remaining 30% were used as validation cohorts. Data from our center were used as an external validation cohort. Finally, we evaluated the model performance using receiver operating characteristic curves (ROC), area under the ROC curve (AUC), accuracy, specificity, and F1-scores.
Results
The mean survival times of patients in the SEER and Chinese cohort were 21.8 months and 37.0 months, respectively. Age, marital status, grade, T stage, N stage, tumor size, brain metastasis, liver metastasis, lung metastasis, and surgery were included in the ML model. We observed that all four ML algorithms performed well in predicting the 1-year and 3-year OS of patients with ccRCC-BM.
Conclusion
ML is useful in predicting the survival of patients with ccRCC-BM, and ML models can play a positive role in clinical applications.
Introduction
Renal cell carcinoma is one of the most common malignant tumors; it is the sixth most common cancer among men and the tenth most common cancer among women worldwide. 1 Clear cell renal cell carcinoma (ccRCC) is the most common pathological type of kidney cancer, accounting for 80%–90% of all renal cell carcinomas 2 along with papillary renal cell carcinoma and chromophobe renal cell carcinoma. ccRCC tends to have worse prognosis than other pathological types of RCC.3,4 At the time of the initial diagnosis, approximately 20%–30% of patients have a combination of distant metastases, of which bone metastasis (BM) is one of the most common sites.5,6 The occurrence of bone metastases from renal cell carcinoma has become increasingly common as patients with renal cell carcinoma live longer due to the rapid development of targeted therapies and immunotherapy. 7 The International Kidney Cancer Working Group states that overall survival (OS) is significantly worse for bone and liver metastases compared to other metastatic sites. 8 Although major advances have been made in the treatment of primary renal cell carcinoma, the overall survival of patients with RCC who develop bone metastases is only 19.7 months. 9 Bone metastases from kidney cancer mainly occur in the mid-axis bones, and common metastatic sites include the pelvis, spine, and limbs. Altogether, 71% of patients with bone metastases from kidney cancer had multiple bone metastases. Osteolytic lesions (approximately 80%) were predominant in kidney cancer bone metastases, with a small number of osteogenic lesions (7%), while the rest were mixed lesions (13%). 10 BM is usually associated with skeletal-related events (SREs) such as pain, hypercalcemia, pathological fractures, spinal cord compression, and even paraplegia, which greatly reduce the patient's quality of life and result in a worse prognosis. 11
Machine learning (ML) is the science of artificial intelligence (AI). It has been used in many different scientific and medical applications in recent years because of its powerful data mining and analysis capabilities.12–14 Marostica et al. 15 used deep convolutional neural networks to detect and diagnose kidney cancer with good results. Zhu et al. 17 used convolutional neural networks for blood cell classification 16 and malaria image classification, and their models were superior to existing methods. ML has been proven to have more accurate predictive power than traditional statistical methods. 18 Several studies have found that the prognosis of patients with bone metastases from renal cell carcinoma is significantly correlated with age, tumor, nodes, and metastases (TNM) stage, other organ metastases, and targeted therapy.19,20 Models using traditional statistical methods to predict the prognosis of patients with clear cell renal cell carcinoma bone metastases (ccRCC-BM) have been reported;21,22 however, there have been no studies on ML algorithms.
This study aimed to investigate the application of ML to predict the prognosis of patients with bone metastases from renal cell carcinoma. This will help identify high-risk patients and guide clinical interventions.
Materials and Methods
Data Source and Selection of Variables
The original data in this study were extracted from the Surveillance, Epidemiology, and End Results (SEER) database (SEER Research Plus Data 18 Registries, Nov 2020 Sub (2000–2018)) and our hospital. The SEER database agreement was signed and permission was obtained to access the information (accession username: 12906-Nov 2021). This retrospective study did not include personal information; therefore, institutional review board approval or informed consent were not required. All the methods used in our study complied with the rules of the SEER database. This study conformed to transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guidelines. 23
All patients with ccRCC diagnosed between 2010 and 2015 were considered for this study, and the exclusion criteria were as follows: (1) No bone metastasis; (2) Tumor size unknown; (3) Unknown TNM stage unknown; (4) Survival month unknown or < 1 month; (5) Presence of brain/liver/lung metastasis unknown; (6) Age at diagnosis < 18 years. Altogether, 1490 patients from the SEER database were included. These patients were randomly divided into validation and training cohorts. The external validation cohort consisted of patients with a confirmed diagnosis of ccRCC-BM at our hospital between 2010 and 2016 (n = 42). These patients were contacted via telephone to obtain verbal informed consent.
To take full advantage of the ML algorithm, several demographic and clinical variables were selected for data analysis. Demographic variables included sex, age, marital status, and insurance purchase, whereas clinical variables included tumor size, laterality, TNM stage, pathological grade, brain metastasis, liver metastasis, lung metastasis, surgery of primary site, radiotherapy, and chemotherapy. The endpoints of the predictive models were the 1- and 3-year overall survival rates.
Statistics Analysis
All statistical analyses were performed using R software version 4.1.3 (https://www.r-project.org/) and Python (version 3.10; Python Software Foundation). Pearson correlation tests were performed between all the variables, and the results are presented in the form of a heat map. To compare baseline characteristics, Pearson's chi-square test was used for categorical variables and Mann–Whitney U test was used for continuous variables. The results were considered statistically significant when the P value was less than 0.05 (two-sided).
Establishment and Validation of Models
Patients from the SEER database were randomly divided into training and validation cohorts in a 7:3 ratio. The chi-square test was used to compare the association between the two groups. Cox proportional hazards model was used for univariate and multivariate analyses to identify independent risk factors for ccRCC-BM. These independent predictors were used to construct machine-learning models to predict the 1- and 3-year overall survival rates. We used four machine-learning algorithms to build the models: logistic regression (LR), random forest (RF), extreme gradient boosting (XGB), and Naive Bayes (NB). Receiver operating characteristic (ROC) curve, area under the curve (AUC), accuracy, specificity, and F1-scores were then used to compare the predictive power of the models.
Results
Demographic and Clinicopathological Characteristics
After screening, 1532 patients diagnosed with ccRCC-BM were included in our study, including 1490 patients from the SEER database and 42 patients from our center. We divided the 1490 patients from the SEER database into training (n = 1043) and validation (n = 477) cohorts in a 7:3 ratio, with the remaining 42 patients in the external validation group. The demographic and clinical characteristics of the patients are shown in Table 1. There were significant differences between the three groups in terms of race, insurance, pathological grade, surgery, radiotherapy, chemotherapy, and survival time (P < 0.05). In the external validation cohort, the race was classified as “other,” and a greater proportion of patients had insurance, a greater propensity for pathological grade II, more patients who underwent surgery for the primary site, and a smaller proportion of patients who received radiotherapy or chemotherapy. This may have led to a longer survival time in these patients. The remaining variables did not differ significantly among the three groups. The mean age was 63.3 years, patients were predominantly elderly, 1089 (71.1%) were male, 954 (62.3%) were married, and the mean tumor size was 7.91 cm. The number of patients with T1, T2, T3, T4, and TX stages were 417 (27.2%), 284 (18.5%), 550 (35.9%), 138 (9.0%), and 143 (9.3%), respectively. Regarding N stage, the number of patients with N0, N1, and NX were 1019 (66.5%), 367 (24.0%), and 146 (9.5%), respectively. Most patients had no brain metastases (n = 1378, 89.9%) or liver metastases (n = 1335, 87.1%), while 840 (54.8%) patients had no lung metastases.
Demographic Characteristics and Clinicopathological Features of ccRCC-BM Patients.
Independent Risk Factors of BM in ccRCC Patients
The Pearson correlation analysis heat map (Figure 1) showed that all variables existed independently of each other, did not affect each other, and could be included in the subsequent statistical analysis. In univariate analysis, the following variables were significantly associated with the prognosis of ccRCC-BM patients (P < 0.05) (Table 2): age, marital status, grade, laterality, T stage, N stage, tumor size, brain metastasis, lung metastasis, liver metastasis, and surgery. Multivariate analysis based on these significant variables demonstrated that age, marital status, grade, T stage, N stage, tumor size, brain metastasis, lung metastasis, liver metastasis, and surgery were independent predictors of overall survival (Table 3).
The heat map shows that all variables are independent of each other.
Univariate Analysis for the Prognosis Factors of ccRCC-BM Patients.
Multivariate Analysis for the Prognosis Factors of ccRCC-BM Patients.
Model Construction and Performance of ML Algorithms
Based on the training set, we applied four ML algorithms to predict the 1-year and 3-year OS rates. The validation set was applied to examine and compare the predictive performance of all the models. We used ROC, AUC (Figures 2 and 3), accuracy scores, specificity, and F1-scores to evaluate and compare the performance of the models. Finally, the models were externally validated using our dataset. In predicting the 1-year OS, the XGB algorithm model performed best in the training set (AUC value of 0.891), in the validation set (AUC value of 0.711), and in the external validation cohort (AUC value of 0.812). For the LR algorithm model, the AUC value in the training set was 0.745, and the AUC values in the validation set and external validation cohort were 0.737 and 0.845, respectively. The other two algorithm models also performed well. The AUC values in the training, validation, and external validation sets of the RF model were 0.762, 0.749, and 0.801, respectively, while those of the NB model were 0.737, 0.739, and 0.809, respectively. For predicting the 3-year OS, the AUC value of the XGB algorithm model in the training set was 0.848, and the AUC values of the other three algorithms, LR, RF, and NB were 0.779, 0.790, and 0.776, respectively. The AUC values in the validation set were 0.777 for LR, 0.787 for RF, 0.762 for NB, and 0.767 for XGB. In the external validation cohort, the RF model performed best, with an AUC value of 0.706, while the other three algorithms LR, NB, and XGB had AUC values of 0.666, 0.679, and 0.674, respectively.
ROC curves of the four models: lr, rf, nb and XGB. (A) training group, (B) validation group and (C) external validation group of 1-year OS predictive model.
ROC curves of the four models: lr, rf, nb and XGB. (A) training group, (B) validation group and (C) external validation group of 3-year OS predictive model.
Finally, the accuracy, specificity, and F1 scores of the four algorithms were determined (Table 4). Compared to the other three models, the XGB model had the highest accuracy, specificity, and F1 score of 0.792,0.806,0.854, respectively. The LR model had an accuracy of 0.680, specificity of 0.805, and F1 score of 0.756. The RF model had an accuracy of 0.698, specificity of 0.777, and F1 score of 0.775. The accuracy, specificity, and F1 score of the NB model were 0.752, 0.680, and 0.827, respectively.
Comparison Prediction Performances of Different Models.
Discussion
Bone is a common site of metastasis in ccRCC, 24 and bone metastases are often osteolytic lesions. SREs, such as pathological fractures, pain, and spinal cord compression caused by bone metastases, not only lead to reduced quality of life and increased medical costs, but also to a worse prognosis. In this study, we first used traditional univariate and multivariate analyses to identify independent risk factors associated with the prognosis of patients with ccRCC-BM, and then constructed prognostic models using several different ML algorithms. We evaluated the performance of the four algorithms and confirmed the feasibility and accuracy of the ML algorithms in constructing prognostic models for patients with ccRCC-BM.
The role of age as a prognostic risk factor for patients with renal cell carcinoma has been controversial.25–27 Our study showed that age was an independent risk factor for the prognosis of patients with ccRCC-BM. Marital status was also an independent prognostic factor, which is consistent with previous findings. Several studies have shown that marriage is a protective factor in the prognosis of oncology patients.28–30 This may be attributed to the psychological and financial support provided by the spouse as well as daily living assistance. Regarding clinicopathological features, tumor size, T stage, N stage, pathological grade, brain metastasis, liver metastasis, lung metastasis, and surgery of the primary lesion were confirmed as independent risk factors for ccRCC-BM. The effects of different metastatic sites on the prognosis of patients with renal cell carcinoma have also been widely reported. Lin et al. 31 reported that patients with isolated bone metastases had the highest overall survival rate, and that patients with multiple bone metastases had higher survival rates than those with pulmonary metastases. McKay et al. 8 showed that liver metastases negatively impacted the prognosis of patients with renal cell carcinoma receiving targeted therapy. In addition, some studies have suggested that brain metastases are a risk factor for increased risk of death and worse prognosis in patients with renal cell carcinoma.32,33
We selected the above factors to construct ML models for predicting the 1-year and 3-year overall survival of patients with ccRCC-BM. Internal validation was performed using the SEER dataset, and external validation was performed using data from our center. The performance of the models was then evaluated using the AUC values, accuracy, specificity, and F1 scores. The models constructed using all four machine learning algorithms performed well, but the XGBoost algorithm had the best performance; this new artificial intelligence algorithm has several advantages in terms of preventing overfitting, cross-validation, and handling missing values. In the medical industry, this new ML algorithm is becoming increasingly popular and is frequently used for disease prediction and early diagnosis. To the best of our knowledge, there are no ML applications for predicting the prognosis of patients with bone metastases from renal cell carcinoma. In this study, we validated the feasibility of ML using four algorithmic models. This facilitates the advancement of ML applications in medicine.
However, this study had some limitations. First, this was a retrospective study; therefore, bias in case selection could not be avoided and further validation with prospective data is needed. Second, the data were obtained from the SEER database, and some clinical features, such as the number of bone metastases, SREs, and time to bone metastases, were not available from the SEER database. Third, immunotherapy and targeted therapies that are recommended for patients with ccRCC, were not recorded in the SEER database. Finally, although we performed external validation, the number of patients was not sufficiently large.
In future studies, we will collect more cases to validate the model and improve the algorithm to increase the stability of the model. In addition, we will attempt to use ML for diagnostic and prognostic studies of other diseases.
Conclusion
We developed ML models to predict overall survival of patients with ccRCC-BM. All the models performed well. We also identified the risk factors affecting the prognosis of patients with bone metastases from renal clear-cell carcinoma, which will provide relevant information for clinicians and help in clinical decision-making.
Footnotes
Abbreviations
Acknowledgments
We'd like to express our gratitude to the SEER database for allowing us to access free and open data.
Data Availability Statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Declaration of Conflicting Interests
The author(s) declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Ethical Approval
The study was approved by the ethics committee of Peking University People's Hospital, Approval Number: 2022PHB394-001.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
