Sage Journals: Discover world-class research

Abstract

Study Design

Retrospective study.

Objectives

Our objective is to create comprehensible machine learning (ML) models that can forecast bone cement leakage in percutaneous vertebral augmentation (PVA) for individuals with osteoporotic vertebral compression fracture (OVCF) while also identifying the associated risk factors.

Methods

We incorporated data from patients (n = 425) which underwent PVA. To predict cement leakage, we devised six models based on a variety of parameters. Evaluate and juxtapose the predictive performances relied on measures of discrimination, calibration, and clinical utility. SHapley Additive exPlanations (SHAP) methodology was used to interpret model and evaluate the risk factors associated with cement leakage.

Results

The occurrence rate of cement leakage was established at 50.4%. A binary logistic regression analysis identified cortical disruption (OR 6.880, 95% CI 4.209-11.246), the basivertebral foramen sign (OR 2.142, 95% CI 1.303-3.521), the fracture type (OR 1.683, 95% CI 1.083-2.617), and the volume of bone cement (OR 1.198, 95% CI 1.070-1.341) as independent predictors of cement leakage. The XGBoost model outperformed all others in predicting cement leakage in the testing set, with AUC of .8819, accuracy of .8025, recall score of .7872, F1 score of .8315, and a precision score of .881. Several important factors related to cement leakage were drawn based on the analysis of SHAP values and their clinical significance.

Conclusion

The ML based predictive model demonstrated significant accuracy in forecasting bone cement leakage for patients with OVCF undergoing PVA. When combined with SHAP, ML facilitated a personalized prediction and offered a visual interpretation of feature importance.

Keywords

bone cement leakage osteoporotic vertebral compression fracture percutaneous vertebral augmentation machine learning interpretable model SHapley Additive exPlanations values

Introduction

Percutaneous vertebral augmentation (PVA), including vertebroplasty (PVP) and kyphoplasty (PKP), is an established procedure for treating osteoporotic and partially tumor-related vertebral fractures, although it carries the risk of potential catastrophic outcomes.¹ Osteoporotic vertebral compression fracture (OVCF) pose a significant hazard and are a major cause of disability and mortality in elderly patients.² PVA is widely utilized in patients with OVCF due to its minimally invasive nature, ability to relieve pain, capacity to maintain or restore vertebral height, and promotion of early ambulation. However, several complications associated with PVA have been identified in previous studies, including bone cement leakage,³ fractures at adjacent or remote levels,⁴ infections, and hidden blood loss.⁵ Among these complications, the leakage of bone cement is receiving increasing attention as the most common complication of PVA, with prevalence ranging from 5% to 80%.³ While the majority of bone cement leakages are asymptomatic, they can occasionally lead to serious consequences, such as nerve or spinal canal compression, as well as intravascular embolism, particularly pulmonary embolism, which can be catastrophic and even fatal. The long-term prognosis of asymptomatic bone cement leakage remains uncertain. Therefore, early recognition and intervention are crucial for bone cement leakage. Unfortunately, effective methods for early identification of bone cement leakage are still lacking in clinical practice.

Clinical prediction models, which assess the probability of specific outcomes based on patient characteristics,⁶ are increasingly popular in orthopedics for predicting complications associated with surgical procedures. With the recent advancements in artificial intelligence (AI), machine learning (ML) algorithms have been introduced into prediction models. Some researchers have developed ML models to predict bone cement leakage and have selected the best-performing model to create a web calculator.⁷ However, previous models did not incorporate comprehensive evaluations of ML models’ performance and neglected the inclusion of multifaceted related variables. Furthermore, the interpretability of ML models is limited due to their “black box” nature, where only input and output are visible without any insight into the intermediate process. Consequently, their applicability in clinical practice is restricted because they lack intuitive interpretation.

To address these issues, our study integrates additional radiology and surgery-related information to develop ML models, utilizing SHapley Additive exPlanations (SHAP) to provide intuitive explanations of individual predictions and risk factors. Our aim is to build more robust ML-based predictive models for bone cement leakage in PVA and assist clinicians in recognizing the early-stage risk of bone cement leakage, analyzing risk factors, and implementing interventions to reduce its occurrence.

Method

Study Participants

We conducted a retrospective analysis of patients (n = 425) who underwent PVA for OVCF at the Department of Spine Surgery, Zhongda Hospital affiliated with Southeast University, from May 2020 to December 2022. This study was approved by the Institutional Ethics Committee of Zhongda Hospital affiliated with Southeast University (reference number: 2022ZDKYSB121) and the application for exemption from informed consent was granted.

Inclusion criteria were as follows: (1) diagnosis of OVCF based on clinical manifestations and radiological characteristics; (2) presence of back pain, with or without trauma, related to OVCF; (3) meeting the indications for PVP/PKP.

Exclusion criteria were as follows: (1) age below 45 years; (2) presence of spinal cord, cauda equina, or nerve root injuries; (3) chronic vertebral compression fracture with negative MRI results, burst fractures, or malignant fractures; (4) Loss and fault of raw clinical data and incomplete imaging data (complete data including preoperative plain radiography, CT and MRI, postoperative plain radiography or CT).

Clinical and Radiological Data

All variables were obtained from patients’ medical records, radiological data (including DXA, preoperative MRI, CT scan, and plain anteroposterior and lateral radiography of the spine), and operative documents. The variables collected were as follows: (1) Baseline characteristics: age, sex, height, weight, body mass index (BMI), and bone mineral density (BMD) T-scores obtained from dual-energy X-ray absorptiometry (DXA); (2) Medical history-related variables: time interval from injury to admission, time interval from admission to surgery, history of OVCF, surgical history of PVP/PKP, history of chronic diseases (including hypertension, diabetes, coronary heart disease, cerebrovascular disease), use of steroid medication, use of anti-osteoporosis medication, surgical history of pedicle screw fixation, number of fractured vertebral bodies, number of surgically treated vertebral bodies, and location of fracture (thoracic, thoracolumbar (T11-L2), lumbar; (3) Operation-related variables: type of operation (PVP or PKP), puncture approach (unipedicular or bipedicular technique), the volume of bone cement, length of surgery time, and anesthesia technique. (4) Radiology-related variables: intervertebral cleft (IVC), cortical disruption, fracture type and severity, basivertebral foramen sign, the intrusion of the posterior wall.

The radiology-related variables were assessed by three spinal experts from the Department of Spine Surgery. The experts were blinded to both the participants’ characteristics and the statistical analysis.

Surgery Procedures

All surgeries were performed by skilled and experienced spinal surgeons from the Department of Spine Surgery, Zhongda Hospital affiliated with Southeast University. Preoperative preparation involved positioning the patient in a prone position and determining the puncture point and angle for the responsible vertebral body by preoperative fluoroscopic imaging. The skin was marked, followed by routine disinfection and draping. Anesthesia approaches included local anesthesia, intravenous anesthesia, and general anesthesia. The vertebroplasty needle was inserted into the vertebral body using either the unipedicular or bipedicular technique, based on the surface projection of the pedicles. Once the needle position was deemed satisfactory under lateral fluoroscopy, the needle core was withdrawn.

For patients undergoing PVP, the appropriate cement material (polymethyl methacrylate, PMMA) was injected into the responsible vertebral body using a pressure syringe under C-arm fluoroscopic guidance. For patients undergoing PKP, a working cannula was inserted into the vertebral body using the PKP surgical system. A balloon was then placed through the working cannula and inflated with a contrast medium until the vertebral body achieved the desired height. Once the desired height was reached, the balloon was deflated and withdrawn, and bone cement was introduced to fill the space in the vertebral body.

During cement injection, the “wiredrawing” or “toothpaste-like” phase was initiated. The distribution of bone cement was confirmed to be satisfactory under C-arm fluoroscopy.

The evaluation of bone cement leakage was based on intraoperative fluoroscopy.

Reference Standard

Definition of Radiology-Related Variables

1. IVC defined as an intravertebral transverse, linear, or cystic radiolucent shadow on preoperative CT; as a hypointense area similar to air on T1-weighted MRI sequences and on T2-weighted MRI sequences, or as a hyperintense area similar to cerebrospinal fluid on MRI STIR sequences.⁸

2. Cortical disruption was characterized by clear discontinuity observed at the endplates or vertebral body wall on preoperative MRI or CT scans.

3. Fracture type and severity were assessed using Genant’s semi-quantitative classification based on lateral X-ray imaging.⁹ Fracture type was classified into three types: wedge, biconcave, and crush. Fracture severity was categorized into mild, moderate, and severe based on the reductions of 20-25%, 26-40%, and >40%, respectively. In this study, reductions of less than 20% were also classified as mild.

4. The basivertebral foramen sign is characterized by the passage of the vertebrobasilar vein through the center of the vertebral body’s posterior wall, resulting in a triangular or irregular quadrilateral bone defect on CT sagittal images or a porous defect in the bone on CT axial images.¹⁰

5. The intrusion of the posterior wall considered when the part of the posterior wall of the vertebral body protruded into the spinal canal on the axial CT image.¹¹

Definition of Bone Cement Leakage

Cement leakage was characterized as the presence of any cement outside the vertebral body.³ Intraoperative fluoroscopy was performed for all patients during the procedure, and plain anteroposterior and lateral radiography or CT scans of the spine were conducted postoperatively to assess the occurrence of bone cement leakage.

Data Processing

Prior to developing the ML models, it was necessary to address missing data and normalize the data. For missing data imputation, variables with a missing value percentage exceeding 30% were excluded, while those below 30% were imputed using the k-nearest neighbors imputation strategy with a value of k set to 5. The dataset was then randomly split into a training set and a testing set, with an 8.5:1.5 ratio. Data normalization was performed by standardizing quantitative variables to have a mean of zero and a unit variance. The testing set was standardized based on the parameters derived from the training set. Categorical variables were encoded using the LabelEncoder functions from Scikit-Learn library.

Model Development and Interpretation

Model Development

We developed six ML models to predict bone cement leakage in PVA procedures. The models utilized the Extreme Gradient Boost classifier (XGBoost), Decision tree classifier (Decision Tree), Random Forest classifier (Random Forest), Gaussian Naive Bayes Classifier (GaussianNB), and support vector machines classifier (SVM), implemented using the Sci-Kit Learn package in Python (version 3.7.6).

To optimize the models’ performance, we employed a grid search approach to tune the parameters using the training set. The models were built based on the best parameters obtained from the grid search. Specifically, GaussianNB, Decision Tree, Random Forest, and SVM models were tuned manually, while the XGBoost model required parameter tuning for each parameter individually. Additionally, we implemented artificial neural networks (ANN) using the TensorFlow (version 2.3) Keras package. The ANN architecture comprised dense layers with a ReLU activation function. Dropout layers were incorporated after the dense layers to prevent overfitting. The gradient descent method was employed to minimize the loss function in the ANN model.

Evaluation of Performance

Various performance measures were calculated to assess the models’ performance on the test set: These measures included accuracy, precision, F1-score, recall, and the area under the receiver operating characteristic curve (AUC), which collectively provided a comprehensive evaluation of the discrimination ability of the ML models. To evaluate the predictive accuracy, calibration plots were utilized to compare the actual probability with the predicted probability. Furthermore, clinical decision curve analysis (DCA) was employed to evaluate the clinical utility of the ML models.

Interpretation and Visualization of the Model

Due to the inherent “black-box” nature of ML algorithms, understanding the underlying process and obtaining meaningful interpretations can be challenging. To address this limitation, we utilized SHapley Additive exPlanations (SHAP) to provide both global and local interpretations of the model.

For the best-performing model, we calculated the SHAP values for all features and obtained the mean SHAP values. These values were then utilized to create visual representations, including single sample force plots and summary plots. These plots were instrumental in visualizing the relationship between the prediction outcomes and each individual feature.

Statistical Analysis

The normality of distribution for continuous variables was assessed using the Shapiro-Wilk or Kolmogorov-Smirnov test. If a continuous variable followed a normal distribution, it was expressed as the mean ± standard deviation (SD), and the independent-sample t-test was used for the comparative analysis. However, if the variable did not conform to a normal distribution, it was presented as the median and interquartile range (IQR), and the Mann-Whitney U test was employed for the comparative analysis. Categorical variables were presented as frequencies and percentages, and the chi-square test was used for analysis. To identify independent risk factors for the occurrence of bone cement leakage, binary logistic regression analysis was conducted. A P-value <.05 was considered statistically significant. IBM SPSS Statistics version 24.0 (SPSS Inc, Chicago, IL, USA) was employed to perform the aforementioned statistical analyses.

Result

Patient Characteristics

A total of 425 patients, comprising 512 vertebral bodies, were included in this study based on the defined inclusion and exclusion criteria. Among these patients, there were 91 males and 421 females, with a mean age of 75 years (range: 49-97 years). Postoperative bone cement leakage occurred in 258 cases, resulting in an overall incidence rate of 50.4% (258/512), as presented in Table 1.

Table 1.

Demographic Characteristics of the Study Participants who Underwent PVA for OVCF.

Number of participants, n		425
Number of veterbraes, n		512
Baseline characteristics	Age (range, years)	75 (49-97)
	Sex
	F/M	421/91
	Height (range, m)	1.59 (1.32-1.83)
	weight(range, Kg)	58.6 (35-96)
	BMI (range,kg/m²)	23.05 (13.81-36.07)
	BMD (range)	−4.1 (−0.6-−8.6)
Medical history related variables	Injury to admission time interval(range, days)	20 (1-240)
	Admission to surgery time interval(range, days)	3 (1-20)
	The history of OVCF
	Y/N	140/372
	Surgical history of PVP/PKP
	Y/N	81/431
	Surgical history of pedicle screw fixation
	Y/N	29/483
	Chronic diseases
	Y/N	348/164
	Medication of steroid
	Y/N	20/492
	Medication of anti-osteoporosis
	Y/N	6/506
	Fracture location
	T/TL/L	61/335/116
	Number of fractured vertebral body(range)	1 (1-4)
	Number of surgical vertebral body(range)	1 (1-4)
Operation related variables	The method of operation
	PVP/PKP	78/434
	Puncture approach
	Unipedicular/bipedicular	205/307
	Volume of bone cement(range, mL)	6.7 (2-12)
	Length of surgery(range, min)	46 (15-205)
	Anesthesia techniques
	Local/Intravenous/General	248/239/25
Radiology related variables	IVC
	Y/N	160/352
	Cortical disruption
	Y/N	356/156
	Basivertebral foramen sign
	Y/N	114/398
	The intrusion of the posterior wall
	Y/N	43/469
	Fracture severity
	Mild/moderate/severe	296/111/105
	Fracture type
	Wedge/ biconcave/crush	461/12/39
Bone cement leakage	Y/N	258/254

Risk Factors

Table 2 presents a comparison of variables between patients with bone cement leakage and those without. Statistically significant differences were observed in various variables, including IVC, cortical disruption, basivertebral foramen sign, intrusion of the posterior wall, fracture severity, fracture type, fracture location, puncture approach, volume of bone cement, number of fractured vertebral bodies, and number of surgical vertebral bodies. In the binary logistic regression analysis, the variables that showed significant differences in the univariate analysis were included. The OR value indicated the relative risk of bone cement leakage. The results demonstrated that cortical disruption (OR 6.880, 95% CI 4.209-11.246), basivertebral foramen sign (OR 2.142, 95% CI 1.303-3.521), fracture type (OR 1.683, 95% CI 1.083-2.617), and volume of bone cement (OR 1.198, 95% CI 1.070-1.341) were independent predictors for bone cement leakage. All of these variables were statistically significant predictors of bone cement leakage (P < .05), as shown in Table 3.

Table 2.

Comparison of Variables Between the Bone Cement Leakage Group and Non-Bone Cement Leakage Group.

Variable		Bone cement leakage (n = 258/512)	Non-Bone cement leakage (n = 254/512)	t/X²	P
Baseline characteristics	Age	74.37 ± 9.77	75.20 ± 9.00	−1.007	0.314
	Sex
	M	46	45	0.001	0.973
	F	212	209	0.001	0.973
	Height	1.59 ± 0.07	1.59 ± 0.07	0.191	0.849
	Weight	59.04 ± 9.97	58.22 ± 9.13	0.97	0.332
	BMI	23.24 ± 3.62	22.92 ± 2.93	1.116	0.265
	BMD	−4.02 ± 0.99	−4.15 ± 0.77	1.589	0.113
Medical history related variables	Injury to admission time interval	19.79 ± 34.87	19.68 ± 33.10	0.039	0.969
	Admission to surgery time interval	2.91 ± 2.09	2.91 ± 1.43	−0.038	0.97
	The history of OVCF
	Y	71	69	0.008	0.928
	N	187	185	0.008	0.928
	Surgical history of PVP/PKP
	Y	37	44	0.854	0.355
	N	221	210	0.854	0.355
	Surgical history of pedicle screw fixation
	Y	14	15	0.055	0.815
	N	244	239	0.055	0.815
	Chronic diseases
	Y	167	181	2.508	0.113
	N	91	73	2.508	0.113
	Medication of steroid
	Y	8	12	0.899	0.343
	N	250	242	0.899	0.343
	Medication of anti-osteoporosis
	Y	3	3	<0.01	0.985
	N	255	251	<0.01	0.985
	Fracture location
	T	26	35	6.014	0.049
	TL	182	153
	L	50	66
	Number of fractured vertebral body	1.34 ± 0.64	1.52 ± 0.78	−2.844	0.005
	Number of surgical vertebral body	1.30 ± 0.55	1.47 ± 0.75	−2.934	0.003
Operation related variables	Operation method
	PVP	32	46	3.228	0.072
	PKP	226	208	3.228	0.072
	Puncture approach
	Unipedicular	92	113	4.156	0.041
	Bipedicular	166	141	4.156	0.041
	Volume of bone cement	7.03 ± 2.19	6.39 ± 2.15	3.334	0.001
	Length of surgery time	45.67 ± 21.19	45.42 ± 19.60	0.142	0.887
	Anesthesia techniques
	Local	132	116	1.748	0.417
	Intravenous	113	126
	General	13	12
Radiology related variables	IVC
	Y	99	61	12.279	<0.01
	N	159	193	12.279	<0.01
	Cortical disruption
	Y	228	128	87.141	<0.01
	N	30	126	87.141	<0.01
	Basivertebral foramen sign
	Y	74	40	12.371	<0.01
	N	184	214	12.371	<0.01
	The intrusion of the posterior wall
	Y	32	11	10.841	0.001
	N	226	243	10.841	0.001
	Fracture severity
	Mild	126	170	17.426	<0.01
	Moderate	66	45
	Severe	66	39
	Fracture type
	Wedge	222	239	14.494	0.001
	Biconcave	5	7
	Crush	31	8

Table 3.

Results of Binary Logistic Regression Analysis to Bone Cement Leakage after PVA.

	OR	P	95% CI
	OR	P	lower limit	upper limit
IVC	1.195	0.459	0.745	1.918
Cortical disruption	6.880	0.000	4.209	11.246
Basivertebral foramen sign	2.142	0.003	1.303	3.521
The intrusion of the posterior wall	1.536	0.297	0.686	3.440
Fracture severity	1.062	0.688	0.791	1.425
Fracture type	1.683	0.021	1.083	2.617
Fracture location	0.724	0.101	0.492	1.065
Number of fractured vertebral body	1.500	0.379	0.608	3.705
Number of surgical vertebral body	0.544	0.221	0.205	1.443
Puncture approach	0.941	0.804	0.584	1.516
Volume of bone cement	1.198	0.002	1.070	1.341

Performance of Prediction Model

The developed model utilized the aforementioned variables, and Table 4 presents the discrimination ability of five ML algorithms and one deep learning algorithm on the testing set. Figure 1 illustrates the comparison of the AUROC for the predictive models between the training set and the test set. Among them, XGBoost exhibited the highest performance in predicting bone cement leakage following PVP/PKP in the testing set. (AUC = .8819, accuracy score = .8025, recall score = .7872, F1 score = .8315, precision score = .881).

Table 4.

Results of Models for Bone Cement Leakage in PVA for OVCF.

	XGBoost	Decision tree	Random forest	SVC	GaussianNB	ANN
Accurancy	0.8052	0.7013	0.8312	0.7662	0.7273	0.7792
Recall	0.7872	0.7021	0.8936	0.6809	0.7012	0.7447
F1-score	0.8315	0.7416	0.866	0.7805	0.7586	0.8046
Precision	0.881	0.7857	0.84	0.9143	0.825	0.875

Figure 1.

Receiver Operating Characteristic (ROC) Curves for 5 Machine Learning Models and 1 Deep Learning Model. The XGBoost model achieved a better Area Under the ROC Curve (AUROC) compared to the other models. (a) ROC curve for the training set; (b) ROC curve for the testing set.

The calibration plot in Figure 2 demonstrates a favorable predictive accuracy between the actual probability and the predicted probability.

Figure 2.

Calibration Plots of 5 mL Models and 1 dL Model.

To assess the clinical utility of the predictive model, the DCA curve was employed. Figure 3 presents the comparison of the six models.

Figure 3.

Decision Curve Analysis for 5 mL Models and 1 dL Model. (a) ANN; (b) Decision tree; (c) GaussianNB; (d) Random Forest; (e) SVM; (f) XGBoost.

In summary, the XGBoost model exhibited the best performance in this study.

Visualized Interpretation of Feature Importance and Personalized Predictions

Visualized Interpretation of Feature Importance

The XGBoost model was selected to calculate the SHAP values for all features using the SHAP package. Based on these results, summary plots and force plots were generated to visually explore the relationship between the prediction outcomes and each individual feature. Figure 4(a) depicts all the evaluated risk factors by their average absolute SHAP values, while Figure 4(b) showcases the top 10 most important features in our model.

Figure 4.

Model Interpretation. (a) Importance ranking of the top 10 risk factors with stability and interpretation using the optimal model. (b) Importance ranking of the top 10 variables based on the mean |SHAP value.

According to the summary plot, features such as cortical disruption, volume of bone cement, height, BMI, injury to admission time interval, IVC, fracture severity, weight, basivertebral foramen sign, and BMD played decisive roles in the predictive models.

Risk factors were assessed by examining the scatter plot generated by the SHAP package in combination with their clinical significance. The scatter plot revealed relatively high absolute SHAP values for cortical disruption, IVC, and basivertebral foramen sign, with clear distinctions in the distribution of SHAP values. It is important to note that a SHAP value of zero for a feature indicates that it had no impact on the model’s output.

Interpretation of Personalized Predictions

The force plot for a single patient, depicted in Figure 5, illustrates the influence of each feature on the predictive outcomes in terms of high risk and low risk of bone cement leakage. The blue and red arrows represent the impact of each feature. A blue arrow indicates that the feature reduces the risk of bone cement leakage, while a red arrow indicates an increased risk. The width of the arrow corresponds to the contribution made to the predictive value. To demonstrate the interpretability of the model, we provide two random examples: one patient who experienced bone cement leakage following PVA, and another patient who did not.

Figure 5.

Interpretation of Model Prediction Results Using Two Sample Cases (a) a patient who didn’t experience bone cement leakage following PVA; (b) a patient who experienced bone cement leakage following PVA.

Discussion

This study developed interpretable ML models that integrate clinical features and radiographic signs to predict the risk of bone cement leakage after PVA in OVCF. All models demonstrated favorable performance, with the XGBoost model outperforming GaussianNB, Decision Tree, Random Forest, SVM, and ANN models. The superior predictive performance of the XGBoost model indicates its potential for accurate and reliable risk assessment in clinical settings. By computing SHAP values and mean values, the XGBoost model enabled both global and local interpretations. The key features found to play a decisive role in the predictive models, considering their clinical significance, include cortical disruption, volume of bone cement, BMI, injury to admission time interval, IVC, fracture severity and type, basivertebral foramen sign, and BMD. Additionally, two examples are provided to illustrate the model’s capability in predicting cement leakage and assessing the relative importance of each variable. These findings empower clinicians to identify high-risk bone cement leakage patients, enhance their understanding of the significance of features in overall and individual prediction models, and support clinical decision-making aimed at reducing the incidence of bone cement leakage while improving patient outcomes.

As populations age, there is a concomitant rise in the prevalence of osteoporosis and the subsequent risk of osteoporotic fractures. PVA is a minimally invasive surgical procedure that is recommended and extensively employed for patients experiencing acute symptomatic OVCF. Nevertheless, the postoperative complication of bone cement leakage poses a significant concern, given its potential to result in severe complications or even fatalities when symptoms emerge. This drawback contradicts the procedure's benefits, such as minimal invasiveness, rapid recovery, and early discharge from bed rest. Literature reports an incidence of bone cement leakage varying between 5-82%.⁷ Our study, which retrospectively evaluated 512 fractured vertebrae, found an incidence of bone cement leakage at 50.4%. This result aligns with studies that utilized postoperative radiography to assess leakage, although it is slightly less than those that exclusively relied on postoperative CT scanning.

Numerous prior studies that have compared the incidence of bone cement leakage by employing different surgical strategies¹² or by accounting for specific factors (like the volume and viscosity of bone cement¹³) have identified a lower occurrence of cement leakage in certain situations. This suggests a potential for prevention. Consequently, the preoperative prediction of bone cement leakage carries substantial clinical importance. The study by Li et al demonstrated that predictive models using nomograms and ML can be developed to forecast bone cement leakage after PVP, based on the associated risk factors.^7,14 Deng et al retrospectively examined 374 cases of thoracolumbar OVCFs treated with single-level PVP. They sifted through data and variables using four ML algorithms and formulated a nomogram prediction model based on six variables, which demonstrated reliable predictive power.¹⁵ There was a similar study on prediction models for constructing nomograms for intradiscal type leakage.¹⁶

Previous models, however, do not integrate radiographic signs, which align more closely with the clinical process, and lack comparison among various deep learning models. In our research, we included 28 features, encompassing radiographic-related variables, into our predictive models. The AUC in the testing set spanned from .7926 to .8819. The predictive performance of our model resembles the ML model used in earlier studies,⁷ surpassing the performance of nomogram predictive models.^14,15 The SHAP package underscored the importance of features, with imaging indicators distinctly contributing in a manner that diverges from prior models. Examining the imaging features carrying significant weight in the predictive results reveals that adjusting the puncture angle and depth, along with optimizing the timing and volume of cement injection, could effectively minimize the risk of bone cement leakage during PVA.

In this study, 5 ML model classifiers and 1 deep learning model classifier were used, and the above models are widely used in spinal diseases, including the field of diagnosis, treatment and prognosis for spinal degenerative diseases, trauma, spinal tumors, etc. In diagnosis, the application of ML algorithm achieved high diagnostic accuracy in disease prediction and differentiation, primarily utilizing radiological imaging. Hao Zhang et al¹⁷ established Gaussian naïve Bayes (GNB), K-nearest neighbor (KNN), logistic regression, multilayer perceptron (MLP), Decision Tree, linear discriminant, and gradient-boosting decision tree classifiers models, combined radiomics to differentiate benign from malignant indistinguishable vertebral compression fractures with improved diagnostic efficacy. Jun Zhang et al have differentiated acute and chronic vertebral compression fractures by similar ML models combined radiomics features and deep transfer learning features. ML has been incorporated into treatment decision-making processes such as surgical grade, instrumental vertebrae, and correction range.¹⁸ For surgical correction of thoracolumbar adult spinal deformity, an ANN successfully mimicked lead surgeons’ decision making in selection of the upper instrumented vertebra.¹⁹An increasing number of ML models have been used to predict prognosis, such as clinical outcomes and complications. Sung Tan Cho et al²⁰ have used Decision Tree and Random Forest models to analysis prognostic factors for the progression of OVCF following conservative treatment. Zhang et al²¹ have built XGBoost, logistic regression, Random Forest, AdaBoost, GNB, ComplementNB, MLP, SVM, and KNN machine learning models to predict delirium in patients after surgery for degenerative spinal disease. Among them, XGBoost models was practical and had a good performance. Ren et al²² also established similar ML models to predict Recurrent Lumbar Disc Herniation Following Percutaneous Endoscopic Lumbar Discectomy. Based on the previous literatures review and the experience of our group, we chose the above models to predict bone cement leakage. Of the models used in our study, the XGBoost model displayed superior predictive performance, even surpassing deep learning algorithms. Despite deep learning models’ strengths in areas such as images, language, and audio, some researchers have posited that, with tabular data, it is simpler to achieve robust predictions using tree-based methods rather than deep learning.²³

Binary logistic regression analysis in our research showed that cortical disruption, basivertebral foramen, fracture type, and the volume of bone cement were independently linked with bone cement leakage post-PVA in OVCF. The interpretation of SHAP demonstrated that features such as cortical disruption, volume of bone cement, BMI, injury to admission time interval, IVC, fracture severity, basivertebral foramen, and BMD play a pivotal role in the predictive model. This does not completely align with the results of previous studies, as shown in the feature importance analysis.^7,24,25 Radiology-related variables such as cortical disruption, IVC, fracture severity and type, and basivertebral foramen hold a key position in the predictive model, underscoring the necessity of their inclusion.

When considering risk factors for bone cement leakage following PVA, prior studies have not provided definitive evidence, leaving some risk factors contentious. Drawing from the results of binary logistic regression analysis and feature importance analysis conducted in this research, in conjunction with their clinical significance, we can deduce that factors such as cortical disruption, IVC, fracture severity and type, basivertebral foramen, BMI, BMD, volume of bone cement, and injury to admission time interval play a significant role in contributing to bone cement leakage in PVA.

Cortical disruption emerged as the primary risk factor, aligning with previous research findings.²⁶ A higher degree and certain types of fractures indicate more severe cortical disruption which amplifies the risk of bone cement leakage. This is particularly the case with end plate-associated intradiscal types, which heighten the risk of new adjacent vertebral fractures.⁶ The concept of IVC was initially proposed in relation to Kummell’s disease, where it is considered a hallmark feature,²⁷ indicating avascular necrosis of the vertebral body and severe collapse. According to Zhang, K et al, IVC is not only a risk factor for type-C leakage but also acts as a protective factor against type-S leakage (segmental venous leakage).¹⁰ This is possibly attributable to the lower stress within the vertebral body spaces during cement injection as compared to a compressed vertebral body, thereby allowing room for the cement to fill, and consequently reducing cement leakage.²⁸ This phenomenon may also be associated with the phase of IVC, which contains liquid or gas.²⁹ The basivertebral foramen is an independent risk factor for type-B leakage,¹⁰ as the bone defect in the posterior wall of the vertebral body provides a natural passage for bone cement leakage.³⁰ A study by Wang et al³¹ uncovered a link between the basivertebral foramen and IVC, which can directly cause bone cement leakage after PKP in elderly OVCF patients. The volume of bone cement emerged as a risk factor for bone cement leakage in both binary logistic regression analysis and feature importance ranking. There is no positive correlation between the volume of bone cement and clinical effectiveness, including pain relief and restoration of vertebral body height. However, an increase in bone cement volume elevates the risk of bone cement leakage. As such, selecting the optimal volume of bone cement is imperative to maximize clinical effectiveness and minimize bone cement leakage. The volume fraction percentage (VF%), which represents the ratio of intravertebral cement volume (IVCV) to fractured vertebral body volume (FVBV), has been proposed to assess the optimal volume of bone cement.³² Nieuwenhuijse et al³³ suggested an optimal value of 24%. Clinicians should undertake an individualized evaluation of each patient to determine the suitable volume of bone cement. A lower BMD, indicative of a higher grade of osteoporosis, suggests larger trabecular space for bone cement injection and lower injection pressure, thus, facilitating the reduction of bone cement occurrence.³⁴ However, a higher degree of osteoporosis also leads to more severe fracture damage, potentially resulting in leakage from the weaker cancellous bone into the venous system.³⁵ In recent years, the Subcutaneous fat index (SFI) has been proven to be a more important evaluated variable for obesity than BMI.³⁶ It was not used in the study because previous literatures did not analyze it as a risk factor for bone cement leakage. Further studies will be conducted to objectively evaluate the image-related features including SFI, and create more efficient predictive and generalized model by using deep learning and radiomics techniques.

This study does come with several limitations. Firstly, as a single-center retrospective study, the data were exclusively sourced from the Department of Spine Surgery, Zhongda Hospital affiliated with Southeast University, warranting further validation of generalizability and potential recalibration when implemented by other medical institutions. Secondly, data from multiple centers should be incorporated into the model development and its internal and external validation. Thirdly, in this study, considering the high-risk population of osteoporotic vertebral compression fracture, only over 45 years of age patients were included in the analysis, which resulted in a certain degree of bias, and subsequent studies will further expand the population to improve the generalizability of the model. Fourthly, Fewer male patients were included in this retrospective study, as a proof of concept, the limited gender proportion and sample size demonstrated the effectiveness of the ML model, but a more rational gender ratio and the sample size expansion need in future research. Fifthly, this study predominantly assessed bone cement leakage using postoperative radiography, with CT not routinely employed. Although postoperative radiography can evaluate bone cement leakage, the gold standard for bone cement leakage assessment remains postoperative CT, which should be more frequently applied in future studies.

Conclusion

In our investigation, we formulated six distinct prediction models for bone cement leakage following PVA in OVCF patients. The XGBoost model demonstrated superior predictive capacity for bone cement leakage. This interpretable model can offer a visual representation of feature importance and individualized prediction. The predictive outcomes could aid medical practitioners in preoperative assessment of the risk of bone cement leakage, allowing comprehensive consideration of clinical characteristics. This, in turn, could help the development of more tailored treatment strategies or preoperative interventions to decrease the occurrence of cement leakage.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The work was supported by the National Natural Science Foundation of China (82202768), Jiangsu Provincial medical Key Discipline (Laboratory) Cultivation Unit (JSDW202222).

ORCID iDs

Yi-Li Hu

Pei-Yang Wang

References

Amans

Carter

Chandra

Shah

Hirsch

. Vertebral augmentation. Handb Clin Neurol. 2021;176:379-394. doi:10.1016/b978-0-444-64034-5.00017-1

Nuti

Brandi

Checchia

, et al. Guidelines for the management of osteoporosis and fragility fractures. Intern Emerg Med. 2019;14(1):85-102. doi:10.1007/s11739-018-1874-2

Zhu

Zhong

Chen

. Risk factors for bone cement leakage in percutaneous vertebroplasty: A retrospective study of four hundred and eighty five patients. Int Orthop. 2016;40(6):1205-1210. doi:10.1007/s00264-015-3102-2

Dai

Liang

Zhang

Dong

Zhou

. Risk factors of vertebral re-fracture after PVP or PKP for osteoporotic vertebral compression fractures, especially in Eastern Asia: A systematic review and meta-analysis. J Orthop Surg Res. 2022;17(1):161. doi:10.1186/s13018-022-03038-z

Qin

Tian

Zhang

. Risk factors for hidden blood loss associated with vertebroplasty or kyphoplasty for osteoporotic vertebral compression fracture: A systematic review and meta-analysis. World Neurosurg. 2023;175:47-56. doi:10.1016/j.wneu.2023.03.114

Zaniletti

Larson

Lewallen

Berry

Maradit Kremers

. How to develop and validate prediction models for orthopedic outcomes. J Arthroplasty. 2022;38(4). doi:10.1016/j.arth.2022.12.032

Wang

Liu

, et al. Machine learning applications for the prediction of bone cement leakag e in percutaneous vertebroplasty. Front Public Health. 2021;9:812023. doi:10.3389/fpubh.2021.812023

Tang

Chen

, et al. The impact of intravertebral cleft on cement leakage in percutaneous vertebroplasty for osteoporotic vertebral compression fractures: a case-control study. BMC Musculoskelet Disord. 2021;22(1):805. doi:10.1186/s12891-021-04685-9

Genant

van Kuijk

Nevitt

. Vertebral fracture assessment using a semiquantitative technique. J Bone Miner Res. 1993;8(9):1137-1148. doi:10.1002/jbmr.5650080915

10.

Zhang

She

Zhu

Wang

. Risk factors of postoperative bone cement leakage on osteoporotic vertebral compression fracture: A retrospective study. J Orthop Surg Res. 2021;16(1):183. doi:10.1186/s13018-021-02337-1

11.

Mumford

Weinstein

Spratt

Goel

. Thoracolumbar burst fractures. The clinical efficacy and outcome of nonoperative management. Spine (Phila Pa 1976). 1993;18(8):955-970.

12.

Wang

Sribastav

, et al. Comparison of percutaneous vertebroplasty and balloon kyphoplasty for the treatment of single level vertebral compression fractures: A meta-analysis of the literature. Pain Physician. 2015;18(3):209-222.

13.

Tan

Cheng

Zhang

. High-viscosity versus low-viscosity cement for the treatment of vertebral compression fractures: A meta-analysis of randomized controlled trials. Medicine (Baltimore). 2022;101(46):e31544. doi:10.1097/md.0000000000031544

14.

Wang

Dong

, et al. Establishment and validation of a nomogram and web calculator for the risk of new vertebral compression fractures and cement leakage after percutaneous vertebroplasty in patients with osteoporotic vertebral compression fractures. Eur Spine J. 2022;31(5):1108-1121. doi:10.1007/s00586-021-07064-z

15.

Deng

Zhu

, et al. Application of machine learning in prediction of bone cement leakage during single-level thoracolumbar percutaneous vertebroplasty. BMC Surg. 2023;23(1):63. doi:10.1186/s12893-023-01959-y

16.

Fan

Wang

, et al. A predictive nomogram for intradiscal cement leakage in percutaneous kyphoplasty for osteoporotic vertebral compression fractures combined with intravertebral cleft. Front Surg. 2022;9:1005220. doi:10.3389/fsurg.2022.1005220

17.

Zhang

Yuan

Wang

, et al. Differentiation of benign versus malignant indistinguishable vertebral compression fractures by different machine learning with MRI-based radiomic features. Eur Radiol. 2023;33(7):5069-5076. doi:10.1007/s00330-023-09678-x

18.

Ren

Xie

, et al. Current applications of machine learning in spine: from clinical view. Global Spine J. 2022;12(8):1827-1840. doi:10.1177/21925682211035363

19.

Lafage

Ang

Alshabab

, et al. Predictive model for selection of upper treated vertebra using a machine learning approach. World Neurosurg. 2021;146:e225-e232. doi:10.1016/j.wneu.2020.10.073

20.

Cho

Shin

Kim

Yoon

Il Lee

Lee

. Prediction of progressive collapse in osteoporotic vertebral fractures using conventional statistics and machine learning. Spine (Phila Pa 1976) 2023;48(21):1535-1543. doi:10.1097/brs.0000000000004598

21.

Zhang

Wan

Chen

, et al. Automated machine learning-based model for the prediction of delirium in patients after surgery for degenerative spinal disease. CNS Neurosci Ther. 2023;29(1):282-295. doi:10.1111/cns.14002

22.

Ren

Liu

Zhang

, et al. Machine learning predicts recurrent lumbar disc herniation following percutaneous endoscopic lumbar discectomy. Global Spine J. 2022;2:21925682221097650. doi:10.1177/21925682221097650

23.

Grinsztajn

Oyallon

Varoquaux

. Why do tree-based models still outperform deep learning on tabular data? 2022. arXiv:2207.08815v1.

24.

Zhan

Jiang

Liao

Tan

Yang

. Risk factors for cement leakage after vertebroplasty or kyphoplasty: A meta-analysis of Published evidence. World Neurosurg. 2017;101:633-642. doi:10.1016/j.wneu.2017.01.124

25.

Tang

Cui

Chen

Liu

. Risk factors for cement leakage in percutaneous vertebroplasty for osteoporotic vertebral compression fractures: An analysis of 1456 vertebrae augmented by low-viscosity bone cement. Spine (Phila Pa 1976) 2021;46(4):216-222. doi:10.1097/brs.0000000000003773

26.

Ding

Zhang

Zhu

, et al. Risk factors for predicting cement leakage following percutaneous vertebroplasty for osteoporotic vertebral compression fractures. Eur Spine J. 2016;25(11):3411-3417. doi:10.1007/s00586-015-3923-0

27.

Chen

Lou

. Comparison of intravertebral clefts between Kümmell disease and acute osteoporotic vertebral compression fracture: A radiological study. Orthop Surg. 2021;13(7):1979-1986. doi:10.1111/os.13025

28.

Hiwatashi

Ohgiya

Kakimoto

Westesson

. Cement leakage during vertebroplasty can be predicted on preoperative MRI. AJR Am J Roentgenol. 2007;188(4):1089-1093. doi:10.2214/ajr.06.0903

29.

Linn

Birkenmaier

Hoffmann

Reiser

Baur-Melnyk

. The intravertebral cleft in acute osteoporotic fractures: Fluid in magnetic resonance imaging-vacuum in computed tomography? Spine (Phila Pa 1976). 2009;34(2):E88-93. doi:10.1097/BRS.0b013e318193ca06

30.

Wang

Jin

Zhang

. Morphological features of basivertebral foramen among different age groups: Recognition in spine. Front Surg. 2023;10:1115654. doi:10.3389/fsurg.2023.1115654

31.

Wang

Fan

Liu

Suyou

Shan

Zhao

. Basivertebral foramen could be connected with intravertebral cleft: A potential risk factor of cement leakage in percutaneous kyphoplasty. Spine J. 2014;14(8):1551-1558. doi:10.1016/j.spinee.2013.09.025

32.

Sun

Jing

Liu

Wang

Hai

. The optimal volume fraction in percutaneous vertebroplasty evaluated by pain relief, cement dispersion, and cement leakage: A prospective cohort study of 130 patients with painful osteoporotic vertebral compression fracture in the thoracolumbar vertebra. World Neurosurg. 2018;114:e677-e688. doi:10.1016/j.wneu.2018.03.050

33.

Nieuwenhuijse

Bollen

van Erkel

Dijkstra

. Optimal intravertebral cement volume in percutaneous vertebroplasty for painful osteoporotic vertebral compression fractures. Spine (Phila Pa 1976). 2012;37(20):1747-1755. doi:10.1097/BRS.0b013e318254871c

34.

Xie

Jin

, et al. Cement leakage in percutaneous vertebral augmentation for osteoporotic vertebral compression fractures: Analysis of risk factors. Clin Spine Surg. 2016;29(4):E171-176. doi:10.1097/bsd.0000000000000229

35.

Gao

Zong

Wang

Cao

Zou

. Analysis of risk factors causing short-term cement leakages and long-term complications after percutaneous kyphoplasty for osteoporotic vertebral compression fractures. Acta Radiol. 2018;59(5):577-585. doi:10.1177/0284185117725368

36.

Berikol

Ekşi

Aydın

Börekci

Özcan-Ekşi

. Subcutaneous fat index: A reliable tool for lumbar spine studies. Eur Radiol. 2022;32(9):6504-6513. doi:10.1007/s00330-022-08775-7

Interpretable Machine Learning Model to Predict Bone Cement Leakage in Percutaneous Vertebral Augmentation for Osteoporotic Vertebral Compression Fracture Based on SHapley Additive exPlanations

Abstract

Study Design

Objectives

Methods

Results

Conclusion

Keywords

Introduction

Method

Study Participants

Clinical and Radiological Data

Surgery Procedures

Reference Standard

Definition of Radiology-Related Variables

Definition of Bone Cement Leakage

Data Processing

Model Development and Interpretation

Model Development

Evaluation of Performance

Interpretation and Visualization of the Model

Statistical Analysis

Result

Patient Characteristics

Risk Factors

Performance of Prediction Model

Visualized Interpretation of Feature Importance and Personalized Predictions

Visualized Interpretation of Feature Importance

Interpretation of Personalized Predictions

Discussion

Conclusion

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iDs

References