External Validation Confirms Validity of a Simple Model to Predict Bowel Outcome After Traumatic Spinal Cord Injury

Abstract

Background. The prediction of functional outcomes after spinal cord injury (SCI) is essential to plan the rehabilitation phase and the social reintegration. Recently, 2 models to predict independent and reliable bowel management 1 year after traumatic SCI have been derived and validated in 2 cohorts of patients included in the European Multicenter Study about Spinal Cord Injury (EMSCI). Objective. We aimed to validate 2 prediction models for bowel outcome after traumatic SCI in a patient sample external to EMSCI. Methods. The simplified model (based on a single predictor, the International Standards for Neurological Classification of Spinal Cord Injury [ISNCSCI] total motor score) and the full model (based on 2 predictors, the ISNCSCI total motor score and item 3a of the Spinal Cord Independence Measure) were applied to the retrospectively collected data of 111 patients with traumatic SCI. Results. The simplified and the full models showed excellent discrimination with an area under the receiver operating characteristic curve of .939 (95% confidence interval (CI) .87-1.00) and .922 (95% CI 0.85-.99), respectively. Both models displayed similar results for sensitivity and negative predictive values; however, the simplified model showed higher values for specificity, positive predictive values, and accuracy. The calibration analysis showed a partial overlap between predicted probabilities and observed proportion, with better and acceptable calibration for the simplified model. Conclusions. Using an independent sample, our study demonstrates the validity of a simple model to predict independent and reliable bowel management 1 year after traumatic SCI.

Introduction

The prediction of functional outcomes after spinal cord injury (SCI) plays a central role in the definition of proper rehabilitative objectives shared among team, patient and caregivers. This process favors a consistent development of the rehabilitation phase and an accurate planning of discharge. ¹

Over the last years, several models have been introduced for the prediction of the main functional outcomes after SCI, such as upper limb function, ambulation, and bladder control.^2-5 Our group has recently derived and validated in patients with traumatic SCI included in the European Multicenter Study about Spinal Cord Injury (EMSCI; http://www.emsci.org; ClinicalTrials.gov Identifier: NCT01571531) 2 models to predict independent and reliable bowel management 1 year after traumatic SCI. ⁶ However, in order to use and implement the predictive models, their generalizability in new populations external to EMSCI should be evaluated. ⁷

The aim of this study was to verify the prediction performance of our models in an independent cohort of patients with traumatic SCI.

Methods

Patients

We included in the study all patients with acute traumatic SCI admitted at the Spinal Cord Rehabilitation Unit of Santa Lucia Foundation, Rome, Italy, before this center entered the EMSCI network in February 2013. Data were retrospectively retrieved by a physician who was blinded to the prediction model characteristics. Inclusion criterion was the availability of data related to predictors within 40 days and to outcome 1 year after SCI. The cohort of patients considered for this study is the same evaluated in a previous study to verify the performance of 2 models for the prediction of bladder outcome 1 year after traumatic SCI. ⁴

Prediction Models

In our previous study, we derived 2 models to predict a positive bowel outcome 1 year after traumatic SCI. ⁶ A positive bowel outcome was defined as independent bowel management with regular bowel movements and appropriate timing with no or rare accidents (ie, fecal incontinence less than twice a month), as assessed through item 7 of Spinal Cord Independence Measure (SCIM) 1 year after SCI. ⁸ A positive bowel outcome was identified by a score of 10 in SCIM version II and 8 or 10 in SCIM version III. The simplified model was based on a single predictor collected within 40 days from injury, the International Standards for Neurological Classification of Spinal Cord Injury (ISNCSCI) total motor score, a cumulative score (range 0-100) derived from the sum of 5 key muscle strength value for each limb. The full model relied on 2 predictors, the ISNCSCI total motor score and the SCIM 3a score, an item assessing the independence in dressing the upper part of body (range 0-3 in version II and 0-4 in version III).

Statistical Analysis

Patient characteristics were expressed as mean (±standard deviation) or number (percentage), as appropriate. Receiver operating characteristic (ROC) analysis was used to describe the discrimination performance of models. Visual inspection, Brier score, and Spiegelhalter z-test were used to assess the calibration of the model. To evaluate the potential clinical utility of the model, we compared the predicted probability with the real outcome of our study. Furthermore, we chose the best cutoff point that maximized both sensitivity and specificity in order to correctly classify most of the patients. A P-value <.05 was considered statistically significant. Stata version 13.1 was used to perform the analysis.

Results

A total of 135 patients evaluated between 2004 and 2013 were considered for the study. Among these, 24 (18%) were lost at follow-up. The final sample consisted of 111 patients, mean age 40 (±16) years, 102 (92%) men. The neurological level was cervical in 45 (41%) patients, thoracic in 57 (51%), lumbar in 8 (7%), and sacral in 1 (1%). The ASIA Impairment Scale grade was A in 63 (57%), B in 12 (11%), C in 12 (11%), and D in 24 (22%).

The ROC analysis showed excellent values of area under the ROC curve for the simplified and the full models: .939 (95% confidence interval (CI) .87-1.00) and .922 (95% CI .85-.99), respectively (Figures 1(A) and 1(B)). Considering a cutoff point of .60, the simplified and the full model showed the following values: sensitivity both 91%, specificity 90% and 74%, positive predictive values 70% and 48%, negative predictive values 98% and 97%, and accuracy 90% and 77%, respectively.

Figure 1.

Receiver operating characteristic curve and calibration plot. Area under the receiver operating characteristic curve of the simplified (A) and full (B) model and calibration plot of the simplified (C) and full (D) model.

The calibration curves for the simplified and the full models are presented in Figures 1(C) and 1(D). The simplified model showed an intercept value of −1.754, a slope of 3.147, and a Brier score of .213 (Spiegelhalter’s z-statistic = .434; P = .332). The full model had an intercept of −1.953, a slope of 2.437, and a Brier score of .235 (Spiegelhalter’s z-statistic = 2.362; P = .0091).

Finally, based on sensitivity analysis of the simplified model (Figure 2(A)), we defined a cutoff value which enables to stratify patients into 2 groups with low (P < .6) or high (P ≥ .6) probability to recover a positive bowel outcome at 1 year (Figure 2(B)).

Figure 2.

Sensitivity analysis of the simplified model and stratification of patients. Based on the sensitivity analysis of the simplified model (A), we identified a cutoff value of probability of positive outcome (P), which enables us to stratify the sample in 2 groups (B) patients with P < .6 are considered at low probability, while patients with P ≥ .6 are considered at high probability to recover a positive bowel outcome 1 year after spinal cord injury.

Discussion

Our study demonstrates in an independent sample an excellent discrimination of our models in the prediction of bowel outcome 1 year after traumatic SCI, in line with the findings by Khan and colleagues. ⁹ The simplified model showed higher specificity, positive predictive values, and accuracy. Concerning calibration, for both models, visual inspection revealed a partial overlap between predicted probabilities and observed proportion. The nonsignificant P-values for Spiegelhalter statistics indicated a better and acceptable calibration for the simplified model. ¹⁰ Therefore, based on statistical and clinical considerations, we suggest to use our simplified model for prediction of bowel outcome 1 year after traumatic SCI. The sensitivity analysis of the simplified model allowed us to identify a cutoff value to stratify patients in 2 groups of probability of bowel outcome recovery.

Our study is limited by the retrospective data collection and by the relatively small number of patients included for the analysis.

The validation in independent samples represents a mandatory step to assess the use and the implementation of a prediction model. ⁷ Our data, together with the results by Khan and collaborators ⁹ further support the validity of our simplified model for the prediction of bowel outcome after traumatic SCI.