Abstract
Background:
This study evaluated the use of several risk prediction models in estimating short- and long-term mortality following hip fracture in an Australian population.
Methods:
Data from 195 patients were retrospectively analysed and applied to three models of interest: the Nottingham Hip Fracture Score, the Age-Adjusted Charlson Comorbidity Index and the Physiological and Operative Severity Score for enUmeration of Mortality and Morbidity. The performance of these models was assessed with receiver operating characteristic curve as well as logistic regression modelling.
Results:
The median age of participants was 83 years and 69% were women. Ten percent of patients were deceased by 30 days, 25% at 6 months and 31% at 12 months post-operatively. While there was no statistically significant difference between the models, the Age-Adjusted Charlson Comorbidity Index had the largest area under the receiver operating characteristic curve for within 30 day and 12 month mortality, while the Nottingham Hip Fracture Score was largest for 6-month mortality. There was no evidence to suggest that the models were selecting a specific subgroup of our population, therefore, no indication was present to suggest that using multiple models would improve mortality prediction.
Conclusions:
While there was no statistically significant difference in mortality prediction, the Nottingham Hip Fracture Score is perhaps the best suited clinically, due to its ease of implementation. Larger prospective data collection across a variety of sites and its role in guiding clinical management remains an area of interest.
Background
An increase in the incidence of hip fractures is projected in Australia, largely due to the aging population.1,2 Although there has been progress in reducing short-term mortality risk among elderly patients, acute care costs and long-term mortality and morbidity risk remain significant.2–8
Although many scoring systems are available for use in hip fracture patients, there is relatively little data comparing the predictive value of these systems. The Age-Adjusted Charlson Comorbidity Index (AACCI), 9 the Physiological and Operative Severity Score for enUmeration of Mortality and Morbidity (POSSUM) 10 and the Nottingham Hip Fracture Score (NHFS) are scores of interest that have been previously evaluated in an orthopaedic setting. 11 To our knowledge, the Charlson Comorbidity Index is one among these systems that has been evaluated on an Australian hip fracture population. 12
This study addresses the lack of Australian-based populations for review of these systems. It aims to identify an optimal score to assess a patient’s fitness for surgery, provide risk information to patients and patient families and, most importantly, identify high-risk patients which may be of use to structure the provision of resources during these admissions.
Methods
Ethics approval
Approval was granted by the Princess Alexandra hospital Human Research and Ethics Committee and Site Governance (HREC/14/QPAH/240 and SSA/14/QPAH/201), with a waiver for patient consent granted.
This study was performed at the Princess Alexandra Hospital, a tertiary hospital in Queensland Australia. Data were stored on the Princess Alexandra Hospital information system and encrypted to preserve patient confidentiality and de-identification.
Data were entered in an IBM SPSS, version 21.0, Statistics for Windows spreadsheet (Armonk, NY) and used for most statistical analyses. Collected data were split into two sets: the calibration set composed of 130 patients and the validation set comprised of 65 patients. To validate the effectiveness of the models, the calibration set (or training set) was used for most analyses. The validation data set was reserved for analysis of the scores at an individual level. The predicted outcomes of the calibration set were then compared against the observed outcomes to assess the models’ predictive power in terms of sensitivity and specificity. Patients were randomly assigned a number to generate these data sets (calibration and validation).
As this was an exploratory analysis, we did not inflate the sample size nor was the significance value threshold adjusted (p value = 0.05) to correct for multiple hypothesis testing.
Data were retrospectively collected from patients aged 65 years and above who underwent an operation for a fractured neck of femur between January 2010 and 2013. All hip fracture surgeries in this time period were randomised and 198 individuals selected (by assigning random patient ID numbers). Three cases were excluded from the study because they did not match the age criteria (age < 65 years). Data were collected on factors related to the criteria of the three tested scores, NHFS, AACCI and POSSUM (see Appendix 1). Total possible scores for NHFS (0–10), AACCI (0–39) and POSSUM (18–132) were calculated. In addition, further physiological and operative data were collected including type of anaesthetic used, cause of fractured neck of femur, and operation performed. The POSSUM scoring system has several criteria related to surgical pathology, which was not uniformly available. In particular, intraoperative blood loss was inconsistently recorded. An estimation was made, in line with the haemoglobin balance method described by Gao et al. 13 where a decrease of 10 g/L from pre-operative to post-operative haemoglobin was considered equivalent to 500 mL of blood loss. Similarly, if the patient received a transfusion, each transfusion was thought equivalent to an equal volume (500 mL).
Mortality data in the form of month and year of death were collected from the Queensland Death Register and linked to within 30-day mortality, 6-month mortality and 12-month mortality. Patients not present on the register were assumed to remain alive at the time of data collection.
Performance of scores of interest
The AACCI, POSSUM and NHFS systems were analysed in several ways. Area under the receiver operating characteristic (ROC) curve was analysed for within 30-day, 6-month and 12-month mortality. 14 Six- and 12-month morbidity data (complicated vs uncomplicated) were likewise assessed. Second, binary logistic regression was performed on the total score from each model for within 30-day, 6-month and 12-month mortality with performance assessed by Nagelkerke R2 and sensitivity and specificity analysis. A cut-off value of 0.5 was taken for each score to categorise a patient into predicted deceased and alive groups (i.e. if a patient’s predicted mortality was found to be >50% by any of the scores they were grouped as ‘more likely to die than not’).
Screening for prognostic variables
Variables within the three scoring systems of interest, in addition to anaesthetic used, cause of fractured neck of femur and operation performed were analysed. Initially, scale variables from the entire data set were analysed with correlation tables for 30-day, 6-month and 12-month mortality. Variables excluded in this process were those with p values exceeding 0.1.
Of the remaining variables a non-parametric independent sample analysis was performed, again excluding variables with p-values exceeding 0.1. Categorical variables were selected with use of crosstabs analysis, excluding variables with p-values greater than 0.1. Finally, the remaining variables from this process were individually analysed using univariate logistic regression.
Determining model variance at individual level
A Venn diagram was generated using InteractiVenn, a web-based tool developed by Heberle et al. 15 on validation data, to identify if each model was predicting a similar group of patients within our population that were expected to be deceased 12 months post-operatively. In addition, McNemar cross-tabulation analysis was performed on predicted outcomes between models to determine if they were significantly different from one another (p < 0.05).
Results
Of the 195 patients recruited in this study, 69% were women with a median age of approximately 83 years of age. On average patients had an admission haemoglobin of 118 g/L, greater than 1 comorbidity with a mean admission mini-mental test score (MMTS) of seven. Sixty-seven patients came from institutions (34%). Of all comorbidities, coronary artery disease (CAD) was by far the most common, with 73 patients (37%) having known disease. One-hundred and fifty-four patients (79%) had normal pre-operative chest radiographs. General anaesthetic was used in 147 cases (75%), with regional anaesthetic performed in 40 (21%) and a combination used in 8 (4%) of all cases. Nineteen (10%) patients were deceased by 30 days, 48 (25%) of patients were deceased at 6 months and 61 (31%) patients were dead at 12 months post their operation date. Baseline characteristics of our population are presented in Table 1.
Characteristics of participants within the calibration and validation data sets.
Hb: admission haemoglobin (g.L−1); MMTS: mini-mental test score (out of 10); CAD: coronary artery disease; CXR: chest x-ray; NHFS: Nottingham Hip Fracture Score; AACC: Age-Adjusted Charlson Comorbidity Index; POSSUM: Physiological and Operative Severity Score for enUmeration of Mortality. Values are mean (SD), median (IQR (range)) or number (proportion).
Ninety-six patients remained alive at the time of follow-up. The three models demonstrated areas under the curve greater than 0.7 for mortality, indicating they are capable predictors of mortality at each time point (Table 2).
Receiver operating characteristic analysis on calibration data set (n = 130) for mortality prediction.
AUC: area under the curve; CI: confidence interval; NHFS: Nottingham Hip Fracture Score; AACC: Age-Adjusted Charlson Comorbidity Index; POSSUM: Physiological and Operative Severity Score for enUmeration of Mortality.
Sensitivity and specificity analysis for the three pre-developed scores was completed with ROC curves for three time periods, 30-day, 6-month and 12-month mortality, graphically represented in Figures 1–3.
Receiver operating characteristic curve for within 30-day mortality prediction on calibration data set (n = 130).
Receiver operating characteristic curve for 6-month mortality prediction on calibration data set (n = 130).
Receiver operating characteristic curve for 12-month mortality prediction on calibration data set (n = 130).
The AACCI had the largest area under the ROC for mortality at both 30 days and 12 months with area under the curve (AUC) of 0.773 and 0.791, respectively. The NHFS had the largest AUC for 6-month mortality of 0.761. However, the three models’ areas under the ROC curve were not statistically significant from one another (Hanley & McNeil comparison p > 0.05; Table 3).
Area under ROC curve comparison in mortality prediction. P values presented below.
ROC: receive operating curve; NHFS: Nottingham Hip Fracture Score; AACC: Age-Adjusted Charlson Comorbidity Index; POSSUM: Physiological and Operative Severity Score for enUmeration of Mortality. Using the Hanley and McNeil method on calibration data set (n = 130).
Sensitivity and specificity analysis via logistic regression for the scores at the various time points has been presented in Table 4 below. Within 30-day mortality sensitivity was low for all scores as the models were predicting most, if not all, patients to remain alive in this time period. The scores corresponding to a predicted mortality of 50% (the threshold value for determining ‘more likely to die than not’) are presented in Table 5.
Sensitivity, specificity and positive predictive value and negative predictive value of scoring systems for mortality.
NHFS: Nottingham Hip Fracture Score; AACC: Age-Adjusted Charlson Comorbidity Index; POSSUM: Physiological and Operative Severity Score for enUmeration of Mortality; PPV: positive predictive value; NPV: negative predictive value. Calibration data set utilised (n = 130).
Score values corresponding to predicted mortality of 50% at varying time points via logistic regression.
NHFS: Nottingham Hip Fracture Score; AACC: Age-Adjusted Charlson Comorbidity Index; POSSUM: Physiological and Operative Severity Score for enUmeration of Mortality. Scores rounded to nearest integer. Calibration data set utilised (n = 130).
Variables of significance differed slightly at each time point. For clarity, this is presented in Appendix 2.
Multivariate logistic regression was performed on the identified variables of significance, determining that the following were predictors of 12-month mortality: age, sex (male), admission haemoglobin (g/L), admission MMTS (0–10), presence of CAD, presence of renal disease and chest radiograph findings. Age, admission haemoglobin and mental test score were evaluated as both categorical and continuous variables, where the variables in their continuous form improved the predictive power of the model. Chest radiographs were classified as either normal or abnormal - where findings of cardiac changes (i.e. cardiomegaly), presence of emphysematous change, pulmonary oedema, pleural effusion, malignancy, would be classified as abnormal. The coefficients and odds ratios of these variables are shown in Table 6. Increasing age and male sex conferred a worse risk prediction, while a higher admission haemoglobin and MMTS resulted in lower risk scores.
Multivariate logistic regression analysis of available variables.
CI: confidence interval; Hb: haemoglobin (g.L−1); MMTS: mini-mental test score (out of 10); CAD: coronary artery disease; CXR: chest x-ray.
Nagelkerke R2 values indicate the level of variance in the data explained by each model (Table 6).
Calibration validation was performed with a Hosmer–Lemeshow goodness of fit, whereby values of less than 0.05 demonstrate a significant lack of fit. Each of the models, therefore, demonstrated a reasonable level of fit to the data (Table 7).
Logistic regression analyses of models on calibration data set (n = 130) for mortality prediction.
NHFS: Nottingham Hip Fracture Score; AACC: Age-Adjusted Charlson Comorbidity Index; POSSUM: Physiological and Operative Severity Score for enUmeration of Mortality.
Venn diagram analysis reveals that the individuals within the validation population appear to be similarly identified and grouped by the four models. Of the 65 participants, 51 (79%) were placed in the same groups (alive or deceased) by the three models at 12 months post-operatively. The model predictions were analysed against one another using McNemar cross-tabulation, which determined that there was no significant difference in predictions at the individual level between models (p value > 0.05; Table 8).
McNemar cross-tabulation of variation between models at individual level on validation data set (n = 65) for 12-month mortality prediction.
NHFS: Nottingham Hip Fracture Score; AACC: Age-Adjusted Charlson Comorbidity Index; POSSUM: Physiological and Operative Severity Score for enUmeration of Mortality.
Type of anaesthesia used, general versus spinal or a combination, had no impact on mortality at any time point (chi-squared p > 0.05; Table 9).
General anaesthetic versus regional versus combination on mortality outcome.
Results from chi-square analysis of entire data set (n = 195) presented above.
Discussion
This study evaluated the performance of three well-known predictive models for mortality in elderly patients following hip fracture. All scores assessed were predictive of mortality prediction at all time points. There was no statistically significant difference of the area under ROC curve between the models at any time point. Of the three pre-developed scores, the AACCI had the highest sensitivity and specificity at 30 days and 12 months with AUC of 0.773 and 0.791, respectively. The NHFS yielded the highest sensitivity and specificity at 6 months, AUC of 0.761. The NHFS model explained the greatest proportion of variability in the mortality data at every time point, with the largest Nagelkerke R2 (Table 7).
Prior to this study, there was a lack of direct comparison of the many available mortality prediction systems in an orthopaedic setting. Burgos et al. 16 evaluated six different scoring systems, including the CCI and POSSUM, in an elderly hip fracture patient population. Following this study, direct, independent comparison of multiple scoring systems on one population has been rare. Karres et al. 17 investigated six prediction models, including the three we have evaluated in our study, for 30-day mortality prediction. They found that all models, except POSSUM, demonstrated an acceptable level of discriminative power for short-term mortality prediction. 17
To our knowledge, our study represents one of the few independent comparative analyses of available risk models for both short- and long-term mortality prediction. Direct comparisons of our scores of interest are presented in Table 7, where the Nagelkerke R2 value indicates the level of variance explained by each score. There was no statistically significant difference between each of the models.
Table 6 examines the significance of the variables of all of the scores of interest in our population. The variables that remain in Table 6 are those that correlated at a significant (p < 0.05) level in our population with 12-month mortality. It is possible that these variables could form the basis of a new scoring system specific to our population and should be examined in future research across a variety of Australian clinical sites.
In clinical practice, it is important that a score is easy to implement, quick to use and depends on as few variables as possible, while maintaining its predictive accuracy. If a score is to be used for assessment of fitness for surgery or resource allocation during surgery or in the perioperative period, it is further desirable that the score be only reliant on variables that can be obtained prior to the surgery taking place. Taking this into account, it would seem that the NHFS is best suited for clinical use, as it is composed of only seven variables and validated in several populations.11,18–21 Indeed, Marufu et al. 18 determined via meta-analysis of several scores, including those assessed in this article, that the NHFS may be best suited for clinical implementation.
Comparing the application of these scores at an individual level must also be considered. Although two separate scoring systems may predict a similar mortality at a population level, the makeup of these predictions at an individual level may differ. If the models investigated in this study were predicting mortality for different subgroups of our population, it would be reasonable to suggest the use of multiple scores to predict the mortality risk of an individual. To evaluate this, a chi-square analysis of individuals predicted to die at 12 months was performed between our available models. There was no statistically significant difference between the models, indicating that they are predicting a similar subgroup of our population at risk. This is graphically represented in a Venn diagram (Figure 4).
Venn diagram depicting the overlap of predicted categories (alive vs deceased) of the three models for an individual (n = 65).
There remains some debate as to the utility of such scoring systems in the clinical setting. We argue that they are of use for several reasons. Scoring systems allow anaesthetists and surgeons alike to identify higher risk patients, which may alter management in the intraoperative and perioperative period, such as the implementation of multidisciplinary teams. The ‘shared-care’ model of orthogeriatric teams has been shown to improve both short- and long-term mortality. 22 Given the significant cost of hip fractures in Australia and worldwide, 3 identifying lower risk patients who may not require the same level of invasive monitoring in the perioperative period can have important implications from a health economic point of view. Whether varying the instituted measures for patients according to risk categorisation alters outcome remains to be seen and represents an area of interest for future research.
Our findings suggest that there is no difference in post-operative mortality (short or long term) when comparing general versus regional anaesthesia. This is in keeping with previous literature concluding there is no difference in mortality or post-operative complications including myocardial infarction, cerebrovascular incidents or delirium dependent on anaesthesia used.23–26
The limitations of our study largely revolve around its relatively small sample size and retrospective design. Intraoperative blood loss was poorly recorded, resulting in the use of an estimate of this variable. Unfortunately, the data relating to the post-operative recovery period was lacking, mostly due to the loss of contact details of the patients. This study was limited to a single centre, with a relatively small data set. To address these concerns, a prospective study over a variety of clinical sites would be beneficial to further compare the available scores in an Australian population. The formation of a nationwide registry, such as present in the United Kingdom, would greatly help in this endeavour.
In conclusion, though all the pre-developed scores were capable of predicting both short- and long-term mortality, the NHFS demonstrates qualities that facilitate its ease of implementation in clinical practice, therefore it is our preferred score of choice. The formation of an Australian registry would greatly assist in future research endeavours which should aim to prospectively identify an optimal scoring system for use in this population and determine its impact in helping clinicians guide management and resource distribution.
Footnotes
Appendix
Identified variables of significance for mortality prediction at various time points.
| Variables | Within 30 days | 6 month | 12 month |
|---|---|---|---|
| Age | ✓ | ✓ | ✓ |
| Sex | ✓ | ✓ | ✓ |
| Admission Hb | ✓ | ✓ | ✓ |
| Admission MMTS | ✓ | ✓ | ✓ |
| Radiograph findings | ✓ | ✓ | ✓ |
| Cause of fracture | ✓ | ✓ | ✓ |
| Dementia | ✓ | ✘ | ✘ |
| CHF | ✓ | ✓ | ✘ |
| Renal impairment | ✓ | ✘ | ✓ |
| Number of comorbidities | ✘ | ✓ | ✘ |
| ECG findings | ✘ | ✘ | ✓ |
| CAD | ✘ | ✘ | ✓ |
Hb: admission haemoglobin (g.L−1); MMTS: mini-mental test score (out of 10); CAD: coronary artery disease; CXR: chest x-ray; COAD; chronic obstructive airways disease; CHF: congestive heart failure.
Authors contributions
P.S. formed the aims and methodology for the article and provided literature analysis of pre-existing data. M.J.N. gathered, analysed and interpreted data utilised in this study. J.S. played a vital role in the statistical analysis of the data. M.J.N. was a major contributor in writing of the manuscript. All authors contributed to editing and final drafting. All authors read and approved the final manuscript.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Ethical approval and consent to participate
Approval was granted by the Princess Alexandra hospital Human Research and Ethics Committee and Site Governance (HREC/14/QPAH/240 and SSA/14/QPAH/201), with a waiver for patient consent granted. Ethical approval for this study was obtained from Metro South Health Human Research Ethics Committee – HREC/14/QPAH/240.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Informed consent
Waiver of consent approved.
Availability of data and material
The data sets generated and/or analysed during the current study are not publicly available as this was not a condition put forward in initial ethics approval, but are available from the corresponding author on reasonable request.