Abstract
Background:
Several different indices summarize patient comorbidity using health care data. An accurate index can be used to describe the risk profile of patients, and as an adjustment factor in analyses. How well these indices perform in persons with chronic kidney disease (CKD) is not well known.
Objective:
Assess the performance of 5 comorbidity indices at predicting mortality in 3 different patient groups with CKD: incident kidney transplant recipients, maintenance dialysis patients, and individuals with low estimated glomerular filtration rate (eGFR).
Design:
Population-based retrospective cohort study.
Setting:
Ontario, Canada, between 2004 and 2014.
Patients:
Individuals at the time they first received a kidney transplant, received maintenance dialysis, or were confirmed to have an eGFR less than 45 mL/min per 1.73m2.
Measurements:
Five comorbidity indices: Charlson comorbidity index, end-stage renal disease-modified Charlson comorbidity index, Johns Hopkins’ Aggregated Diagnosis Groups score, Elixhauser score, and Wright-Khan index. Our primary outcome was 1-year all-cause mortality.
Methods:
Comorbidity indices were estimated using information in the prior 2 years. Each group was randomly divided 100 times into derivation and validation samples. Model discrimination was assessed using median c-statistics from logistic regression models, and calibration was evaluated graphically.
Results:
We identified 4111 kidney transplant recipients, 23 897 individuals receiving maintenance dialysis, and 181 425 individuals with a low eGFR. Within 1 year, 108 (2.6%), 4179 (17.5%), and 17 898 (9.9%) in each group had died, respectively. In the validation sample, model discrimination was inadequate with median c-statistics less than 0.7 for all 5 comorbidity indices for all 3 groups. Calibration was also poor for all models.
Limitations:
The study used administrative health care data so there is the potential for misclassification. Indices were modeled as continuous scores as opposed to indicators for individual conditions to limit overfitting.
Conclusions:
Existing comorbidity indices do not accurately predict 1-year mortality in patients with CKD. Current indices could be modified with additional risk factors to improve their performance in CKD, or a new index could be developed for this population.
What was known before
Several different indices have been developed that summarize patient comorbidity using health care data. How well these indices perform in persons with chronic kidney disease is not well known.
What this adds
Existing comorbidity indices do not accurately predict 1-year mortality in patients with chronic kidney disease.
Introduction
The burden of kidney disease has increased worldwide, with the population prevalence of chronic kidney disease now exceeding 10% in many countries. 1 Even mild chronic kidney disease associates with a higher risk of mortality, and these individuals often have multimorbidity that influences their survival. 2 Kidney disease is understudied, despite this increased mortality risk and its associated high health care cost and resource utilization.3,4 There is an interest in efficient care and further research for this important segment of the population.
An index or risk score that can accurately prognosticate mortality based on readily available administrative health data would be beneficial for multiple situations. Including a prognostic index in a set of baseline characteristics can help describe the risk profile of a group of persons under study. Similarly, adjustment for confounding is often performed based on a person’s comorbidity. An accurate index could efficiently adjust for confounding due to comorbid illness. Performance reports and benchmarks are becoming increasingly common to improve the quality of care patients receive, with reimbursement for the care of dialysis patients in the United States now directly tied to quality of care measures. 5 When contrasting performance across physicians, regions, or facilities, as in the End-Stage Renal Disease Quality Incentive Program, a prognostic index could be used when there is a need to adjust for case-mix. A number of methods have been developed and used to classify individuals’ level of comorbidity, with many indices accurately predicting mortality in the general population and various patient groups.6-8
The Charlson comorbidity index was originally derived from medical charts at a single hospital to predict in-hospital mortality by levels of comorbidity in a general medicine population.
9
The index has since been adapted for administrative data, where based on 17 conditions a score can range from 0 (no comorbidity) to 33 (high comorbidity).
10
Similarly, Elixhauser identified 30 factors that quantify the level of comorbidity in hospitalized individuals using administrative databases.
11
The Charlson comorbidity index has also been modified for use in end-stage renal disease patients.
12
This modified score uses the same conditions as in the original Charlson study but incorporates new weights. The Johns Hopkins ACG
How well most of these indices perform in the kidney disease population is not well understood. Accurate estimates within this population would provide reassurances regarding adequate comorbidity adjustment, while poor performance of these indices would indicate the need for a better alternative. In this study, we assessed the performance of these comorbidity indices at predicting 1-year all-cause mortality in 3 different patient groups with chronic kidney disease: incident kidney transplant recipients, maintenance dialysis patients, and individuals with low estimated glomerular filtration rate (eGFR).
Materials and Methods
Data Sources
We conducted an observational cohort study using linked, administrative health care databases in Ontario, Canada. In Ontario, residents have universal access to physician and hospital services. The conduct and reporting of this study followed guidelines recommended for studies evaluating prediction models, with the TRIPOD checklist available in the appendix (Table S1). 16 We obtained data from 6 databases, and these databases were linked using unique, encoded identifiers and analyzed at the Institute for Clinical Evaluative Sciences (ICES). The use of data in this project was authorized under section 45 of Ontario’s Personal Health Information Protection Act, which does not require review by a Research Ethics Board. The Canadian Organ Replacement Register has complete data on all transplant and dialysis activity in the province. 17 We used this database to identify incident kidney transplant recipients and individuals initiating maintenance dialysis. We used the Ontario Laboratory Information System (OLIS) database to identify adults with low eGFR, defined using serum creatinine laboratory records. OLIS is a province-wide repository of laboratory results, with 91% coverage of the province’s total annual laboratory testing as of 2016. 18 We obtained demographic and vital status information from the Registered Persons Database. The Canadian Institute for Health Information (CIHI) Discharge Abstract Database (DAD) has information on all hospital admissions, and their associated diagnoses and procedures. Diagnoses were coded using the 10th revision of the International Classification of Diseases. Similarly, the CIHI National Ambulatory Care Reporting System (NACRS) database has information on all emergency department visits in the province. We used the Ontario Health Insurance Plan database to ascertain physician billing claims and associated diagnoses.
Study Population
We accrued 3 groups of patients in Ontario, Canada, between the years 2004 and 2014, at the time they first received a kidney transplant, received maintenance dialysis, or were confirmed to have an eGFR less than 45 mL/min per 1.73 m2. The date of transplant or dialysis initiation was considered the index date (date of cohort entry). Due to the availability of serum creatinine data, individuals with low eGFR were accrued from January 1, 2007, onward. We restricted the low eGFR group to those with an eGFR <45 mL/min/1.73m², indicative of moderate to severe chronic kidney disease. 19 To ensure kidney function was determined based on stable, outpatient measurements, we required individuals to have at least 2 outpatient eGFR values, separated by 90 to 365 days, within 5 units or 5% of each other, with both tests having an eGFR <45 mL/min/1.73m². The date of the confirmatory eGFR value was considered the index date. To facilitate analyses that would be representative of these entire patient populations, we allowed individuals to be potentially included in multiple kidney disease groups. Restricting individuals to their first eligible group or excluding those who appeared in multiple groups would lead to biased sampling. We excluded all non-Ontario residents and those aged 18 years or younger on the index date. Across all 3 groups, we excluded any individuals with evidence of a prior kidney transplant to ensure that only patients with incident kidney disease were captured. Similarly, we excluded anyone in the dialysis and low eGFR groups with evidence of prior maintenance dialysis. In the kidney transplant recipients, we excluded anyone with evidence of a multi-organ transplant, to limit the analyses to kidney-only transplants. Kidney transplant recipients incorporated both preemptive transplants and prevalent dialysis patients receiving their first renal graft. We have previously studied these patient populations and in validation studies our coding algorithms accurately identified these groups.17,20-23
We identified 5 comorbidity indices that could be accurately estimated using our administrative data. Other potential indices were identified, but could not be appropriately ascertained using our data sources.24,25 These indices included measures developed in the general population, as well as those developed specifically in chronic kidney disease populations. All comorbidity indices were calculated using information available in the 2 years before the index date. The Charlson comorbidity index, the end-stage renal disease-modified Charlson comorbidity index, and Elixhauser score were obtained using hospital discharge data. The Wright-Khan index used data from hospital discharges and physician service claims. The Johns Hopkins’ ADGs were determined using data from hospital discharges, emergency department records, and physician service claims.
The primary outcome was 1-year all-cause mortality. The rate of emigration from the province was very low (0.5% per year) and represented the only reason for loss to follow-up. 26
Statistical Analysis
To evaluate the predictive performance of the various comorbidity indices, we randomly divided the cohort into 2 roughly equal-sized samples: a derivation sample and a validation sample. Parameter estimates (i.e., regression coefficients) associated with the comorbidity indices were computed in the derivation sample, and applied to the individuals in the validation sample. Such an approach is necessary as the previous studies in which the comorbidity indices were derived did not present the regression coefficients, and this maintains the relative weighting of the comorbidities as in the original derivation studies. The split-sample approach enables evaluation of model performance of the indices in a sample independent of the data from which parameter estimates are obtained. The Wright-Khan index was fit as a categorical variable (low, medium, high) with “low” used as the reference category. All other indices were modeled as linear continuous variables, with the Elixhauser index and ADGs converted to continuous variables based on weights that have been developed previously.27,28 If an individual had no hospitalizations during the lookback window, their Charlson and ESRD-modified Charlson were coded as “0”. A model’s ability to discriminate 1-year mortality was assessed using the c-statistic. C-statistics were determined using logistic regression, with 1-year mortality treated as a binary outcome, and the only covariate in the model was the comorbidity index. The c-statistic can range from 0.5 to 1.0, representing chance and perfect discrimination, respectively. A c-statistic exceeding 0.7 is generally regarded as adequate, with a value exceeding 0.8 indicating excellent discrimination. 29 Model calibration was assessed graphically using calibration plots and a smoothing loess algorithm. 30 A well-calibrated model would have a calibration plot with an approximately straight line at a 45-degree angle, indicating the predicted probabilities are similar in magnitude to the observed probabilities. Split-sample validation has previously been shown to have a high-degree of variability. 31 As such, we repeated the random splitting of the cohort 100 times into derivation and validation samples as done in prior validation studies of comorbidity indices. 6 This allowed us to obtain a range of c-statistics across the 100 repetitions and a median (interquartile range [IQR]) c-statistic for each comorbidity index. All analyses were conducted using SAS 9.4 (SAS Institute, Cary, NC) and R 3.1.2. 32
Results
After exclusions, we identified 209 433 individuals with chronic kidney disease (Figure 1). Specifically, 4111 kidney transplant recipients, 23 897 initiated on maintenance dialysis, and 181 425 with low eGFR. Of the 209 433 individuals, 46% were male, with a median (IQR) age of 78 (69-85) years (Table 1). We observed 22 185 (10.6%) deaths in the 1 year follow-up. This varied by group, with 108 (2.6%) of kidney transplant recipients, 4179 (17.5%) of maintenance dialysis users, and 17 898 (9.9%) of individuals with low eGFR dying during follow-up.
Patient selection for the 3 groups.
Characteristics at the Time of Cohort Entry.
Note. eGFR = estimated glomerular filtration rate; IQR = interquartile range; ESRD = end-stage renal disease; ADG = Aggregated Diagnosis Group.
Defined as residence within a municipality with population <10 000.
Across all 3 chronic kidney disease groups, the comorbidity indices in the validation sample displayed inadequate discrimination for the outcome of 1-year all-cause mortality (Table 2). In kidney transplant recipients, the median c-statistic ranged from 0.55 for the Elixhauser score, to 0.63 for the Wright-Khan index. Similarly, among incident users of dialysis, the median c-statistic ranged from 0.61 for the Charlson comorbidity index, to 0.64 for the Johns Hopkins’ ADG score. The indices also demonstrated generally poor discrimination in the individuals with low eGFR, with the median c-statistics ranging from 0.63 for the Elixhauser score, to 0.66 for the Johns Hopkins’ ADG score. These results did not substantially differ from the c-statistics observed in the derivation samples (Table S2).
Median (Interquartile Range) C-Statistics in the Validation Sample.
Note. eGFR = estimated glomerular filtration rate; ESRD = end-stage renal disease; ADG = Aggregated Diagnosis Groups.
The c-statistic can range from 0.5 to 1.0, representing chance and perfect discrimination, respectively. A c-statistic exceeding 0.7 is generally regarded as adequate, with a value exceeding 0.8 indicating excellent discrimination.
Overall, calibration with the various comorbidity indices was also poor in the validation samples, across the 3 kidney disease groups (Figure 2). In kidney transplant recipients, the indices tended to underestimate the probability of one-year mortality. For individuals receiving dialysis or with low eGFR, the indices appeared to overestimate the probability of 1-year mortality in higher risk individuals.
Calibration plots comparing predicted and observed probabilities of one-year mortality.
Discussion
In this population-based study of individuals with kidney disease in Ontario, Canada, existing comorbidity indices did not adequately predict 1-year mortality. In kidney transplant recipients, individuals receiving maintenance dialysis, and individuals with low eGFR, the comorbidity indices all had c-statistics less than 0.7, indicating inadequate discrimination. Model calibration was also poor for the studied indices. Our findings suggest existing comorbidity indices are inadequate and inaccurate in predicting 1-year mortality. These results differ from prior studies of comorbidity indices in different subsets of individuals in Ontario, where calibration and discrimination have been very good.6-8,28 Researchers and clinicians should be cautious when using existing comorbidity indices in this chronic kidney disease population, and cognizant of their limitations. As the indices performed comparably across all 3 groups, an investigator would likely be reasonable in choosing whichever comorbidity index is most readily available with their data.
Our c-statistics are marginally lower than those found in prior studies.12,33-35 These differences may be at least partially explained through differences in setting, sample size, study design, and era of study. Specifically, these 4 prior studies each consisted of less than 2000 dialysis patients with follow-up ending in 2004, while we were able to accrue more than 23 000 individuals on dialysis up to March 31, 2014. Smaller studies with fewer events may produce unstable estimates of the c-statistic, and performance of these indices may have worsened over the last decade due to changes in patient characteristics and coding practices.
Prior studies evaluating the ability of the Charlson comorbidity index to predict mortality in individuals receiving dialysis have seen c-statistics range from 0.67 to 0.76.12,35 This range is higher than the 0.61 median c-statistic observed in our validation sample. The study that observed a c-statistic of 0.76 accrued individuals based on an ICD-10 code for chronic renal failure that potentially identified a different cohort of individuals than those accrued in our study. This ICD-10 diagnosis code required a hospitalization, which may have detected a select group of individuals compared to our analysis where we captured all individuals at the time of dialysis initiation. In a study of 237 individuals initiating dialysis between 1999 and 2000 in Alberta, Canada, the Charlson index had a c-statistic of 0.72, and their ESRD-modified Charlson had a c-statistic of 0.73. This is slightly better than the c-statistics of 0.61 and 0.63 we observed, respectively. The mean scores were higher in the Alberta study (4.70 and 4.84) compared to the scores observed in our study (Charlson mean 2.13 (SD 2.29), ESRD-modified Charlson mean 1.41 (SD 2.01)), suggesting their study may have been able to assess comorbidity more accurately than with our administrative databases. In the Alberta study, patient comorbidity was abstracted and manually reviewed at the time of dialysis. This method likely captures diagnoses more comprehensively than database coding algorithms, and could explain the improved performance observed in Alberta.
Two prior studies evaluated the Wright-Khan index in dialysis patients, and observed c-statistics comparable to the median of 0.63 seen in our validation sample. The study using United States data from 1997 to 2000 found a c-statistic of 0.68, in a model that incorporated the Wright-Khan index and age. 33 Age is likely to be a significant predictor of mortality, which may describe the 0.05 increase in the c-statistic observed in this American study. In a Dutch prospective cohort study of dialysis patients, they observed a c-statistic of 0.62, comparable to our median c-statistic of 0.63. 34 These similar findings occur despite a higher burden of comorbidity observed in our cohort (20% categorized as “low,” compared to 39% categorized as “low” in the Dutch study). These studies, in addition to our results, suggest that the Wright-Khan index does not adequately predict 1-year mortality in the population of individuals with kidney disease.
The strengths of our study include a large population-based cohort of incident kidney transplant recipients, patients newly starting maintenance dialysis, and individuals with a laboratory-confirmed low eGFR in a universal health care setting. We were able to evaluate the predictive performance of comorbidity indices in a much larger sample than any other prior study. Our study was comprehensive, incorporating 5 different comorbidity indices, across all spectrums of kidney disease. Study of these indices in individuals with kidney disease, particularly the Johns Hopkins’ ADG score, have been lacking to date and our study provides stable, methodologically sound estimates of their performance.
Our study is not without limitations. Performance of these comorbidity indices may change dependent on the method of data collection and comorbidity ascertainment. Prior studies that compared these indices based on more detailed data derived from medical charts observed slightly higher c-statistics. As with any study that uses existing health care databases, there is the potential for misclassification. Wherever possible, we classified individuals using validated coding algorithms. To avoid overfitting, particularly in the kidney transplant recipients, we modeled the Johns Hopkins’ ADG and Elixhauser indices continuously. Prior studies have shown that modifying these indices into continuous scores provided similar predictive ability for mortality in the general population.27,28 However, categorizing these indices into continuous scores as opposed to modeling them using individual indicators for each condition as in the original studies may have contributed toward the poor discrimination observed in our study.
Individuals with kidney disease are a complex population, associated with a high burden of comorbid illness and mortality.2,36 The cause of this heightened risk of mortality is multifactorial and potentially difficult to elucidate through diagnosis codes alone. Our 1-year mortality rates ranged from 3% in the transplant recipients, to more than 17% in those initiating dialysis. With a 1-year mortality rate in the general adult population of 0.8%, our observed rates were much higher, but mortality was also highly variable across the kidney disease continuum. 8 It is unsurprising that the discriminatory ability of these various indices were inadequate, as these indices were either developed in a general ambulatory population, or despite being developed in a population with kidney disease, did not comprehensively include risk factors for mortality specific to this population. These indices were also developed to predict shorter term mortality or resource utilization, and this may partially explain the poor discrimination when predicting 1-year mortality. In kidney transplant recipients, transplant-specific factors known to be associated with posttransplant mortality, such as donor characteristics, time on dialysis, and pretransplant panel reactive antibody score are not incorporated in the comorbidity indices, but would likely provide substantial gains when trying to predict transplant recipients at greatest risk for mortality after transplantation. 37 Similarly for individuals receiving dialysis, potential predictors such as modality, access type, and cause of kidney failure are not incorporated into these indices, but are likely important factors when predicting mortality. 38 Previous studies have shown that regularly monitored laboratory tests can accurately predict progression of kidney disease and mortality in individuals with reduced kidney function.39,40 Incorporating these routinely collected tests into comorbidity scores has the potential to provide substantial improvement to the results observed in our current study. The continued adoption of electronic medical records globally should improve accessibility to these laboratory measures, in turn also increasing the feasibility of using these laboratory values for research and clinical decision making. Future work should focus on the development of a comorbidity index that incorporates risk factors specific to the chronic kidney disease population and can accurately predict mortality.
Our study found existing comorbidity indices do not sufficiently predict one-year mortality in individuals with kidney disease. Predictive performance may be improved by incorporating risk factors specific to this population with chronic kidney disease. Modification of an existing comorbidity index or the creation of a new index, that can achieve good discrimination and calibration when predicting mortality, would be valuable for researchers, clinicians, and policy makers.
Supplemental Material
Appendix_2018_06_05 – Supplemental material for Comparing Five Comorbidity Indices to Predict Mortality in Chronic Kidney Disease: A Retrospective Cohort Study
Supplemental material, Appendix_2018_06_05 for Comparing Five Comorbidity Indices to Predict Mortality in Chronic Kidney Disease: A Retrospective Cohort Study by Eric McArthur, Sarah E. Bota, Manish M. Sood, Gihad E. Nesrallah, S Joseph Kim, Amit X. Garg and Stephanie N. Dixon in Canadian Journal of Kidney Health and Disease
Footnotes
Acknowledgements
This study was supported by the Institute for Clinical Evaluative Sciences (ICES) Western site in London, Ontario, Canada. ICES is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). Core funding for ICES Western is provided by the Academic Medical Organization of Southwestern Ontario (AMOSO), the Schulich School of Medicine and Dentistry (SSMD), Western University, and the Lawson Health Research Institute (LHRI). The research was conducted by members of the ICES Kidney, Dialysis and Transplantation team, at the ICES Western facility, who are supported by a grant from the Canadian Institutes of Health Research (CIHR). The opinions, results and conclusions are those of the authors and are independent from the funding sources. No endorsement by ICES, AMOSO, SSMD, LHRI, CIHR, ODPRN, or the MOHLTC is intended or should be inferred. Parts of this material are based on data and/or information compiled and provided by CIHI. However, the analyses, conclusions, opinions, and statements expressed in the material are those of the author(s), and not necessarily those of CIHI. Dr Amit Garg was supported by the Dr Adam Linton Chair in Kidney Health Analytics, and a Clinician Investigator Award from the Canadian Institutes of Health Research.
Ethics Approval and Consent to Participate
The use of data in this project was authorized under section 45 of Ontario’s Personal Health Information Protection Act, which does not require review by a Research Ethics Board.
Consent for Publication
All authors consent to publication.
Availability of Data and Materials
The dataset from this study is held securely in coded form at the Institute for Clinical Evaluative Sciences (ICES). While data sharing agreements prohibit ICES from making the dataset publicly available, access can be granted to those who meet pre-specified criteria for confidential access, available at 
.
Author Contributions
EM, SND: Research idea and study design.
EM, SEB: data acquisition.
EM, SEB, MMS, GEN, SJK, AXG, SND: data analysis/interpretation.
EM: statistical analysis.
SND: supervision or mentorship.
Each author contributed important, intellectual content during manuscript drafting or revision and accepts accountability for the overall work.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
