Abstract
Objective
Sepsis is the leading cause of in-hospital mortality in the United States (US). Quality improvement initiatives for improving sepsis care depend on accurate estimates of sepsis mortality. While hospital 30-day risk-standardized mortality rates have been published for patients hospitalized with acute myocardial infarction, heart failure, and pneumonia, risk-standardized mortality rates for sepsis have not been well characterized. We aimed to construct a sepsis risk-standardized mortality rate map for the United States, to illustrate disparities in sepsis care across the country.
Methods
This cross-sectional study included adults from the US Nationwide Inpatient Sample who were hospitalized with sepsis between 1 January 2010 and 30 December 2011. Hospital-level risk-standardized mortality rates were calculated using hierarchical logistic modelling, and were risk-adjusted with predicted mortality derived from (1) the Sepsis Risk Prediction Score, a logistic regression model, and (2) gradient-boosted decision trees, a supervised machine learning (ML) algorithm.
Results
Among 1,739,033 adults hospitalized with sepsis, 50% were female, and the median age was 71 years (interquartile range: 58–81). The national median risk-standardized mortality rate for sepsis was 18.4% (interquartile range: 17.0, 21.0) by the boosted tree model, which had better discrimination than the Sepsis Risk Prediction Score model (C-statistic 0.87 and 0.78, respectively). The highest risk-standardized mortality rates were found in Wyoming, North Dakota, and Mississippi, while the lowest were found in Arizona, Colorado, and Michigan.
Conclusions
Wide variation exists in sepsis risk-standardized mortality rates across states, representing opportunities for improvement in sepsis care. This represents the first map of state-level variation of risk-standardized mortality rates in sepsis.
Background
Over the past two decades, the Surviving Sepsis Campaign 1 and a multitude of quality improvement initiatives have aimed to standardize sepsis management and improve bundle compliance.2,3 Central to the outcome evaluation of these initiatives are accurate estimates of sepsis mortality, as mortality reflects quality of care, and is the ultimate meaningful outcome to patients. Risk-standardized mortality rates (RSMRs) can reduce the confounding effect of case mix (e.g. age, comorbidity burden) and assess regional variation in sepsis mortality more precisely.
State-based differences in RSMR for acute myocardial infarction (AMI), heart failure (HF), and pneumonia (PNA) have been visualized based on public reporting from the Centers for Medicare and Medicaid Services (CMS),4–7 but have not been performed for sepsis (eTable 1). Using similar methods, several studies demonstrated the inequality in sepsis care by reporting the mortality variation between hospitals. Hatfield et al. reported a difference of 11.5 percentage points in median sepsis mortality rate between the highest and lowest decile of hospitals. 8 Wang et al. calculated the observed-to-expected mortality ratio in severe sepsis using administrative data from the University Health System Consortium. 9 Darby et al. revealed that a hospital-specific RSMR for sepsis ranged from 12.2% to 24.5% using hospital discharge data only from the state of Pennsylvania. 10 Walkey et al. also found a wide difference in RSMR in septic shock by both implicit diagnosis and explicit ICD codes. 11 However, these studies used mortality rates that were not risk-adjusted, employed traditional statistical regression techniques, and were restricted to populations at specific medical centers or age groups.
Thus, our proposed method differs from previous sepsis mortality measurement in two fundamental aspects. First, our RSMR estimates were based on a nationwide database with the integration of a supervised machine learning (ML)-based model allowing for relationships not based on conventional regression. Second, unlike previous studies which performed comparisons at a hospital level, we constructed a national sepsis mortality map of the United States to illustrate the geographic variation in sepsis quality of care.
Methods
Data sources
We extracted data from the 2010 and 2011 years of the US Nationwide Inpatient Sample (NIS), the largest all-payer, public claims database of inpatient encounters. The NIS contains a 20% stratified sample of all short-term, non-federal, non-rehabilitation hospitals, representing 5–8 million discharges per year. We applied the trend weights developed by the Healthcare Cost and Utilization Project to weight patient-level discharge data, to derive estimates for >35 million hospitalizations nationally. We could not use more recent data because the NIS did not provide state-specific identifiers after 2011.
Ethics approval
This study was approved by the Institutional Review Board at National Taiwan University Hospital, which waived the requirement for informed consent from patients because of the anonymous nature of this public database.
Case selection and definitions
In accordance with the Sepsis-3 definition, sepsis was defined as life-threatening organ dysfunction caused by a dysregulated host response to infection. 12 Sepsis hospitalizations were identified by Martin and Moss’ validated International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis codes or procedure codes from the electronic health record. 13 We selected patients with any diagnosis of sepsis (995.92 severe sepsis, 785.52 septic shock) or systemic infection (038.* septicemia, 020.0 sepsis, 790.7 bacteremia, 117.9 disseminated fungal infection, 112.5 disseminated candidiasis, or 112.81 disseminated fungal endocarditis) and a diagnosis of acute organ dysfunction in any of seven organ systems: cardiovascular, respiratory, central nervous, hematologic, hepatic, renal and metabolic systems. We excluded patients not meeting Martin's sepsis criteria, missing mortality information, below 18 years of age, or who were transferred to a short-term hospital against medical advice. We collected demographic information, pre-existing comorbidities, and in-hospital mortality from claims records of the index hospitalization. Comorbidities, based on Gagne's validated ICD-9-CM codes, 14 were not included in the risk adjustment unless they were also present in the 12 months before admission. Since the NIS lacks a code for ICU admission, mechanical ventilation >1 day was used as a proxy for ICU stay. 15
Outcome
Thirty-day in-hospital crude mortality rate was calculated based on length of stay and in-hospital mortality data for each patient. The primary outcome was 30-day in-hospital risk-standardized all-cause mortality rate, as described below.
Descriptive statistics
Continuous variables were summarized as mean with standard deviation or median with interquartile range (IQR). Categorical variables were summarized as number and proportion. All analyses were performed in R (version 3.5, Vienna, Austria).
Risk-standardized mortality rates
Hierarchical logistic regression models were used to estimate hospital-level 30-day all-cause RSMRs for sepsis. This approach takes into account the two-level (patient and hospital) hierarchical structure of the data to account for patient clustering within hospitals. Each model includes age, sex, hospital-specific random effects (representing baseline mortality risk at that hospital), predicted mortality and an intercept. Higher RSMRs represent worse outcomes. RSMRs in our study are expressed as percentages except where otherwise noted as fractions of one.
Feature analysis and selection
For the reference model, we fit a logistic regression model using the widely-used Sepsis Risk Prediction Score model (SRS), which assigns patients a different number of points for each variable listed in Table 1. 15 In the feature analysis step of the construction of the ML model, we used the same variables in Table 1 and performed multiple logistic regression to obtain adjusted odds ratios (aOR) in our sample. For each variable, all remaining covariates were mutually adjusted.
Baseline characteristics of inpatients with sepsis in 2010–2011, with adjusted odd ratios for all-cause mortality by demographic and clinical characteristics.
Early mechanical ventilation = 0–2 days from admission.
Late mechanical ventilation > 2 days from admission.
Selection, training, and validation of the ML model
Gradient boosted decision tree (GBDT; hereafter referred to as the “boosted tree”) is an ensemble supervised learning method that builds an additive model of decision trees in a “greedy” stage-wise process estimated by gradient descent. 16 “Boosting” refers to the combination of weak prediction trees to produce a stronger prediction. We constructed a GBDT model with the xgboost package in R to predict state-level RSMR using the variables identified above to associate with mortality. 17 We used a maximum depth of 6 and a gamma of 0. We used 580 variables as estimators during training, and 100 variables during testing. The 2010 data was used for training with 5-fold primary validation. The 2011 data was used for secondary validation (eFigure 4). To determine the degree of each variable's influence in the boosted tree model, we aggregated the reduction in error resulting from use of that variable as the splitting variable over all trees.
Evaluation of the two models
The goodness of fit of the two models was examined using Harrell's C-statistic, the R-square, and the area under the receiver operator characteristic curve (AUROC).
Construction of national RSMR maps
Hospital-level RSMRs were analyzed for 45 states. Data for five states (Idaho, Alabama, Hawaii, New Hampshire, and Delaware) was unavailable or incomplete in 2010–2011. States were classified by their Bureau of Economic Analysis regions (eTable 4) as the US Census administrative divisions were not adequately descriptive in name and did not adequately describe the “sepsis belt.” Maps were plotted using our in-house dashboard. 18 For each state, inter-model RSMR differences were calculated by subtracting RSMRs of the SRS model from RSMRs of the boosted tree model (eFigure 3).
Hospital-level comparisons
The range of crude mortality rate and RSMR were visualized with box plots. Hospitals were divided into deciles by RSMR to examine the range in RSMR. Hospitals were divided into quintiles by annual number of sepsis admissions to examine the range in hospital volume, visualized by histograms. Differences in RMSR and sepsis admission counts across quintiles of sepsis admissions were tested using the Kruskal–Wallis test.
Results
Study population and encounter characteristics
Among 1,739,033 inpatients with sepsis identified in the NIS (eFigure 1), 15% were African-American, 8% were Hispanic, and 77% were White or Other; 50% were female (Table 1). Mortality increased with age, with 80–89-year-olds carrying an aOR for mortality of 4.17 (confidence interval (CI): 3.95–4.40) compared to <40-year-olds (Table 1). Mortality was slightly higher in women than men (aOR 1.07; CI: 1.05–1.09), and in African-American patients than White patients (aOR 1.05; CI: 1.02–1.08). Mortality was higher in patients requiring early mechanical ventilation (aOR 4.63; CI: 4.53–4.75) and late mechanical ventilation (aOR 5.72; CI: 5.43–6.02) than patients not requiring ventilation. Mortality was also higher in patients who had shock (aOR 2.06; CI: 2.02–2.10) or who required hemodialysis (aOR 1.43; CI: 1.39–1.48).
Two models of RSMR
Using the variables above, we constructed a reference model for predicted mortality rate using SRS logistic regression, and an alternative model for predicted mortality rate using boosted trees, for estimating 30-day RSMR of sepsis. In this boosted tree model, the variable with the highest influence on prediction was the need for intubation and ventilation, with a weight of 0.24 (eFigure 4), followed by age, respiratory failure, cardiac arrest and ventricular fibrillation, and shock (eTable 3).
After risk adjustment, the range of the hospital-level mortality narrowed from 0–100% to 4.7–58.2% (eFigure 6). The median RSMR for the top and bottom decile of hospitals by RSMR was 13.5% and 25.8%, respectively (eFigure 7). In the SRS model, median RSMR was 18.5% (IQR: 17.1–20.6%, representing a 3.4% difference between the 25th and 75th percentiles; Table 2). In the boosted tree model, median RSMR was 18.4% (IQR 17.0–21.0%, representing a 4.0% difference). RMSRs higher than 18.5% and 18.4% in the respective models indicate that the mortality from sepsis is higher than expected after risk adjustment.
Assessment of model fit for the sepsis risk prediction score (SRS) model and the boosted tree model in predicting median 30-day risk-standardized mortality rate (RSMR) of sepsis among inpatients hospitalized with sepsis in 2011.
Abbreviations: AUPRC: area under the precision-recall curve; PPV: positive predictive value; %ile: percentile; SD: standard deviation.
Under a predefined cutoff of 0.5.
Goodness of fit of RSMR models
The boosted tree model demonstrated better calibration than the SRS model (eFigure 8). The boosted tree model also had better discrimination (AUROC: 0.87 vs 0.78; AUPRC: 0.62 vs 0.47; Table 2 and eFigure 2) and higher positive predictive value (0.69 vs 0.60) than the SRS model. Having established its improved calibration and discrimination compared to the reference SRS model, RSMR estimates in the Discussion will refer to those done by the boosted tree method, unless otherwise specified.
State-level median 30-day risk-standardized mortality rate (RSMR) of sepsis predicted by the Sepsis Risk Prediction Score (SRS) model (left) and the boosted tree model (right) among inpatients hospitalized with sepsis in 2011, ranked in order of decreasing RSMR.
Geographic variation in risk-standardized all-cause mortality by state
Next, we calculated median 30-day RSMR at the state level using both the SRS model and the boosted tree model (Table 3). State-level RSMRs ranged from 14% to 23% in both models. The SRS model found clusters of high RSMR states in the Southeast (West Virginia, Virginia, Arkansas, Mississippi), New England (Massachusetts, Connecticut), and individual states such as Wyoming (Mountain), Ohio (Great Lakes), North Dakota (Plains), and Alaska (Far West). (Figure 1(a)). The boosted tree found clusters of high RSMR states in the northern Mountain states of Wyoming and Montana, the Southeast (Mississippi, West Virginia, Arkansas, Kentucky), and New England (Rhode Island, Maine, and Massachusetts), and individual states such as North Dakota (Plains) and Alaska (Far West) (Figure 1(b)). West Virginia, Massachusetts, Wyoming, Ohio, North Dakota, Connecticut, Arkansas, New York, Missouri, Alaska, Georgia, Rhode Island, Montana, Utah, Maryland, Nevada, Maine, Tennessee, Minnesota, Oklahoma, Kentucky, Indiana, New Jersey, and Vermont were above the median national RSMR in both models. The lowest RSMR states were identical in both models: Arizona (Southwest), Colorado (Mountain), and Michigan (Great Lakes). Differences between crude mortality and RSMR for each state are shown in Table 4.
State-level median 30-day risk-standardized mortality rate (RSMR) of sepsis predicted by the sepsis risk prediction score (SRS) model (top) and boosted tree model (bottom) among inpatients hospitalized with sepsis in 2011, color coded by sextile. Note that data for Idaho, Alabama, Hawaii, New Hampshire, and Delaware was unavailable or incomplete in 2010–2011 and are thus colored in white.
Difference between crude and risk-standardized 30-day all-cause mortality for each state in 2011 using the boosted tree model.
Hospital volume and risk-standardized all-cause mortality
To determine if RSMR was affected by hospital volume, we divided hospitals into quintiles by their number of sepsis admissions per year. Hospitals in the lowest and highest quintiles had a median of 3 (IQR: 1–4) and 547 (IQR: 547) sepsis admissions per year, respectively (eFigure 7). The median 30-day RSMR differed significantly across the 5 quintiles (p < 0.001), but this association was not linear (eTable 2, eFigure 5).
Discussion
In this study of 1,739,033 inpatients with sepsis drawn from a nationally-representative database, we constructed the first state-level estimates of risk-standardized all-cause mortality rates for sepsis patients in the United States. We employed a traditional logistic SRS model as the reference and constructed a supervised ML model to increase the accuracy of risk prediction, the latter of which demonstrated higher model discrimination and calibration. We further built an online dashboard to illustrate time-dependent RSMR changes by state. 18 We found significant inter-state variation in RSMR, suggesting that quality of sepsis care is not uniform across the country. These risk-standardized findings suggest that there is opportunity to improve sepsis-related survival in many states, and can inform policymaking.
Because sepsis outcomes are highly dependent on patient characteristics and comorbidities, it is important to develop accurate estimates of mortality risk with which to standardize observed mortality, in order to differentiate between whether differences in sepsis-related mortality are due to differences in patient distributions or differences in quality of care. Investment to improve sepsis care should be prioritized in states such as Wyoming, North Dakota, Mississippi, Rhode Island, Maine, Massachusetts, and Montana, which are in the top sextile of RSMR. At the extremes, Wyoming had an RMSR of 23.2% while Arizona had an RSMR of 14.3%, which were 4.8% above and 4.1% below the national median RSMR, respectively. Arizona, Colorado, Michigan, Illinois, Wisconsin, Washington, and South Carolina, in the bottom sextile of RSMR, should be studied to identify success factors and processes that can be replicated at hospitals in other states.
Our work geographically refines RSMR estimates by Hatfield, 8 who coded facilities at the four-region level (e.g. Midwest, Northeast, Southwest, and West), and consequently does not identify individual high-performing and low-performing states. Wang's 19 state-level analysis of mortality was age-standardized but not risk-standardized to comorbidities or in-hospital factors, and identified a cluster of states in the Southeast and Mid-Atlantic (Arkansas, Louisiana, Mississippi, Alabama, Georgia, Tennessee, North Carolina, Virginia, Maryland, Delaware, and New Jersey) with the highest age-standardized sepsis mortality (ASMR). These states were all found to have RSMR above the national median in our analysis (except Alabama for which we had no data). Moore's 20 county-level analysis of ASMR found three hotspots with the highest ASMR: Central Appalachia (Kentucky, Tennessee, Virginia), Middle Georgia, and Mississippi Valley (Mississippi, Arkansas, Louisiana). These hotspots, termed the “sepsis mortality belt,” 21 were also identified in our analysis as areas of high RSMR. We additionally identified individual states in the Northwest, including Montana, Wyoming, and North Dakota, with high RSMR that were not identified in Wang's or Moore's ASMR analyses.
The variation of care quality between states and hospitals is an important issue for policymakers to address for reducing health disparities. The advent of standardized measurement, required public reporting, Joint Commission accreditation requirements, Medicare payment incentives, and the threat of Medicare payment penalties, have dramatically improved mortality in AMI, HF, and PNA. 22 Specific hospital-based initiatives have been shown to be associated with lower AMI RSMR 23 There is increasing support for reporting HF readmission rates in the context of RSMR. 24 There is reason to be optimistic that focusing RSMR as the outcome indicator of sepsis care quality will improve patient health. In New York, the implementation of mandatory hospital reporting of adherence to sepsis protocol initiation and completion of 3 hour and 6 hour sepsis bundle requirements led to increased compliance with these metrics, and reduced RSMR from 28.8% to 24.4%. 25 More rapid completion of the 3 hour sepsis bundle was associated with reduced adjusted mortality. 26 Besides improving the accessibility and the quality of health care, policymakers should also pay attention to other factors that will affect the regional disparity of sepsis outcome, such as the community poverty rate. 27 Moore et al. showed that counties in Sepsis Cluster or Sepsis Belt regions had larger numbers of physicians per 100,000 persons, suggesting the existence of financial, social, cultural barriers for those people to access health care. 27
Although speculation about reasons for poor sepsis outcomes in specific states is beyond the scope of this study, higher poverty rate, larger uninsured population, lower education, and higher unemployment have been suggested as factors explaining high age-adjusted sepsis mortality rate in poor-performing counties. 20 Uninsured critically-ill patients are less likely to receive critical care procedures and post-acute care compared to insured critically ill patients, and thus have higher mortality rates. 28 Genetics and environmental exposures, not included in our risk standardization, may alter sepsis outcomes.29,30 Although protocoled resuscitation strategies have improved mortality in sepsis,3,26,31–33 differential adherence to these protocols may account for variation in RSMR. Our finding that RMSR differed across quintiles of hospital volume should not necessarily be interpreted as higher volumes corresponding to improved RSMR, because the relationship was not monotonously decreasing. Some groups have shown that hospitals with higher case volumes had improved adjusted sepsis survival, 34 but regional inter-hospital transfer of sepsis patients from low to high sepsis case volume hospitals has not improved outcomes, possibly due to delays in treatment and adverse events from care transitions. 35 Meanwhile, interventions focusing on improving coordination of existing hospital responses to suspected severe sepsis cases have shown promise. 36 Implementation of evidence-based sepsis processes of care was more likely at hospitals with higher sepsis case volumes, but did not strongly mediate case volume-mortality associations. 37
Our results should be interpreted in light of several limitations. First, the NIS does not allow data linkage between hospitals, rendering information about subsequent mortality unavailable for transferred patients. The NIS did not provide state-specific data after 2011, limiting our ability to provide current estimates. Nevertheless, our study features the largest sample size to date in estimating sepsis RSMR. Second, because the data was administrative, variation in coding practices among hospitals may limit accuracy. Moreover, we did not employ advanced deep learning method in both the recognition of sepsis and the prediction of mortality as described in previous studies.38–42 However, standardized clinical criteria has been shown to have good reliability for identifying sepsis in the EMR43,44 and trade-off between data complexity and model interpretation also exists in deep learning algorithm.45,46 Third, in-hospital mortality could be biased by hospital discharge practice and length of hospital stay, 47 and may not necessarily reflect the quality of care.48,49 In what has been termed the “early post-discharge mortality” phenomenon, 50% of older patients who died from PNA within 30 days of admission did not die in hospital but after discharge. 50 This can be addressed by examining a different mortality time frame or examining the risk adjusted readmission rate, 4 which are areas of future investigation in our group. Fourth, although we utilized a robust risk-adjustment model, we cannot assure that differences in RSMRs between states are purely owing to quality differences, as we cannot identify, for example, patients who are admitted expressly for hospice care. Our estimates are strengthened by the use of complementary models to estimate predicted sepsis mortality. Compared to generalized logistic regression, boosted trees have more freedom because they do not assume any a priori hypothesis about the feature-prediction relationship; instead, they deduce that relationship from the data. To make our final result simpler and interpretable for policy makers, we have made our tool publicly available at https://sepsismap.shinyapps.io/index2/. 18
Conclusion
In conclusion, high RSMRs persist for sepsis. The highest RSMRs were found in Wyoming, North Dakota, and Mississippi, while the lowest RSMRs were found in Arizona, Colorado, and Michigan. To our knowledge, this is the first description of state-level variation in RSMR of sepsis patients. Most importantly, we constructed a national map of sepsis RSMR, using a dashboard which we are making freely available to any investigator or clinician. We hope that identification of these high-RMSR hotspots will facilitate investment and innovation in sepsis mortality-reduction strategies, improving sepsis mortality and healthcare expenditures.
Supplemental Material
sj-docx-1-dhj-10.1177_20552076211072400 - Supplemental material for Risk-standardized sepsis mortality map of the United States
Supplemental material, sj-docx-1-dhj-10.1177_20552076211072400 for Risk-standardized sepsis mortality map of the United States by Jiun-Ruey Hu, Chia-Hung Yo, Hsin-Ying Lee, Chin-Hua Su, Ming-Yang Su, Amy Huaishiuan Huang, Ye Liu, Wan-Ting Hsu, Matthew Lee, Yee-Chun Chen and Chien-Chang Lee in Digital Health
Footnotes
Acknowledgments
We thank the staff of the Core Labs, the Department of Medical Research, and National Taiwan University Hospital for technical support, and Medical Wizdom, LLC, USA for technical assistance in statistical analysis. We also thank Shih-Chuan Chou, MD, MPH at the Department of Emergency Medicine, Brigham and Women's Hospital, Boston, MA, USA for reviewing the article and his insightful comments.
Contributorship
C-CL has full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis, concept and design, critical revision of the manuscript for important intellectual content, obtaining funding and supervision. J-RH and C-HY were responsible for drafting the manuscript, interpretation of the data, and critical revision of the manuscript for important intellectual content. H-YL was responsible for drafting the manuscript and interpretation of the data. C-HS was responsible for statistical analysis. Y-CC, ML, MYS, AHH, YL were responsible for critical revision of the manuscript for important intellectual content. W-TH was responsible for drafting the manuscript, statistical analysis, interpretation of the data and critical revision of the manuscript. All authors read and approved the final manuscript.
Conflict of interest
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Ethical approval
Not applicable, because this article does not contain any studies with human or animal subjects.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work were supported by the Taiwan Ministry of Science and Technology (Grant 105-2811-B-002-031) and Far Eastern Memorial Hospital, Taiwan (Grant FEMH-2021-C-031).
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
