Abstract
Sepsis is an inflammatory response triggered by infection, with a high in-hospital mortality rate. Early recognition and treatment can reverse the inflammatory response, with evidence of improved patient outcomes. One challenge clinicians face is identifying the inflammatory syndrome against the background of the patient’s infectious illness and comorbidities. An approach to this problem is implementation of computerized early warning tools for sepsis. This multicenter retrospective study sought to determine clinimetric performance of a cloud-based computerized sepsis clinical decision support system (CDS), understand the epidemiology of sepsis, and identify opportunities for quality improvement. Data encompassed 6200 adult hospitalizations from 2012 through 2013. Of 13% patients screened-in, 51% were already suspected to have an infection when the system activated. This study focused on a patient cohort screened-in before infection was suspected; median time from arrival to CDS activation was 3.5 hours, and system activation to diagnostic collect was another 8.6 hours.
Keywords
Sepsis is an uncontrolled inflammatory response to an infection. The systemic inflammatory response syndrome (SIRS) may be triggered by an infection, trauma, burn, or other offense. Sepsis occurs in the presence of an infection with a SIRS response and can strike anyone at any age. 1 When treatment is delayed, sepsis can rapidly advance to septic shock, multiple organ dysfunction syndrome (MODS), and death. The estimated 5-year mortality rate for patients with severe sepsis is about 60%; survivors experience much lower physical quality of life. 2 The interval from diagnosis to treatment affects short- and long-term patient outcomes. 3
A clinical decision support system (CDS) offers a systematic application of health-related knowledge and analysis of available data. At the enterprise level, a CDS facilitates achievement of key performance goals. 4 The CDS also may integrate with early warning systems and rapid response teams designed to achieve earlier intervention. 5 Adoption of enterprise electronic health record (EHR) systems, which may include embedded CDS and early warning systems, enables digital surveillance of the patient. 6 Ideal CDS is available in real time and at the point of care, integrating cloud-based CDS with the host EHR system in a patient-centric clinical workflow. 7
Early-generation sepsis screening tools were typically stand-alone applications implemented in specified patient care units. As front-runners to computerized sepsis CDS, screening tools were often paper-based algorithms and protocols. In the intensive care unit (ICU), severe sepsis screening tools guided nurses and intensivists during admission assessment. These tools performed relatively well because the populations in which they were being applied were selected for patients with infections and organ dysfunction.8-10 The emergency department (ED) is another venue for effective sepsis screening tools.11,12 An initiative in the United Kingdom demonstrated validity of an early-warning score in the ED for identification of patients with higher risk for ICU care and/or mortality. 13 A study of electronic surveillance for SIRS criteria and hypotension demonstrated effectiveness of provider alerts in increasing early goal-directed therapy. 14 With growing recognition of sepsis outside the ED and ICU settings, CDS tools also have been studied in general care/medical units.15-18
Venue-centric screening tools contain localized definitions of sepsis. Diagnostic utility may suffer when these tools are applied to other areas because of differences in the underlying populations. As a consequence, broad adoption of sepsis screening tools, or the more advanced surveillance/detection/alerting systems, remains a challenge. A clinimetrics analytic framework offers a quantitative approach to describe clinical findings and assess accuracy of diagnostics (eg, vital signs, laboratory tests, imaging studies). The framework can be used for development of measurement instruments and for measurement of the quality of performance in clinical practice.19,20 The objective of this study was to determine the clinimetric performance of a cloud-based sepsis CDS for early recognition of sepsis in the hospital; understand sepsis prevalence, incidence, and outcomes by applying the sepsis CDS; and identify potential opportunities for quality improvement and enhancement of the sepsis CDS.
Methods
Patients and Data Collection
This multiple center retrospective cohort study was performed at 5 different medical centers in 4 distinct geographic regions across the United States. Facilities included a level 1 trauma center, a level 2 trauma center, a women’s and children’s hospital, and 2 community hospitals. All facilities had an enterprise EHR system (Millennium: Cerner Corporation, Kansas City, MO). Data were included for adults (≥18 years old) who were hospitalized during 2012 and 2013. The US Department of Health and Human Services Office for Human Research Protections clarified that quality improvement activities, described herein, often qualify for institutional review board exemption and do not require individual informed consent. 21
The sepsis CDS is a cloud-based computerized system running in real time to identify patients with SIRS or severe SIRS criteria captured in the EHR (St John Sepsis: Cerner Corporation, Kansas City, MO). Encounter data from the EHR systems are electronically abstracted to and analyzed within the cloud platform. Surveillance begins when a patient’s initial clinical events are documented in the EHR system, regardless of patient location. The sepsis CDS has distinct activation definitions for SIRS and severe SIRS. If clinical criteria align, an alert is activated, and criteria are saved with time stamp information. When the sepsis CDS is in live surveillance mode, notifications are delivered to providers for action. For this quality improvement study, the sepsis CDS ran in silent mode with encounters flagged but without provider notification. EHR charts for patients flagged by the sepsis CDS were examined manually to understand the relationship and timing between alerts and clinical indications of SIRS.
Detailed reviews applied a day-in-the-life of a patient paradigm from arrival to hospital discharge. Reviews encompassed digital registration and room/bed assignment, clinical documents, vital signs, orders and results for laboratory and imaging, and orders for and administration of medications. Among patients screened-in for SIRS, a pattern analysis was conducted to identify combinations of SIRS criteria associated with confirmed infections. Shock index was calculated with patient data on arrival.22,23 The corrected apparent strong ion difference (SIDa) also was calculated on arrival because a patient’s pre-resuscitation SIDa, when aberrant, offers an indicator of electrolyte abnormality, metabolic disturbance, and evolving disease process.24-27 The SIDa normal range was taken as 40 ± 2 mmol/L.
Among patients screened-in, time stamps for key data were examined in detail. Patients were segmented into one of 3 cohorts based on intervals between first alert activation and initial orders for diagnostics and interventions. Patient cohort A (early recognition by the sepsis CDS) included patients with a first alert activated before being suspected of infection, with later ordering of diagnostics and interventions. Patient cohort B (provider suspected infection before activation of first alert) included patients already suspected of infection when the first alert was activated. Patient cohort C (diagnostics not resulted) included patients with an activated alert during hospitalization, without later ordering of diagnostics or antibiotics.
All encounters screened-in were examined for presence of sepsis-related International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis codes at discharge. Detailed clinical documentation using keywords of sepsis, severe sepsis, or septic shock was expected for Health Information Management (HIM) assignment of sepsis-associated ICD-9-CM discharge codes. Discharge codes for all patient encounters were reviewed to identify false-negative performance for patients with coded sepsis but no CDS alert.
The focus of this study is on patient cohort A to better understand the population of patients screened-in for possible sepsis before a provider suspected infection. The manual chart review of cohort A included diagnostic tests to determine presence of infection and, if present, the pathogen, susceptibilities, and source. Infection was confirmed by diagnostic test results from microbiology cultures or clinical documentation of suspected viral or bacterial infection supported by imaging. Community-acquired and health care–associated infections were not differentiated in this study. Patient encounter data were further examined for evidence of organ dysfunction.
Definitions
Sepsis and severe sepsis were defined per the American College of Chest Physicians/Society of Critical Care Medicine Consensus Conference.2,28-30 Sepsis was defined as suspected or confirmed infection with clinical evidence of SIRS, whereas severe sepsis additionally required evidence of organ system dysfunction. MODS was defined as sepsis with 2 or more organ system dysfunctions. Thresholds for SIRS were established when ≥3 of the following 5 criteria were satisfied: (1) temperature >38.3°C or <36°C; (2) heart rate >95 beats/min; (3) respiratory rate >22 breaths/min; (4) white blood cell count >12 000 cells/mm3 or <4000 cells/mm3, or >10% immature (band) forms; or (5) glucose 141 to <200 mg/dL. The threshold for severe SIRS was established when ≥2 SIRS criteria were present and ≥1 of the following 4 organ system dysfunction criteria were satisfied: (1) cardiovascular system, systolic blood pressure <90 mm Hg and/or mean arterial pressure <65 mm Hg; (2) tissue perfusion, serum lactate >2.0 mmol/L; (3) hepatic system, total bilirubin ≥2.0 mg/dL and <10.0 mg/dL; and (4) renal system, serum creatinine Δ↑0.5 mg/dL from baseline. A look-back period consisted of 12 hours for serum lactate, 30 hours for the other criteria, and 72 hours for Δ↑serum creatinine.
Elevated shock index (heart rate divided by systolic blood pressure, normalized by age and sex) was defined when a patient’s initial shock index was ≥97th percentile. 22 The calculation for a corrected SIDa was derived from first lab results. Because lactate is an independent determinant of mortality in critically ill patients, the SIDa was “partitioned” into inorganic ion difference [(Na+ + K+ + Mg2+ + Ca2+) − Cl−] and lactate plasma level.31,32 Lactate plasma level was removed from the calculation because it was typically unavailable with initial laboratory results. Because Ca2+ and Mg2+ levels were not always resulted with the initial laboratory order sets, a summative value of 1.85 mmol/L was incorporated into the SIDa model, based on reported results of ionized calcium and magnesium among critically ill patients: Ca2+ = 1.11 mmol/L 33 and Mg2+ = 0.78 mmol/L. 34 Thus, the calculation for the corrected SIDa = [(Na+ + K+ + 1.85) − Cl−] was established on patient arrival (ie, the first set of tests resulted). Electrolyte abnormality and metabolic disturbance were based on corrected SIDa ≤34.0 or >42.0 mmol/L. It is important to note that the baseline laboratory results used to compute corrected SIDa preceded orders for blood gas analysis. Sepsis-related ICD-9-CM diagnosis codes at discharge included 038.xx, 995.91, 995.92, and 785.52.
Statistical Analysis
Reported results included frequencies, proportions, means, standard deviation, and medians with interquartile range (IQR). All analyses were conducted using Statistica software, version 8.0 (StatSoft, Inc, Tulsa, Oklahoma). A confusion matrix was applied to report sepsis prevalence rate, correct classification rate and odds ratio, sensitivity and specificity, and positive predictive value (PPV) and negative predictive value (NPV) metrics for the sepsis CDS.
Results
Of 6200 hospitalizations examined, the cloud-based sepsis CDS screened-in 817 (13%) patients corresponding to a daily activation rate of 10.0 patients per day activating a first alert within a 500-bed hospital. Table 1 describes demographics of the 3 established patient cohorts, the median and IQR hours from activation of first alert to diagnostics, and prevalence rates of confirmed infection via chart review and sepsis-associated diagnosis. Patient cohort A comprised the 1 in 4 (n = 195, 24%) patients recognized by the sepsis CDS before a provider suspected infection and for whom the provider ordered diagnostics and interventions (ie, antibiotics) after the sepsis CDS alert. The first activated alert to diagnostic collect had a median 8.6 (IQR = 1.7 to 38.7) hours; confirmed infection rate was 65%. Patient cohort B comprised the 417 (51%) patients already suspected of having infection, with orders for diagnostics or treatment interventions already submitted when the first alert was activated. The time to first alert for this group is shown in Table 1. Patient cohort C comprised 1 in 4 (n = 205, 25%) patients who had an alert activated during their hospitalization but without orders for diagnostics to identify a potential source of infection or orders for antibiotics indicating a suspicion of infection. Of the 5383 patients without an alert, 80 patients (not included in Table 1) had a sepsis-associated discharge code documented on their encounter, for 5303 true negatives and 80 false negatives.
Characteristics of Patient Cohorts. a
Abbreviations: CDS, clinical decision support; IQR, interquartile range; NA, not applicable; SD, standard deviation; SIRS, systemic inflammatory response syndrome.
There were 6200 patient hospitalizations; n = 817 (13.2%) had an activated alert. Of the remaining 5383 patients who did not have an activated alert, 80 patients had a sepsis diagnosis code documented at discharge from hospital. Prevalence of sepsis using diagnosis code methodology was 4.6%, increasing to 7.4% when applying a blended chart review and diagnosis code methodological approach. Regarding cohort C, a source of infection as a possible causal factor of SIRS could not be determined.
Clinimetric performance of the sepsis CDS was established by incorporating these 6200 patient-level results into a confusion matrix, from which several accuracy metrics were then derived. Prevalence of sepsis using the diagnosis code method was 4.6%, increasing to 7.4%, reflecting a Δ↑61% when using a blended chart review and diagnosis code methodological approach. Under this latter construct depicting 7.4% prevalence, the cloud-based sepsis CDS correctly classified 5682 of 6200 patients (ie, 92% correct classification with an odds ratio of 57), accurately detected 379 of 459 patients (83% sensitivity), and appropriately rejected 5303 of 5741 patients (92% specificity); activation was appropriate in 379 of 817 patients (46% PPV—ie, 54% false-positive rate), and absence of activation was appropriate in 5303 of 5383 patients (99% NPV). Of special note, these accuracy metrics included those 1 in 4 (n = 205, 25%) patients with an activated alert (ie, screened-in) but who had no diagnostics collected or antibiotics ordered (ie, patient cohort C).
Early Recognition by the Sepsis CDS
Characteristics of the 195 patients comprising cohort A are shown in Table 2. Approximately half the patients had an elevated shock index ≥97th percentile, while 4 in 10 patients had an abnormal corrected SIDa. The median time from arrival to activation of first alert was 3.5 (IQR = 1.0 to 14.3) hours. Approximately 60% of patients had a first activated alert of SIRS versus severe SIRS alert.
Characteristics of Patient Cohort A (Early Recognition by the Sepsis CDS).
Abbreviations: CDS, clinical decision support; IQR, interquartile range; LOS, length of stay; SD, standard deviation; SIRS, systemic inflammatory response syndrome.
Clinimetric performance was examined further by considering the 195 cohort A patients for whom the system activated for indications of SIRS or severe SIRS. Specific criteria for each first-activated alert were studied; 126 of 195 (65%) patients had confirmed infection, whereas 69 (35%) patients did not have a confirmed infection. Table 3 illustrates patients with a first-activated SIRS alert; two thirds of patients had a confirmed infection. Four patterns were responsible for activating the first SIRS alert; the top 2 patterns accounted for 84% of patients. Table 3 also reports the second type of activated alert (ie, severe SIRS). Results illustrate the frequency of specific organ system dysfunction associated with a confirmed infection among patients whose first-activated alert indicated severe SIRS. Regarding activation of severe SIRS alert, more than 60% of patients had a confirmed infection. Cardiovascular or tissue perfusion organ system involvement was responsible for 74% of first-activated severe SIRS alerts (Table 3).
Relationship Between SIRS/Severe SIRS Criteria and Confirmed Infection for Patient Cohort A (Early Recognition by the Sepsis CDS). a
Abbreviations: CDS, clinical decision support; HR, heart rate; RR, respiratory rate; SIRS, systemic inflammatory response syndrome; T, temperature; WBC, white blood cell.
SIRS alert criteria: any combination ≥3 criteria within profile; “●” must be included in the profile; inclusion of “○” indications to meet algorithm definition.
Among the 126 of 195 (65%) patients with confirmed infections, infection source included respiratory (n = 46, 37%), urinary tract (n = 41, 33%), bloodstream (n = 25, 20%), skin/soft tissue (n = 11, 9%), and abdominal (n = 3, 2%). Table 4 lists pathogens from microbiology cultures and imaging studies. Also, 31 (25%) patients had 2 or more concomitant infectious pathogens according to results.
Diagnostic Test Results for Patient Cohort A (Early Recognition by the Sepsis CDS). a
Abbreviation: CDS, clinical decision support.
Patients with a microbiology or radiology imaging diagnostic test that was positive or with clinical finding and impression documented, with at least one of the pathogens in the table. Data given as n (%). Multiple responses per patient were possible.
Citrobacter, Enterobacter, Proteus, Acinetobacter, Haemophilus, Prevotella oralis, Lactobacillus sp, and Helicobacter pylori antigent.
Hemolysis, Streptococcus salivarius, Streptococcus sp (Mitis/oralis), Viridans streptococci, Bacteriodes fragilis, Actinomyces sp, and Hafnia alvei. Additional pathogens include parasitic Blastocystis hominis as well as diabetic foot infection being documented.
Table 5 reports the incidence of SIRS, severe SIRS of noninfectious origin, sepsis, severe sepsis, and MODS as well as relationship to ICU admission, in-hospital mortality, hospital length of stay (LOS), and documented discharge code for sepsis. Of the 113 patients with a first-activated SIRS alert, 75 had confirmed infection (ie, sepsis), with nearly half (n = 36 of 75, 48%) progressing to severe sepsis or MODS. In comparison, 51 of the 82 patients with a first-activated severe SIRS alert had confirmed infection—that is, severe sepsis (n = 51 of 82, 62%)—with two thirds (n = 34 of 51, 67%) progressing to MODS. In all, 80% of patients with MODS were admitted to the ICU, compared with 45% patients with severe sepsis and 33% patients with sepsis. Of those patients with abnormal shock index and abnormal corrected SIDa at baseline, 7 in 10 received ICU care during their hospitalization. None of the patients with sepsis or severe sepsis died in hospital. Of 56 patients with MODS, 12 (21%) died in hospital. A similar mortality rate among patients with abnormal shock index and abnormal corrected SIDa at presentation was found as well; these patients accounted for half (n = 6, 50%) of the in-hospital deaths. The median hospital LOS for patients with sepsis was 3.7 days, with LOS increasing with severity of sepsis. Only 54 of 126 (43%) patients with confirmed infections had a diagnosis code for sepsis, severe sepsis, or septic shock at discharge. Patients without a sepsis diagnosis code did not have provider-supporting clinical documentation at discharge, which was necessary for HIM to apply diagnosis codes.
Incidence and Outcomes for Patient Cohort A (Early Recognition by the Sepsis CDS). a
Abbreviations: CDS, clinical decision support; ICD-9-CM, International Classification of Diseases, Ninth Revision, Clinical Modification; ICU, intensive care unit; IQR, interquartile range; LOS, length of stay; SIDa, apparent strong ion difference; SIRS, systemic inflammatory response syndrome.
ICU admission anytime during hospitalization; mortality: in-hospital; a sepsis-related ICD-9-CM diagnosis code was assigned at discharge. SIDa indicates corrected SIDa.
Types of organs failed: multiple responses per patient possible.
Discussion
The cloud-based sepsis CDS began silent surveillance when a patient’s initial diagnostics were resulted in their EHR. The sepsis CDS methodology was consistent with a flow sheet paradigm for capturing results, clinical events, and time stamps. Definitions of SIRS and severe SIRS alerts are robust, as reported by the patient population-level clinimetric performance results described in the confusion matrix. The system’s most powerful impact was in identifying patients for whom providers did not yet suspect or recognize SIRS or sepsis. An examination of activated alerts and time stamps stored in the cloud showed the capability of the sepsis CDS to inform disease state and progression over time. The sepsis CDS presents opportunities to recognize sick patients earlier and trigger a postalert notification to begin sepsis assessment, accelerating orders for diagnostics, antibiotics, and other treatments. The research team identified opportunities to improve predictive qualities by examining relationships between serious infections, shock index, corrected SIDa, and clinical patterns responsible for the first-activated SIRS and severe SIRS alerts.
The cloud-based sepsis CDS identified many different patterns of sepsis among patients in cohort A. More than half of the patients experienced deteriorating health status during their hospitalization. Nearly two thirds of patients with severe sepsis progressed into MODS. The organ systems involved included cardiovascular (87%), tissue perfusion (47%), hepatic (36%), and renal (32%). The danger of MODS was evident, illustrated by 1 in 5 patients dying in hospital. It is interesting to note that 1 in 3 patients diagnosed with severe sepsis had shock index ≥97th percentile and abnormal corrected SIDa at presentation; these patients contributed to half the deaths involving MODS.
This study has a number of limitations. It was a multiple-center study performed at 5 hospitals located in 4 different geographic regions in the United States; hospitals were diverse in capability and capacity. The definitions of SIRS were modified from consensus definitions for inclusion into the cloud-based sepsis CDS. Numerical thresholds for vital signs that activated the SIRS alert varied slightly between hospitals because of version upgrades unrelated to this study. The sepsis CDS screened-in a large number of patients, with infection already suspected in about half of the patients prior to their first-activated alert. Among patients who had an infection confirmed, the distribution pattern of organisms/pathogens showed geographic variation. In 1 in 4 patients with an activated alert, no testing or treatment specific to sepsis was ordered.
Study findings have several implications for quality improvement. To begin with, this study’s approach is premised on the relationship between serious infection as causal factor of SIRS and the timing between the sepsis CDS detection and providers’ suspicion of infection among sick patients. This study found that 9 in 10 patients in the cohort analyzed first presented to the ED, with most activating a first alert within just a few hours from triage check-in. Although many very sick patients were admitted to the ICU during their hospitalization, a majority of patients with sepsis disease were managed in general care/medical units.
Conclusions
The cloud-based sepsis CDS integrated with enterprise EHR systems is an effective approach toward early recognition of sepsis in a hospital setting. These systems are indeed difficult to design, build, and implement; it is also difficult to achieve broad and sustained adoption by providers. Study findings indicate that a 500-bed hospital can anticipate a demand of 10 patients per day (24-hour period of time) who activate a first alert; most of these patients will have arrived to the ED. The first alert is the most important alert for action, with optional functionality to suppress serial alerts.
The sepsis CDS enables providers to accelerate diagnostic and therapeutic interventions and offers transparency to hospital-wide care management, with an objective of reversing a patient’s sepsis syndrome to potentially avert serious complications such as organ system dysfunction and death. Assessing clinimetric performance of the system requires screened-in patients to have diagnostics resulted to rule out or confirm serious infection. Thus, although some activated alerts may be interpreted as false positives, uncertainty remains. The system’s measure of accuracy may be improved by providers carrying out 2 or 3 different types of diagnostics (eg, microbiology cultures, serology, chemistry, lactic acid). Despite this near-term gap between screened-in patients and diagnostics resulted, a fairly clean windshield into the sepsis CDS promotes the early recognition of patients, with a respectable degree of accuracy. By extending this visibility to providers, an opportunity to extend the influence beyond early adopters of the sepsis CDS to the larger provider population becomes possible.
Future quality improvement initiatives include application of the sepsis CDS across patient care processes and venues of care. The system’s activation rate appears to be acceptable and not too burdensome while simultaneously achieving good clinimetric performance. The research team does not foresee a need to modify the base system; rather, it can offer information through provider education forums and rapid cycle feedback mechanisms. Because time is the enemy for patients with sepsis, process measure considerations include time intervals from alert activation to completion of provider sepsis assessments, and use and intensity of diagnostics and therapeutic interventions for patients screened-in. Moreover, the research team seeks a better understanding of patient outcomes as well as outcomes for those who survive their index hospitalization but then are readmitted to hospital shortly thereafter. Finally, the importance of clearly articulated clinical documentation at discharge to enable HIM coding should not be forgotten; this also has substantial revenue cycle implications.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.