Abstract
Objective
To examine the diagnostic accuracy of a two-stage clinical decision support system for early recognition and stratification of patients with sepsis.
Design
Observational cohort study employing a two-stage sepsis clinical decision support to recognise and stratify patients with sepsis. The stage one component was comprised of a cloud-based clinical decision support with 24/7 surveillance to detect patients at risk of sepsis. The cloud-based clinical decision support delivered notifications to the patients’ designated nurse, who then electronically contacted a provider. The second stage component comprised a sepsis screening and stratification form integrated into the patient electronic health record, essentially an evidence-based decision aid, used by providers to assess patients at bedside.
Setting
Urban, 284 acute bed community hospital in the USA; 16,000 hospitalisations annually.
Participants
Data on 2620 adult patients were collected retrospectively in 2014 after the clinical decision support was implemented.
Main outcome measure
‘Suspected infection’ was the established gold standard to assess clinical decision support clinimetric performance.
Results
A sepsis alert activated on 417 (16%) of 2620 adult patients hospitalised. Applying ‘suspected infection’ as standard, the patient population characteristics showed 72% sensitivity and 73% positive predictive value. A postalert screening conducted by providers at bedside of 417 patients achieved 81% sensitivity and 94% positive predictive value. Providers documented against 89% patients with an alert activated by clinical decision support and completed 75% of bedside screening and stratification of patients with sepsis within one hour from notification.
Conclusion
A clinical decision support binary alarm system with cross-checking functionality improves early recognition and facilitates stratification of patients with sepsis.
Keywords
Introduction
Sepsis is an uncontrolled inflammatory response to an infection. 1 A spectrum of sepsis exists and mortality increases as the severity of sepsis increases. Early recognition is paramount in improving sepsis outcomes as early intervention and resuscitation have been proven to improve patient outcomes, including survival. 2 Without early recognition, early intervention is impossible. When treatment is delayed, sepsis can rapidly advance to septic shock, multiple organ dysfunction syndrome, and death. 3 Sepsis may be associated with one in two in-hospital deaths. 4 The interval from diagnosis to treatment affects longer term patient outcomes too.5,6 A sepsis programme enabled by clinical decision support (CDS) functionality offers a systematic application of health-related knowledge, which may be integrated with an electronic health record (EHR) system to achieve earlier intervention, as well as provide real-time surveillance and analysis.7–9
Given the adverse consequences of sepsis among hospitalised patients, a hospital-based sepsis programme enabled by a two-stage CDS was established to accurately identify and stratify patients at risk of sepsis, and increase and expedite diagnostics and treatments. This study examined the programme’s accuracy in identifying patients at risk of sepsis, performance of clinical processes, and clinical outcomes.
Methods
Patients and data collection
This was an observational cohort study of prospectively screened patients admitted to a 284-bed urban, non-profit community hospital with more than 16,000 annual admissions. The hospital had an enterprise electronic health record (EHR) system (Millennium: Cerner Corporation, Kansas City, MO) and wanted to demonstrate the effectiveness of a sepsis programme enabled by cloud-based surveillance and computerised CDS to detect and stratify patients with sepsis. The U.S. Department of Health and Human Services’ Office for Human Research Protections clarified that quality improvement activities, described herein, often qualify for Institutional Review Board exemption and do not require individual informed consent. 10
The sepsis programme was designed to improve early recognition and intervention of sepsis. The sepsis protocol was guided by the Surviving Sepsis Campaign resuscitation and management bundles. 11 Education regarding the ‘magic four’ as a simple concept highlighted clinical events and processes key to saving lives, i.e. STAT lactic acid, obtain cultures prior to antibiotics, early administration of antibiotics, and early administration of fluids in patients with severe sepsis or septic shock. An alert system was developed by combining resources of a sepsis CDS with the resources of the onsite IT department. The CDS is consistent with a human factors design for a binary (off/on) alarm system with user access to additional clinical information that can be used to cross-check the validity of the alarm before responding with a decision. Binary alarm systems are notable for high sensitivity but lower specificity. A postalarm cross-check activity has shown to improve specificity, but cross-checking can be time consuming, 12 which has implications for provider adoption.
The hospital population surveillance screening tool component was a cloud-based early recognition system (St. John Sepsis Rescue Agent: Cerner Corporation; Kansas City, MO). The base system definition13,14 and its performance characteristics are described in a previous publication.
15
The alert system’s crawler values and suppression parameters are described in the ‘Definitions’ section. Essentially, the alert system applied a binary alarm system paradigm with two alerts: (1) indications of SIRS (proxy for sepsis) and (2) indications of Severe SIRS (proxy for severe sepsis). The system, running continuously to monitor patient diagnostics from arrival until discharge, was integrated into the EHR and clinical workflow. The project team defined positions and relationships to present alerts to specific nursing and provider groups; designed rules to drive alerts, diagnosis, orders, and documentation; and built forms and plans of care for clinical documentation. The core enabling content for clinical decision-making includes an alert notification generated in the cloud and delivered to a nurse (Figure 1), a rules-driven alert notification delivered to a provider (Figure 2), and a sepsis screening and stratification form presented to a provider (Figure 3).
Alert notification to nurse. Alert notification to provider. Sepsis screening and stratification form.
The clinical workflow included the following process: alert notifications with clinical indications were delivered to a designated nurse16,17 who became responsible for contacting a provider within 5 minutes of receiving the alert. The provider was responsible for conducting the sepsis assessment and stratification within 15 minutes of contact, and, if indicated, submitting orders for the suggested sepsis plan of care, including initiating the initial resuscitation bundle as delineated in the surviving sepsis guidelines. A provider–nurse relationship was established to ensure completion of the resuscitation bundle.
After receiving notification from nursing that an alert had ‘fired’, providers could either complete the EHR sepsis screening and stratification form or bypass it. This cross-check option allows providers to establish whether the alert does or does not meet criteria, and if indicated, document the severity of sepsis. The form, developed to meet Surviving Sepsis bundle compliance, pulled in details of the alerting criteria. Based on severity of sepsis, the provider was prompted to place appropriate orders and/or consider critical care consultation. Also, based on level of sepsis risk stratification, a patient’s sepsis-related International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis code was added to the EHR Problems and Diagnosis tab, and the suggested plan of care was added to the Orders Tab as well. The provider could also submit orders once the sepsis plan of care was initiated.
Provider education with regards to the importance of early recognition was paramount when compared to a standalone technical solution. 18 Provider education was conducted prior to go-live; which included a live lecture series, an online teaching tool, navigating the EHR with the locally developed sepsis screening and stratification tool, and highlighting clinimetric performance of the CDS related to sensitivity and specificity.
Data collection
The study included adult (≥ 18 years old) hospitalised patients following implementation of the sepsis programme in 2014. The clinical process applied a day in the life of a patient paradigm from arrival to hospital discharge. Source data included EHR registration, vital signs, laboratory, pharmacy, and clinical orders. Discharge codes for all encounters were reviewed to identify false negative system performance; i.e. documented sepsis-related International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis code but no CDS alert notification delivered. The study then focused on a cohort of patients screened-in by the sepsis CDS and then screened and stratified by a provider at the patient bedside.
Definitions
Sepsis and severe sepsis were defined per the American College of Chest Physicians/Society of Critical Care Medicine Consensus Conference.13,14 Sepsis was defined as suspected or confirmed infection with clinical evidence of SIRS; while Severe Sepsis additionally required evidence of organ system dysfunction. ‘Suspected infection’ gold standard required at least one microbiology culture be obtained (e.g. blood, urine, sputum, ORSA/MRSA, or soft tissue) and administration of at least one IV/PO anti-infective antibiotics (e.g. antibacterial/fungal medication). Thresholds for SIRS were established when ≥ 3 of the following five criteria were satisfied: (1) temperature > 38.3℃ or < 35℃; (2) heart rate > 95 beats/min; (3) respiratory rate > 22 breaths/min; (4) white blood cell count > 12,000 cells/mm3, or < 4000 cells/mm3, or > 10% immature (band) forms; or (5) glucose 141 < 200 mg/dL. (Note: the threshold for temperature was lowered to < 35℃ because the surveillance alerting system was unable to distinguish core from non-core temperature.) Threshold for severe SIRS was established when ≥ 2 SIRS criteria present, and ≥ 1 of the following four organ system dysfunction criteria were satisfied: (1) cardiovascular system: SPB < 90 mmHg and/or MAP < 65 mmHg; (2) tissue perfusion: serum lactate > 2.0 mmol/L; (3) hepatic system: total bilirubin: ≥ 2.0 mg/dL and < 10.0 mg/dL; and (4) renal system: serum creatinine: Δ↑0.5 mg/dL from baseline. A look-back period consisted of 12 hours for serum lactate, 30 hours for the other criteria, and 72 hours for Δ↑ serum creatinine. Alert notifications for patients in an ICU location were not delivered to providers. Sepsis-related ICD-9-CM diagnosis codes at discharge included 038.xx, 995.91, 995.92, and 785.52.
Statistical analysis
Data were retrospectively analysed. A confusion matrix was applied to report sepsis prevalence, sensitivity and specificity, and positive (PPV) and negative (NPV) predictive values for the sepsis programme. Unadjusted analyses applied Fisher’s exact and chi-square (two-tail, p-value) for dichotomous variables in 2 × 2 and 2 × n contingency tables, respectively. All analyses were conducted using SPSS v21 (IBM, Inc., Armonk, NY).
Results
A total of 2620 patients with 14,907 hospitalisation days were included in the study; 417 (16%) patients were screened-in by the CDS, corresponding to an alert activation rate of 28 patients per 1000 patient days ([417/14,907 days] × 1000). Of the 417 patients screened-in, 210 (50%) were women. Patients’ age was median 68 (IQR = 54–83) years. Nearly all (94%) patients arrived at the emergency department. The CDS first alert activated median 3.9 (IQR = 1.3–31.9) hours after arrival; 89% (n = 370 of 417) patients were screened and stratified by providers. Stratification was completed median 19 (IQR = 6–50) min after alert activation. Patients’ hospitalisation duration was median 6.0 (IQR = 3.0–10.1) days.
Response to two-stage CDS screen and stratification.
Table 1(a) presents a 16% prevalence of sepsis when applying the blended ‘suspected infection’ gold standard and diagnosis code methodological approach. Approximately one in four (n = 116 of 420, 28%) patients had a sepsis diagnosis code documented, but not an activated alert, were considered simple sepsis because they remained below the threshold to activate an alert. Under this construct, the cloud-based sepsis CDS correctly classified 2391 of 2620 patients (91% correct classification) and accurately detected 304 of 420 patients (72% sensitivity) and appropriately rejected 2087 of 2200 patients (95% specificity). Activation was appropriate in 304 of 417 patients (73% positive predictive value) and absence of activation was appropriate in 2087 of 2203 patients (95% negative predictive value). Of special note, these metrics included about one in 11 (n = 39 of 417, 9%) patients with an activated alert (i.e. CDS screened-in), but did not have an order (or combination of orders) for lactic acid, or microbiology cultures, or antibiotics; these 39 (9%) patients screened-in by the CDS, but missing diagnostics and antibiotics, were designated as false positives.
Sepsis screening and stratification by provider.
SS/SS: severe sepsis/septic shock. Cultures and antibiotics: ‘Suspected Infection’ gold standard; severe SIRS alert includes ≥ 2 SIRS criteria and organ dysfunction.
The ensuing clinical process analysis showed that patients with a Severe SIRS alert (i.e. SIRS with Organ Dysfunction criteria) as the first activated alert were more likely to be stratified into the Severe Sepsis/Septic Shock risk group (p = .001). Stratification clearly illustrated patients with sepsis were resource intensive compared to sepsis rule-out; they received a significantly higher volume of diagnostic orders for lactic acid and microbiology cultures, and interventions of IV/PO antibiotics and ICU admission (p = .001). On a continuum, higher acuity sepsis patients were more likely to have supporting clinical documentation for Health Information Management to apply sepsis diagnosis codes (p = .001). Outcomes analysis showed patients with Severe Sepsis/Septic Shock compared to other patients with sepsis were nearly three times at greater risk of an adverse outcome (i.e. death or referred to hospice) (odds ratio = 2.6, 95% CI [1.1–6.1], p = .028), and among sepsis survivors they were 60% less likely to be discharged home (odds ratio = 0.4, 95% CI [0.2–1.2], p = .10).
Clinimetric performance of screening and stratification of patients with sepsis is presented in Table 1(b). First, 417 (16%) of 2620 patients were screened-in by the CDS. Second, applying ‘suspected infection’ gold standard reported in Table 2, all 417 patients screened-in by the CDS were mapped into a second confusion matrix (i.e. from Table 1(a) to (b)). Under this construct, screening and stratification by providers accurately detected 246 of 304 patients (81% sensitivity) and activation was appropriate in 246 of 261 patients (94% positive predictive value).
Regarding outcomes, 17% (n = 71 of 417) of patients screened-in by the CDS either expired or were discharged to hospice. Subgroup analysis identified 17% (n = 45 of 261) of patients screened-in and ruled-in either expired or were discharged to hospice, compared to 17% (n = 26 of 156) of patients screened-in by the CDS, but ruled-out by providers. This latter outcome was driven primarily by patients with sepsis rule-out but suspected of infection (n = 14 of 58, 24%) versus patients with sepsis rule-out and not suspected of infection (n = 12 of 98, 12%). Among patients without an activated CDS alert but with documentation of sepsis at discharge, 12% (n = 14 of 116) expired or were discharged to hospice.
In summary, the sepsis cohort comprised one in five (n = 533 of 2620, 20%) patients when including all hospitalisations, with one in six (n = 88 of 533, 17%) patients either expired or discharged to hospice. However, these patients accounted for one in two (n = 88 of 152, 58%) deaths or discharges to hospice among all 2620 hospitalised patients. The two-stage CDS for early recognition and stratification of patients with sepsis resulted in sepsis prevalence of 14% (n = 377 of 2620), comprised of 261 patients screened-in by the CDS and ruled-in by providers, and another 116 patients who did not activate an CDS alert but did have documentation of sepsis in their EHR. These (n = 377 of 2620, 14%) patients’ discharge disposition showed one in seven (n = 62 of 377, 16%) patients had either died in-hospital or discharged to hospice.
Discussion
Statement of principle findings
The sepsis programme increased awareness among providers and nursing, enabled by active surveillance and CDS to accurately screen-in patients at risk of sepsis; providers then conducted a near-immediate sepsis screening and stratification. Providers screened and risk stratified nine in 10 patients, and completed the assessment within one hour from alert activation for 75% of patients. By using the sepsis CDS in this manner, patients screened-in were successfully stratified into risk groups, with differentiated intensity of diagnostics and interventions. Clearly, ‘suspected infection’ gold standard is a reliable measure of provider decision based on a patient’s clinical indications and stratification, and also supports estimation of sepsis prevalence and outcomes.
Strengths and limitations
Study findings speak to the validity of the sepsis CDS alert definition for early recognition, and the fidelity of the sepsis screening and stratification protocol. Providers’ compliance with the CDS was likely predicated upon a positive association with the sepsis screening and stratification definitions because an alert demonstrating a high positive predictive value (PPV) increases user compliance. 19 The initial alert activated on 16% of patients, with screening achieved 72% sensitivity and 73% PPV, and stratification by providers increased accuracy to 81% sensitivity and 94% PPV. Of the nine in 10 patients who were risk stratified, 91% had at least one diagnostic ordered or antibiotic administered. A substantial finding is that of the remaining 39 (one in 11, 9%) patients without orders, 37 of them were ruled out by a provider, while the other two patients were sepsis rule-in. Moreover, nearly all patients (n = 123 of 127, 97%) screened-in by the CDS and stratified into higher acuity sepsis had orders for lactic acid, microbiology cultures, and IV/PO antibiotics. To place this discussion in proper context, in a previously published study unaffiliated with this current study, with a sepsis CDS running in silent surveillance mode (i.e. alerts were not delivered to a provider), one in four patients recognised by the system did not have diagnostics resulted or antibiotics administered. 15
There are some limitations to this study to consider. First, the setting was a single centre initiative at a 284-bed urban non-profit community hospital, which may not be generalisable to other clinical settings. Second, while the CDS was built on current evidence and good best practices for a binary alarm system with cross-check and validation functionality, the application was developed to promote broad adoption across the hospital’s provider and nursing groups; other health systems may have different circumstances. Third, the programme’s adoption by providers may not be fully known because the study began a few weeks after the CDS go-live date; some variance in usability and fidelity may exist because the sepsis programme enabled by the two-stage CDS was relatively new to providers. Fourth, the study design incorporated a retrospective analysis of cohort data after launch of the sepsis programme, which may have introduced some selection bias associated with real-world clinical practice and processes, temporal relationship of the sepsis programme’s maturation, and missing data in the patient EHR system (e.g. sepsis screening and stratification tool). Fifth, although the sepsis programme is grounded in current guidelines for recognition, assessment, and treatment of patients with sepsis, these guidelines may evolve over time and the sepsis programme should evolve too.
Interpretation of the findings in light of previous research
This study is unique because education regarding the ‘magic four’ instilled a simple concept that illuminated clinical events and processes key to saving lives (i.e. STAT lactic acid, obtain cultures prior to antibiotics, early administration of antibiotics, and early administration of fluids in patients with severe sepsis or septic shock). Realising ‘time is tissue’, the sepsis CDS was localised modestly to appeal to clinical workflow and achieve broad adoption. Cloud-based surveillance began when a patient’s initial vital signs or other diagnostics were resulted in their EHR, with continuous 24/7 monitoring of clinical results until discharge, thereby mitigating problems reported elsewhere. 20 Nearly all patients with an activated alert arrived via the emergency department, which supports findings from prior studies. 21 The demographic profile of screened-in patients showed they were remarkably similar, which, coupled with expediency of prospective stratification by providers, minimised confounders to study findings. A number of patients with sepsis were admitted to the ICU. A majority of them, however, were managed in general medical/surgical units. Findings from the stratification accentuate this situation. A picture has emerged where surveillance can detect patients at risk of sepsis; diagnostic specificity is improved when the CDS includes postalert screening and stratification. A reasonable line of demarcation exists between patients with rule-out versus rule-in sepsis, severe sepsis, and septic shock. On a progressive continuum, higher acuity sepsis patients were clinically intense, more likely to require ICU services, and more likely to have a poor outcome. These findings on service intensity are consistent with other recent studies.22,23
Implications for policy and practice
A sepsis programme enabled by cloud-based surveillance and computerised CDS can expedite accurate recognition and stratification of patients with sepsis, with goals of improving hospital survival and increasing the likelihood of discharge to home. Indeed, CDS systems provide a platform for discussion, debate, and innovation among sepsis programmes with an eye on improving patient outcomes. Moreover, sepsis is under-documented in the EHR and code assignment suffers. By adopting CDS, opportunities to improve revenue cycle and value-based programmes exist.
The CDS design incorporating a socio-technical systems approach for rule-in/out is consistent with a human factors design for a two-stage binary alarm system that activates an alert when threshold criteria are met, delivers notifications containing criterion variance, and includes an option to cross-check and validate the alert, as well as consider other pertinent criteria before responding with a medical decision. Leveraging existing human capital to improve outcomes, rather than adopting a rapid response team model, may also be possible by emphasising provider education; incorporating evidence-based CDS and surveillance alert notification functionality into the EHR; standardising nursing and provider workflows; mandating risk stratification; and if indicated, ordering of diagnostics, interventions, and consultation.
Direction for future research
An area of future research includes a focus on improving outcomes of patients within sepsis risk groups, as well as survivors at risk of early readmission because many unfortunate patients stricken with sepsis syndrome experience loss of productivity following hospitalisation. 24 A two-stage sepsis CDS as described in this study can be applied for demand planning and facilitate future study. Further research would add depth to our understanding of transition care, the care continuum, and patient health and wellness.