A Real-Time Automated Machine Learning Algorithm for Predicting Mortality in Trauma Patients: Survey Says it’s Ready for Prime-Time

Abstract

Background

Though artificial intelligence (“AI”) has been increasingly applied to patient care, many of these predictive models are retrospective and not readily available for real-time decision-making. This survey-based study aims to evaluate implementation of a new, validated mortality risk calculator (Parkland Trauma Index of Mortality, “PTIM”) embedded in our electronic healthrecord (“EHR”) that calculates hourly predictions of mortality with high sensitivity and specificity.

Methods

This is a prospective, survey-based study performed at a level 1 trauma center. An anonymous survey was sent to surgical providers and regarding PTIM implementation. The PTIM score evaluates 23 variables including Glasgow Coma Score (GCS), vital signs, and laboratory data.

Results

Of the 40 completed surveys, 35 reported using PTIM in decision-making. Prior to reviewing PTIM, providers identified perceived top 3 predictors of mortality, including GCS (22/38, 58%), age (18/35, 47%), and maximum heart rate (17/35, 45%). Most providers reported the PTIM assisted their treatment decisions (27/35, 77%) and timing of operative intervention (23/35, 66%). Many providers agreed that PTIM integrated into rounds and patient assessment (22/36, 61%) and that it improved efficiency in assessing patients’ potential mortality (21/36, 58%).

Conclusions

Artificial intelligence algorithms are mostly retrospective and lag in real-time prediction of mortality. To our knowledge, this is the first real-time, automated algorithm predicting mortality in trauma patients. In this small survey-based study, we found PTIM assists in decision-making, timing of intervention, and improves accuracy in assessing mortality. Next steps include evaluating the short- and long-term impact on patient outcomes.

Introduction

Artificial intelligence and machine learning have become integrated in direct patient care for years and for a variety of applications, but with mixed durability and barriers to implementation. Traditionally, machine learning algorithms utilize information input from a clinician or extracted from the electronic health record (EHR) to develop a result to a specific query. This technology has been applied to a wide variety of clinical scenarios, including risk of surgical complications, burn burden assessment, and mortality amongst trauma patients.^1,2 While many of these machine learning algorithms have been developed and validated, few have created a meaningful and lasting impact on provider practices and patient care. This is largely due to the retrospective nature of many existing machine learning algorithms and barriers to implementation (lack of integration into the EHR). Effective use of an algorithm requires real-time acquisition of data and prediction of outcomes to effectively deliver prophylactic treatment and adjust treatment courses. ³ In this prospective, survey-based study, we evaluated provider practices on implementation of a real-time, automated machine learning algorithm embedded in the EHR to help predict mortality. We hypothesized that it would help improve communication between teams and impact utilization of resources and timing of interventions.

Methods

The Parkland Trauma Index of Mortality (PTIM) was developed and validated to predict the mortality risk of trauma patients over the next 48 hours from calculating a score based on objective variables (Figures 1 and 2). ⁴ This tool is novel in that it is integrated directly into the EHR, extracts the data automatically, and calculates a PTIM score, requiring no input from the clinician (Figure 3). The clinician can then utilize this mortality prediction for planning, including operative intervention timing, deciding between damage control or definitive management, and goals of care. Surgical intensive care unit (SICU) providers, in addition to surgeon-specialists, were briefed on the implementation and utility of the PTIM score. For each instance, if the PTIM score was utilized in medical decision-making, an anonymous survey could be completed by the clinician via REDCap (Research Electronic Data Capture, Vanderbilt University, Nashville, TN). The survey first queried what top 3 measurable factors (eg, Glasgow Coma Score (GCS), age, creatinine, and hemoglobin) the clinician felt contributed most to the patient’s PTIM score. Next, the survey queried whether the score assisted in guiding the clinician’s recommended treatment plan. Finally, the survey addressed the clinician’s perceived ease of use, current utility, and future use. Summary statistics of survey outcomes were collected and organized using Stata IC 15 (StataCorp, College Station, TX). This study was approved by our institution’s IRB and performed in compliance with the Declaration of Helsinki. Given the retrospective nature of patient data collection, informed consent was waived as the survey was anonymous; informed consent was waived by providers volunteering to participate.

Figure 1.

23 variables included in the Parkland Trauma Index of Mortality (PTIM) model. Sample scoring with most significant variable listed at the top.

Figure 2.

PTIM schematic. (Top) Patient encounters were divided into 12-hour time intervals from presentation to death or discharge. (Bottom) For each twelve-hour time interval, electronic health record data was used as PTIM input to predict mortality every hour in the next 48 hours.

Figure 3.

Sample patient dashboard, snapshot of PTIM metric, and top 5 predictors with hourly calculations (age, GCS, maximum potassium, maximum creatinine, and maximum AST).

Results

A total of 40 surveys were submitted, 35 of which reported using the PTIM score in medical decision-making. The factors that clinicians perceived as the top 3 predictors of mortality included GCS (22/38, 58%), age (18/35, 47%), and maximum pulse rate (17/35, 45%) which correlated with the algorithm’s calculated actual most significant predictors of mortality (Figure 4(a) and (b)). Providers were also asked to provide what they felt were top predictors not scored on the algorithm, and they included “elevated intra-cranial pressure,” “massive transfusion,” “on-going bleeding,” and “rising lactate.” (Figure 5).

Figure 4.

a, b proposed top 3 predictors of mortality (Glasgow Coma Score, age, and maximum pulse) vs actual top 3 predictors of mortality (Glasgow Coma Score, age, and maximum pulse).

Figure 5.

Top variables not accounted for by the model and ranked high by clinicians (elevated ICP, massive transfusion, concern for bleeding, rising lactate, coagulopathy, and new-onset oliguria).

Respondents reported that the PTIM score assisted in determining the course of the treatment plan (27/35, 77%) and surgical intervention timing (23/35, 66%). When assessing the ease of use, clinicians reported (agree or strongly agree) that the PTIM score was easily integrated into rounds and patient assessment (22/36, 61%) and that it improved efficiency in assessing patients’ potential mortality (21/36, 58%). Several clinicians reported that they would either continue to use the PTIM score when assessing patients (21/36, 58%) or were neutral on utilizing the score (15/36, 41%).

Discussion

Artificial intelligence, machine learning, and neural networks have been applied to patient care for over almost 3 decades, ranging from melanoma and burn wound recognition, predicting sepsis and transfusion needs, and mortality in trauma patients. These methods are becoming significantly more sophisticated and “intelligent,” often surpassing the clinicians’ ability to diagnose and treat. However, several of these predictive models are retrospective and are not readily available for real-time decision-making.

Several studies have applied teaching learning algorithms to retrospectively collected data on trauma and emergency general surgery patients, utilizing large databases from the National Trauma Data Bank and Trauma Quality Improvement Program, amongst smaller institutional databases. Methodologies vary from random-forest, logistic regression, and backwards elimination. At present, we have robust models to help predict mortality,^5,6 venous thromboembolism, and sepsis.^1,7 The data, however, is most useful when it is delivered in real-time and reduces the “noise” and time required for manual calculations. One example of prospective use of machine learning algorithms in patient care can be seen with algorithms utilized for predicting a patient’s likelihood of developing sepsis. ⁸

In the trauma realm, there may be a decision point between definitive management and damage control for a patient’s injuries. The indications for when damage control tactics might be utilized are still debated.^9,10 When to operate for hemorrhage control is less controversial; however, once damage control surgery is done, when is it safe to proceed with other clinically significant injuries (eg, open femur fracture and pelvic fracture) that if not fixed or washed out in a timely fashion can result in infection, non-union, or pain? For example, some patients in this situation, and also at end-of-life, are either too high-risk or may not want to pursue invasive procedures given the risk over benefit. Several calculators exist to describe perioperative risk in surgical patients, such as the American College of Surgeons’ National Surgical Quality Improvement Program or the non-linear POTTER surgical-risk calculator. ^{2 , 11 –} 13 However, these are lengthy, may require manual input, or are retrospective, only describing after the care of the patient that they were likely to die or have complications. To our knowledge, there is no live, fully automated, and integrated machine learning algorithm that prospectively estimates mortality risk in trauma patients.

Many existing trauma mortality scoring systems or risk assessments are retrospective, or may require lengthy calculations that normally do not occur in real-time and are calculated retrospectively. Scoring systems including the TRISS, ¹⁴ Trauma Score, ¹⁵ the Injury Severity Score, Abbreviated Injury Scales, and ASCOT ¹⁶ are critical in assessing severity of disease but may require lengthy calculations, are external to the EHR, and may not be not be as helpful with real-time evaluation. In trauma and acute care, there is the historically described “golden hour” which encompasses the time prior to arrival and initial presentation to a trauma center where the focus to is to resuscitate, perform life-saving maneuvers, and stabilize the patient. ¹⁷ The PTIM score informs clinicians about the patient’s mortality risk over the next 48 hours, by synthesizing multiple physiologic and laboratory data every hour. Additionally, because the algorithm utilizes objective data gathered from the EMR, it reduces the amount of human or user error that occurs with manual input (tracking utilization of massive transfusion and diagnosis of acute respiratory distress syndrome). This allows the PTIM algorithm to be utilized with minimal effort by clinicians, allowing for more time spent planning and delivering patient care.

The PTIM algorithm can be distinguished from a more common “risk calculator,” which can be based on traditional statistical methods or machine learning approaches. The distinction lies in the method used to derive the predictions or calculations.

A traditional risk calculator is typically developed using statistical methods, often based on regression models and the underlying model is usually simpler and interpretable. For example, the Framingham Risk Score for cardiovascular disease is based on a regression model using factors like age, cholesterol levels, and blood pressure. In the field of trauma, the ISS, RTS, and TRISS are examples of traditional risk and/or severity calculators.

A machine learning-based risk calculator utilizes machine learning algorithms to predict risk. These algorithms can range from simple linear models to more complex structures like neural networks, decision trees, or ensemble methods. Further, these models “learn” from data. Given a sufficiently large dataset, they capture intricate patterns or interactions between variables that might be harder to model with traditional statistical methods. The model can be updated or retrained as new data becomes available.

The algorithm for PTIM was developed utilizing a balanced bagging ensemble of decision tree classifiers on a large dataset from a level 1 trauma center. Thus, it is a machine learning model. Further, the model implementation methodology allows for model updates and retraining if the model performance falls out of acceptable bounds. Further details about the machine learning algorithm driving the risk calculator are available to the public. ¹⁸

Our study is not without limitations. Our sample size was small, and as such, may not represent all the clinicians that rotate through or work in our ICU. Additionally, our study is likely affected by selection bias in the responses of the clinicians that completed the survey. Some of the surveys were completed by the same clinician, as clinicians could complete a survey each time they utilized the PTIM tool during their care of a patient. Thus, some responses may be from the same clinician, leading to possible selection bias in our results. Lastly, the PTIM score was likely to play a more impactful role in the care of complex patients, which also contributes to a selection bias.

Despite these limitations, the PTIM tool was useful in the majority of instances it was used. The tool is already integrated into the EHR and because it is updated automatically, requires no input from clinicians on their part save for noting the hourly score. Machine learning algorithms used for predicting patient mortality or outcomes prospectively have been developed in other areas of medicine, such as acute kidney injury prediction in pediatric and adult populations, or neurosurgical interventions.^19-21

Conclusion

In this study, we prospectively surveyed clinicians on the use of a real-time, hourly, automated machine learning algorithm embedded into the electronic healthrecord for more effective triage, management, and mortality risk assessment of trauma patients. We found concordance between perceived predictors and actual predictors of mortality scored by PTIM. Providers found it easy to use and assisted in treatment and prediction of mortality. We hope to address clinician barriers to utilizing the PTIM tool, and standardizing how it is incorporated into patient care—where to look for the tool in notes and interpretation of the score across all specialties. To date, the tool has been used mostly for communication between the SICU and orthopedic surgery services and establish “readiness” for the operating room. Lastly, we hope to answer the critical question—how does this help our patients? Can we improve our resuscitation and utilization of resources and decrease ICU length of stay, complications, and mortality? The answer may lie in more data and through challenging the algorithm with more diverse patients through our current multi-center work.