Abstract
Objective
Delirium is commonly reported from the inpatients with Coronavirus disease 2019 (COVID-19) infection. As delirium is closely associated with adverse clinical outcomes, prediction and prevention of delirium is critical. We developed a machine learning (ML) model to predict delirium in hospitalized patients with COVID-19 and to identify modifiable factors to prevent delirium.
Methods
The data set (n = 878) from four medical centers was constructed. Total of 78 predictors were included such as demographic characteristics, vital signs, laboratory results and medication, and the primary outcome was delirium occurrence during hospitalization. For analysis, the extreme gradient boosting (XGBoost) algorithm was applied, and the most influential factors were selected by recursive feature elimination. Among the indicators of performance for ML model, the area under the curve of the receiver operating characteristic (AUROC) curve was selected as the evaluation metric.
Results
Regarding the performance of developed delirium prediction model, the accuracy, precision, recall, F1 score, and the AUROC were calculated (0.944, 0.581, 0.421, 0.485, 0.873, respectively). The influential factors of delirium in this model included were mechanical ventilation, medication (antipsychotics, sedatives, ambroxol, piperacillin/tazobactam, acetaminophen, ceftriaxone, and propacetamol), and sodium ion concentration (all p < 0.05).
Conclusions
We developed and internally validated an ML model to predict delirium in COVID-19 inpatients. The model identified modifiable factors associated with the development of delirium and could be clinically useful for the prediction and prevention of delirium in COVID-19 inpatients.
Introduction
Delirium is a neuropsychiatric condition characterized by altered cognition, agitation, and hallucinations, and it is frequently reported in patients with Coronavirus disease 2019 (COVID-19), with prevalence ranging from 10% to 70%.1–4 Amplified inflammatory immune response, hypometabolism in the cerebral cortex, and restricted environment are considered possible causes of delirium in COVID-19 patients.5–7 As delirium is associated with poor clinical outcomes, including long hospital stays, high medical costs, dementia, and death8,9 in critically ill patients, and a significant association has been found between delirium and mortality in patients with COVID-19, 10 predicting high-risk patients and identifying related factors are crucial for prevention and early intervention. In fact, active preventive interventions can reduce the occurrence of delirium and related poor outcomes, including mortality. 11 Most of the known risk factors for delirium, including age, comorbidities, and smoking and drinking history, are not modifiable, and factors such as sedation and mechanical ventilation cannot be avoided during the course of treatment.12–14 Rather, medication-related factors offer a potential prevention strategy through drug adjustments or substitutions.
A significant association has been found between delirium and certain types of medications in patients with COVID-19, including antipsychotics, benzodiazepines, and opioids. 13 Despite these findings, few studies have investigated medication as a risk factor for delirium. Machine learning (ML) techniques have emerged as an effective tool in healthcare for predicting disease incidence and prognosis based on patterns identified in extensive clinical datasets comprising electronic health records (EHRs). 15 Recently, ML algorithms developed to predict neuropsychiatric outcomes successfully predicted delirium and identified associated risk factors.16,17 While current ML models for delirium provide some utility, their reliability is found lacking, particularly in disease-specific cases like COVID-19.18,19 A recent study developed and validated a prediction model for delirium in COVID-19 patients 20 ; yet the model does not suggest prevention measures. The COVID-19 pandemic has amplified the consequences of delirium, making the delivery of delirium-reducing interventions more challenging due to infection control precautions and resource conservation efforts, ML-based model could facilitate prediction of delirium and classification of high-risk patients, saving resources of healthcare professionals. 21
We argue for the development of a COVID-19-specific ML model, that not only predicts delirium and classifies high-risk patients, but also identifies modifiable factors like medication for prevention. In this retrospective study, our focus is the development of such an ML model, targeting medication, especially psychotropic drugs and COVID-19 treatment agents, as modifiable influential factors for delirium.18,19 The aim of this study is to leverage individual patient EHRs to predict delirium and classify high-risk patients, thereby optimizing healthcare resources. Our hypothesis suggests that this targeted, ML-based prediction model could be implemented in other medical institutions to enhance patient care.
Methods
Study population
This retrospective study included adults diagnosed with COVID-19 admitted to one of four medical centers in the Republic of Korea from January 2020 to August 2021. The data set was comprised of inpatients from four medical centers (Seoul National University Bundang Hospital, Seoul National University Hospital, National Medical Center, and Seoul Medical Center). COVID-19 was diagnosed based on positive polymerase chain reaction results and all patients were isolated in negative pressure units of each medical center for treatment. Patients with a previous diagnosis of dementia or cognitive impairment were excluded. The study protocol was approved by the institutional review board of each medical center and the requirement for informed consent was waived.
Procedures
As part of the routine clinical protocol in all hospitals operating COVID-19 isolation wards, clinical, radiological, and laboratory tests were performed on the first day of admission for all COVID-19 inpatients, for the purpose of clinical evaluation and determining treatment plans. These data were retrospectively obtained from medical charts of each medical center and included laboratory data, chest X-ray, and chest computed tomography (CT) findings. The obtained data was all from the admission date of each patient, and these were used as indicators of illness of severity reflecting conditions such as pneumonia or sepsis. Chest X-ray and CT findings were categorized using a ternary coding system, as follows: 0, normal findings (no active lesions in the lungs); 1, moderate problems (mild patchy opacity or haziness in both lungs); and 2, severe problems (multifocal patchy consolidation or patchy opacities in both lungs). The primary outcome was the onset of delirium during hospitalization. All patients hospitalized at each medical center were routinely evaluated by board-certified psychiatrists (HuP, HoP, SNK, SHL). As members of the Liaison team, psychiatric staffs were assigned to COVID-19 isolation units at each center, and all admitted patients were evaluated through tele-monitoring and phone calls. Delirium evaluation was conducted using the confusion assessment method (CAM), and delirium was diagnosed based on the established criteria of the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5). 22 To screen for delirium using CAM, it was conducted as a routine clinical practice under the supervision of the psychiatric consultation team and evaluated daily by the medical staff responsible for the isolation ward. Patients who were in deep sedation state or unable to be evaluated were excluded.
Variables
Initially, 93 variables were selected as influential factors based on findings of previous studies. 14 Variables with missing data in >15% of patients were excluded, these variables were either factors with decreased clinical importance or overlapping clinical implications (such as the bed number in the ward, the type of alcohol predominantly consumed, family history of psychiatric diseases), or were related to outcomes of delirium (such as mortality). The remained 78 variables in the final analysis included demographic characteristics (age and sex), type of insurance, presence or absence of windows in the isolation room, comorbidities (assessed using the Charlson Comorbidity Index), smoking and alcohol drinking status, vital signs, mechanical ventilation, and medication received during entire hospitalization. Medications were divided into two categories: those known to be associated with delirium onset, including opioids, sedatives, antipsychotics, antidepressants, antiepileptics, cholinergics, and anticholinergics,18,19 and those administered to treat COVID-19 or COVID-19-related symptoms, including antivirals, antibiotics, anti-inflammatory medications, steroids, mucolytics, and expectorants. The medications for COVID-19-related symptoms are shown in Supplementary Table 1.
Statistical analyses
Statistical analyses were performed using R software (R Core Team, Vienna, Austria). Extreme gradient boosting (XGBoost) is an ML-based gradient boosting model that combines weak classifiers to form a strong classifier. The ML model was designed using Python 3.7. XGBoost is a gradient boosting model based on ML that employs a series of decision trees, with each tree correcting the residuals of its predecessors. To determine the need for additional tree branching, accuracy and complexity scores are measured. 23 Also, we referenced key results from the logistic regression analysis for each variable (Supplementary Table 2). As a result, XGBoost has emerged as a powerful ML technique that learns quickly while preventing overfitting. A built-in GBM algorithm was used to impute missing values. The dataset was randomly divided into two subgroups for model development: a training group (n = 615, 70%) and a validation group (n = 263, 30%). We employed two feature selection methods, sequential feature selection and forward elimination. The 78 candidate variables were classified as either categorical or continuous variables. For each variable subgroup, a stratified fivefold cross-validation was applied to the training group as part of the feature selection process. The final model included the top three variables identified by the algorithm. In each variable subgroup, the 4th, 5th, and 6th ranked variables suggested by the algorithm were assessed alongside six preselected variables. The variable that optimized internal validation performance was incorporated into the final model, which comprised a total of 10 variables. Candidate evaluation metrics included accuracy, precision, recall, and F1 score, which were calculated using the Scikit-learn library in Python. The 95% confidence intervals (CIs) of evaluation metrics were also defined using the mean and standard deviation of stratified fivefold cross-validation scores in the training set and the 1000 scores from bootstrapping in the validation set. Ultimately, the area under the curve (AUC) of the receiver operating characteristic (ROC) curve was chosen as the evaluation metric due to its widespread use in ML research. We employed the Youden index to select an optimal threshold on the ROC curve, enhancing model performance by balancing sensitivity and specificity. This selection, while independent of the AUROC metric, ensures practical applicability and effectiveness of the model beyond its overall discriminatory power. Referring to previous studies utilizing similar ML modeling 24 and literature on sample size for ML modeling research, 25 it was determined that a minimum of 560 samples would be required to construct the predictive model with 10 key variables.
Results
Total of 878 patients, 57 (6.5%) of whom had delirium (Table 1). Of the 78 predictor variables, nine key input variables selected by recursive feature elimination were included in the model: mechanical ventilation, medication (sedatives, antipsychotics, ambroxol, piperacillin/tazobactam, acetaminophen, ceftriaxone, and propacetamol), and sodium ion concentration (Table 2). The feature importance plot for the gradient boost model and the CIs expressed as error bars are shown in Figure 1. The CIs of the feature importance were defined using the mean and standard deviation of stratified fivefold cross-validation scores. The stratified fivefold AUC was 0.873 (CI = 0.830–0.917) of the trained model (Figure 2(a)). The optimal ROC threshold for the final model was 0.5, and the sensitivity and specificity were 42.1% and 98.1%, respectively. In the validation set, the AUC was 0.842 (CI = 0.752–0.928; Figure 2(b)), and the sensitivity and specificity were 70.6% and 97.7%, respectively. Other evaluation metrics of models are shown in Table 3. The calibration plot and histogram showed favorable results that the predicted probabilities were close to observed delirium (Figure 3).
Feature importance plot of the machine learning model with 95% confidence interval reported as error bars.
The AUROC of the training set (a) and validation set (b) AUROC, area under the receiver operating characteristic.
The calibration plot (a) and histogram (b) of the machine learning model.
Significant differences in clinical characteristics between delirium group and non-delirium group.
CCI, Charlson comorbidity index; BMI, body mass index; ECMO, extracorporeal membrane oxygenation.
*p < 0.05, **p < 0.01, ***p < 0.001.
Selected features and odds ratio in training set and validation set.
*p < 0.05, **p < 0.01, ***p < 0.001.
Various evaluation metrics for the machine learning model. Each metric is the mean value with 95% confidence interval.
CI, confidence interval; AUC, area under the curve.
Discussion
Compared with a recent meta-analysis that reported a 24.3% of pooled prevalence rate for delirium in COVID-19 patients, 26 the rate was considerably lower than expected in our study, at 6.5%. This low prevalence could be attributed to the clinical setting of treating COVID-19 patients in South Korea, especially during the early stages of the pandemic. All patients with COVID-19 infection were hospitalized in medical centers regardless of the severity of illness, meaning that the majority of inpatients had mild symptoms. Indeed, the spectrum of clinical manifestations due to COVID-19 infection is quite varied. Another possible explanation for this discrepancy is that the previous studies mostly included elderly patients, whereas we included adults of all ages. The exclusion of patients diagnosed with dementia may also have contributed to the low incidence of delirium in our study. Dementia is a well-known risk factor for delirium; thus, the exclusion of these patients may have reduced the incidence of delirium and biased our sample. As dementia and delirium have overlapping symptoms, including memory problems and disorientation, 27 differential diagnosis can be challenging, particularly in retrospective review studies. Prospective studies are needed to clarify the effects of dementia on the likelihood of delirium in patients with COVID-19.
We constructed an ML algorithm-based model to predict delirium using the EHR data of inpatients diagnosed with COVID-19. Among various ML models including decision tree, random forest, and logistic regression, we chose XGBoost due to its superior and stable performance with changing variables. XGBoost, a decision-tree-based ensemble model leveraging a gradient boosting framework, appears to overcome limitations of previous models, and many prior ML studies employing XGBoost have demonstrated good performance.28,29 We selected nine input variables for the predicting model, including mechanical ventilation, sodium ion concentration, and the use of various psychotropic drugs and medications to treat COVID-19 symptoms. The AUC of the model was 0.873, and this model was internally validated (AUC = 0.842). ML algorithms learn patterns from data to make inferences and predict outcomes. These types of predictive models are considered to be useful especially when large datasets or several complex predictors are involved. 30 ML-based models also can accommodate missing data using advanced techniques such as the XGBoost algorithm.23,24 In spite of these strengths, ML-based predictive models still have some limitations. This model was developed based on data collected from four independent institutions using the same protocol during the COVID-19 pandemic at the same point in time. This aspect enhances its performance and generalizability for clinical use across various medical settings. Our model intends to automatically calculate their risk of developing delirium during hospitalization by using each patient's medical information. Furthermore, this model predicts individual risk for delirium to identify high-risk patients who should be prioritized for healthcare resources, and it also suggests factors that can be adjusted to reduce the risk within a few hours of admission, i.e. after EHRs are obtained. 24
Our delirium prediction model revealed several influential factors for delirium in patients with COVID-19, some of which can be modified. The identification of possible factors that can be used in a delirium prevention protocol is the major strength of our study. Normalizing sodium concentration can be beneficial, and previous research has also shown that electrolyte abnormalities, including hyponatremia, are known risk factors for delirium. 31 However, the impact of each medication on delirium is complex, as it may act as both a risk factor and a protective factor, mediated by the improvement of underlying conditions. Therefore, we did not categorize variables as either risk or protective factors during the learning process of the model. Instead, we examined the overall effect of each variable, and some were found to increase delirium while others decreased it. We found that psychotropic medications, including antipsychotics and sedatives are associated with the onset of delirium in COVID-19 inpatients. Benzodiazepines (e.g. alprazolam, lorazepam, and midazolam) are sedatives associated with sleep problems and anxiety in intensive care unit patients, which consequently makes them vulnerable to delirium.32,33 In addition to the pharmacological effects of sedatives, these medications may be administered to severely ill COVID-19 patients more prone to delirium than those with milder disease. Antipsychotics are frequently used to treat the symptoms of delirium because they are highly effective for agitation in delirium 34 ; however, a recent cohort study found that use of antipsychotics was associated with a higher risk of delirium in patients with COVID-19. 13 Indeed, the challenge of classifying antipsychotic use as either a predictor or part of delirium management cannot be easily addressed by considering antipsychotic use solely before delirium onset. In instances of subsyndromal delirium, antipsychotics are often prescribed even when symptoms do not meet the severity criteria for a delirium diagnosis. 35 This issue could be better addressed through a prospective study that includes daily screening of delirium symptoms.
A considerable number of medications related to COVID-19 are already known to be associated with delirium, having been previously used for other infectious diseases or respiratory conditions. 5 Although antibiotics such as ceftriaxone and piperacillin/tazobactam also contribute to delirium, 36 the association between these medications and delirium may depend more on the severity of the patient's condition than the direct actions of the drugs. Nevertheless, our analysis adjusted for the effects of illness severity by controlling variables such as mechanical ventilation, radiological findings, and vital signs, implying that medications positively associated with delirium are targets for preventative interventions. Moreover, antibiotics were generally not administered alone. As interactions between multiple drugs or a high prevalence of comorbidities that require multiple medications are risk factors for delirium, 37 our finding of an association between antibiotic use and delirium suggests that modifying these medications could aid the management of delirium. Acetaminophen was associated with a lower risk of delirium in this study. It has been recommended for managing fever of pain in COVID-19 patients, 38 this effect for fever and pain could be associated with a reduced risk of delirium. In contrast, propacetamol was associated with an increased risk of delirium. Such contradictory results may not only be due to the influence of the drug itself but also relate to the complex state of the patients. For instance, propacetamol is primarily administered as an intravenous formulation to more severe patients who cannot take oral medication. Although we adjusted for other variables such as patient severity, there is a possibility that these patient factors contributed to these results. Therefore, the actual clinical association between acetaminophen and delirium could be more complex and contribute differently depending on the patient's condition. 39 Ambroxol is primarily a mucolytic drug acting on the respiratory system, with little known about its central nervous system effects. However, literatures suggest that ambroxol's anti-inflammatory potency and its modulation effects on sodium channels can influence neurotransmission and neuronal degeneration. 40 Therefore, it is worthwhile to further explore the potential relations of acetaminophen and ambroxol with delirium in future studies.
Our study had several limitations. Despite our attempts to overcome the limitations imposed by the small sample size and the low incidence of delirium, concerns regarding sensitivity, positive predictive value, and overfitting persist. Due to the study's modest sample size and lower-than-expected prevalence, external validation was not feasible, which necessitates future validation efforts with larger, independent cohorts. A second limitation of our study was that we were unable to establish causality between factors associated with delirium and the incidence thereof. Although we concluded that modifiable factors, such as certain psychotropic drugs and COVID-19 medications, are closely related to the development of delirium, our findings are not sufficient to definitively prove that modification of these factors will reduce the likelihood of delirium. Randomized controlled trials or studies with detailed pre-admission medication data are required to determine causal relationships between modifiable medication factors and delirium incidence. Third, given the study's focus on medication impact, other variables were measured on admission day, while medication use spanned the entire hospital stay. This approach may introduce bias, so caution is advised when interpreting the results. The last limitation includes the use of retrospective data for the ML-based delirium model, potentially introducing bias as robust risk factors like dementia and frailty were omitted due to isolation admission circumstances. Future prospective studies are necessary to minimize bias and maximize predictive power.
Conclusion
We constructed a ML model to predict the development of delirium in hospitalized patients with COVID-19. The model included nine primary variables including modifiable factors as various psychotropic medications and drugs targeting COVID-19 symptoms. In this study, we presented a model for predicting delirium in patients experiencing exceptional circumstances, such as isolation during unexpected events like the COVID-19 pandemic. Clinically, such models enable the individual risk prediction of hospitalized patients, allowing for a more concentrated allocation of medical resources to high-risk patients. Moreover, the influential factors presented in this study can serve as the basis for subsequent research related to prevention. While additional refinements may be necessary, ML-based models have the advantage of continuous performance enhancement as more data is gathered. Further, despite the decline in COVID-19 prevalence, the potential for sudden outbreaks of new isolation-requiring diseases like Middle East respiratory syndrome coronavirus or severe acute respiratory syndrome-associated coronavirus persists, underscoring the importance of strategies for predicting and preventing delirium in high-risk groups. Although excluded from the key features of the final model, this study incorporated specific variables related to COVID-19 isolation treatment that were not addressed in previous studies. For example, we included factors such as the presence of windows in the patient's room, access to medical care, and the presence of remote contact with caregivers. Therefore, studies on relevant evidence remain vital and can be extended to delirium prevention in other isolation-required respiratory infections, such as tuberculosis.
Supplemental Material
sj-docx-1-dhj-10.1177_20552076231223811 - Supplemental material for Predicting delirium and the effects of medications in hospitalized COVID-19 patients using machine learning: A retrospective study within the Korean Multidisciplinary Cohort for Delirium Prevention (KoMCoDe)
Supplemental material, sj-docx-1-dhj-10.1177_20552076231223811 for Predicting delirium and the effects of medications in hospitalized COVID-19 patients using machine learning: A retrospective study within the Korean Multidisciplinary Cohort for Delirium Prevention (KoMCoDe) by So Hee Lee, Hyun Jung Hur, Sung Nyun Kim, Jang Ho Ahn, Du Hyun Ro, Arum Hong, Hye Yoon Park, Pyoeng Gyun Choe, Back Kim and Hye Youn Park in DIGITAL HEALTH
Footnotes
Acknowledgements
We are grateful for technical support by Korean Multidisciplinary Cohort for Delirium prevention research group (KoMCoDe).
Contributorship
HYP has received funding and conceived the study. SNK and PGC were involved in data acquisition. HYP, JHA, BK and DHR analyzed the data and interpreted results. AH was involved in gaining ethical approval of this study. LSH and HHJ interpreted results and wrote the draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version of this manuscript.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Ethical approval
The study protocol was approved by the institutional review board of each medical center and the requirement for informed consent was waived (joint approval no. 2103-675-302).
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by a grant of Patient-Centered Clinical Research Coordinating Center (PACEN) funded by the Ministry of Health & Welfare, Republic of Korea (grant number : HC20C0040).
Guarantor
HYoun Park
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
