Abstract
Globus pallidus necrosis (GPN) is one of typical neurological imaging features in patients with carbon monoxide (CO) poisoning. Current clinical guideline recommends neurological imaging examination for CO-intoxicated patients with conscious disturbance rather than routine screening, which may lead to undiagnosed GPN. We aimed to develop an artificial intelligence algorithm for predicting GPN in CO intoxication patients. We included CO intoxication patients with neurological images between 2000 and 2019 in Chang Gung Memorial Hospital. We collected 41 clinical and laboratory parameters on the first day of admission for algorithm development. We used fivefold cross validation and applied several machine learning algorithms. Random forest classifier (RFC) provided the best predictive performance in our cohort. Among the 261 patients with CO intoxication, 52 patients presented with GPN. The artificial intelligence algorithm using the RFC-based AI model achieved an accuracy = 79.2 ± 2.6%, sensitivity = 77.7%, precision score = 81.9 ± 3.4%, and F1 score = 73.2 ± 1.8%. The area under receiver operating characteristic was approximately 0.64. Top five weighted variables were Platelet count, carboxyhemoglobin, Glasgow Coma scale, creatinine, and hemoglobin. Our RFC-based algorithm is the first to predict GPN in patients with CO intoxication and provides fair predictive ability. Further studies are needed to validate our findings.
Introduction
Carbon monoxide (CO) poisoning is a major global public health problem. An estimation based on the Global Health Data Exchange registry showed 137 patients per million would suffer from CO intoxication and 4.6 CO-intoxicated patients per million would be fatal in 2017. 1 Such problem have become more severe since the beginning of the 21st century when the media widely reported charcoal burning as new painless suicide method. 2 In consequence, charcoal-burning suicide rate increased dramatically, especially in Hong Kong, Taiwan, Japan, Korea, and Singapore. 3 In fact, the incidence of charcoal-burning suicide increased 29.5-fold in 2006 compared to in 1999 and 33.5% of suicide deaths in Taiwan were caused by this single suicide method. 4 In addition to CO intoxication caused by charcoal-burning suicide, the incidence of accidental CO poisoning has also increased due to usage of indoor water heaters during cold weather. 5 Therefore, CO intoxication is an issue that should not be ignored by clinicians.
Delayed neuropsychiatric syndrome (DNS) is one of the most debilitating sequelae after CO poisoning. 6 Survivors may still experience cognitive impairment, akinetic mutism, sphincter incontinence, gait ataxia and extrapyramidal syndromes even after the acute stage. 7 Imaging tools such as computer tomography (CT) and magnetic resonance imaging (MRI) have been developed to stratify high-risk CO-intoxicated patients that would develop DNS. Currently, the pathophysiology of CO-related DNS has not been fully elucidated. Several mechanisms, including impaired oxygen delivery, impaired oxygen utilization, oxidative stress with lipid peroxidation, and ischemia-reperfusion injury have been proposed for CO-related brain injury.6,8–12 Hypoxic injury caused by impaired oxygen delivery and utilization are attributed to hemoglobin preferentially binding to CO instead of oxygen, prevention of oxygen release from hemoglobin into hypoxic environment, and shutdown of electron transportation chain due to CO binds to heme a3 subunit two of cytochrome C oxidase in mitochondria.6,13,14 Notably, globus pallidus may be more susceptible to CO intoxication due to high iron content, high oxygen consumption, and poor anastomotic blood supply.15,16 Globus pallidus necrosis (GPN) is a common brain lesion and associated with the development of DNS, which presents as symmetric hypodensity over globus pallidus in CT and low intensity on T1-weighted images but high intensity on T2-weighted images over medial portion of the globus pallidus on MRI.17–20 Current guideline recommends brain CT or MRI for CO-intoxicated patients with loss of consciousness rather than routine screening. 21 However, typical neurological imaging finding may still occur in asymptomatic CO-intoxicated patients. 22 A point-of-care clinical prediction tool may help clinician efficiently sort out patients who are more likely to have GPN before performing the neuroimaging.
There has been tremendous interest regarding the utility of artificial intelligence (AI) in medical research. The utility of AI in the field of acute kidney injury (AKI) has been studied not only in bedside application, but also in improving our understanding of the disease. 23 Different AI algorithms have been utilized for different medical research with various degrees of accuracy, including toxicology studies.24–27 AI prediction has the advantage of incorporating multiple parameters to improve predicative ability. Currently, there is no literature utilizing AI for predicting GPN in CO intoxication patients. We aim to determine the feasibility of using AI for predicting GPN in CO intoxication patients and determine which algorithm provides the best predictive ability.
Materials and methods
Ethics statement
This study complied with the guidelines of the Declaration of Helsinki and was approved by the Medical Ethics Committee of Chang Gung Memorial Hospital, Linkou, Taiwan. The institutional review board number was 201701109B0.
Patients
We included all patients with CO intoxication treated at Chang Gung Memorial Hospital between 2000 and 2019. Chang Gung Memorial Hospital is Taiwan's largest tertiary medical center, with 3700 beds and a specialized clinical poison center. The patients with available neurological imaging were included in the development and validation of artificial intelligence algorithms. We included a total of 261 patients in the cohort; the medical records of these patients, including demographic, clinical parameters, and laboratory data were integrated into the analysis.
Clinical management
The diagnosis of carbon CO intoxication was based on the exposure history and a blood carboxyhemoglobin level of more than 10% for smokers or 3% for non-smokers. 28 Supportive treatment and oxygen therapy with non-rebreathing facemask were provided for all CO intoxication patients. Due to hyperbaric oxygen (HBO) therapy is not covered in health reimbursement in Taiwan, the indication for HBO was not standardized in all patients. However, HBO therapy would not provide for patients with contraindications such as untreated pneumothorax, asthma, chronic obstructive pulmonary disease, seizure, pregnancy, high fever, upper respiratory tract infections, active malignancy, congenital spherocytosis, claustrophobia, and patients need intensive care.
Definitions of globus pallidus necrosis
The diagnosis of globus pallidus necrosis was based on typical radiographic imaging characteristics. 18 The typical findings included symmetric hypodensity over globus pallidus in computer tomography, and low intensity on T1-weighted but high intensity on T2-weighted images over the bilateral globus pallidus area in MRI.
Statistical analysis
Data are expressed as the mean ± standard deviation or number (percentage), unless otherwise stated. t test was used to compare the means of continuous variables and normally distributed data. Categorical data was analyzed using the chi-square test. All statistical tests were two-tailed, with p values < .05 considered statistically significant. Data were analyzed with SPSS 25.0 software for Windows (SPSS, Inc., Chicago, IL, USA).
Data collection and feature engineering
The algorithm development process included three steps. First, demographics, vital signs, laboratory values, interventions, medications, and nurse documentation were accessed through the hospital information system of Chang Gung Memorial Hospital. The data collected is presented in the case report form (Supplementary Table S1). For missing values in the other recording data, we filled them with the median values of each corresponding feature. This process guarantees the quality and consistency of the data as well as the model. Next, we conducted standard normalization of every feature, alleviating the influence of a different range of values. We selected 41 clinical and laboratory features recorded on the first day of admission for use in algorithm development. Third, during feature engineering, the 261 patients were divided into training data (n = 209) and testing data (n = 52). The RFC-based AI model was developed using 80% of the records as the training cohort and 20% of the records as the testing cohort. All development and analyses associated with AI were performed using Jupyter Notebook Version 6.4.8.
Model selection
We applied several machine-learning methods, such as random forest classifier (RFC), K-nearest neighbor classification (KNN), logistic regression (LR), support vector machine, eXtreme gradient boosting (XGB), and neural network, and used fivefold cross-validation for the assessment of model performance. Cross-validation is a resampling procedure used to evaluate machine learning models, and “5” refers to the number of groups the given data sample was split into. We found that RFC, a decision-tree-based algorithm had the best performance. Therefore, we adopted RFC and LR for the following artificial intelligence algorithm development.
Performance evaluation
After we used random forest classifier and logistic regression to fit the training dataset and obtained the probability threshold, we computed the globus pallidus necrosis probability of the testing dataset as well as the accuracy, recall (sensitivity), specificity, precision and F1 score given that specific threshold. The F1 score provides an equal consideration to both precision and recall by calculating their harmonic mean into a single metric. 29 More importantly, we computed the receiver operating characteristic (ROC) curve for performance evaluation. We also performed a feature weight analysis to differentiate the contribution of each feature to the performance of the algorithm to evaluate the algorithm’s fitness for the clinical scenario. Feature weighting is closely associated with the AUROC, sensitivity, and precision of an algorithm and can be obtained during cross-validation in model training.
Results
Study population characteristics
Clinical data of patients with carbon monoxide intoxication, stratified according to the status of globus pallidus necrosis (GPN) (n = 261).
AKI: acute kidney injury; ALT: alanine aminotransferase; AMI: acute myocardial injury; AST: aspartate aminotransferase; BE: base excess; COHb: carboxyhemoglobin; GCS: glascow coma scale; HBO: hyperbaric oxygen; HCO3: bicarbonate; PCO2: partial pressure of carbon dioxide; PO2: partial pressure of oxygen; WBC: white blood cell.
The performance of developed AI algorithms
Identification of top ten weighted features for globus pallidus necrosis (GPN) using artificial intelligence approach.
COHb: carboxyhemoglobin; GCS: glascow coma scale; PO2: partial pressure of oxygen; RBC: red blood cell; WBC: white blood cell.

Ranking the weight of each variable in the random forest classifier algorithm. AMI: acute myocardial injury; ARF: acute renal failure; BE: base excess; COHb: carboxyhemoglobin; CVSym: cardiovascular symptoms; DM: diabetes mellitus; GCS: glascow coma scale; GIsym: gastrointestinal symptoms; HBO: hyperbaric oxygen; HCO3: bicarbonate; hepatitisALT2X: patients with alanine aminotransferase increase more than 2 times of upper limit; ICU: intensive care unit admission; K: potassium; MusculoSym: musculoskeletal symptoms; neurologicalD: baseline neurological comorbidities; NeuroSym: neurological symptoms; PCO2: partial pressure of carbon dioxide; PH: blood pH value; PO2: partial pressure of oxygen; RBC: red blood cell; renalD: baseline renal comorbidities; ResFailure: respiratory failure; respiratoryD: baseline respiratory comorbidities; WBC: white blood cell bicarbonate; PCO2: partial pressure of carbon dioxide; PO2: partial pressure of oxygen; WBC: white blood cell.
Discussion
To our best knowledge, this is the first study to predict globus pallidus necrosis after carbon monoxide intoxication using RFC-based artificial intelligence. Our results demonstrated that the established algorithm can achieve an accuracy of 79.2% and precision of 81.9%. The top five weighted variables for predicting GPN in our algorithm are platelet count, carboxyhemoglobin, Glasgow Coma scale, creatinine, and hemoglobin (Figure 1 and Table 2).
The development of neurological sequela is a long-term burden for survivors of CO intoxication. Ku et al. reported 59.7% of patients developed GPN after CO intoxication in Taiwan, but the incidence rate ranging from 18.7% to 100% has been reported by different studies.20,30,31 Current guideline from Centers for Disease Control and Prevention recommends brain imaging workup for CO-intoxicated patients with loss of consciousness or cardiopulmonary signs and symptoms. 21 However, typical neurological imaging findings may still occur in asymptomatic CO-intoxicated patients. 22 Considering DNS is associated with GPN, identification of those who are at higher risk of developing GPN is crucial for CO intoxication patients. Although several predictors have been investigated by previous group,30,32 no AI-based computerized tools have been developed to predict GPN until this study. Our algorithm provides an adequate prediction ability to predict GPN with clinical characteristics that can be easily acquired by clinicians and therefore the implementation into clinical practice would be easy. The features used in this AI algorithm are clinical or laboratory data obtained during the routine management of CO intoxication patients, without imposing any additional burden on frontline healthcare providers in the emergency department. Furthermore, by inputting these variables, the AI algorithm can provide a recommendation to clinicians on whether a neuroimaging examination is necessary within minutes, thereby alleviating the clinical workload for busy healthcare providers. Additionally, this algorithm has also opened up possibilities for future AI-related research in toxicology. It has confirmed the feasibility of using clinical features to predict toxicological outcomes through AI, paving the way for further studies in this area. Future studies are required to validate our results.
Platelet count, carboxyhemoglobin, GCS, creatinine, and hemoglobin are the top five weighted variables in the RFC model for GPN prediction. Despite our incomplete understanding of why these variables are weighted in this particular order in our algorithm, they have all been either reported to be predictors for GPN or pathophysiological related.30,32,33 For example, creatinine has been reported as a predictor for GPN in charcoal-burning CO intoxication patients during univariate analysis. 30 In addition, higher platelet count has been reported in patients with CO intoxication compared to those without. 33 CO displaces NO from platelet surface hemoproteins and the displaced NO subsequently forms peroxynitrite, which leads to platelet activation.34,35 Increased thrombotic events such as deep vein thrombosis have been reported in patients with CO intoxication.36–38 Platelet activation also elicits an inflammatory effect via neutrophil activation. The level of cerebrospinal fluid myelin basic protein, a product of lipid peroxidation after neutrophil activation, is significantly higher in CO-intoxicated patients with DNS compared to those without.39,40 Moreover, a meta-analysis including 2328 patients from 10 studies demonstrated that patients with initial low GCS score had higher risk for developing DNS compared to those with higher GCS score. 41 Future studies may explore the different contributions of each variable to GPN.
CO exhibits an affinity for hemoglobin that is 200 times higher than that of oxygen, resulting in the formation of carboxyhemoglobin and causing a leftward shift in the oxyhemoglobin dissociation curve. 42 Carboxyhemoglobin level is important for the diagnosis of CO intoxication but the level can be affected by the alveolar ventilation, blood volume, metabolic activity, smoking and duration of exposure. 42 While a carboxyhemoglobin level above 50% is generally considered fatal, it does not reliably indicate the severity of systemic effects, poor prognosis, or the development of DNS.42,43 The disparity between carboxyhemoglobin levels and the extent of CO poisoning could be attributed to variations in the time interval between patients' exposure to CO and their arrival at the hospital. 44 In other word, patients who experience lower carboxyhemoglobin levels but develop GPN might have experienced a longer delay in reaching the hospital and consequently had a lengthier exposure to ambient oxygen. However, obtaining the exact time of CO intoxication is challenging, if not impossible, as is the case in all real-world studies of CO poisoning. 45 Therefore, prediction of development GPN solely based on carboxyhemoglobin level is unreliable and our algorithm based on multiple clinical features can more effectively predict the occurrence of GPN.
Although not weighted in the top five, other previous reported predictors including acute myocardial injury, serum bicarbonate concentration, and age are also moderately weighted in our model. Patient’s underlying comorbidities, such as neurological disease, respiratory disease and renal disease, are also weighted in our model. The advantage of using AI is the ability to take all these variables into one prediction algorithm to improve predictive ability.
Our method of using an AI algorithm in analyzing intoxication patients has several advantages compared to using traditional statistics. First, all variables used in our algorithm are clinically relevant data and would be rapidly acquired while CO-intoxicated patients presented to the emergency department. In addition, the established algorithm is ready for clinical utility and swiftly predicts the risk of GPN. Implementation into clinical practice is easy. Moreover, the algorithm uses feature weighting to approximate the influence of individual features during AI classifier development, there is no need to manually discard variables as in the traditional statistical methods. Finally, the predictive power using AI is based on multiple variables rather than single variable as in traditional statistical method, which may provide better predictive ability. Although promising, our study is limited by the small sample size. Another limitation is that our study cannot extrapolate the relationship or causality among all clinical and laboratory features. Further studies are needed to validate our findings.
Conclusion
Our AI algorithm using the RFC method demonstrates fair ability to predict GPN in patients with CO intoxication. To our knowledge, this is the first AI-based prediction tool for patients with CO intoxication. The top five weighted features for GPN are Platelet count, carboxyhemoglobin, GCS, creatinine, and hemoglobin.
Supplemental Material
Supplemental Material - An artificial intelligence algorithm for analyzing globus pallidus necrosis after carbon monoxide intoxication
Supplemental Material for An artificial intelligence algorithm for analyzing globus pallidus necrosis after carbon monoxide intoxication by Ming-Jen Chan, Ching-Chih Hu, Wen-Hung Huang, Ching-Wei Hsu, Tzung-Hai Yen, and Cheng-Hao Weng in Human & Experimental Toxicology.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by grants from Chang Gung Memorial Hospital Research Program (CMRPG5L0201 and CMRPG3L0991).
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
