Identifying metabolic dysfunction-associated steatotic liver disease in patients with hypertension and pre-hypertension: An interpretable machine learning approach

Abstract

Objective

Metabolic dysfunction-associated steatotic liver disease (MASLD) is one of the most prevalent liver diseases and is associated with pre-hypertension and hypertension. Our research aims to develop interpretable machine learning (ML) models to accurately identify MASLD in hypertensive and pre-hypertensive populations.

Methods

The dataset for 4722 hypertensive and pre-hypertensive patients is from subjects in the NAGALA study. Six ML models, including the decision tree, K-nearest neighbor, gradient boosting, naive Bayes, support vector machine, and random forest (RF) models, were used in this study. The optimal model was constructed according to the performances of models evaluated by K-fold cross-validation (k = 5), the area under the receiver operating characteristic curve (AUC), average precision (AP), accuracy, sensitivity, specificity, and F1. Shapley additive explanation (SHAP) values were employed for both global and local interpretation of the model results.

Results

The prevalence of MASLD in hypertensive and pre-hypertensive patients was 44.3% (362 cases) and 28.3% (1107 cases), respectively. The RF model outperformed the other five models with an AUC of 0.889, AP of 0.800, accuracy of 0.819, sensitivity of 0.816, specificity of 0.821, and F1 of 0.729. According to the SHAP analysis, the top five important features were alanine aminotransferase, body mass index, waist circumference, high-density lipoprotein cholesterol, and total cholesterol. Further analysis of the feature selection in the RF model revealed that incorporating all features leads to optimal model performance.

Conclusions

ML algorithms, especially RF algorithm, improve the accuracy of MASLD identification, and the global and local interpretation of the RF model results enables us to intuitively understand how various features affect the chances of MASLD in patients with hypertension and pre-hypertension.

Keywords

Metabolic dysfunction-associated steatotic liver disease hypertension pre-hypertension interpretable machine learning SHAP

Introduction

Metabolic dysfunction-associated steatotic liver disease (MASLD), formerly known as nonalcoholic fatty liver disease, is defined as the presence of hepatic steatosis along with one cardiometabolic risk factor, and no other secondary causes.¹ It originates from increased lipid storage in more than 5% of primary hepatocytes.² MASLD has become one of the most prevalent liver diseases, with a global prevalence of about 30%.^3,4 In recent years, due to rapid changes in dietary structure and lifestyle, the incidence of MASLD has been increasing year by year.⁵ In addition, if not intervened promptly, MASLD will gradually develop into liver fibrosis or even liver cancer.^6,7 These will pose a significant threat to human health and have widespread social and economic impacts.⁸

Liver biopsy has always served as the definitive diagnostic method for MASLD.⁹ Nevertheless, because of its invasive nature and high risk, it cannot be widely used for clinical diagnosis and general population screening. In addition, inconsistencies in the observation and interpretation of liver biopsy specimens by different clinical-pathological diagnosticians can result in diagnostic discrepancies.^9,10 Consequently, there is an urgent need to find an assistant method that can effectively replace liver biopsy while maintaining a high level of accuracy in identifying MASLD.

Recently, due to the development of statistical theory and computer technology, the application of machine learning (ML) in clinical practice has received widespread attention and recognition.¹¹ ML algorithms enable computers to autonomously acquire knowledge, analyze data, and build predictive models, thereby facilitating the prediction of previously unknown data.¹² The new ML technology has been widely used in diverse disease prediction models, showcasing remarkable performances.¹³

MASLD is significantly correlated with blood pressure levels.¹⁴ Research has shown that even without other metabolic risk factors, early hypertension may lead to MASLD, and hypertension is also an independent risk factor for MASLD.¹⁵ In addition, controlling blood pressure may help prevent or limit MASLD in hypertensive patients.¹⁵ Pre-hypertension increases the risk of hypertension and cardiovascular events,¹⁶ while hypertension is associated with cardiovascular and all-cause mortality in MASLD patients.¹⁷

However, there is currently a lack of relevant research on identifying MASLD in pre-hypertensive and hypertensive populations. Considering the advantages of ML, our research aims to develop interpretable ML models using data from pre-hypertensive and hypertensive patients in the NAGALA study to effectively identify MASLD.

Methods

Study populations and data collection

This research is a secondary analysis of data from subjects in the NAGALA study from 2004 to 2015 in a cross-sectional design. The purpose of the NAGALA study was to conduct an epidemiological study of diabetes and fatty liver disease using the examination data of the general population who underwent health checkups at Murakami Memorial Hospital in Japan. The previous study was approved by the Ethics Committee of Murakami Memorial Hospital and authorized with informed consent from the subjects.¹⁸ This longitudinal cohort data has been uploaded to the Dryad database for public sharing by Okamura et al.¹⁹ According to the data sharing policy, all researchers are free to download the original data and use the data for secondary analysis without harming the rights and interests of the authors. Following the policy, we needed to cite data sources when using these data.¹⁹

Based on our research objectives, patients with liver diseases (alcoholic fatty liver disease and viral hepatitis), normal blood pressure [systolic blood pressure (SBP) < 120 mmHg and diastolic blood pressure (DBP) < 80 mmHg], type 2 diabetes or fasting blood glucose over 6.1 mmol/L, drugs use, heavy drinking habits (over 140 g/w for females or 210 g/w for males), and missing data were excluded in our study.

As previously described,¹⁸ age, gender, exercise habits, smoking status, and drinking habits were documented using standardized questionnaires. Exercise habits were defined as engaging in any form of physical activity more than once a week. Drinking status was categorized as no drinking or small (<40 g/w) intake, light (40–139 g/w) intake, and moderate (140–209 g/w) intake based on weekly alcohol consumption. The smoking status was classified as non-smoking, past smoking, and current smoking by querying subjects about their smoking history before the baseline survey. Height, weight, waist circumference (WC), SBP, and DBP were measured using standard methods by trained clinicians. Following an overnight fast, the venous blood samples of the subjects were collected, and the biochemical parameters were analyzed by an automatic analyzer, including alanine aminotransferase (ALT), aspartate transaminase (AST), γ-glutamyltransferase (GGT), high-density lipoprotein cholesterol (HDL-C), total cholesterol (TC), triglyceride (TG), fasting plasma glucose (FPG), and hemoglobin A1c (HbA1c).

Diagnosis, definition, and calculation

Experienced gastroenterologists evaluated and made a diagnosis by abdominal ultrasound without knowing the purpose of the study and the clinical information of the subjects. MASLD is diagnosed if the total score of blurred blood vessels (0–1 points), liver brightness (0–4 points), deep attenuation (0–2 points), and liver and kidney echo contrast (0–4 points) exceeds 2.²⁰

According to the Japanese Hypertension Society Hypertension Management Guidelines (JSH 2019), hypertension is defined as SBP ≥ 140 and/or DBP ≥ 90 mmHg; elevated blood pressure, SBP 130–139 and/or DBP 80–89 mmHg; and high normal blood pressure, SBP 120–129 mmHg and DBP < 80 mmHg. In this study, elevated blood pressure and high normal blood pressure were defined as pre-hypertension.²¹

In the present study, body mass index (BMI) was calculated as weight (kg)/[height (m)]², and LDL-C was calculated as 0.9 × non-HDL-C (mg/dL) − 0.1 × TG (mg/dL) [non-HDL-C = TC (mg/dL) − HDL-C (mg/dL)].²²

Preprocessing, supervised ML, and cross-validation

The quantitative variables of the collected dataset were standardized, scaled to the unit variance, and normalized to the unit norm by using standard ML practices. Binary representations were created from categorical variables by using a label encoder. The original dataset was randomly divided into training and test sets at a ratio of 7:3.

After data preprocessing, the chances of MASLD were modeled as outcomes for prediction. Six ML models were used in this study, including the decision tree (DT) model,²³ K-nearest neighbor (KNN) model, extreme gradient boosting (XGB) model,²⁴ support vector machine (SVM) model,²⁵ random forest (RF) model, and naive Bayes (NB) model. Notably, there is an inherent sparsity of diseased cases in the dataset and to enhance the classifier's characterization capability of sparse samples, a weighted loss approach was employed during the model training to tackle the issue of data imbalance. K-fold cross-validation (k = 5) was performed before model training, with the study dataset randomly divided into the internal model validation set (75%) to ensure robust performance, and the model evaluation set (25%) for final model evaluation. For the internal validation set, models were trained using each of the 4 folds as train data and subsequently validated using the remaining fold of data.²⁶ K-fold cross-validation results were used to evaluate the models’ stability and robustness in complex data. The model with the best K-fold cross-validation results was ultimately implemented in the proposed dataset.²⁷ Model predictive capacities were assessed using the area under the receiver operating characteristic curve (AUC), average precision (AP), accuracy, sensitivity, specificity, and F1. Data preprocessing, supervised ML modeling, and cross-validation were computed using the scikit-learn 1.0.2 package in Python 3.8.

Feature importance, clustering, and interaction effects with Shapley additive explanation (SHAP) values

The SHAP values were calculated to interpret the results of the RF model.^28,29 Hierarchical clustering was performed using the SHAP value feature weightings to compare the feature similarities and differences in the population subsets across the study populations. The SHAP interaction values were calculated to quantitatively study the interaction effects between features. All SHAP values were computed using the SHAP 0.41.0 package in Python 3.8.

Results

After applying the inclusion and exclusion criteria, 4722 patients were ultimately selected for analysis (Supplemental Figure S1). A total of 1469 (31.1%) participants were diagnosed with MASLD. The prevalence of MASLD in hypertensive patients and pre-hypertensive patients was 44.3% (362 cases) and 28.3% (1107 cases), respectively. Compared with the non-MASLD controls, MASLD patients had higher BMI, WC, SBP, DBP, ALT, AST, GGT, HDL-C, LDL-C, TC, TG, and FPG levels, as well as a higher proportion of males. Additionally, they had lower levels of HDL-C and engaged in less physical exercise (all P < 0.05) (Table 1). Figure 1(a) intuitively illustrates the patients with and without MASLD in the reduced three-dimensional space. The overall data distribution demonstrates a distinct separation boundary between two categories: MASLD and non-MASLD. The correlation analysis of all variables in the dataset is shown in Figure 1(b).

Figure 1.

Visualization of the dataset. (a) The three-dimensional scatter plot of all data in the dataset. (b) Correlation matrix for all features in the dataset. MASLD, metabolic dysfunction-associated steatotic liver disease; BMI, body mass index; WC, waist circumference; SBP, systolic blood pressure; DBP, diastolic blood pressure; ALT, alanine aminotransferase; AST, aspartate aminotransferase; GGT, gamma-glutamyl transferase; HDL-C, high-density lipoprotein cholesterol; LDL-C, low-density lipoprotein cholesterol; TC, total cholesterol; TG, triglyceride; FPG, fasting plasma glucose; HbA1c, hemoglobin A1c.

Table 1.

Baseline characteristics of the study populations.

	Non-MASLD (N = 3253)	MASLD (N = 1469)	P-Value
Age (year)	45 (38, 53)	43 (38, 51)	<0.001
Sex (n, %)			<0.001
Male	2019 (62.1)	1234 (84.0)
Female	1234 (37.9)	235 (16.0)
BMI (kg/m²)	22.4 (20.7, 24.3)	25.8 (24.0, 28.2)	<0.001
WC (cm)	78.0 (72.6, 83.0)	87.0 (82.1, 92.2)	<0.001
Exercise habits (n, %)			<0.001
Yes	608 (18.7)	206 (14.0)
No	2645 (81.3)	1263 (86.0)
Drinking status (n, %)			0.001
Non/small	2463 (75.7)	1182 (80.5)
Light	534 (16.4)	187 (12.7)
Moderate	256 (7.9)	100 (6.8)
Smoking status (n, %)			<0.001
None	1859 (57.2)	722 (49.1)
Past	739 (22.7)	393 (26.8)
Current	655 (20.1)	354 (24.1)
SBP (mmHg)	127.0 (123.0, 133.0)	130.5 (124.5, 137.5)	<0.001
DBP (mmHg)	80.5 (76.5, 84.5)	82.5 (78.5, 88.0)	<0.001
ALT (U/L)	17.0 (13.0, 22.0)	28.0 (21.0, 41.0)	<0.001
AST (U/L)	18.0 (14.0, 21.0)	21.0 (17.0, 27.0)	<0.001
GGT (U/L)	16.0 (12.0, 23.0)	24.0 (18.0, 36.0)	<0.001
HDL-C (mmol/L)	1.4 (1.2, 1.7)	1.1 (1.0, 1.3)	<0.001
LDL-C (mmol/L)	2.7 (1.5, 3.4)	3.3 (2.6, 3.9)	<0.001
TC (mmol/L)	5.2 (4.6, 5.8)	5.5 (4.9, 6.0)	<0.001
TG (mmol/L)	0.8 (0.6, 1.1)	1.3 (0.9, 1.9)	<0.001
FPG (mmol/L)	5.2 (5.0, 5.6)	5.4 (5.2, 5.7)	<0.001
HbA1c (%)	5.2 (5.0, 5.4)	5.3 (5.1, 5.5)	<0.001

Values were expressed as medians (quartile interval) or n (%). A two-sided P < 0.05 was considered statistically significant.

MASLD, metabolic dysfunction-associated steatotic liver disease; BMI, body mass index; WC, waist circumference; SBP, systolic blood pressure; DBP, diastolic blood pressure; ALT, alanine aminotransferase; AST, aspartate aminotransferase; GGT, gamma-glutamyl transferase; HDL-C, high-density lipoprotein cholesterol; LDL-C, low-density lipoprotein cholesterol; TC, total cholesterol; TG, triglyceride; FPG, fasting plasma glucose; HbA1c, hemoglobin A1c.

ML models in predicting chances of MASLD

K-fold cross-validation on ML models

Figure 2(a)–(f) illustrates the comparison of the performances of various models on the proposed dataset. The RF and the XGB model both had superior performances in the entire dataset, but the RF model had a slightly higher average AUC value of 0.847 compared with the XGB model. Considering the stability and generalization of the RF model against outliers and noise, the RF model was built in the final stage.

Figure 2.

The comparison of the performances of various models on the proposed dataset. (a–f) Five-fold cross-validation test for DT, KNN, XGB, NB, SVM, and RF models. (g–l) ROCs, PRCs, and the confusion matrixes of six ML models with all features. (m) Radar graph showing the comprehensive prediction performances of six ML models with all features. DT, decision tree; KNN, K-nearest neighbor; XGB, extreme gradient boosting; NB, naive Bayes; SVM, support vector machine; RF, random forest; MASLD, metabolic dysfunction-associated steatotic liver disease; ROC, receiver operating characteristic; AUC, the area under the ROC curve; AP, average precision.

The comprehensive evaluations for ML models with all features

Figure 2(g)–(l) shows the specific receiver operating characteristic (ROC) curves, precision-recall curves (PRCs), and confusion matrixes in the prediction process. The RF model had the highest AUC of 0.889 (95% CI, 0.865, 0.910), and AP of 0.800 (95% CI, 0.757, 0.839) compared with the other five models.

Figure 2(m) shows the result of radar plot analysis, which is a blanket, clear, intuitive, and easy-to-judge analysis and is suitable for comprehensive evaluation as it can show the AUC, AP, accuracy, sensitivity, specificity, and F1 values of different models in multiple dimensions to more clearly reflect the performances of the models. The results suggested that the RF model with all features performed best. More specific values are shown in Table 2.

Table 2.

The comparison of the performances of various models with all features.

	AUC	AP	Accuracy	Sensitivity	Specificity	F1
DT	0.712 (0.680, 0.745)	0.462 (0.412, 0.511)	0.748 (0.717, 0.776)	0.625 (0.569, 0.684)	0.799 (0.765, 0.829)	0.592 (0.546, 0.641)
KNN	0.833 (0.802, 0.861)	0.710 (0.655, 0.765)	0.807 (0.782, 0.831)	0.632 (0.578, 0.689)	0.880 (0.855, 0.904)	0.658 (0.612, 0.702)
XGB	0.879 (0.855, 0.901)	0.759 (0.707, 0.809)	0.824 (0.799, 0.849)	0.740 (0.686, 0.789)	0.859 (0.833, 0.886)	0.712 (0.665, 0.754)
NB	0.858 (0.833, 0.882)	0.696 (0.635, 0.757)	0.798 (0.771, 0.823)	0.715 (0.662, 0.766)	0.832 (0.803, 0.860)	0.675 (0.628, 0.717)
SVM	0.869 (0.846, 0.893)	0.725 (0.665, 0.782)	0.805 (0.782, 0.831)	0.755 (0.703, 0.801)	0.826 (0.799, 0.856)	0.694 (0.651, 0.737)
RF	0.889 (0.865, 0.910)	0.800 (0.757, 0.839)	0.819 (0.793, 0.842)	0.816 (0.770, 0.859)	0.821 (0.790, 0.847)	0.729 (0.687, 0.768)

DT, decision tree; KNN, K-nearest neighbor; XGB, extreme gradient boosting; NB, naive Bayes; SVM, support vector machine; RF, random forest; AUC, the area under the receiver operating characteristic curve; AP, average precision.

Interpretable RF model

Hyperparameters of the RF model

The optimal hyperparameters were obtained by the GridSearchCV. Table 3 shows that the parameter combination of criterion = entropy, n_estimators = 500, min_sample leaf = 6, min_sample split = 5, and max_depth = 6 was the optimal setting for the RF model in this research.

Table 3.

The RF model's general hyperparameters configurations and optimal hyperparameters settings of the dataset.

Model	Hyperparameters	Optimal hyperparameters
RF model	criterion: [gini, entropy] n_estimators: [50, 100, 200, 300, 500, 600, 800, 1000] min_samples leaf: [1, 3, 5, 6, 8, 10] min_samples split: [3, 5, 7] max_depth: [1, 3, 5, 6, 8, 10]	criterion = entropy n_estimators = 500 min_samples leaf = 6 min_samples split = 5 max_depth = 6

RF, random forest.

criterion: the measure for splitting the decision tree nodes; n_ estimators: the number of decision trees in the forest; min_samples leaf: the minimum number of observations required in the terminal node or leaf; min_ samples split: the minimum number of observations required to split a node; max_depth: the maximum depth of the decision tree.

Establishment and global interpretation of the RF model

Figure 3(a) shows the results of abilities to predict the chances of MASLD in different models when using all variable data. The RF model performed best with the largest AUC of 0.889. Then, the SHAP values of the RF model were calculated to determine the global feature importance rank across the study populations.

Figure 3.

Global interpretation of the RF model on the test dataset. (a) The ROC curves of different models predicting the chances of MASLD. (b) SHAP summary histogram plots showing the most important features for predicting good functional outcomes for patients with hypertension and pre-hypertension. (c) The scatter plot of the feature density according to the important ranking. The red dot represents the high feature value of the patient, while the blue dot indicates the low potential feature value. For binary categorical features, a “low” feature value indicates its absence and a “high” value indicates its presence. (d) Visualization of the sixth decision tree in the RF model. MASLD, metabolic dysfunction-associated steatotic liver disease; SVM, Support Vector Machine; ROC, receiver operating characteristic; AUC, the area under the ROC curve; SHAP, Shapley Additive Explanation; BMI, body mass index; WC, waist circumference; SBP, systolic blood pressure; DBP, diastolic blood pressure; ALT, alanine aminotransferase; AST, aspartate aminotransferase; GGT, gamma-glutamyl transferase; HDL-C, high-density lipoprotein cholesterol; LDL-C, low-density lipoprotein cholesterol; TC, total cholesterol; TG, triglyceride; FPG, fasting plasma glucose; HbA1c, hemoglobin A1c; RF, random forest.

Figure 3(b) demonstrates that the most important features notably affected the outcomes of the model were ALT, BMI, WC, HDL-C, and TG. The features were sorted by the mean absolute values of SHAP which represent the importance of SHAP global feature.

Figure 3(c) displays the feature density according to the important ranking. The HDL-C exhibited a strong negative correlation with the final results, whereas ALT had a strong positive correlation with the final results. On the other hand, exercise habits, drinking status, and smoking status were ambiguous in the final results. Therefore, the hypertensive and pre-hypertensive patients with high ALT or low HDL-C levels had a higher incidence of MASLD. In addition, the remaining features were positively correlated with the model outcomes.

Figure 3(d) presents the sixth DT of the RF model, providing an interpretive framework for distinguishing patients with or without MASLD. It elaborates on the whole process of how the DT in the RF model could judge whether the hypertensive and pre-hypertensive patient has MASLD according to the given values of the features. When the node is colored yellow, it indicates that the patient suffers from MASLD, and the darker the color, the higher the incidence of the disease. On the contrary, if the node is blue, it implies that the patient has no MASLD, and the darker the color, the higher the chances of non-MASLD. However, if the nodes of the whole judgment process have no child nodes, it means the process is over. In this decision-making process, the value of FPG was first judged, and LDL-C, BMI, ALT, GGT, and TG were recognized as the dominant features, further confirming the correctness of the features’ important ranking in Figure 3(b) and (c) and explaining the working principle of the RF model from a macro perspective.

Local interpretation of the RF model

The principal component analysis (PCA) was used to reduce all data in the dataset into two-dimensional data, and the corresponding scatter diagram was drawn in Figure 4(a). The “force plot” function in the SHAP package was used to determine the overall SHAP value for each patient. Combining the features that push up (red, contributing to a higher chance of MASLD) and push down (blue, contributing to a lower chance of MASLD) the overall SHAP value, the overall SHAP value of this patient as a functional outcome of being MASLD was obtained. Specifically, four typical sample cases were selected, with functional outcomes of 0.08, 0.22, 0.55, and 0.83. It can be further observed from these four typical patients that ALT, BMI, and WC exerted a strong driving effect on predicting the chances of MASLD, and ALT, BMI, WC, AST, HDL-C, and TG played a dominant role in the final output of the model (Figure 4(b)).

Figure 4.

Local interpretation of the RF model. (a) Two-dimensional scatter diagram of the dataset by principal component analysis (PCA) dimensionality reduction. (b and c) Hierarchical study population clustering by using SHAP value feature weightings to quantify the patient similarity. The figure is available in color online only. MASLD, metabolic dysfunction-associated steatotic liver disease; BMI, body mass index; WC, waist circumference; SBP, systolic blood pressure; DBP, diastolic blood pressure; ALT, alanine aminotransferase; AST, aspartate aminotransferase; GGT, gamma-glutamyl transferase; HDL-C, high-density lipoprotein cholesterol; LDL-C, low-density lipoprotein cholesterol; TC, total cholesterol; TG, triglyceride; FPG, fasting plasma glucose; HbA1c, hemoglobin A1c.

Then, the force plot was rotated counterclockwise by 90°, and the process was repeated for a sample of 500 patients in the test dataset, providing a global graph of the probability of being MASLD in the test dataset clustered by similar risk factors combinations (Figure 4(c)). The common features of patient subgroups with high (red) or low (blue) predicted probabilities are shown below the plot. From the figure, it can be seen that the driving factors of the RF model for high-risk patients were mainly ALT, WC, BMI, TG, AST, and HDL-C.

Consistent with the summary plot of importance ranking, hierarchical clustering shows that the abnormal levels of ALT, WC, BMI, TG, AST, and HDL-C tended to constitute a subset with a higher incidence of MASLD. However, when HDL-C, WC, ALT, and BMI were at normal levels, the prediction results were more likely to be disease-free for hypertensive and pre-hypertensive patients.

More importantly, normal or abnormal levels of some important features do not always determine whether a pre-hypertensive or hypertensive patient has MASLD. For example, patients with normal levels of TG, BMI, and HDL-C still had a higher incidence of MASLD, which may be due to higher ALT or AST levels dominating this trend. On the contrary, patients with high levels of TG and GGT had a lower incidence of MASLD, which may be due to features like WC, BMI, and ALT alleviating the effect of the former on induction.

Variable correlation analysis and interaction effects

Figure 5(a) illustrates the correlation analysis among the five important features mentioned in the above analysis, namely ALT, BMI, HDL-C, AST, and TG. HDL-C had a strong negative correlation with TG, with a correlation coefficient of −0.45, and all other features also had a negative correlation with HDL-C. AST and ALT had a strong positive correlation with a correlation coefficient of 0.78. In addition, the correlation coefficients between AST and BMI, AST and TG were 0.26 and 0.19 respectively.

Figure 5.

Variable correlation analysis and interaction effects on the RF model. (a) Scatter plot and histogram matrix for the correlation of the top 5 important features. (b) Partial dependence plot (PDP) of top 10 important features according to the SHAP summary histogram plots. (c) Variable interaction effects between sex and HDL-C quantified by SHAP interaction values. (d) Variable interaction effects between sex and ALT quantified by SHAP interaction values. MASLD, metabolic dysfunction-associated steatotic liver disease; BMI, body mass index; WC, waist circumference; SBP, systolic blood pressure; DBP, diastolic blood pressure; ALT, alanine aminotransferase; AST, aspartate aminotransferase; GGT, gamma-glutamyl transferase; HDL-C, high-density lipoprotein cholesterol; LDL-C, low-density lipoprotein cholesterol; TC, total cholesterol; TG, triglyceride; FPG, fasting plasma glucose; HbA1c, hemoglobin A1c; SHAP, Shapley additive explanation; RF, random forest.

Figure 5(b) illustrates the impact of each feature on the disease prediction for individual patients and the overall average influence of the respective feature across the entire sample. Except for HDL-C, the top ten features had strong positive correlations with the result. This can be explained that when the outcome of the model was 0.83, the lower the HDL-C level of patients, the higher the chances of MASLD.

When examining the SHAP interaction values, the strongest and most consistent interactions were related to sex. Figure 5(c) shows the interaction of sex with HDL-C. Particularly, female patients with lower HDL-C levels had a higher incidence of MASLD. Figure 5(d) illustrates the interaction of sex with the pivotal feature ALT. The likelihood of MASLD in males exhibits a swift escalation in tandem with increasing ALT levels. Visualizing the response function and bias analysis of a single feature in the model can enable us to comprehend the relationship between features and response variables, thus enhancing the interpretability and visualization of the RF model.

Numbers of variable selection analysis

To explore the coupling between the features, the influence of the number of important features on the RF model outcomes was further studied. Figure 6(a) shows that when considering only the top five features, the RF model performed with an AUC value of 0.876 (95%CI, 0.851–0.899). When the number of features increased to the top 10, the AUC value increased to 0.887 (95%CI, 0.866–0.907) which implied increasing the number of features to the top 10 indeed improved the effectiveness of identifying MASLD. However, when the number of features increased to the top 15, the AUC value was 0.888 (95%CI, 0.867–0.910), demonstrating that the five new features (gender, SBP, DBP, TC, and drinking status) slightly improve the MASLD identification ability. In general, when the top five or ten features were used, the model exhibited a notably enhanced accuracy, further emphasizing the importance of these specific features in MASLD diagnosis. Although the remaining variables had a small improvement on the results, their contribution was significantly reduced compared to the top ten features. When all indicators were used, the RF model achieved the highest AUC value of 0.889 (95%CI, 0.865–0.910).

Figure 6.

The performances of the RF model with different features. (a) ROC curves of the RF models with different features for the test set. (b) Radar graph showing the comprehensive prediction performances of the RF models with different features for the test set. RF, random forest; MASLD, metabolic dysfunction-associated steatotic liver disease; ROC, receiver operating characteristic; AUC, the area under the ROC curve.

Figure 6(b) and Table 4 detailedly describe the AUC, accuracy, sensitivity, specificity, and F1 values of each RF model with different features. The results suggested that the RF model performed best when all features were used, which was the model we used in this research.

Table 4.

The performances of the RF models with different features.

	AUC	Accuracy	Sensitivity	Specificity	F1
All features	0.889 (0.865, 0.910)	0.819 (0.793, 0.842)	0.816 (0.770, 0.859)	0.821 (0.790, 0.847)	0.729 (0.687, 0.768)
Top 15 features	0.888 (0.867, 0.910)	0.815 (0.788, 0.840)	0.816 (0.767, 0.860)	0.815 (0.783, 0.843)	0.724 (0.683, 0.762)
Top 10 features	0.887 (0.866, 0.907)	0.810 (0.784, 0.833)	0.812 (0.766, 0.856)	0.808 (0.779, 0.837)	0.718 (0.678, 0.756)
Top 5 features	0.876 (0.851, 0.899)	0.796 (0.771, 0.821)	0.794 (0.747, 0.838)	0.796 (0.765, 0.829)	0.699 (0.659, 0.740)

RF, random forest; AUC, the area under the receiver operating characteristic curve.

Discussion

In this study, we developed several ML models for identifying MASLD in hypertensive and pre-hypertensive patients and provided global and local interpretation for the model results using the SHAP values, elaborating how various features affect the chances of MASLD. To the best of our knowledge, this is the first study about the identification of MASLD in pre-hypertensive and hypertensive patients via ML.

Hypertension is closely related to the onset, progression, and prognosis of MASLD. Despite numerous studies suggesting a bidirectional relationship between MASLD and hypertension, a confirmed causal relationship has not been established.³⁰ The underlying mechanism of the relationship between hypertension and MASLD might be insulin resistance, systemic inflammation, activation of the renin-angiotensin-aldosterone system and sympathetic nervous system, oxidative stress, gut microbiota disbalance, sex hormone, and genetic factors.³¹ In hypertensive patients, the prevalence of hepatic steatosis exhibits a direct correlation with the severity of hypertension.³² Additionally, a history of hypertension increases the risk of developing MASLD.³³ In our study, the incidence of MASLD was higher in hypertensive patients compared to pre-hypertensive patients. Compared to previous studies, the prevalence of MASLD in these two groups significantly exceeded that of the general population in the same region.³⁴ A meta-analysis revealed that the presence of hypertension at the time of initial liver biopsy was identified as a risk factor for progressive fibrosis in patients with MASLD.³⁵ Epidemiological studies have indicated that patients with both hypertension and MASLD had poor prognosis.³⁰ In comparison to patients with MASLD alone, those with concomitant hypertension and MASLD had a higher risk of cardiovascular disease death.¹⁷ Timely control of hypertension to achieve the target value of blood pressure can decrease the likelihood of MASLD by more than 40% and reduce the risk of liver fibrosis progression.¹⁵ In addition, dietary approaches to stop hypertension diet improve hepatic fibrosis, steatosis, liver enzymes, oxidative stress, and body composition in patients with MASLD.^36,37

Early identification is crucial for effectively preventing, treating, and delaying the disease progression of MASLD,³⁸ especially in hypertensive and pre-hypertensive patients. Compared with traditional models, advanced ML technologies provide researchers with enhanced capabilities to develop optimized disease prediction models.³⁹ Liu et al. evaluated selected clinical and biochemical factors to predict MASLD using various ML algorithms, among which the AUC, accuracy, sensitivity, specificity, and F1 of the optimal model were at relatively high levels akin to our research, showcasing the good performance of the models.⁴⁰ However, this study lacked a detailed local and global explanation. Deng et al. used a time series deep learning model with longitudinal health examination records to predict the onset of MASLD in the future and continuously updated the model by incorporating the new examination records for dynamic prediction, but the AUC of the model did not reach a very high level.⁴¹ What is more, despite numerous ML studies on MASLD, there is currently no MASLD identification model targeting the specific high-risk population of hypertensive and prehypertensive patients, and our research just fills this gap. Previous research found that the identification of novel predictors for fibrosis and steatohepatitis and the improvement of the prediction accuracy of fibrosis in MASLD can be facilitated via the RF model.^42,43 In addition, the five-gene prediction model seeking MASLD candidate therapeutic targets based on minimum depth RF had good calibration and performance.⁴⁴ Our study demonstrated that among the six models used on the proposed dataset, the RF model exhibited the best performance in predicting the chances of MASLD in pre-hypertensive and hypertensive patients.

One of the foremost challenges in MASLD identification and management is understanding which features are associated with satisfactory model outcomes. We endeavored to address this problem by applying predictive ML models to a large-scale patient dataset before using the models and explaining their predictions with game theory-based solution concepts. By using ML, we identified several features strongly associated with MASLD in pre-hypertensive and hypertensive patients. The ALT was the most important feature because it invariably influenced the RF model outcome. The ALT, a standard and widely used indicator of liver function, is closely related to glucose and lipid metabolism disorders,⁴⁵ inflammation, and oxidative stress.⁴⁶ Elevated ALT concentration increases heart rate,⁴⁷ arterial stiffness,⁴⁸ and the risk of carotid atherosclerosis,⁴⁹ all of which are pathophysiological mechanisms of hypertension. Individuals with elevated ALT levels had a higher prevalence of hypertension compared to those with normal levels.⁵⁰ Additionally, there was also a linear and robust correlation between ALT level and SBP or DBP. SBP and DBP increased with the elevation of serum ALT levels in both hypertensive and normotensive people.⁵⁰ Furthermore, ALT has also been implicated in liver fat accumulation, the pathogenesis of MASLD.⁵¹ ALT levels are normally low in the serum of healthy individuals, and elevated ALT levels usually indicate hepatocellular injury or apoptosis.⁵² Previous studies have also shown that a higher ALT level at baseline was an independent risk factor for the development of MASLD, even in healthy subjects, and the ALT level was proved as a surrogate marker in the diagnosis of MASLD with high sensitivity and specificity,^53,54 which confirmed that this ML-based approach seems to produce valid results. In addition, in a general population-based ML study of MASLD prediction, ALT was identified as the second most crucial predictor, affirming that ALT is also an important predictor of MASLD in the general population.⁴¹

Additionally, BMI, WC, TG, and HDL-C are crucial features that warrant particular attention. MASLD is characterized by the massive accumulation of lipids in the liver and the subsequent increase of inflammatory cytokines and adipokines, in which TG and HDL-C play important roles.⁵⁵ BMI and WC are important indicators to evaluate obesity, which is considered chronic low-grade inflammation and is closely related to MASLD.⁵⁶ With the increasing number of obese people, the disease burden of MASLD in Western countries and Asia is projected to increase two to three times by 2030.⁵⁷ Pan et al. identified that high BMI, WC, and TG were risk factors for MASLD, while high HDL-C levels were protective.⁵⁸ Similarly, Peng et al. found that the visceral adiposity index, calculated by BMI, WC, TG, and HDL-C, ranked first in testing the best predictive characteristics of risk factors in patients with MASLD. In our study, BMI, WC, TG, and HDL-C were the top five variables, ranking second to ALT.⁵⁵ Moreover, patients with pre-hypertension and hypertension tend to have higher BMI and lipid levels compared with normotensive individuals.⁵⁹ Given that MASLD is a metabolism-related disease, and BMI, WC, TG, and HDL-C are all metabolism-related indicators, their high importance in our research is aligned with the metabolic nature of MASLD.⁶⁰

Under the guidance of ML-based variable selection, this study applied a new propensity score matching analysis to the data of the previous study, providing another data for the literature on this topic. However, formal data with well-defined patient groups and clinical features are needed to shift the current paradigm for MASLD research because it provides an optimal opportunity to understand some of the fundamental features associated with clinicians’ decisions on whether patients suffer from MASLD. Our study improves an intuitive understanding of how the features affect the ML models’ predicted outcomes. The findings may aid in the development of identifying MASLD, enabling clinicians to more accurately diagnose MASLD and optimize clinical variables usage.

There are some limitations of this study that deserve to be mentioned. First, due to its unsatisfactory sensitivity, ultrasonography is not the gold standard for the diagnosis of MASLD. However, in this investigation, the liver biopsy could not be performed on all populations, which might influence the reliability of the outcomes. Second, although high-quality data from large sample sources may increase the power of our analysis, these data were obtained from specific regional populations, which may limit the external validity of our findings. Finally, restricting the analysis to the variables available in the clinical dataset may prevent the detection of other features that affect MASLD.

Conclusion

ML algorithms, especially RF algorithm, improve the accuracy of MASLD identification in patients with hypertension and pre-hypertension, which can help clinicians make early diagnosis and prevention in the clinic. Meanwhile, global and local interpretation of the results of the RF model enables us to intuitively understand how various features affect the chances of MASLD in this kind of patient.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076241233135 - Supplemental material for Identifying metabolic dysfunction-associated steatotic liver disease in patients with hypertension and pre-hypertension: An interpretable machine learning approach

Supplemental material, sj-docx-1-dhj-10.1177_20552076241233135 for Identifying metabolic dysfunction-associated steatotic liver disease in patients with hypertension and pre-hypertension: An interpretable machine learning approach by Chen Chen, Wenkang Zhang, Gaoliang Yan and Chengchun Tang in DIGITAL HEALTH

Footnotes

Acknowledgments

The authors thank the investigators and the participants of the original study for their contributions.

Availability of data and materials

The datasets analyzed during the current study are available in the DRYAD repository, .¹⁹

Contributorship

Chen Chen contributed to data collection, analysis, and interpretation. Wenkang Zhang contributed to the conception and design, data analysis, and interpretation. Gaoliang Yan contributed to the conception and design. Chengchun Tang provided administrative support. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

The ethics committee of Murakami Memorial Hospital approved this study.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Natural Science Foundation of China (82170433).

Guarantor

Chengchun Tang.

ORCID iD

Wenkang Zhang

Supplemental material

Supplemental material for this article is available online.

References

Rinella

Lazarus

Ratziu

, et al. A multisociety Delphi consensus statement on new fatty liver disease nomenclature. J Hepatol 2023; 79: 1542–1556.

Roeb

. Excess body weight and metabolic (dysfunction)-associated fatty liver disease (MAFLD). Visc Med 2021; 37: 273–280.

Baez

, et al. Global incidence of non-alcoholic fatty liver disease: a systematic review and meta-analysis of 63 studies and 1,201,807 persons. J Hepatol 2023; 79: 287–295.

Younossi

Golabi

Paik

, et al. The global epidemiology of nonalcoholic fatty liver disease (NAFLD) and nonalcoholic steatohepatitis (NASH): a systematic review. Hepatology 2023; 77: 1335–1347.

Powell

Wong

Rinella

. Non-alcoholic fatty liver disease. Lancet 2021; 397: 2212–2224.

Zoller

Tilg

. Nonalcoholic fatty liver disease and hepatocellular carcinoma. Metabolism 2016; 65: 1151–1160.

Younossi

Tacke

Arrese

, et al. Global perspectives on nonalcoholic fatty liver disease and nonalcoholic steatohepatitis. Hepatology 2019; 69: 2672–2682.

Golabi

Paik

AlQahtani

, et al. Burden of non-alcoholic fatty liver disease in Asia, the Middle East and North Africa: data from global burden of disease 2009–2019. J Hepatol 2021; 75: 795–809.

Yin

Guo

Liu

, et al. Advances in the diagnosis and treatment of non-alcoholic fatty liver disease. Int J Mol Sci 2023; 24: 2844.

10.

Davison

Harrison

Cotter

, et al. Suboptimal reliability of liver biopsy evaluation has implications for randomized clinical trials. J Hepatol 2020; 73: 1322–1332.

11.

Sim JZ

Fong

Huang

, et al. Machine learning in medicine: what clinicians should know. Singapore Med J 2023; 64: 91–97.

12.

Moll

Qiao

Regan

, et al. Machine learning and prediction of all-cause mortality in COPD. Chest 2020; 158: 952–964.

13.

Haag

Lynch

, et al. Comparison of the tree-based machine learning algorithms to Cox regression in predicting the survival of oral and pharyngeal cancers: analyses based on SEER database. Cancers (Basel) 2020; 12: 2802.

14.

Vasunta

Kesäniemi

Ylitalo

, et al. High ambulatory blood pressure values associated with non-alcoholic fatty liver in middle-aged adults. J Hypertens 2012; 30: 2015–2019.

15.

Aneni

Oni

Martin

, et al. Blood pressure is associated with the presence and severity of nonalcoholic fatty liver disease across the spectrum of cardiometabolic risk. J Hypertens 2015; 33: 1207–1214.

16.

Egan

Stevens-Fabry

. Prehypertension–prevalence, health risks, and management strategies. Nat Rev Cardiol 2015; 12: 289–300.

17.

Wong

Chew

NWS

, et al. Hypertension is prevalent in non-alcoholic fatty liver disease and increases all-cause and cardiovascular mortality. Front Cardiovasc Med 2022; 9: 942753.

18.

Okamura

Hashimoto

Hamaguchi

, et al. Ectopic fat obesity presents the greatest risk for incident type 2 diabetes: a population-based longitudinal study. Int J Obes (Lond) 2019; 43: 139–148.

19.

Okamura

Hashimoto

Hamaguchi

, et al. Ectopic fat obesity presents the greatest risk for incident type 2 diabetes: a population-based longitudinal study, Dryad, Dataset, https://doi.org/10.5061/dryad.8q0p192

20.

Hamaguchi

Kojima

Itoh

, et al. The severity of ultrasonographic findings in nonalcoholic fatty liver disease reflects the metabolic syndrome and visceral fat accumulation. Am J Gastroenterol 2007; 102: 2708–2715.

21.

Umemura

Arima

, et al. The Japanese Society of Hypertension Guidelines for the Management of Hypertension (JSH 2019). Hypertens Res 2019; 42: 1235–1481.

22.

Chen

Zhang

Pan

, et al. A modified formula for calculating low-density lipoprotein cholesterol values. Lipids Health Dis 2010; 9, 52.

23.

Horng

Huang

, et al. Identifying discriminative amino acids within the hemagglutinin of human influenza A H5N1 virus using a decision tree. IEEE Trans Inf Technol Biomed 2008; 12: 689–695.

24.

Dash

Chakraborty

Mahapatra

, et al. Gradient boosting machine and efficient combination of features for speech-based detection of COVID-19. IEEE J Biomed Health Inform 2022; 26: 5364–5371.

25.

Arora

Dubey

Jaffery

, et al. Bag of feature and support vector machine based early diagnosis of skin cancer. Neural Comput Appl 2022; 34: 8385–8392.

26.

Zhang

Chen

, et al. Predictive analytics with ensemble modeling in laparoscopic surgery: a technical note. Laparosc Endosc Surg Sci2022; 5: 25–34.

27.

Greener

Kandathil

Moffat

, et al. A guide to machine learning for biologists. Nat Rev Mol Cell Biol 2022; 23: 40–55.

28.

Pang

, et al. Use of machine learning method on automatic classification of motor subtype of Parkinson’s disease based on multilevel indices of RS-fMRI. Parkinsonism Relat Disord 2021; 90: 65–72.

29.

Baptista

Goebel

Henriques

EMP

. Relation between prognostics predictor evaluation metrics and local interpretability SHAP values. Artif Intell 2022; 306: 103667.

30.

Lonardo

Nascimbeni

Mantovani

, et al.

Hypertension, diabetes, atherosclerosis and NASH: cause or consequence?

J Hepatol 2018; 68: 335–352.

31.

Golubeva

Sheptulina

Elkina

, et al.

Which comes first, nonalcoholic fatty liver disease or arterial hypertension?

Biomedicines 2023; 11: 2465.

32.

Wang

Zeng

Lin

, et al. Hypertension and non-alcoholic fatty liver disease proven by transient elastography. Hepatol Res 2016; 46: 1304–1310.

33.

Huang

Zhong

, et al. Association between hypertension and nonalcoholic fatty liver disease: a cross-sectional and meta-analysis study. J Hum Hypertens 2023; 37: 313–320.

34.

Ito

Ishigami

Zou

, et al. The epidemiology of NAFLD and lean NAFLD in Japan: a meta-analysis with individual and forecasting analysis, 1995–2040. Hepatol Int 2021; 15: 366–379.

35.

Singh

Allen

Wang

, et al. Fibrosis progression in nonalcoholic fatty liver vs nonalcoholic steatohepatitis: a systematic review and meta-analysis of paired-biopsy studies. Clin Gastroenterol Hepatol 2015; 13: 643–e40.

36.

Sangouni

Nadjarzadeh

Rohani

, et al. Dietary approaches to stop hypertension (DASH) diet improves hepatic fibrosis, steatosis and liver enzymes in patients with non-alcoholic fatty liver disease: a randomized controlled trial. Eur J Nutr 63, 2024, 95–105.

37.

Zare

Hosseinzadeh

Sharuni

, et al. The effect of the dietary approaches to stop hypertension diet on total antioxidant capacity, superoxide dismutase, catalase, and body composition in patients with non-alcoholic fatty liver disease: a randomized controlled trial. Front Nutr 2023; 10: 1163516.

38.

Xie

Huang

Liu

, et al. The associations between modifiable risk factors and nonalcoholic fatty liver disease: a comprehensive Mendelian randomization study. Hepatology 2023; 77: 949–964.

39.

Churpek

Yuen

Winslow

, et al. Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards. Crit Care Med 2016; 44: 368–374.

40.

Liu

Cen

, et al. Comparison and development of advanced machine learning tools to predict nonalcoholic fatty liver disease: an extended study. Hepatobiliary Pancreat Dis Int 2021; 20: 409–415.

41.

Deng

, et al. A dynamic machine learning model for prediction of NAFLD in a health checkup population: a longitudinal study. Heliyon 2023; 9: e18758.

42.

Tan

Lee

Jumat

, et al. Non-obese non-alcoholic fatty liver disease (NAFLD) in Asia: an international registry study. Metabolism 2022; 126: 154911.

43.

Sowa

Heider

Bechmann

, et al. Novel algorithm for non-invasive assessment of fibrosis in NAFLD. PLoS One 2013; 8: e62439.

44.

Wen

Zhang

, et al. Comprehensive analysis of NAFLD and the therapeutic target identified. Front Cell Dev Biol 2021; 9: 704704.

45.

Sookoian

Castaño

Scian

, et al. Serum aminotransferases in nonalcoholic fatty liver disease are a signature of liver metabolic perturbations at the amino acid and Krebs cycle level. Am J Clin Nutr 2016; 103: 422–434.

46.

Kerner

Avizohar

Sella

, et al. Association between elevated liver enzymes and C-reactive protein: possible hepatic contribution to systemic inflammation in the metabolic syndrome. Arterioscler Thromb Vasc Biol 2005; 25: 193–197.

47.

Liu

Hamnvik

Chamberland

, et al. Circulating alanine transaminase (ALT) and γ-glutamyl transferase (GGT), but not fetuin-A, are associated with metabolic risk factors, at baseline and at two-year follow-up: the prospective Cyprus metabolism study. Metabolism 2014; 63: 773–782.

48.

Uemura

Katsuura-Kamano

Yamaguchi

, et al. Relationships of elevated levels of serum hepatic enzymes and alcohol intake with arterial stiffness in men. Atherosclerosis 2015; 238: 83–88.

49.

Wang

Lin

Tseng

, et al. Elevation of serum aminotransferase activity increases risk of carotid atherosclerosis in patients with non-alcoholic fatty liver disease. J Gastroenterol Hepatol 2009; 24: 1411–1416.

50.

Jia

Yang

Liu

, et al. The association between serum alanine aminotransferase and hypertension: a national based cross-sectional analysis among over 21 million Chinese adults. BMC Cardiovasc Disord 2021; 21: 145.

51.

Mochizuki

Miyauchi

Misaki

, et al. Accumulation of visceral fat is positively associated with serum ALT and γ-GTP activities in healthy and preclinical middle-aged Japanese men. J Nutr Sci Vitaminol (Tokyo) 2011; 57: 65–73.

52.

Kim

Flamm

Di Bisceglie

, et al. Serum activity of alanine aminotransferase (ALT) as an indicator of health and disease. Hepatology 2008; 47: 1363–1370.

53.

Chang

Ryu

Sung

, et al. Higher concentrations of alanine aminotransferase within the reference interval predict nonalcoholic fatty liver disease. Clin Chem 2007; 53: 686–692.

54.

Miyake

Kumagi

Hirooka

, et al. Metabolic markers and ALT cutoff level for diagnosing nonalcoholic fatty liver disease: a community-based cross-sectional study. J Gastroenterol 2012; 47: 696–703.

55.

Peng

Duan

Pan

, et al. Development and validation of machine learning models for nonalcoholic fatty liver disease. Hepatobiliary Pancreat Dis Int 2023; 22: 615–621.

56.

Milić

Lulić

Štimac

. Non-alcoholic fatty liver disease and obesity: biochemical, metabolic and clinical presentations. World J Gastroenterol 2014; 20: 9330–9337.

57.

Estes

Anstee

Arias-Loste

, et al. Modeling NAFLD disease burden in China, France, Germany, Italy, Japan, Spain, United Kingdom, and United States for the period 2016–2030. J Hepatol 2018; 69: 896–904.

58.

Pan

Xie

Peng

, et al. Risk prediction for non-alcoholic fatty liver disease based on biochemical and dietary variables in a Chinese Han population. Front Public Health 2020; 8: 220.

59.

Moussouni

Sidi-Yakhlef

Hamdaoui

, et al. Prevalence and risk factors of prehypertension and hypertension in Algeria. BMC Public Health 2022; 22: 1571.

60.

Huang

Jin

Mao

. Predicting the 5-year risk of nonalcoholic fatty liver disease using machine learning models: prospective cohort study. J Med Internet Res 2023; 25: e46891.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.06 MB