Sage Journals: Discover world-class research

Abstract

Background:

The Rome proposal provides an objective assessment tool for severity of acute exacerbation of chronic obstructive pulmonary disease (AE-COPD) but requires validation.

Objectives:

We aimed to evaluate the predictive performance of the Rome proposal in patients with AE-COPD.

Design:

This observational study assessed the patients who visited the emergency room (ER) or were hospitalized due to AE-COPD between January 2010 and December 2020.

Methods:

We compared the performance of the Rome Proposal with that of the DECAF score or GesEPOC 2021 criteria in predicting intensive care unit (ICU) admission, need for non-invasive ventilation (NIV) or invasive mechanical ventilation (IMV), and in-hospital mortality.

Results:

A total of 740 events of ER visit or hospitalization due to AE-COPD were reviewed and classified into mild (30.9%), moderate (58.6%), or severe (10.4%) group according to the Rome proposal. The severe group had a higher rate of ICU admission, required more NIV or IMV, and had a higher in-hospital mortality than the mild and moderate groups. The predictive performance of the Rome proposal was significantly better for ICU admission [area under the receiver operating characteristic curve (AU-ROC) = 0.850 versus 0.736, p = 0.004] and need for NIV or IMV (AU-ROC = 0.870 versus 0.770, p = 0.004) than that of the GesEPOC 2021 criteria but better than that of the DECAF score only in female patients. There was no significant difference in predicting the in-hospital mortality between the Rome proposal and DECAF score or GesEPOC 2021 criteria.

Conclusion:

External validation of the Rome Proposal in Korean patients demonstrated excellent performance for ICU admission and need for NIV or IMV and an acceptable performance for in-hospital mortality.

Keywords

chronic obstructive pulmonary disease exacerbation intensive care units mechanical ventilation mortality predictive value of tests symptom validation study

Introduction

Acute exacerbation of chronic obstructive pulmonary disease (AE-COPD) is a leading cause of disease-associated morbidity and mortality among patients with COPD.¹ AE-COPD accounts for one in eight of the total hospital admissions and is associated with accelerated lung function decline, poor health-related quality of life, and increased risk of mortality in patients with COPD.^1,2 Prediction model for prognosis of AE-COPD can provide risk-stratified clinical management, including avoidance of unnecessary hospitalization or excessive examination for low-risk groups, and timely intensive care for high-risk groups.^3,4 However, the current severity classification of AE-COPD suggested by the Global Initiative for Chronic Obstructive Lung Disease (GOLD) has not provided practical information for making clinical decisions, because it is usually assessed retrospectively according to the medical resources for dealing with the event.¹ In addition, severity of AE-COPD may be classified differently according to the regions owing to the use of different medical resources or healthcare system.^5,6

The Rome proposal has been recently presented as an expert recommendation to provide an objective assessment of AE-COPD severity.⁷ The Rome proposal assessed AE-COPD severity using routinely measured physiological parameters without the need for chest radiograph or electrocardiogram, which are needed for the DECAF score.⁸ However, since severity stratification of AE-COPD by the Rome proposal is based on Delphi methodology, predictive performance of the Rome proposal needs to be validated in a real-world setting.

Therefore, we aimed to elucidate whether the Rome proposal can predict the clinical outcomes in patients with AE-COPD, including in-hospital mortality, need for non-invasive ventilation (NIV) or invasive mechanical ventilation (IMV), and intensive care unit (ICU) admission. Furthermore, we compared the predictive performance of the Rome Proposal with other prediction models including the DECAF score⁹ and GesEPOC 2021 criteria.¹⁰

Materials and methods

We followed the guidelines presented by the Strengthening the Reporting of Observational Studies in Epidemiology statement.¹¹

Study design and participants

The present observational study included patients who visited the emergency room (ER) or were admitted to hospital owing to AE-COPD in the Seoul Metropolitan Government–Seoul National University Boramae Medical Center between January 2010 and December 2020. The eligibility criteria were as follows: (1) adults over 40 years of age; (2) post-bronchodilator forced expiratory volume in 1 s (FEV₁)/forced vital capacity (FVC) <0.7; (3) an event of exacerbation characterized by acute worsening of dyspnea (⩾5 using a 0–10 scale checked by initial nursing assessment) over <14 days; and (4) available information on six parameters to assess the severity of AE-COPD, including respiratory rate (RR), heart rate (HR), arterial pH, arterial partial pressure of carbon dioxide (PaCO₂), arterial oxygen saturation (SaO₂) at rest, and serum C-reactive protein (CRP). We excluded (1) patients with missing data, (2) patients who were hospitalized owing to more serious medical conditions other than AE-COPD, and (3) patients with definite lobar pneumonia that could be improved without steroid or bronchodilator.

Variables

Baseline information included age, sex, body mass index, smoking history, previous exacerbation history, Charlson comorbidity index (CCI), underlying disease, respiratory comorbidities, and inhaled therapy. Clinical features including respiratory symptoms, modified Medical Research Council (mMRC) scale, vital signs, laboratory examination results, radiologic findings, adjuvant treatments, and lung function test were obtained.

Definition of AE-COPD severity

Based on the Rome proposal, exacerbation events were categorized into three severity groups. The criteria for judging severity in the Rome proposal were visual analog scale (VAS) ⩾5, RR ⩾24, HR ⩾95, Resting SaO₂ ⩾92%, and CRP ⩾1 mg/dl.⁷ Since the eligible patient’s complaint dyspnea VAS ⩾5, we defined mild severity as the case that met ⩽2 criteria and moderate severity as the case that met >2 criteria. The presence of hypercapnia (PaCO₂ >45 mmHg) and respiratory acidosis (pH <7.35) were defined as severe severity.

The DECAF score classified AE-COPD into low, intermediate, and high risk for in-hospital mortality using five variables including extended Medical Research Council dyspnea (eMRCD) score, eosinopenia, consolidation, acidemia, and atrial fibrillation.⁹ We determined eMRCD 5a/5b based on daily activities, such as washing and dressing difficulties, based on the medical records and described by the attending nurse. Three degrees of risk were obtained by categorizing the total scores as follows: low risk, 0 to 1 point; intermediate risk, 2 points; and severe risk, ⩾3 points.

The GesEPOC 2021 criteria classified AE-COPD into mild, moderate, severe, and very severe according to the degree of dyspnea using the mMRC scale, level of consciousness, RR, and gas exchange.¹⁰ Mild severity was defined as mMRC ⩽2, alert consciousness, RR <24, and SaO₂ ⩾92%. A case was considered of moderate severity if it met any of the following criteria: RR of 24–30 or SaO₂ of 90–94%; severe if mMRC >3, drowsiness, RR ⩾30, or SaO₂ <90% was present; and very severe if stupor or coma or pH <7.30 or PaCO₂ ⩾60 mmHg was present.

Main outcomes

We evaluated the prognostic performance of the Rome proposal to predict the following outcomes: ICU admission, need for NIV or IMV, and in-hospital mortality in patients with AE-COPD. Furthermore, we compared the prognostic accuracy of the Rome proposal, DECAF score, and GesEPOC 2021 criteria to predict the aforementioned outcomes.

Statistical analyses

Data are presented as means with standard deviations (SDs) or medians with standard errors for continuous variables and numbers with percentages for categorical variables. Analysis of variance or Kruskal–Wallis test was used to examine continuous variables. The chi-square or Fisher’s exact test was used to analyze categorical variables. Kaplan–Meier curves and log-rank tests were performed to compare the clinical outcomes among the mild, moderate, and severe groups. For the receiver operating characteristic (ROC) curves, we determined the sensitivity and specificity under optimal cut-offs of the Rome proposal, DECAF score, and GesEPOC 2021 criteria to predict the clinical outcomes. Two-proportion Z-test was performed to compare the sensitivity and specificity between the different AE-COPD severity classifications. Predictive performances of the Rome proposal, DECAF score, and GesEPOC 2021 criteria were evaluated using the area under the ROC curve (AU-ROC). We considered AU-ROC <0.7, 0.7–0.79, 0.8–0.89, and ⩾0.9 as suboptimal performance, acceptable, excellent, and outstanding performance, respectively.¹² Prognostic accuracy resulting from optimal cut-off for the highest AU-ROC was compared among the different prediction models using DeLong’s test.¹³ Concordance between the different AE-COPD severity classifications was analyzed using Cohen’s kappa coefficients. We assessed the level of agreement as none, minimal, weak, moderate, and strong with κ C0.2, κ = 0.21–0.39, κ = 0.40–0.59, κ = 0.60–0.79, and κ = 0.80–0.90, respectively.¹⁴ Statistical significance was set at p < 0.05. All the statistical analyses were performed with pROC and PredictABEL packages using R statistical software, version 4.1.0 [R Core Team (2020), Vienna, Austria].

Results

Among the 801 patients with AE-COPD, 61 patients with missing data were excluded. We found 740 events of ER visit or hospitalization owing to AE-COPD and classified them into mild (n = 229, 30.9%), moderate (n = 434, 58.6%), and severe groups (n = 77, 10.4%) based on the Rome proposal. There were 28 events of ICU admission with time to event of median 1 [interquartile range (IQR) = 1–5] day, 33 events of NIV or IMV with time to event of median 3 (IQR = 0–7) days, and 10 events of in-hospital mortality with time to event of median 29 (IQR = 20–42) days.

Baseline characteristics and clinical features

The baseline characteristics are summarized according to the severity of AE-COPD based on the Rome proposal in Table 1. Current smokers were lesser found in the severe group; however, a positive association was observed between smoking pack-year and severity of AE-COPD. Severity of AE-COPD was increased with any previous exacerbation rate. Although CCI was not related to the severity of AE-COPD, more patients with malignancy had a greater severity of AE-COPD. More patients who used triple therapy (inhaled corticosteroid/long-acting β2-agonist/long-acting muscarinic antagonist) experienced a greater severity of AE-COPD.

Table 1.

Baseline characteristics according to AE-COPD severity based on the Rome proposal.

	Mild (n = 229, 30.9%)	Moderate (n = 434, 58.6%)	Severe (n = 77, 10.4%)	Total (N = 740)	p for linear trend	p
Age, years, mean ± SD	69.67 ± 0.69	71.15 ± 0.44	71.44 ± 0.86	70.72 ± 0.35	0.057	0.121
Male, n (%)	203 (88.65)	391 (90.09)	65 (84.42)	659 (89.05)	0.611	0.330
Body mass index, mean ± SD	21.21 ± 0.22	20.62 ± 0.17	21 ± 0.46	20.85 ± 0.13	0.223	0.122
Smoking history, n (%)
Never	27 (11.79)	45 (10.37)	16 (20.78)	88 (11.89)	0.187	0.034
Former	108 (47.16)	224 (51.61)	43 (55.84)	375 (50.68)	0.147	0.349
Current	94 (41.05)	165 (38.02)	18 (23.38)	277 (37.43)	0.017	0.020
Pack-year, mean ± SD	35.56 ± 1.73	40.95 ± 1.45	44.37 ± 3.89	39.63 ± 1.09	0.008	0.027
Previous history of any exacerbation, n (%)	156 (68.1)	293 (67.4)	55 (71.4)	504 (68.1)	0.744	0.794
Previous any exacerbation rate, /year	2.4 ± 0.22	3.02 ± 0.19	4.43 ± 0.65	2.98 ± 0.15	<0.001	0.001
Emergency room visit	139 (60.7)	335 (77.2)	67 (87)	541 (73.1)	<0.001	<0.001
Discharge from emergency room^a	53 (40.5)	69 (20.6)	9 (13.4)	131 (24.2)	<0.001	<0.001
Charlson comorbidity index, mean ± SD	3.87 ± 0.12	4.12 ± 0.09	4.01 ± 0.17	4.04 ± 0.07	0.241	0.237
Underlying disease, n (%)
Myocardial infarction	5 (2.18)	14 (3.23)	0 (0)	19 (2.57)	0.676	0.233
Heart failure	17 (7.42)	23 (5.3)	9 (11.69)	49 (6.62)	0.617	0.097
Peripheral vascular disease	29 (12.66)	40 (9.22)	2 (2.6)	71 (9.59)	0.011	0.032
Cerebrovascular disease	18 (7.86)	39 (8.99)	4 (5.19)	61 (8.24)	0.747	0.520
Dementia	6 (2.62)	13 (3)	3 (3.9)	22 (2.97)	0.590	0.849
Peptic ulcer	10 (4.37)	14 (3.23)	3 (3.9)	27 (3.65)	0.640	0.752
Chronic liver disease	17 (7.42)	25 (5.76)	6 (7.79)	48 (6.49)	0.780	0.630
Diabetes	32 (13.97)	59 (13.59)	7 (9.09)	98 (13.24)	0.386	0.520
Chronic kidney disease	24 (10.48)	44 (10.14)	5 (6.49)	73 (9.86)	0.418	0.572
Malignancy	62 (27.07)	157 (36.18)	30 (38.96)	249 (33.65)	0.015	0.036
Respiratory comorbidities, n (%)
Asthma	65 (28.38)	103 (23.73)	17 (22.08)	185 (25)	0.164	0.346
Pulmonary tuberculosis	79 (34.5)	139 (32.03)	24 (31.17)	242 (32.7)	0.496	0.776
NTM lung disease	11 (4.8)	14 (3.23)	0 (0)	25 (3.38)	0.050	0.126
Bronchiectasis	12 (5.24)	17 (3.92)	1 (1.3)	30 (4.05)	0.139	0.308
TB destroyed lung disease	20 (8.73)	33 (7.6)	6 (7.79)	59 (7.97)	0.675	0.876
Interstitial lung disease	1 (0.44)	6 (1.38)	0 (0)	7 (0.95)	0.785	0.324
Inhaled therapy, n (%)	228 (99.56)	426 (98.16)	76 (98.7)	730 (98.65)	0.283	0.328
LAMA	76 (33.19)	126 (29.03)	15 (19.48)	217 (29.32)	0.030	0.072
LABA	7 (3.06)	13 (3)	2 (2.6)	22 (2.97)	0.864	0.978
LAMA/LABA	85 (37.12)	184 (42.4)	31 (40.26)	300 (40.54)	0.349	0.420
ICS/LABA	42 (18.34)	75 (17.28)	9 (11.69)	126 (17.03)	0.254	0.396
ICS/LABA/LAMA	153 (66.81)	321 (73.96)	64 (83.12)	538 (72.7)	0.004	0.014

AE-COPD, acute exacerbation of chronic obstructive pulmonary disease; ER, emergency room; ICS, inhaled corticosteroid; LABA, long-acting β2-agonist; LAMA, long-acting muscarinic antagonist; NTM, non-tuberculosis mycobacterium; SABA, short-acting β2-agonist; SD, standard deviation; TB, tuberculosis.

Data presented as n (%) or mean ± SD.

The percentage of patients in each group was calculated in those who visited ER.

The clinical features are presented according to the AE-COPD severity based on the Rome proposal in Table 2. The mild and moderate groups were more likely to have an mMRC score of 2; scores ⩾3 were more frequently in the severe group. Considering the vital signs, the moderate and severe groups had higher systolic blood pressures, faster HRs and RRs, and lower SpO₂ compared with the mild group. The severe group had lower arterial pH, higher PaCO₂ or HCO₃⁻, and lower SaO₂ compared with mild or moderate group. The complete blood counts demonstrated higher leukocytosis, higher segmented neutrophil counts, and higher hs-CRP among those in moderate and severe groups compared with the mild group. Considering the baseline lung function, the severe group had significantly lower FEV₁ and FVC values than the mild or moderate group in Rome proposal. The patients in the severe group were more likely to receive antibiotics and steroids.

Table 2.

Clinical features of patients according to AE-COPD severity based on the Rome proposal.

	Mild (n = 229, 30.9%)	Moderate (n = 434, 58.6%)	Severe (n = 77, 10.4%)	Total (N = 740)	p for linear trend	p
Respiratory symptoms, n (%)
Cough	99 (43.23)	228 (52.53)	27 (35.06)	354 (47.84)	0.931	0.004
Sputum	132 (57.64)	291 (67.05)	50 (64.94)	473 (63.92)	0.057	0.055
mMRC scale, n (%)
1	4 (1.7)	3 (0.7)	0 (0)	7 (0.9)	0.001	0.272
2	105 (45.9)	183 (42.2)	23 (29.9)	311 (42.0)	0.019	0.049
⩾3	120 (52.4)	248 (57.1)	54 (70.1)	422 (57.0)	0.007	0.025
Vital sign, mean ± SD
Systolic blood pressure, mmHg	132.2 ± 1.61	134.6 ± 1.23	142.7 ± 2.95	134.7 ± 0.93	0.005	0.008
Diastolic blood pressure, mmHg	78.6 ± 1	78.7 ± 0.72	79.8 ± 1.61	78.8 ± 0.55	0.627	0.826
Heart rate, per minute	83.8 ± 0.96	105.9 ± 0.8	99.3 ± 1.96	98.5 ± 0.7	<0.001	<0.001
Respiratory rate, per minute	20.6 ± 0.14	24.3 ± 0.25	27.6 ± 1.14	23.5 ± 0.21	<0.001	<0.001
Body temperature, °C	36.6 ± 0.09	37.1 ± 0.17	37.5 ± 0.83	37 ± 0.13	0.029	0.091
SpO₂, %	95.2 ± 0.16	92.8 ± 0.23	86 ± 1.62	92.8 ± 0.24	<0.001	<0.001
Laboratory examination, mean ± SD
Arterial blood gas analysis
pH	7.4 ± 0	7.4 ± 0	7.3 ± 0.01	7.4 ± 0	<0.001	<0.001
PaCO₂, mmHg	35.8 ± 0.47	37.3 ± 0.49	66.2 ± 1.95	40.9 ± 0.6	<0.001	<0.001
PaO₂, mmHg	81.3 ± 1.66	70.8 ± 1.13	81.5 ± 5.71	74.8 ± 1.14	0.356	<0.001
HCO₃⁻, mmol/L	23.6 ± 0.25	24.2 ± 0.25	31.1 ± 0.77	25 ± 0.22	<0.001	<0.001
SaO₂, %	95.1 ± 0.17	91.9 ± 0.36	88.3 ± 1.21	92.2 ± 0.29	<0.001	<0.001
Complete blood count
WBC, /ml	8834 ± 352	10,702 ± 277	10,021 ± 548	10,057 ± 206	0.003	<0.001
Segmented neutrophil count, /ml	6348 ± 319	8293 ± 255	7598 ± 537	7621 ± 190	0.001	<0.001
Lymphocyte count, /ml	1665 ± 59.91	1608 ± 69	1603 ± 121	1625 ± 46	0.604	0.845
Neutrophil–lymphocyte ratio	4.85 ± 0.38	13.81 ± 5.53	7.61 ± 0.93	10.4 ± 3.25	0.476	0.446
Eosinophil count, cells/mm³	214 ± 16	205 ± 14	196 ± 29	207 ± 10	0.597	0.869
Eosinophil >300 cells/mm³, n (%)	45 (19.91)	89 (20.65)	18 (23.38)	152 (20.71)	0.565	0.810
hs-CRP, mg/dl	4.1 ± 0.4	9.7 ± 0.5	5.7 ± 1.0	7.6 ± 0.4	<0.001	<0.001
Pneumonic consolidation in chest X-ray, n (%)	50 (21.83)	169 (38.94)	23 (29.87)	242 (32.7)	0.004	<0.001
Baseline lung function
Post-bronchodilator
Bronchodilator response, yes	15 (6.6)	39 (9)	7 (9.2)	61 (8.3)	0.534	0.547
FEV₁, L	1.37 ± 0.48	1.27 ± 0.45	0.94 ± 0.41	1.27 ± 0.47	<0.001	0.104
FEV₁, % predicted	58.7 ± 20.44	54.88 ± 19.09	41.46 ± 16.06	54.67 ± 19.8	<0.001	0.105
FVC, L	2.82 ± 0.76	2.71 ± 0.74	2.21 ± 0.89	2.69 ± 0.78	<0.001	0.104
FVC, % predicted	82.53 ± 21.23	79.5 ± 21.22	65.18 ± 22.29	78.96 ± 21.85	<0.001	0.106
FEV₁/FVC ratio	49.31 ± 14.33	47.94 ± 15.03	46.36 ± 17.09	48.2 ± 15.05	0.114	0.104
DL_CO, ml/mmHg/min	11.24 ± 4.24	10.16 ± 3.84	9.58 ± 4.47	10.43 ± 4.06	0.001	0.611
DL_CO, %	74.29 ± 25.96	69.18 ± 24.8	63.26 ± 24.09	70.15 ± 25.26	0.001	0.611
DL_CO/VA, L	2.65 ± 1	2.39 ± 0.92	2.7 ± 1.3	2.5 ± 0.99	0.274	0.461
DL_CO/VA, %	71.73 ± 26.05	65.38 ± 24.02	73.75 ± 34.76	68.15 ± 26.06	0.391	0.460
Radiologic finding
Bronchiectasis	52 (26.9)	96 (24.2)	16 (22.5)	164 (24.8)	0.398	0.693
TB-destroyed lung	25 (13)	33 (8.3)	15 (21.1)	73 (11.1)	0.474	0.004
Treatment, n (%)
Antibiotics	174 (75.98)	394 (90.78)	73 (94.81)	641 (86.62)	<0.001	<0.001
Steroid	111 (48.47)	273 (62.9)	65 (84.42)	449 (60.68)	<0.001	<0.001

AE-COPD, acute exacerbation of chronic obstructive pulmonary disease; DL_CO, diffusing capacity of the lungs for carbon monoxide; FEV₁, forced expiratory volume in 1 s; FVC, forced vital capacity; hs-CRP, high-sensitivity C-reactive protein; mMRC, Modified Medical Research Council dyspnea scale; SD, standard deviation; WBC, white blood cell.

Data presented as n (%) or mean ± SD.

Clinical outcomes

Clinical outcomes according to AE-COPD severity based on the Rome proposal are described in Table 3. A greater number of patients in the severe group were admitted to the ICU (24.7% versus 2.1% and 0, p < 0.001) and showed prolonged length of stay (LOS) in the ICU (1.5 days versus 0.1 and 0 day, p < 0.001) than those in the moderate and mild groups. LOS in hospital was significantly prolonged in the severe group compared with the moderate and mild groups (8 days versus 7 and 5 days, p < 0.001). The oxygen requirement, including conventional oxygen therapy or high-flow nasal oxygen, significantly increased with AE-COPD severity. More number of patients received NIV (19.5% versus 1.84% and 0%, p < 0.001) or IMV (18.2% versus 0.5% and 0%, p < 0.001) in the severe group than in the moderate and mild groups. Reintubation (1.3% versus 0% and 0%, p = 0.013) and tracheostomy (2.6% versus 0% and 0%, p < 0.001) were also required more often in the severe group compared with the moderate and mild groups. Furthermore, the severe group had a higher in-hospital mortality (6.5% versus 1.2% and 0%, p < 0.001) than the moderate and mild groups.

Table 3.

Clinical outcomes according to AE-COPD severity based on the Rome proposal.

	Mild (n = 229, 30.9%)	Moderate (n = 434, 58.6%)	Severe (n = 77, 10.4%)	Total (N = 740)	p for linear trend	p
ICU admission, n (%)	0	9 (2.07)	19 (24.68)	28 (3.78)	<0.001	<0.001
ICU LOS, days, mean ± SD	0	0.1 ± 0.05	1.5 ± 0.4	0.2 ± 0.1	<0.001	<0.001
Hospital LOS, days, median (IQR)	5 (2–9)	7 (4–11)	8 (5–17)	6 (3–10)	0.169	<0.001
Oxygen delivery equipment
Conventional oxygen therapy	55 (24.02)	237 (54.61)	56 (72.73)	348 (47.03)	<0.001	<0.001
High-flow nasal oxygen	0	6 (1.38)	3 (3.9)	9 (1.22)	0.008	0.023
Non-invasive ventilation	0	8 (1.84)	15 (19.48)	23 (3.11)	<0.001	<0.001
Mechanical ventilation	0	2 (0.46)	14 (18.18)	16 (2.16)	<0.001	<0.001
Reintubation	0	0 (0)	1 (1.3)	1 (0.14)	0.048	0.013
Tracheostomy	0	0 (0)	2 (2.6)	2 (0.27)	0.005	<0.001
In-hospital mortality	0	5 (1.15)	5 (6.49)	10 (1.35)	<0.001	<0.001

AE-COPD, acute exacerbation of chronic obstructive pulmonary disease; ICU, intensive care unit; IQR, interquartile range; LOS, length of stay; SD, standard deviation.

Data presented as n (%) or mean (SD) and median (IQR).

Kaplan–Meier estimation

Kaplan–Meier estimation with log-rank test demonstrated significantly different time to events of the clinical outcomes according to AE-COPD severity based on the Rome proposal (Figure 1). The severe group had a higher risk of ICU admission (p < 0.001), need for NIV or IMV (p < 0.001), and in-hospital mortality (p < 0.001) compared with the mild or moderate groups. The moderate group had a higher risk of ICU admission (p = 0.039) and need for NIV or IMV (p = 0.039) than the mild group.

Figure 1.

Kaplan–Meier estimation according to the severity of AE-COPD based on Rome proposal: (a) Log-rank test for ICU admission: mild versus moderate, p = 0.039; mild versus severe, p < 0.001; moderate versus severe, p < 0.001. (b) Log-rank test for NIV or IMV: mild versus moderate, p = 0.039; mild versus severe, p < 0.001; moderate versus severe, p < 0.001. (c) Log-rank test for in-hospital mortality: mild versus moderate, p = 0.100; mild versus severe, p < 0.001; moderate versus severe, p = 0.002.

Predictive performance of different AE-COPD severity classifications

Prognostic accuracies to predict ICU admission, need for NIV or IMV, and in-hospital mortality according to the Rome proposal, DECAF score, and GesEPOC 2021 criteria are summarized in Table 4. Sensitivity under optimal cut-offs for ICU admission and need for NIV or IMV were significantly higher in the Rome proposal than in the GesEPOC 2021 criteria (p = 0.008 and 0.003). Specificity under optimal cut-offs for need for NIV or IMV was also significantly higher in the Rome proposal compared with DECAF score (p = 0.047). The Rome proposal showed a lower sensitivity (p = 0.012) and a higher specificity (p = 0.010) for in-hospital mortality compared with GesEPOC 2021 criteria. AU-ROCs of the Rome proposal were 0.850 [95% confidence interval (CI) = 0.789–0.911] for ICU admission, 0.870 (95% CI = 0.817–0.924) for NIV or IMV, and 0.779 (95% CI = 0.666–0.892) for in-hospital mortality (Figure 2). Although the AU-ROCs for ICU admission, NIV or IMV, and in-hospital mortality were higher in the Rome proposal compared with the DECAF score, these were not statistically significant. The Rome proposal had significantly better predictive performances for ICU admission (AU-ROC = 0.850 versus 0.736, p = 0.004) and need for NIV or IMV (AU-ROC = 0.870 versus 0.770, p = 0.004) than GesEPOC 2021 criteria.

Table 4.

Prognostic accuracy of the Rome proposal, DECAF score, and GesEPOC 2021 criteria.

	Severity classification	Optimal cut-off	Sensitivity	Specificity	AU-ROC	95% CI
ICU admission	Rome proposal	Severe	0.679	0.919	0.850	0.789–0.911
	DECAF score	Intermediate	0.821	0.742	0.810	0.729–0.892
	GesEPOC 2021	Very high	0.321	0.985	0.736	0.657–0.814
NIV or IMV	Rome proposal	Severe	0.727	0.925	0.870	0.817–0.924
	DECAF score	Intermediate	0.818	0.745	0.801	0.726–0.875
	GesEPOC 2021	Very high	0.363	0.989	0.770	0.705–0.835
In-hospital mortality	Rome proposal	Severe	0.500	0.901	0.779	0.666–0.892
	DECAF score	Intermediate	0.700	0.726	0.738	0.573–0.903
	GesEPOC 2021	High	1.000	0.326	0.752	0.651–0.854

AU-ROC, area under the receiver operating characteristic curve; CI, confidence interval; DECAF, dyspnea with eMRCD (dyspnea with extended Medical Research Council dyspnea), eosinopenia, consolidation, acidemia, atrial fibrillation; GesEPOC, Spanish COPD Guidelines; ICU, intensive care unit; IMV, invasive mechanical ventilation; NIV, non-invasive ventilation.

Figure 2.

ROC curve of Rome proposal, DECAF score, and GesEPOC 2021 criteria: (a) Delong’s tests for ICU admission: the Rome proposal versus DECAF score, p = 0.294; the ROME proposal versus GesEPOC 2021 criteria, p = 0.004. (b) Delong’s tests for NIV or MV: the Rome proposal versus DECAF score, p = 0.085; the ROME proposal versus GesEPOC 2021 criteria, p = 0.004. (c) Delong’s tests for in-hospital mortality: the Rome proposal versus DECAF score, p = 0.634; the ROME proposal versus GesEPOC 2021 criteria, p = 0.564.

Subgroup analysis

The AU-ROCs of the Rome proposal, DECAF score, and GesEPOC 2021 criteria were estimated according to a subgroup analysis. AU-ROC of the Rome proposal to predict ICU admission was significantly higher among female patients compared with male patients (p < 0.001) and never smokers compared with ever smokers (p = 0.002) (Supplemental Figure S1). The Rome proposal showed better accuracy in predicting ICU admission compared with DECAF score in female patients (p = 0.021). Predictive performance for ICU admission of the Rome proposal was significantly higher compared with GesEPOC 2021 criteria in patients aged <65 years (p = 0.012), male patients (p = 0.002), never smokers (p = 0.010), and ever smokers (p = 0.012). The Rome proposal showed better accuracy in predicting the need for NIV or IMV in patients with age <65 compared with those with age ⩾65 (p < 0.001), in female patients compared with male patients (p = 0.033), and in never smokers compared with ever smokers (p = 0.002) (Supplemental Figure S2). The Rome proposal showed better accuracy in predicting ICU admission compared with DECAF score in male (p = 0.018) and female patients (p = 0.037). Predictive accuracy for NIV or IMV of the Rome proposal was significantly higher compared with GesEPOC 2021 criteria in age <65 years (p = 0.003), male (p = 0.008) and female patients (p < 0.001), and ever smokers (p = 0.017). The Rome proposal showed a better accuracy to predict in-hospital mortality in age <65 compared with age ⩾65 (p < 0.001) (Supplemental Figure S3). We found no significant difference in the predictive performance between the Rome proposal and DECAF score or GesEPOC 2021 criteria in the subgroup analyses.

Concordance between different severity classifications

Agreement profiles between the different severity classifications of AE-COPD are summarized in Supplemental Table S1. The overall agreement rate was 38.9% between the Rome proposal and DECAF score and 30.8% between the Rome proposal and GesEPOC 2021 criteria. Cohen’s kappa coefficients indicated no agreements between the Rome proposal and DECAF score [κ = 0.180 (95% CI = 0.108–0.260)] and the Rome proposal and GesEPOC 2021 criteria [κ = 0.170 (95% CI = 0.129–0.210)].

Discussion

Our study validated the predictive performance of AE-COPD severity classification based on the Rome proposal in Korean patients who visited ER or were hospitalized for AE-COPD. Considering that AE-COPD could be triggered or aggravated by respiratory infection, patients with pneumonia were not excluded. There were more never smokers in the severe group and more current smokers in the mild group. This paradoxical finding could be owing to different health behaviors among the groups with varying severities. The severe group demonstrated more radiologic structural abnormalities and had significantly lower FEV₁, FVC, or DL_CO (diffusing capacity of the lungs for carbon monoxide) values than the mild or moderate groups. Severe exacerbation was most likely contributed by radiologic structural abnormalities or lung function impairment, which are clinical features often found in never smokers.

AE-COPD severity based on the Rome proposal had a significant positive association with the incidence of adverse outcomes including ICU admission, need for NIV or IMV, and in-hospital mortality. Severe group demonstrated a higher risk of ICU admission, need for NIV or IMV, or in-hospital mortality compared with mild and moderate groups. AE-COPD severity classification based on the Rome proposal showed an excellent performance in predicting ICU admission and need for NIV or IMV and an acceptable performance in predicting in-hospital mortality. In predicting ICU admission and need for NIV or IMV, the prognostic performances of the Rome proposal were superior to those of GesEPOC 2021 criteria, while comparable with those of DECAF score. Furthermore, the Rome proposal may have a better predictive performance in patients with age <65 years, female patients, and never smokers. Therefore, AE-COPD severity classification based on the Rome proposal is considered a useful prediction tool for in-hospital mortality, need for NIV or IMV, and ICU admission among patients with AE-COPD.

To the best of our knowledge, our study is the first to validate whether the Rome proposal can predict clinically important outcomes in patients with AE-COPD and to compare the Rome proposal with other AE-COPD severity classifications including DECAF score and GesEPOC 2021 criteria. DECAF score can predict in-hospital mortality with AU-ROC of 0.820 in hospitalized patients with AE-COPD.¹⁵ In a meta-analysis for AE-COPD, DECAF score performed better in predicting in-hospital mortality with AU-ROC of 0.83 compared with other severity scores including CURB-65 (confusion, urea >7 mmol/L, RR ≥30/min, blood pressure systolic <90 mmHg or diastolic <60 mmHg and age >65 years), BAP-65 (blood urea nitrogen, altered mental status, pulse rate and age >65 years), Acute Physiology and Chronic Health Evaluation (APACHE-II), and COPD and Asthma Physiology Score (CAPS).¹⁶ In addition, a study demonstrated that DECAF score may be promising for the prediction of IMV in hospitalized patients with AE-COPD.¹⁷ Spanish COPD guidelines had suggested the GesEPOC 2017 criteria for AE-COPD assessment, which was validated in a prospective study to have an AU-ROC of 0.721 for overall mortality.¹⁸ Recently, the GesEPOC 2017 criteria has been revised to the GesEPOC 2021 criteria by excluding lung function and adding physiologic parameters related to acute episode; however, validation has not been conducted.¹⁰ Our study showed that the Rome proposal had similar predictive performance for in-hospital mortality compared to DECAF score, although AU-ROC was numerically higher in the Rome proposal. The performance of the GesEPOC 2021 criteria for in-hospital mortality was also evaluated in our study and shown to be similar to that of the Rome proposal.

The Rome proposal consists of clinical variables that were different from those included in the DECAF score and GesEPOC 2021 criteria. First, VAS was used to measure the intensity of dyspnea in the Rome proposal while the DECAF score or GesEPOC 2021 criteria used mMRC scale or eMRCD score. The simplicity of VAS facilitates the evaluation of daily variations of dyspnea in the outpatient clinic.¹⁹ VAS has the advantage of quantitatively expressing ventilatory load in a 0–10 scale.¹⁹ Compared with mMRC and the Borg scale, VAS correlated well with spirometric parameters in patients with COPD.²⁰ Second, different physiologic parameters and inflammatory biomarkers were used in the Rome proposal. The Rome proposal used HR as a physiologic parameter to predict the prognosis of AE-COPD. Cardiovascular event during AE-COPD is a risk factor for mortality.²¹ Tachycardia is one of the physiologic parameters aiding the prognostication in patients with AE-COPD. Increased HR is associated with mortality²² and a higher risk of hospitalization²³ during AE-COPD. In addition, serum CRP is an important inflammatory biomarker for diagnosing AE-COPD with AU-ROC of 0.73, which could be improved to 0.97 with a combination of dyspnea severity and blood neutrophil counts.^24,25 Since elevated CRP is associated with the need for antibiotics and higher mortality in patients with AE-COPD, it is considered an important indicator for deciding the treatment and predicting the prognosis.^26,27 However, eosinopenia or atrial fibrillation used in the DECAF score was not applied to the Rome proposal. Third, PaCO₂ and pH were evaluated only to determine the severe group in the Rome proposal, which is different from the DECAF score or GesEPOC 2021 criteria. There is a significant association between acute hypercapnic respiratory failure and mortality.^28,29 In a current guideline, NIV was strongly recommended for persistent hypercapnic respiratory failure in patients with AE-COPD.²⁹ The severe group in the Rome proposal and the need for NIV are highly correlated in that they share same criteria for hypercapnia and respiratory acidosis. Considering that intensive care is necessary for patients with acute hypercapnic respiratory failure, it would be relevant to exclusively categorize the patients with PaCO₂ >45 mmHg and pH <7.35 into the severe group.

The Rome proposal is valuable because of its high specificity. It is important for an AE-COPD severity assessment tool to accurately specify the patients at high risk for adverse outcomes. Particularly, the Rome proposal demonstrated excellent performance for identifying the patients who are at high risk for ICU admission or need for NIV or IMV with specificity ⩾0.9. Therefore, it is reasonable to consider hospitalization or ICU care for the patients classified in the severe group. Moreover, patients with AE-COPD in the mild group did not actually experience any adverse outcome. Consequently, the patients who were categorized into the mild group would not require hospitalization, and early discharge can be considered in case they have already been hospitalized. Therefore, we speculate that the mild group could be treated in outpatient clinic, the moderate group needs hospitalization, and the severe group needs hospitalization and close observation for further intensive care. However, our study discovered that only 40% of the patients in the mild group were discharged and the others were hospitalized. This is mainly because the objective criteria for hospital admission in the patients with AE-COPD are not currently available. AE-COPD severity assessment through the Rome proposal may reduce unnecessary hospitalization.

Since COPD is a heterogeneous disease, the prognosis of AE-COPD also varies ways depending on the clinical phenotypes of COPD. Elderly, male, and ever-smoker patients have been associated with a poor prognosis of AE-COPD.^30–32 Our study demonstrated that these subgroups showed a lower accuracy of AE-COPD severity classification based on the Rome proposal. In these subgroups, the lower accuracy mainly contributed to lower sensitivity in the moderate group, while specificity was maintained as high as the other patients. Therefore, even though AE-COPD is evaluated as moderate severity based on the Rome proposal in elderly, male, and ever-smoker patients, it would be better to be prepared for a worse prognosis. In addition, we need to consider clinical variables that do not directly reflect the severity of AE-COPD, such as comorbidities or concurrent inhalation therapy. The current definition of the Rome proposal should not be considered as a completed version and needs to be improved.

This study has certain limitations. First, our study may have a selection bias because we included the patients who visited the ER or were hospitalized for AE-COPD. In addition, we included the patients with VAS ⩾5, which was assumed as a corresponding degree of dyspnea for ER visit or hospitalization.^7,33–36 This patient selection was to obtain sufficient objective information and to assess the severity of AE-COPD. Therefore, the predictive performance of the Rome proposal is still unclear in less symptomatic patients with AE-COPD. In addition, ICU admission can be affected by preferences or values of the patients or their families, and medical resource capacities of the ICU. Therefore, the predictive performance of the Rome proposal could be altered based on the comorbidity, patient’s refusal on intensive care, and health care resources. Second, the retrospective nature of our study may prohibit accurate measurement of symptom parameters. As VAS and eMRCD scores are not routinely obtained in clinical practice, we assumed the VAS and eMRCD scores from medical records. In cases of altered level of consciousness or inability to express symptoms, the severity of dyspnea was inferred by asking the nearest family member. Considering the association between the severity of dyspnea and mortality in a previous study in the United Kingdom, the severity of dyspnea may be exaggerated in our study, especially in mild group.⁹ Third, it is difficult to suggest optimal medical treatments according to the AE-COPD severity based on the Rome proposal. In the GOLD report, mild severity of AE-COPD has been defined as aggravation of respiratory symptoms with the need for medical therapy other than corticosteroid or antibiotics.¹ However, in our study, more than half of the patients in the mild group used antibiotics or steroids. In addition, although the severe group had a lower CRP than moderate group, greater amount of antibiotics were used in the severe group than the moderate group. Considering that CRP is a good indicator for the use of antibiotics, it may not be appropriate to prescribe antibiotics according to the severity of AE-COPD based on the Rome proposal.³⁷

Conclusion

In this validation study, the Rome proposal for AE-COPD severity assessment demonstrated excellent performance in predicting ICU admission and need for NIV or IMV and acceptable performance in predicting in-hospital mortality. The Rome proposal was superior to the GesEPOC 2021 criteria in predicting ICU admission and need for NIV or IMV. The Rome proposal can be a useful prediction tool for in-hospital mortality, need for NIV or IMV, and ICU admission in patients with AE-COPD.

Supplemental Material

sj-docx-1-tar-10.1177_17534666231172917 – Supplemental material for Validation of the Rome proposal for severity of acute exacerbation of chronic obstructive pulmonary disease

Supplemental material, sj-docx-1-tar-10.1177_17534666231172917 for Validation of the Rome proposal for severity of acute exacerbation of chronic obstructive pulmonary disease by Hyo Jin Lee, Jung-Kyu Lee, Tae Yun Park, Eun Young Heo, Deog Kyeom Kim and Hyun Woo Lee in Therapeutic Advances in Respiratory Disease

Supplemental Material

sj-tif-2-tar-10.1177_17534666231172917 – Supplemental material for Validation of the Rome proposal for severity of acute exacerbation of chronic obstructive pulmonary disease

Supplemental material, sj-tif-2-tar-10.1177_17534666231172917 for Validation of the Rome proposal for severity of acute exacerbation of chronic obstructive pulmonary disease by Hyo Jin Lee, Jung-Kyu Lee, Tae Yun Park, Eun Young Heo, Deog Kyeom Kim and Hyun Woo Lee in Therapeutic Advances in Respiratory Disease

Supplemental Material

sj-tif-3-tar-10.1177_17534666231172917 – Supplemental material for Validation of the Rome proposal for severity of acute exacerbation of chronic obstructive pulmonary disease

Supplemental material, sj-tif-3-tar-10.1177_17534666231172917 for Validation of the Rome proposal for severity of acute exacerbation of chronic obstructive pulmonary disease by Hyo Jin Lee, Jung-Kyu Lee, Tae Yun Park, Eun Young Heo, Deog Kyeom Kim and Hyun Woo Lee in Therapeutic Advances in Respiratory Disease

Supplemental Material

sj-tif-4-tar-10.1177_17534666231172917 – Supplemental material for Validation of the Rome proposal for severity of acute exacerbation of chronic obstructive pulmonary disease

Supplemental material, sj-tif-4-tar-10.1177_17534666231172917 for Validation of the Rome proposal for severity of acute exacerbation of chronic obstructive pulmonary disease by Hyo Jin Lee, Jung-Kyu Lee, Tae Yun Park, Eun Young Heo, Deog Kyeom Kim and Hyun Woo Lee in Therapeutic Advances in Respiratory Disease

Footnotes

Acknowledgements

None.

Declarations

ORCID iD

Hyun Woo Lee

Supplemental material

Supplemental material for this article is available online.

References

Global Initiative for Chronic Obstructive Lung Disease. Global strategy for diagnosis, management and prevention of chronic obstructive pulmonary disease, 2022, https://goldcopd.org/2022-gold-reports/

Global Initiative for Chronic Obstructive Lung Disease. Global strategy for the diagnosis, management and prevention of chronic obstructive pulmonary disease, 2021, https://goldcopd.org/wp-content/uploads/2020/11/GOLD-REPORT-2021-v1.1-25Nov20_WMV.pdf.

Wildman

Sanderson

Groves

, et al. Predicting mortality for patients with exacerbations of COPD and asthma in the COPD and asthma outcome study (CAOS). QJM 2009; 102: 389–399.

Jeppesen

Brurberg

Vist

, et al. Hospital at home for acute exacerbations of chronic obstructive pulmonary disease. Cochrane Database Syst Rev 2012: CD003573.

Calverley

PM.

Minimal clinically important difference-exacerbations of COPD. COPD 2005; 2: 143–148.

Barbosa-Lorenzo

Ruano-Ravina

Fernández-Villar

, et al. COPD prevalence and hospital admissions in Galicia (Spain). Pulmonology 2018; 24: 323–329.

Celli

Fabbri

Aaron

, et al. An updated definition and severity classification of chronic obstructive pulmonary disease exacerbations: the Rome proposal. Am J Respir Crit Care Med 2021; 204: 1251–1258.

Steer

Gibson

Bourke

SC.

The DECAF score: predicting hospital mortality in exacerbations of chronic obstructive pulmonary disease. Thorax 2012; 67: 970–976.

Steer

Norman

Afolabi

, et al. Dyspnoea severity and pneumonia as predictors of in-hospital mortality and early readmission in acute exacerbations of COPD. Thorax 2012; 67: 117–121.

10.

Soler-Cataluña

Piñera

Trigueros

, et al. Spanish COPD guidelines (GesEPOC) 2021 update diagnosis and treatment of COPD exacerbation syndrome. Arch Bronconeumol 2022; 58: 159–170.

11.

Von Elm

Altman

Egger

, et al. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet 2007; 370: 1453–1457.

12.

Mandrekar

JN.

Receiver operating characteristic curve in diagnostic test assessment. J Thorac Oncol 2010; 5: 1315–1316.

13.

DeLong

Clarke-Pearson

DL.

Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988; 44: 837–845.

14.

McHugh

ML.

Interrater reliability: the kappa statistic. Biochem Med 2012; 22: 276–282.

15.

Echevarria

Steer

Bourke

SC.

Comparison of early warning scores in patients with COPD exacerbation: DECAF and NEWS score. Thorax 2019; 74: 941–946.

16.

Huang

Xiong

, et al. DECAF score as a mortality predictor for acute exacerbation of chronic obstructive pulmonary disease: a systematic review and meta-analysis. BMJ Open 2020; 10: e037923.

17.

Sangwan

Chaudhry

Malik

Dyspnea, eosinopenia, consolidation, acidemia and atrial fibrillation score and BAP-65 score, tools for prediction of mortality in acute exacerbations of chronic obstructive pulmonary disease: a comparative pilot study. Indian J Crit Care Med 2017; 21: 671–677.

18.

Cabrera López

Casanova Macario

Marín Trigo

, et al. Prognostic validation using GesEPOC 2017 severity criteria. Arch Bronconeumol 2019; 55: 409–413.

19.

Gift

Narsavage

Validity of the numeric rating scale as a measure of dyspnea. Am J Crit Care 1998; 7: 200–204.

20.

Camargo

Pereira

CA.

Dyspnea in COPD: beyond the modified medical research council scale. J Bras Pneumol 2010; 36: 571–578.

21.

Laratta

Van Eeden

Acute exacerbation of chronic obstructive pulmonary disease: cardiovascular links. Biomed Res Int 2014; 2014: 528789.

22.

Jensen

Marott

Lange

, et al. Resting heart rate is a predictor of mortality in COPD. Eur Respir J 2013; 42: 341–349.

23.

Borel

Pelletier

Taleux

, et al. Parameters recorded by software of non-invasive ventilators predict COPD exacerbation: a proof-of-concept study. Thorax 2015; 70: 284–285.

24.

Celli

Decramer

Wedzicha

, et al. An official American Thoracic Society/European Respiratory Society statement: research questions in COPD. Eur Respir Rev 2015; 24: 159–172.

25.

Hurst

Donaldson

Perera

, et al. Use of plasma biomarkers at exacerbation of chronic obstructive pulmonary disease. Am J Respir Crit Care Med 2006; 174: 867–874.

26.

Prins

Duijkers

van der Valk

, et al. CRP-guided antibiotic treatment in acute exacerbations of COPD in hospital admissions. Eur Respir J 2019; 53: 1802014.

27.

Leuzzi

Galeone

Taverna

, et al. C-reactive protein level predicts mortality in COPD: a systematic review and meta-analysis. Eur Respir Rev 2017; 26: 160070.

28.

Barberà

Roca

Ferrer

, et al. Mechanisms of worsening gas exchange during acute exacerbations of chronic obstructive pulmonary disease. Eur Respir J 1997; 10: 1285–1291.

29.

Rochwerg

Brochard

Elliott

, et al. Official ERS/ATS clinical practice guidelines: noninvasive ventilation for acute respiratory failure. Eur Respir J 2017; 50: 1602426.

30.

Stone

Lowe

Potter

, et al. Managing patients with COPD exacerbation: does age matter. Age Ageing 2012; 41: 461–468.

31.

Aryal

Diaz-Guzman

Mannino

DM.

Influence of sex on chronic obstructive pulmonary disease risk and treatment outcomes. Int J Chron Obstruct Pulmon Dis 2014; 9: 1145–1154.

32.

Xue

, et al. Smoking status affects clinical characteristics and disease course of acute exacerbation of chronic obstructive pulmonary disease: a prospectively observational study. Chron Respir Dis 2020; 17: 1479973120916184.

33.

Scioscia

Blanco

Arismendi

, et al. Different dyspnoea perception in COPD patients with frequent and infrequent exacerbations. Thorax 2017; 72: 117–121.

34.

Marin

Montes

Oca

Rassulo

, et al. Ventilatory drive at rest and perception of exertional dyspnea in severe COPD. Chest 1999; 115: 1293–1300.

35.

Pinto-Plata

Livnat

Girish

, et al. Systemic cytokines, clinical and physiological changes in patients hospitalized for exacerbation of COPD. Chest 2007; 131: 37–43.

36.

Noell

Cosío

Faner

, et al. Multi-level differential network analysis of COPD exacerbations. Eur Respir J 2017; 50: 1700075.

37.

Butler

Gillespie

White

, et al. C-reactive protein testing to guide antibiotic prescribing for COPD exacerbations. N Engl J Med 2019; 381: 111–120.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.48 MB

0.47 MB