Sage Journals: Discover world-class research

Abstract

Background

Metastasis is one of the most significant prognostic factors in osteosarcoma (OS). The goal of this study was to construct a clinical prediction model for OS patients in a population cohort and to evaluate the factors influencing the occurrence of pulmonary metastasis.

Methods

We collected data from 612 patients with osteosarcoma (OS), and 103 clinical indicators were collected. After the data were filtered, the patients were randomly divided into training and validation cohorts by using random sampling. The training cohort included 191 patients with pulmonary metastasis in OS and 126 patients with non-pulmonary metastasis, and the validation cohort included 50 patients with pulmonary metastasis in OS and 57 patients with non-pulmonary metastasis. Univariate logistics regression analysis, LASSO regression analysis and multivariate logistic regression analysis were performed to identify potential risk factors for pulmonary metastasis in patients with osteosarcoma. A nomogram was developed that included risk influencing variables selected by multivariable analysis, and used the concordance index (C-index) and calibration curve to validate the model. Receiver operating characteristic curve (ROC), decision analysis curve (DCA) and clinical impact curve (CIC) were employed to assess the model. In addition, we used a predictive model on the validation cohort.

Results

Logistic regression analysis was used to identify independent predictors [N Stage + Alkaline phosphatase (ALP)+Thyroid stimulating hormone (TSH)+Free triiodothyronine (FT3)]. A nomogram was constructed to predict the risk of pulmonary metastasis in patients with osteosarcoma. The performance was evaluated by the concordance index (C-index) and calibration curve. The ROC curve provides the predictive power of the nomogram (AUC = 0.701 in the training cohort, AUC = 0.786 in the training cohort). Decision curve analysis (DCA) and clinical impact curve (CIC) demonstrated the clinical value of the nomogram and higher overall net benefits.

Conclusions

Our study can help clinicians effectively predict the risk of lung metastases in osteosarcoma with more readily available clinical indicators, provide more personalized diagnosis and treatment guidance, and improve the prognosis of patients.

Mini Abstract

A new risk model was constructed to predict the pulmonary metastasis in patients with osteosarcoma based on multiple machine learning.

Keywords

Osteosarcoma pulmonary metastasis nomogram predictive value of clinical indicators machine learning

Introduction

Osteosarcoma (OS) is the most common primary malignancy of the skeletal system with a high propensity for local invasion and metastasis, characterized by a high risk of tumor progression such as metastasis or recurrence after treatment.¹ Generally speaking, metastasis is one of the most significant factors affecting the prognosis of patients with OS and remains the biggest obstacle to improve the prognosis of OS.^2,3,4 Even after surgery and chemotherapy, there is unfavourable prognosis in patients with metastatic OS.^5,6 The lung is the main site of metastasis in patients with OS.^3,7 Therefore, it’s crucial to assess the status of pulmonary metastasis accurately for improving the long-term survival rates. However, there is no effective method to evaluate the status of pulmonary metastasis and predict the probability of metastasis in patients with OS.⁸

The global annual incidence of osteosarcoma is about 3-4 per million.⁹ There is a substantial amount of patients with OS in the world, but the evidence-based information about OS patients is still limited and controversial, and there are few nomogram models being reported for the prognosis of OS patients, let alone models for predicting the risk of pulmonary metastasis of OS.^10,11 However, they focused primarily on a single factor such as inflammatory factor¹² or radiomic factor.¹³ In practice, a single factor can’t anticipate the disease progression satisfactorily. Most importantly, there is a lack of reliable information about the effects on prognosis. According to statistics, benefit from the age of neoadjuvant chemotherapy and limb salvage, the overall survival rate for non-metastatic OS has increased from less than 20% to about 70%. However, the survival rate is still less than 20% in patients with distant metastasis or local recurrence.¹⁰

Therefore, the purpose of this study is to predict the influencing factors of the occurrence of pulmonary metastasis in OS, so as to provide more help for guiding therapeutic intervention, which is helpful to prolong the overall survival and improve the quality of life of patients. The nomogram is an effective model for visual regression analysis because it can establish scoring criteria based on the regression coefficients of all independent variables.¹⁴ Compared with other prediction models, the prediction based on nomogram is more accurate. This approach has been proposed as an alternative or even a new standard approach, and has been widely used as a prognostic tool for many types of tumors, such as hepatocellular carcinoma, lung cancer, renal cell carcinoma and colorectal cancer.^15–19 However, there are still few researches that build nomogram to predict the risk of pulmonary metastasis in OS with more clinical indicators. Our study was aimed to investigate the prognostic significance of 103 clinical factors including surgery, primary site,

N Stage, clinical stages (AJCC 8th), WBC, coagulation factors, transaminase, bilirubin, platelet, ALP, K, Ca, TSH, FT4, FT3, etc. In addition, we hope to use clinical indicators to establish a more accurate prognostic model based nomogram to assess the risk of pulmonary metastasis in OS.

Materials and methods

Study population

This was a retrospective cohort study. We collected datas from 612 patients with osteosarcoma from January 2013 to June 2021, who were treated in the AAA Hospital, the BBB Hospital and the CCC Hospital, and collected 103 clinical features and routine blood biochemical indexes of each patient at the time of diagnosis. Approved by Ethics Committee of Affiliated Hospital of Guangxi Medical University. The study was carried out in accordance with the Declaration of Helsinki. In consideration of the retrospective nature of the research, informed consent was not obtained. The inclusion criteria were as follows: (a) Primary OS

was confirmed by pathology; (b) There was radiographic evidence of pulmonary metastasis or pathological evidence of lung biopsy; (c) there were no histories of other malignancies or synchronous cancer; (d) Patients with complete hematological results; (e) Patients received standard treatment; (f) Complete follow-up datas. The exclusion criteria were as follows: (a) Patients with Synchronous cancers; (b) Lacked of laboratory results; (c) lost to follow-up. We excluded data from 188 patients based on inclusion and exclusion criteria. Patients were excluded for the following reasons: incomplete data (n = 117); History of Synchronous Neoplasms (n = 26); Lost to follow-up (n = 45). All patients underwent routine blood biochemical tests, OS magnetic resonance imaging;MRI , bone scans;ECT;and chest computed tomography (CT) scans at diagnosis. The final diagnosis and classification were determined by several senior orthopedic surgeons and radiologists after evaluation of the radiographic and pathological diagnosis reports. Ultimately, 424 patients were incorporated into our study. After a year and a half follow-up, the patients were further divided into two

groups according to the presence (MT) or absence (Non-MT) of lung metastases. Patients from the MT group were those with multiple nodules at presentation or nonspecific nodules that increased in number or size during the follow-up on chest CT. Patients from the non-MT group have not demonstrated any suspicious nodules on chest CT exams during a year and a half follow-up.²⁰ All data has been anonymized and de-identified. Figure 1 Compared the medical imaging pictures of one patient with osteosarcoma with pulmonary metastasis and the other without pulmonary metastasis, including the X-ray of osteosarcoma lesions, the X-ray and CT of the lungs, and clearly identified the pulmonary metastatic lesions from the image of the metastatic group.

Figure 1.

Compare osteosarcoma with pulmonary metastasis (a, b and c) to osteosarcoma without pulmonary metastasis (d, e and f). (a) X-ray of osteosarcoma. (b) X-ray of chest. (c) CT of the Lungs. (d) X-ray of osteosarcoma. (e) X-ray of chest. (f) CT of the Lungs.

Data collection and analysis

Laboratory Hematologic data in patients with osteosarcoma included WBC, LYM%, MO%, NEU%, LYM, MO, NEU, RBC, Hb, BLT, EOSR%, BASOR%, EOSR, BASOR, ALT, TP, ALB, TRF, AST, AST/ALT, GGT, ALP, LDH, LPS, SF, K, NA, Cl, Ca, UREA, CREA, Ccr, UA, CK, CK-MB, FBG, hs-TnI, MYO, PT, APTT, FIB, TT, D-dimer, CRP, hs-CRP, ESR, PCT, C3, C4, IgG, IgA, IgM, T-lym, Th-lym, Ts-lym, Th/Ts, NKC, B-lym, CEA, AFP, CA-125, CA-153, CA-199, SCC-Ag, cyfra21-1, TSH, T4, T3, FT4, FT3, TG, TCHO, TAG, HDL-C and LDL-C. In addition, we collected clinical features, such as pulmonary metastasis, Age, Gender, Marriage, Nation, Systolic pressure, Diastolic pressure, BMI, Smoke, Drink, Surgery, Primary site, N Stage, Clinical stages, Neoadjuvant chemotherapy, Postoperative chemotherapy and Radiotherapy, from the patients' medical records.

Statistical analysis

Statistical analysis was performed using IBM SPSS Statistics Version 25.0 (SPSS Inc., Chicago, USA) and R 4.2.1 (http://www.R-project.org). Continuous variables were expressed as mean ± standard deviation or median (quartile distance, IQR) and were compared using the t test (for normal distribution continuous variables) or the Fisher’s exact test (for non-normal distribution continuous variables). The categorical variables were expressed as proportions and compared by chi-square test. Assessed the association between variables and pulmonary metastasis using logistic regression models. All analyses were performed using R software (version 4.2.1) and used multiple R packages (including glmnet, rms, hmisc, ROCR, etc.). The accuracy of the predictive nomogram was evaluated by calibration curve. The discriminability of the established nomogram was calculated by receiver operating characteristic (ROC) curve. Decision curve analysis (DCA) is used to further evaluate the performance of the prediction model. A two-sided p < .05 was considered statistically significant, and confidence intervals (CI) were expressed as 95% confidence levels. Figure 2 showed the data processing process of osteosarcoma patients and the establishment and validation of the prediction model of osteosarcoma lung metastasis.

Figure 2.

The numbers of enrolled patients, and the establishment and validation of the prediction model of osteosarcoma lung metastasis.

Results

Baseline demographic and clinical characteristics

The demographic and clinical characteristics of the studied patients with OS are described in Table 1. After the data were filtered, a total of 424 patients were enrolled in this study. By using random sampling, the patients were randomly divided into training and validation cohorts. The training cohort included 191 patients with pulmonary metastasis in OS and 126 patients with non-pulmonary metastasis, and the validation cohort included 50 patients with pulmonary metastasis in OS and 57 patients with non-pulmonary metastasis.

Table 1.

Clinical characteristics of studied patients with osteosarcoma.

Variables	Pulmonary metastasis (n=191)	Non-pulmonary metastasis (n=126)	P-value	Pulmonary metastasis (n=50)	Validation cohort Non-pulmonary metastasis (n=57)	P-value
Gender			0.330			1.000
Male	108	79		31	36
Femle	83	47		19	21
Marriage			0.480			1.000
Unmarried	114	81		29	33
Married	77	45		21	24
Nation			0.197			0.289
Han	134	81		29	40
Zhuang	55	41		20	16
Yao	0	3		1	0
Miao	1	1		0	0
Mulao	1	0		0	1
Systolic pressure(mmHg)			0.864			1.000
140	181	118		46	53
140-159	10	8		4	4
Diastolic pressure(mmHg)			0.733			0.798
;90	179	116		48	53
≥90	12	10		2	4
BMI			0.870			0.855
18.5	84	57		23	29
18.5-24	82	55		21	21
24	25	14		6	7
Smoke			0.121			0.575
No	179	111		45	54
Yes	12	15		5	3
Drink			0.170			0.519
No	185	117		47	56
Yes	6	9		3	1
Surgery			0.409			0.939
No	34	28		10	10
Yes	157	98		40	47
Primary site			0.117			0.036
Others	7	4		0	4
Limb bone	147	105		44	41
Spine	2	5		2	0
Cranium viscerale	28	10		3	11
Pelvis	7	2		1	1
N Stage			<0.0001			0.012
N0	170	83		34	51
N1	21	43		16	6
Clinical stages (AJCC 8th)			<0.0001			<0.0001
I-II	151	0		0	48
III-IV	40	126		50	9
Neoadjuvant chemotherapy			0.459			0.046
No	82	48		21	36
Yes	109	78		29	21
Postoperativ chemotherapy			0.640			0.539
No	59	35		13	19
Yes	132	91		37	38
Radiotherapy			0.629			0.444
No	165	112		44	46
Yes	26	14		6	11
Age	25.560 (15.552)	26.865 (16.141)	0.472	27.620 (16.264)	28.000 (16.710)	0.906
FBG(mmol/L)	4.641 (0.910)	4.599 (0.746)	0.673	4.538 (0.695)	4.757 (1.222)	0.266
Blood routine
WBC(*109/L)	7.254 (2.607)	7.865 (2.542)	0.040	7.971 (2.812)	7.293 (2.540)	0.193
LYM%	29.628 (11.003)	25.661 (9.488)	0.001	24.614 (8.104)	30.536 (12.632)	0.005
MO%	7.023 (2.859)	7.331 (2.089)	0.299	7.292 (2.043)	6.418 (2.056)	0.030
NEU%	59.073 (12.620)	63.173 (10.124)	0.002	63.558 (9.308)	58.685 (13.534)	0.035
LYM (*109/L)	1.999 (0.708)	1.908 (0.637)	0.242	1.851 (0.653)	2.059 (0.760)	0.135
MO (*109/L)	0.491 (0.201)	0.572 (0.254)	0.002	0.571 (0.252)	0.474 (0.187)	0.025
NEU (*109/L)	4.473 (2.338)	5.126 (2.275)	0.014	5.248 (2.569)	4.466 (2.263)	0.097
RBC(*1012/L)	4.559 (0.758)	4.369 (0.807)	0.034	4.329 (0.742)	4.545 (0.805)	0.154
Hb(g/L)	122.218 (18.481)	117.477 (21.584)	0.038	117.714 (24.332)	124.088 (19.842)	0.139
BLT(*109/L)	332.965 (109.745)	311.471 (108.885)	0.088	315.068 (121.490)	336.199 (125.135)	0.379
EOSR%	3.384 (3.992)	2.858 (3.687)	0.237	3.212 (5.128)	3.613 (4.014)	0.652
BASOR%	0.560 (0.424)	0.470 (0.213)	0.028	0.506 (0.217)	0.502 (0.327)	0.943
EOSR(*109/L)	0.247 (0.423)	0.221 (0.373)	0.576	0.259 (0.556)	0.256 (0.338)	0.971
BASOR(*109/L)	0.038 (0.029)	0.036 (0.022)	0.592	0.041 (0.028)	0.035 (0.022)	0.226
Liver function
ALT(U/L)	21.551 (29.994)	19.132 (16.844)	0.411	19.680 (18.281)	24.018 (28.589)	0.360
TP(g/L)	69.153 (5.915)	67.746 (6.313)	0.044	67.178 (6.284)	69.916 (5.629)	0.019
ALB(g/L)	41.024 (16.216)	38.366 (5.399)	0.077	38.110 (6.226)	40.065 (4.839)	0.071
TRF(g/L)	2.421 (0.507)	3.372 (12.819)	0.306	2.297 (0.516)	2.376 (0.525)	0.435
AST(U/L)	24.797 (12.298)	28.063 (18.162)	0.057	28.078 (23.185)	24.702 (12.866)	0.346
AST/ALT	1.740 (1.093)	1.836 (0.962)	0.425	1.694 (0.972)	1.478 (0.789)	0.206
GGT(U/L)	30.493 (35.935)	36.913 (42.803)	0.150	37.560 (49.421)	37.386 (37.267)	0.984
ALP(U/L)	294.158 (650.850)	529.179 (1134.582)	0.020	183.300 (168.242)	116.421 (35.863)	0.004
LDH(U/L)	276.037 (714.032)	391.874 (568.697)	0.127	385.000 (768.699)	195.667 (58.978)	0.067
LPS(U/L)	33.540 (40.163)	28.082 (10.779)	0.137	25.980 (9.068)	39.544 (67.130)	0.160
SF(ug/L)	196.706 (182.225)	296.224 (293.173)	<0.0001	335.700 (377.987)	206.860 (213.454)	0.030
Electrolyte
K(mmol/L)	4.281 (0.378)	4.163 (0.427)	0.011	4.110 (0.372)	4.349 (0.377)	0.001
NA(mmol/L)	141.681 (19.964)	139.786 (2.998)	0.291	139.760 (3.146)	140.368 (2.623)	0.278
Cl(mmol/L)	100.245 (10.523)	100.048 (3.298)	0.839	100.180 (3.561)	101.737 (2.649)	0.011
Ca(mmol/L)	2.331 (0.160)	2.290 (0.170)	0.032	2.278 (0.183)	2.331 (0.130)	0.087
Renal function
UREA(mmol/L)	5.548 (16.751)	4.346 (1.555)	0.423	4.176 (1.570)	4.305 (1.400)	0.655
CREA(μmol/L)	59.328 (18.211)	60.481 (18.675)	0.585	63.240 (19.456)	62.351 (18.464)	0.809
Ccr(ml/min)	98.625 (23.478)	95.095 (23.319)	0.190	96.198 (25.936)	100.351 (25.261)	0.404
UA(μmol/L)	312.708 (96.301)	319.589 (100.682)	0.541	328.560 (105.568)	315.712 (109.091)	0.539
Cardiovascular function
CK(U/L)	91.476 (144.423)	114.138 (195.553)	0.237	116.290 178.058)	119.156 (255.691)	0.947
CK-MB(U/L)	13.114 (10.450)	16.462 (20.262)	0.055	17.429 (28.220)	11.855 (5.767)	0.148
hs-TnI(pg/ml)	3.576 (2.408)	3.384 (5.076)	0.653	4.002 (7.752)	3.746 (1.846)	0.810
MYO(ng/ml)	79.629 (181.876)	103.650 (203.161)	0.273	89.397 (146.558)	68.950 (163.031)	0.499
Coagulation function
PT(Sec)	12.357 (1.231)	12.675 (1.272)	0.027	12.778 (1.253)	12.356 (1.428)	0.110
APTT(Sec)	31.246 (5.065)	30.877 (5.304)	0.534	31.844 (5.085)	32.086 (5.165)	0.808
FIB(g/L)	3.817 (1.233)	4.065 (1.407)	0.100	4.112 (1.617)	3.686 (1.119)	0.113
TT(Sec)	16.369 (3.069)	17.016 (8.961)	0.358	16.216 (1.437)	16.051 (2.676)	0.699
D-dimer(ug/ml)	1.922 (4.078)	3.201 (4.507)	0.009	3.249 (4.861)	1.089 (1.668)	0.002
Inflammatory factors
CRP(mg/L)	17.373 (23.651)	27.965 (37.227)	0.002	29.751 (44.010)	17.092 (22.844)	0.060
hs-CRP(mg/L)	8.330 (14.051)	11.952 (29.246)	0.142	17.584 (45.122)	7.244 (7.462)	0.091
ESR(mm/h)	37.862 (22.219)	42.466 (25.632)	0.091	45.346 (27.671)	37.443 (21.968)	0.103
PCT(ng/ml)	1.898 (7.379)	1.875 (4.361)	0.975	2.759 (6.779)	3.236 (13.071)	0.817
Humoral immune factors
C3(g/L)	1.181 (0.248)	1.204 (0.240)	0.414	1.185 (0.230)	1.178 (0.210)	0.865
C4(g/L)	0.300 (0.117)	0.298 (0.076)	0.845	0.275 (0.070)	0.299 (0.076)	0.091
IgG(g/L)	12.683 (2.669)	13.507 (10.275)	0.292	14.395 (16.087)	13.010 (2.673)	0.523
IgA(g/L)	2.371 (0.779)	2.398 (0.852)	0.773	2.422 (0.861)	2.503 (0.892)	0.636
IgM(g/L)	1.345 (0.524)	1.248 (0.431)	0.084	1.214 (0.441)	1.295 (0.468)	0.362
Cellular immune factors
T-lym(%)	69.221 (8.816)	70.250 (7.970)	0.292	70.369 (7.780)	68.850 (7.898)	0.320
Th-lym(%)	56.455 (245.843)	39.030 (8.697)	0.427	38.971 (8.790)	38.050 (9.917)	0.614
Ts-lym(%)	25.554 (6.797)	25.148 (6.246)	0.592	25.624 (6.126)	25.033 (6.764)	0.639
Th/Ts	1.548 (0.589)	1.607 (0.584)	0.384	1.558 (0.565)	1.575 (0.632)	0.881
NKC(%)	10.809 (6.756)	9.477 (5.195)	0.062	9.690 (4.665)	12.996 (8.204)	0.013
B-lym(%)	12.627 (5.920)	13.349 (5.843)	0.287	12.977 (5.081)	10.991 (4.712)	0.038
Adenocarcinoma markers
CEA(ng/ml)	1.372 (0.723)	1.297 (0.643)	0.351	1.258 (0.394)	1.530 (0.736)	0.021
AFP(ng/ml)	2.182 (1.702)	1.976 (1.197)	0.239	2.068 (1.363)	2.314 (1.355)	0.352
CA-125(U/ml)	16.769 (15.440)	29.170 (64.697)	0.011	37.117 (89.897)	17.558 (9.086)	0.105
CA-153(U/ml)	11.893 (5.538)	12.567 (8.033)	0.377	11.961 (5.479)	12.638 (5.880)	0.541
CA-199(U/ml)	13.077 (15.749)	15.351 (43.260)	0.508	9.183 (8.465)	14.331 (21.904)	0.121
Squamous carcinoma markers
SCC Ag(ng/ml)	0.820 (0.309)	0.863 (0.455)	0.323	0.899 (0.559)	0.779 (0.195)	0.130
cyfra21-1(ng/ml)	1.791 (0.577)	2.299 (3.027)	0.024	1.841 (0.715)	1.825 (0.501)	0.896
Thyroid function
TSH(uIU/ml)	1.840 (0.646)	2.345 (2.508)
T4(nmol/L)	93.979(15.969)	95.868 (13.059)	0.269	93.741(13.809)	92.625 (13.269)	0.671
T3(nmol/L)	1.556 (0.263)	1.520 (0.280)	0.248	1.437 (0.257)	1.511 (0.091)	0.045
FT4(pmol/L)	13.202 (1.137)	13.270 (1.300)	0.624	13.132 (1.474)	13.175 (0.435)	0.835
FT3(pmol/L)	4.642 (0.612)	4.465 (0.617)	0.012	4.279 (0.611)	4.522 (0.124)	0.004
Serum lipid
TG(mmol/L)	11.928 (10.404)	12.412 (16.361)	0.748	11.241 (4.260)	11.527 (2.930)	0.683
TCHO(mmol/L)	4.378 (0.651)	4.310 (0.664)	0.371	4.303 (0.633)	4.424 (0.820)	0.401
TAG(mmol/L)	1.121 (0.374)	1.095 (0.373)	0.544	1.097 (0.342)	1.187 (0.434)	0.240
HDL-C(mmol/L)	1.160 (0.198)	1.139 (0.205)	0.359	1.117 (0.189)	1.162 (0.235)	0.285
LDL-C(mmol/L)	2.817 (0.540)	2.827 (0.558)	0.865	2.863 (0.567)	2.838 (0.575)	0.824

Note: The bold text means that the P value was < .05. Abbreviations: Han, the Han nationality; Zhuang, the Zhuang nationality; Yao, the Yao nationality; Miao, the Miao nationality; Mulao, the Mulao nationality; BMI, Body Mass Index; FBG, fasting blood glucose; WBC, white blood cell; LYM%, percentage of lymphocytes; MO%, Monocyte percentage; NEU%, neutrophilic granulocyte percentage; LYM, lymphocytes; MO, Monocyte; NEU, neutrophilic granulocyte; RBC, red blood cell; Hb, hemoglobin; BLT, blood platelet; EOSR%, percentage of eosinophils; BASOR%, percentage of basophilic granulocyte; EOSR, eosinophils; BASOR, basophilic granulocyte; ALT, alanine transaminase; TP, total protein; ALB, albumin; TRF, transferrin; AST, aspartate aminotransferase; ALT, glutamic-pyruvic transaminase; GGT, γ -glutamyl transpeptadase; ALP, alkaline phosphatase; LDH, lactic dehydrogenase; LPS, lipase; SF, serum ferritin; K, Serum potassium ion; NA, Serum sodium ion; Cl, Serum chloride ion; Ca, Serum calcium ion; UREA, ureophil; CREA, Creatinine; Ccr, Creatinine Clearance Rate; UA, Uric Acid; CK, Creatine Kinase; CK-MB, Creatine kinase isoenzyme; hs-TnI, Hypersensitive plasma troponin I; MYO, myohemoglobin; PT, prothrombin time; APTT, activated partial thromboplastin time; FIB, fibrinogen; TT, thrombin time; CRP, C-reactive protein; hs-CRP, high-sensitivity C-reactive protein; ESR, erythrocyte sedimentation rate; PCT, procalcitonin; C3, Complement C3; C4,Complement C3; IgG, immunoglobulin G; IgA, immunoglobulin A; IgM, immunoglobulin M; T-lym, cytotoxic T lymphocytes; Th-lym, helper T lymphocytes; Ts-lym, suppressor T cell; NKC, natural lxiller cell; B-lym, B lymphocytes; CEA, carcinoembryonic antigen; AFP, alpha fetoprotein; CA-125, Carbohydrate antigen −125; CA-153 , Carbohydrate antigen −153; CA-199, Carbohydrate antigen −199; SCC Ag, squamous cell carcinoma antigen; cyfra21-1, Human CYFRA21-1 Antigen; TSH, Thyroid Stimulating Hormone; T4, thyroxine; T3, triiodothyronine; FT4, free thyroxine; FT3, free triiodothyronine; TG, triglyceride; TCHO, total cholesterol; TAG, triglyceride; HDL-C, high-density lipoprotein; LDL-C, low-density lipoprotein.

In the training cohort, it included 187 males and 130 females. Patients ranged in age from 4 to 75, with an average age of 26. 126 patients of them had pulmonary metastasis confirmed by pulmonary imaging evidence or pathological evidence of lung biopsy, while 191 patients had no lung metastasis. N Stage, ALP, TSH and FT3 were significantly different between pulmonary metastasis group and non-pulmonary metastasis group. WBC 、NEU%、MO、 NEU 、ALP 、SF 、PT 、D-dimer 、CRP 、CA-125 、cyfra21-1 ans TSH were significantly higher in patients with pulmonary metastasis than those without pulmonary metastasis. The levels of LYM%, RBC, Hb, BASOR%, TP, K, Ca and FT3 were significantly reduced in patients with pulmonary metastasis.

Identify predictors of pulmonary metastasis in OS

By means of logistic regression analysis, we identified variables associated with pulmonary metastasis of osteosarcoma. Univariate analysis determined that there were significant differences among N Stage, WBC, LYM%, NEU%, MO, NEU, RBC, Hb, BASOR%, TP, ALP, SF, K, Ca, PT, D-dimer, CRP, TSH and FT3 between the pulmonary metastasis group and the non-pulmonary metastasis group (Table 2). N Stage, Clinical stages, MO, NEU, Hb, BASOR, TP, ALP, SF, D-dimer, CA.125, cyfra21.1, TSH and FT3 were identified as relevant variables via LASSO regression analysis (Figure 3). Multivariate analysis determined that N Stage (p < .001, OR = 4.376, 95% CI = 2.310–8.29), ALP (p = .034, OR = 1.000, 95% CI = 1.000–1.001), TSH (p = .022, OR = 1.445, 95% CI = 1.055–1.980) and FT3 (p = .042, OR = 0.605, 95% CI = 0.373–0.983) were independent predictors for pulmonary metastasis of osteosarcoma. (Table 3)

Table 2.

Univariate logistic regression analysis.

Variables		Univariate analysis
Variables		OR	95% CI	P-value
N stage	N0	Reference
N stage	N1	4.200	2.334-7.559	0.000
Clinical stages	I-II	Reference
Clinical stages	III-IV	5088747598.247	0.000-.	0.995
WBC(*10⁹/L)		1.095	1.003-1.196	0.043
LYM%		0.962	0.939-0.985	0.001
NEU%		1.032	1.011-1.053	0.003
MO(*10⁹/L)		5.066	1.762-14.564	0.003
NEU(*10⁹/L)		1.129	1.022-1.247	0.017
RBC(*10¹²/L)		0.728	0.541-0.979	0.036
Hb(g/L)		0.988	0.977-0.999	0.039
BASOR%		0.437	0.205-0.931	0.032
TP(g/L)		0.962	0.927-0.999	0.046
ALP(U/L)		1.000	1.000-1.001	0.041
SF(ug/L)		1.002	1.001-1.003	0.001
K(mmol/L)		0.475	0.266-0.846	0.011
Ca(mmol/L)		0.216	0.053-0.884	0.033
PT(Sec)		1.228	1.022-1.476	0.029
D-dimer(ug/ml)		1.076	1.015-1.142	0.015
CRP(mg/L)		1.012	1.004-1.02	0.004
CA-125(U/ml)		1.010	1.000-1.02	0.056
TSH(uIU/ml)		1.333	1.015-1.749	0.038
FT3(pmol/L)		0.602	0.400-0.906	0.015
cyfra21-1(ng/ml)		1.235	0.954-1.600	0.110

Note: The bold text means that the P value was < 0.05. Abbreviations: WBC, white blood cell; LYM%, percentage of lymphocytes; NEU%, neutrophilic granulocyte percentage; MO, Monocyte; NEU, neutrophilic granulocyte; RBC, red blood cell; Hb, hemoglobin; BASOR%, percentage of basophilic granulocyte; TP, total protein; ALP, alkaline phosphatase; SF, serum ferritin; K, Serum potassium ion; Ca, Serum calcium ion; PT, prothrombin time; CRP, C-reactive protein; CA-125, Carbohydrate antigen -125; TSH; Thyroid Stimulating Hormone; FT3, free triiodothyronine; cyfra21-1, Human CYFRA21-1 Antigen

Figure 3.

As a method of machine learning, LASSO (Least absolute shrinkage and selection operator) binary logistic regression model was used to select the predictive features. (a) Identify lambda (λ) or the optimal penalty coefficient in the Lasso model using minimum criteria and 10-fold cross validation. (b) Lasso coefficient profiles of the features. A vertical line was drawn at the selected value using 10-fold cross-validation, and optimal values were obtained by applying the 1 standard error of the minimum criteria (the 1-se criteria) and minimum criteria.

Table 3.

Multivariate logistic regression analysis

Variables		Multivariate analysis
Variables		OR	95% CI	P-value
N Stage	N0	Reference
N Stage	N1	4.376	2.310-8.29	0.000
MO(*10⁹/L)		3.121	0.702-13.883	0.135
NEU(*10⁹/L)		0.984	0.849-1.140	0.828
Hb(g/L)		0.995	0.981-1.010	0.524
BASOR%		0.464	0.203-1.060	0.068
TP(g/L)		0.963	0.919-1.008	0.108
ALP(U/L)		1.000	1.000-1.001	0.034
SF(ug/L)		1.001	1.000-1.002	0.136
D-dimer(ug/ml)		1.008	0.949-1.071	0.798
TSH(uIU/ml)		1.445	1.055-1.980	0.022
FT3(pmol/L)		0.605	0.373-0.983	0.042

Note: The bold text means that the P value was < .05. Abbreviations: MO, Monocyte; NEU, neutrophilic granulocyte; Hb, hemoglobin; BASOR%, percentage of basophilic granulocyte; TP, total protein; ALP, alkaline phosphatase; SF, serum ferritin; TSH; Thyroid Stimulating Hormone; FT3, free triiodothyronine.

Construction and validation of nomogram

A predictive nomogram combining all important independent predictors of pulmonary metastasis of osteosarcoma is shown in Figure 4. Independent factors include N Stage, ALP, TSH and FT3. The total score can be calculated as the sum of the points of relevant predictors and then calculating the probability of pulmonary metastasis in OS. For example, an osteosarcoma patient with lymph node metastasis had ALP of 1000U/L, TSH of 8mU/L and FT3 of 5pmol/ml, with a total score of 75, and the probability of pulmonary metastasis was about 90%.

Figure 4.

The use of nomogram to determine the probability of pulmonary metastasis of osteosarcoma. The nomogram integrated the N Stage and all independent clinical factors (ALP, TSH, and FT3) to predict the likelihood of pulmonary metastasis of OS. The N Stage and all independent clinical factors (ALP, TSH and FT3) were integrated in the nomogram to predict the likelihood of pulmonary metastasis of OS. Each variable is assigned a point on the top axis by drawing a line up. The sum of these numbers is located on the total score axis and a line is drawn down to the probability axis used to determine the risk of pulmonary metastasis in OS.

Index of concordance (C-index) and the slope of the calibration curve were used to verify the accuracy of the nomogram model. The C-index was 0.701 (95% CI, 0.641–0761) to predict pulmonary metastasis of osteosarcoma in the nomogram. The calibration curve of the nomogram showed the best consistency between the predicted values and actual observed values (Figure 5).

Figure 5.

The calibration curve of the nomogram. Calibration curve for the model of predicting pulmonary metastasis in patients with osteosarcoma. The predicted probability of the nomogram is plotted on the x axis and the actual probability is plotted on the y axis.

The AUC (area under ROC curve) is plotted in Figure 6. This nomogram exhibited excellent discrimination with an AUC of 0.701.

Figure 6.

Nomogram ROC curve. The normogram showed excellent discrimination with an AUC of 0.701 (95% CI, 0.641–0761). ROC, receiver operating characteristics; AUC, the area under the ROC curve.

Clinical use

Figure 7 showed the decision curve analysis (DCA) of the nomogram. DCA was performed on our predictive model to estimate the net benefit to patients. The decision curve showed that the nomogram model has significant net benefits for almost all threshold probabilities, especially those between 30-80% (Figure 7). However, in cases that the threshold probability was less than 30%, the nomogram exhibited a net benefit comparable to predicted positive results for all patients. Similarly, CIC (Clinical Impact Curve) evaluated the clinical utility of the model (clinical benefit), and the results showed that there were predictive value and application value in the nomogram (Figure 8).

Figure 7.

The decision analysis curve of the nomogram. The Y-axis represents the net benefits. The Nomogram is represented by the red line. The gray line indicates the net benefit assuming that all patients had pulmonary metastasis in OS. The black line indicates the net benefit assuming that no patients had pulmonary metastasis. The net benefit was estimated by subtracting the proportion of patients who tested false positive from the true positive proportion, weighted by the relative harm of not getting treated in comparison to the adverse outcomes of an unrequired treatment ^21,22. The decision curve indicates that when the threshold probability is >20% and <60%, there is greater advantage using the nomogram to predict pulmonary metastasis of osteosarcoma.

Figure 8.

Clinical Impact Curve of nomogram. The red curve represents the number of people classified by the model as positive (high risk) at each threshold probability (abscissa). The blue curve represents the number of true positives under each threshold probability.

Validation cohort

The prediction model was employed for the validation cohort for verification (Figure 9). In the validation cohort, the AUC of the prediction model was 0.786 (95% CI, 0.700–0.871). The calibration curves exhibited a satisfactory agreement between nomogram predictions and actual probabilities. The decision analysis curve and the clinical impact curve showed that there were predictive value and application value in the nomogram.

Figure 9.

Validation cohort. A Calibration curves for predicting pulmonary metastasis in OS probability. B AUC for predicting pulmonary metastasis in OS probability. C The decision analysis curve for predicting pulmonary metastasis in OS probability. D The clinical impact curve for predicting pulmonary metastasis in OS probability.

Discussion

In the study, N Stage, WBC, LYM%, NEU%, MO, NEU, RBC, Hb, BASOR%, TP, ALP, SF, K, Ca, PT, D-dimer, CRP, TSH, FT3 were important factors for pulmonary metastasis of OS in univariate logistic regression analysis. LASSO regression analysis identified N Stage, Clinical stages, MO, NEU, Hb, BASOR, TP, ALP, SF, D-dimer, CA.125, cyfra21.1, TSH and FT3 as relevant variables. Finally, N Stage, ALP, TSH and FT3 were determined to be the most significant factors after stepwise logistic regression and multivariate analysis.

Osteosarcoma metastasis is usually secondary to hematogenous dissemination, while it’s rare for metastasis to the lymphatic system, and has been identified as being significantly associated with poor prognosis.²³ At N stage, patients with N1 stage; with lymphatic metastasis; were more prone to lung metastases and worse prognosis than those with N0 stage;without lymphatic metastasis;. Patients with lymphatic metastasis (N1) are low proportion in the data set, about 20%. Instead, compared with the non-metastatic group, a higher proportion of N1 stage was found in patients with pulmonary metastasis than N0 stage. Therefore, we believe that the presence of lymphatic metastases indicates that osteosarcoma is very aggressive. A study has found that there was significantly lower survival rates in patients with local lymphatic metastases or distant metastasis than the others.²⁴ The majority of osteosarcoma patients die mainly due to lung metastasis. Osteosarcoma is rare in general oncology, and osteosarcoma with lymphatic metastasis is even rarer. Considering the correlation between lymph node metastasis and lung metastasis, clinicians cannot neglect the examination of lymph node metastasis.

Alkaline phosphatase (ALP) is widely distributed in the human liver, bone, intestine, kidney and placenta, and is excreted by the liver and biliary system. Many studies have evaluated the relationship between alkaline phosphatase (ALP) levels and the prognosis of osteosarcoma. Experimental studies have shown that serum ALP is a valuable tumor marker with high specificity in osteosarcoma and that the diagnostic performance of ALP for osteosarcoma diagnosis and metastasis is better than that of well-known tumor markers in other cancers.^{25, 26} Our study showed that the ALP level was increased in pulmonary metastasis group of OS. Therefore, we believe that osteosarcoma is in the active stage and has a higher risk of metastasis when ALP is increased. In clinical practice, the risk of lung metastasis of osteosarcoma can be evaluated by monitoring the change of serum ALP level.

Clinical studies have shown that euthyroid status in adults is important for bone homeostasis. Hyperthyroidism leads to increased bone resorption, which results in osteopenia and its mineralization. Patients with hypothyroidism exhibit reduced bone turnover due to reduced osteoclast activity. Although the critical role of thyroid hormones in bone homeostasis is well documented, the role of thyroid stimulating hormone (TSH) is still under investigation, and the direct effect of TSH on human bone is controversial.^27,28 The nomogram of this study shows that the increase of TSH and the decrease of FT3 in patients with pulmonary metastasis may indicate that the increase of TSH level is important related to the occurrence of pulmonary metastasis of osteosarcoma. However, we still need to conduct prospective studies on large cohorts to verify the conclusion.

Finally, we developed a nomogram using N Stage, ALP, TSH, and FT3 to predict pulmonary metastasis in OS. Each prognostic factor has a corresponding value based on a set ratio. Based on the personalized information and its corresponding values, we can obtain the total score, which can be used to predict the risk of metastasis. For example, for osteosarcoma patients, the corresponding points can be found in the nomogram according to the patient’s information, all points can be added up, and the total score can be correlated with the event probability we want to predict.

In the era of rapid development of treatment technology, individualized treatment is particularly important. Patients in the high-risk group may have had a higher tumor burden. Therefore, early identification of high-risk patients may be the key to improved survival outcomes and will help to optimize treatment. The nomogram in our study shows great potential for application in clinical practice. Clinicians can use this model to assess the risk of pulmonary metastasis in OS patients, provide more personalized diagnosis and treatment guidance, and improve the prognosis of patients.

The advantages of this study are as follows. First of all, this is the first study ever to report the establishment of a risk model for predicting lung metastasis of OS based on common clinical indicators. With wide coverage and strong representativeness, this study had important reference value which included more than 100 clinical data indicators for analysis and research. The results of this study have certain educational and practical significance. Secondly, the variables involved in the nomogram are readily available in most medical establishments, so the model has wide range of clinical applications. Finally, the nomogram is a visual prediction tool that can be used in clinical practice to help physicians quickly assess prognosis of patients through simple calculations and categorize patients with different levels of severity to help determine individualized treatment options.

Our study also has some limitations. First, our follow-up was not long enough and there were not enough patients due to the key inclusion criteria. If possible, there should be a longer follow-up period and a larger study population. Second, the evidence from the three institutional studies is limited. Prospective studies in large cohorts are needed to validate these conclusions. Furthermore, despite its preponderance in this cohort, inflammatory markers were nonspecific predictors of osteosarcoma and thus inevitably had inherent weaknesses and limitations. However, there are no specific serum markers for osteosarcoma. The nonspecific findings such as N Stage, ALP, TSH and FT3 may help to support suspicion when osteosarcoma was diagnosed clinically and radiographically in some patients. In addition, similar to other retrospective studies, selection bias is inevitable.²⁹ However, this retrospective study is clinically relevant because it provided a reference for the planning of some prospective studies.

In conclusion, our study developed and validated a novel nomogram for predicting pulmonary metastasis in patients with osteosarcoma, involving N Stage, ALP, TSH and FT3. Decision curve analysis (DCA) and clinical impact curve (CIC) demonstrated the clinical value and higher overall net benefit of the nomogram. In the future, prospective studies are needed to extend the clinical usefulness of our model.

Supplemental Material

Supplemental Material - Clinical model of pulmonary metastasis in patients with osteosarcoma: A new multiple machine learning-based risk prediction

Supplemental Material for Clinical model of pulmonary metastasis in patients with osteosarcoma: A new multiple machine learning-based risk prediction by Zhiping Su, Feihong Huang MM, Chunyue Yin, Yuezhao Yu, Chaojie Yu in Journal of Orthopaedic Surgery

Footnotes

Acknowledgements

We thank all the patients who participated in this study. We thank all reviewers for their time and thoughtful critiques to refine our manuscript.

Declaration of conflicting interests

All ICMJE Conflict of Interest Forms for authors and Clinical Orthopaedics and Related Research® editors and board members are on file with the publication and can be viewed on request. All authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the we acknowledge support from The Youth Science Foundation project of Guangxi Natural Science Foundation Project (No: 2022JJB140494), Guangxi Postdoctoral Special Foundation Project, “Two Stations” Project and Guangxi Medical University Postdoctoral Research Foundation (No: 46/02305222008X).

Author’s Note

This work was performed at Guangxi Medical University Cancer Hospital, Guangxi Medical University, Nanning, Guangxi, China.

Ethics approval

Ethical approval for this study was obtained from Guangxi Medical University Cancer Hospital, Nanning, Guangxi, China (number LW2022179). The study was carried out in accordance with the Declaration of Helsinki statement. The present study was approved by the Ethics Committee of Guangxi Medical University Cancer Hospital. The need for written informed consent was waived by Guangxi Medical University Cancer Hospital’s Ethical Review Committee due to retrospective nature of the study. Written informed consent was not obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

Data availability

The original contributions presented in the study are included in the article/supplementary material. Further enquiries can be directed to the corresponding author. The data that support the findings of this study are available upon reasonable request.

ORCID iD

Chaojie Yu

References

Cersosimo

Lonardi

Bernardini

, et al. Tumor-associated macrophages in osteosarcoma: from mechanisms to therapy. Int J Mol Sci. 2020;21(15):5207. doi: 10.3390/ijms21155207

Huang

Zhao Bai , et al. Risk and clinicopathological features of osteosarcoma metastasis to the lung: a population-based study. J Bone Oncol. 2019;16:100230.

Sheng

Gao

Yang

, et al. Osteosarcoma and metastasis. Front Oncol 2021 11:780264. doi: 10.3389/fonc.2021

Wang

Ding

Sun

, et al. A population-based propensity-matched study of regional dissections in patients with metastatic osteosarcoma. J Orthop Surg Res 2020 15(1):107. doi: 10.1186/s13018-020-01592-y

Kager

Zoubek

Pötschger

Cooperative German-Austrian-Swiss Osteosarcoma Study Group , et al. Primary metastatic osteosarcoma: presentation and outcome of patients treated on neoadjuvant cooperative osteosarcoma study group protocols. Am J Clin Oncol 2003 21:2011–2018. doi: 10.1200/JCO.2003.08.132

Hattori

Yamamoto

. Lymph node metastasis of osteosarcoma. Am J Clin Oncol 2012 30:e345–e349. doi: 10.1200/JCO.2012.42.3384

, et al. Osteosarcoma immune prognostic index can indicate the nature of indeterminate pulmonary nodules and predict the metachronous metastasis in osteosarcoma patients. Front Oncol. 2022;12:952228. doi: 10.3389/fonc.2022.952228

Wang

, et al. Lung Immune Prognostic Index Could Predict Metastasis in Patients With Osteosarcoma. Front Surg. 2022;9:923427. doi: 10.3389/fsurg.2022.923427

Rojas

Hubbard

Diessner

, et al. International trends in incidence of osteosarcoma (1988‐2012). International Journal of Cancer. 2021;149(5):1044–1053. doi: 10.1002/ijc.33673

10.

Yao

Tan

Yang

, et al. Prognostic nomogram for predicting 5-year overall survival in Chinese patients with high-grade osteosarcoma. Sci Rep 2021;11(1):18729.

11.

Wang

Yang

Wang

, et al. Survival and prognostic factors in Chinese patients with osteosarcoma: 13-year experience in 365 patients treated at a single institution. Pathol Res Pract 2017;213:119–125. doi: 10.1016/j.prp.2016.11.009

12.

Yang

Wang

. et al. CONUT Score or/and peripheral Blood CD4+/CD8+ ratio-based web dynamic nomograms to predict the individualized survival of patients with advanced osteosarcoma. Cancer Manag Res. 2020;12:4193–4208. doi: 10.2147/cmar.S251814

13.

Chen

Liu

Cheng

et al. Development and external validation of an MRI-based radiomics nomogram for pretreatment prediction for early relapse in osteosarcoma: a retrospective multicenter study. Eur J Radiol 2020;129:109066. doi: 10.1016/j.ejrad.2020.109066

14.

Kattan

Scardino

. Evidence for the usefulness of nomograms. Nat Rev Urol. 2007;4(12):638–639.

15.

Huang

Jing

, et al. A novel and validated prognostic nomogram based on liver fibrosis and tumor burden for patients with hepatocellular carcinoma after curative resection. J of Surg Oncol. 2018;117(4):625–633.

16.

Huang

Liang

. et al. Development and validation of a radiomics nomogram for preoperative prediction of lymph node metastasis in colorectal cancer. Am J Clin Oncol. 2016;34(18):2157–2164. doi: 10.1200/JCO.2015.65.9128

17.

Kattan

Sternberg

Mehmud

, et al. Development and validation of a prognostic nomogram for progression-free survival in patients with advanced renal cell carcinoma treated with pazopanib. Oncology. 2015;89(4):235–241. doi: 10.1159/000430989

18.

Kent

Mandrekar

Landreneau

, et al.

A nomogram to predict recurrence and survival of high-risk patients undergoing sublobar resection for lung cancer: an analysis of a multicenter prospective study (ACOSOG Z4032)

Ann Cardiothorac Surg. 2016;102(1):239–246. doi: 10.1016/j.athoracsur.2016.01.063

19.

Wang

Xia

. et al. Prognostic nomogram for intrahepatic cholangiocarcinoma after partial hepatectomy. Am J Clin Oncol. 2013;31(9):1188–1195. doi: 10.1200/JCO.2012.41.5984

20.

Pereira

Leite Duarte

Ribeiro Damasceno

, et al. Machine learning-based CT radiomics features for the prediction of pulmonary metastasis in osteosarcoma. Brit J Radiol. 2021;94(1124):20201391. doi: 10.1259/bjr.20201391

21.

Friedman

Hastie

Tibshirani

. Regularization paths for generalized linear models via coordinate descent. J Stat Softw 2010;33:1–22.

22.

Shen

Zhou

, et al. Establishment of a predictive nomogram and its validation for severe adenovirus pneumonia in children. Ann Palliat Med. 2021;10(7):7194–7204. doi: 10.21037/apm-21-675

23.

Meazza

Scanagatta

. Metastatic osteosarcoma: a challenging multidisciplinary treatment. Expert Rev Anticancer Ther. (2016) 16:543–556. doi: 10.1586/14737140.2016.1168697

24.

Miller

Cram

Lynch

, et al. Risk factors for metastatic disease at presentation with osteosarcoma. J Bone Jt Surg. 2013;95(13):p. e89. doi: 10.2106/jbjs.L.01189

25.

Hao

Chen

Huang

, et al. Meta-analysis of alkaline phosphatase and prognosis for osteosarcoma. European Journal of Cancer Care 2017;26, e12536(5).

26.

Kim

Shin

Moon

, et al. Reassessment of alkaline phosphatase as serum tumor marker with high specificity in osteosarcoma. Cancer Medicine. 2017;6(6):1311–1322. doi:10.1002/cam4.1022

27.

Boutin

Gershengorn

Neumann

. β-Arrestin 1 in thyrotropin receptor signaling in bone: studies in osteoblast-like cells. Front Endocrinol. 2020;11:312.

28.

Geras-Raaka

Neumann

Gershengorn

. Persistent cAMP signaling by TSH receptors revealed by phosphodiesterase inhibition. Thyroid. 2013;23(11):1484–1489. doi: 10.1089/thy.2013.0089

29.

Euser

Zoccali

Jager

, et al. Cohort studies: prospective versus retrospective. Nephron Clin Prac. 2009;113(3):c214-c217. doi: 10.1159/000235241

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.50 MB