Sage Journals: Discover world-class research

Abstract

Background: Ultrasound (US) based radiomics is susceptible to variations in scanners, sonographers. Objective: To retrospectively investigate the feasibility of an adapted cycle generative adversarial networks (CycleGAN) in the style transfer to improve US based radiomics in the prediction of lymph node metastasis (LNM) with images from multiple scanners for patients with early cervical cancer (ECC). Methods: The CycleGAN was firstly trained to transfer paired US phantom images from one US device to another one; the model was then further trained and tested with clinical US images of ECC by transferring images from four US devices to one specific device; finally, the adapted model was tested with its effects on the radiomics feature harmonization and accuracy of LNM prediction in US based radiomics for ECC patients. Results: Phantom study demonstrated an increased radiomics harmonization using CycleGAN with an average Pearson correlation coefficient of 0.60 and 0.81 for radiomics features extracted from original and generated images in correlation with the target phantom images, respectively. Additionally, the image quality metric Peak Signal-to-Noise Ratio (PSNR) was increased from 11.18 for the original images to 15.45 for the generated image. Clinical US images of 169 ECC patients were enrolled for style transfer model training and validation. The area under curve (AUC) of LNM prediction radiomics models with features extracted from generated images of different style transfer models ranged from 0.73 to 0.85. The AUC was improved from 0.78 with features extracted from original images to 0.85 with style transferred images. Conclusions: The adapted CycleGAN network is able to increase the radiomics feature harmonization for images from different ultrasound equipment based on image domain and improve the LNM prediction accuracy for ECC.

Keywords

harmonization generative adversarial networks ultrasound radiomics lymph node metastasis

Introduction

Ultrasound (US) is one of the most used imaging modalities in clinical practice for breast, abdominal, transrectal, intravascular US, and prenatal diagnosis, obstetrics and gynecology due to its relative safety with non-ionizing radiation, cost effectiveness, real-time display, portability, and accessibility, etc ¹. However, real-time display and diagnosis also render the diagnostic results highly operator dependent with high inter- and intra-observer variability ². Advanced automatic US image analysis methods have been investigated intensively to make US-based diagnosis and interventions more objective, accurate, and intelligent ^{3, 4}. US-based radiomics features have been reported with objective, promising results in characterizing breast biology ⁵, gestational age ⁶, neonatal respiratory morbidity ⁷, thyroid tumor, and lymph node status (LNS) of cervical cancer (CC) ⁸, etc.

Radiomics is valuable for diagnostic, prognostic, and predictive analysis in the era of precision medicine by extracting high-throughput quantitative features from images of magnetic resonance image (MRI), computed tomography (CT), positron emission tomography (PET), and US, etc ⁹. A previous studies utilized radiomics models based on ultrasound (US) for non-invasive prediction of preoperative lymph node metastasis (LNM) in cervical cancer patients.⁴⁷ However, studies indicated that radiomics features are susceptible to variations in scanners, acquisition protocols, and reconstruction settings, which is unavoidable in retrospective and multicenter studies in the current clinical practice ¹⁰. The influences of different scanners and automatic segmentation algorithms in US-based radiomics had also been reported ^11–13. Therefore, different harmonization solutions were proposed to improve the reproducibility and stability of radiomics features ¹⁴.

The solution for feature harmonization could be categorizes two principal methodologies: the feature domain and the image domain approach. In the feature domain, various methods achieved harmonization by identifying and focusing on reproducible features ^15–17. However, this could also result in the neglect of potentially valuable information during feature extraction, and no universally accepted criterion existed for defining high reproducibility. Standardization of acquisition protocols and reconstruction settings were usually considered for the harmonization in the image domain for CT, MRI and PET. US imaging is a unique procedure that relies heavily on sonographers’ knowledge and experience, which renders standardization of acquisition protocols and reconstruction settings impracticable ¹⁸. Deep learning networks, such as convolutional neural networks (CNN) or generative adversarial networks (GAN), had been applied in many studies to harmonize medical images either by image-to-image translations or domain transformations ^{19, 20}. However, studies also indicated that unwanted artifacts may be introduced or the quality of the quantitative information contained in the images may be reduced after style transfer ^{21, 22}. On the other hand, few studies had addressed the harmonization of US images using style transfer. The purpose of this study is to investigate the feasibility and accuracy of an adapted cycle-GAN network in the style transfer for US-based radiomics with images from multiple scanners for patients with ECC.

Materials and Methods

Methodological Overview

Figure 1 illustrates the study design flowchart. A style transfer model based on CycleGAN network was firstly trained using ECC images and adapted to transfer paired US phantom images from one US device to another one; the model was then further trained and tested with clinical US images of ECC by transferring images from four US devices to one specific device; finally, the adapted model was tested with its effects on the accuracy of radiomics study for ECC patients with confirmed LNS.

Figure 1.

The flowchart of the study design. Initially, the style transfer model based on the CycleGAN network will be trained using early cervical cancer images and tested with paired ultrasound phantom images from one ultrasound device to one specific device. Finally, the adapted model was tested with its effects on the accuracy of radiomics study for ECC patients with confirmed LNS.

Data and Preprocessing

Phantom Image

Paired US images were acquired with two different US devices of Voluson-E8 (GE Healthcare) and HI VISION Preirus (Hitachi Ltd) using a CIRS multi-purpose multi-tissue ultrasound phantom (Model 040GSE, CIRS Inc., Virginia, USA). US images of three fixed shapes were acquired by an experienced sonographer according to different combination of the grayscale, anechoic stepped cylinder, the horizontal and vertical distance points in the phantom, as shown in the Figure 2a1-a3. The images were filtered according to the displayed structures, and the “original” and “target” images were paired using landmark registration in 3D Slicer (version 5.0.2).

Figure 2.

Ultrasound images on phantom, a1-a3) Phantom images acquired by Voluson E8 device; b1-b3) Phantom image acquired by HI VISION Preiris device; c1-c3) Generated images with style transfer model (Voluson E8 to HI VISION Preiris).

Clinical Image

We retrospectively collected the ultrasound images of 1707 patients with early-stage cervical cancer at the author's affiliation from 2012 to 2018. These images were obtained from five different scanners: ATL HDI 5000 (Philips), Voluson-E8 (GE Healthcare), Mylab classC (Esaote), ACUSON S2000 (Siemens), and HI VISION Preirus (HITACHI Ltd). The inclusion criteria were as follows: (i) patients who underwent radical hysterectomy and systematic pelvic lymph node dissection; (ii) postoperative histologically confirmed cervical cancer and lymph node status; and (iii) standard ultrasound examination performed within 2 weeks prior to hysterectomy. Exclusion criteria included: (i) incomplete clinical data or inability to perform statistical analysis; (ii) preoperative chemotherapy or radiotherapy; and (iii) patients with a history of malignancy or combined malignancy. Micrometastatic lymph nodes were not considered in this study due to a lack of relevant examinations. According to our inclusion criteria, a total of 169 cases were included in the final analysis, and all the images were staged according to the FIGO classification (2018) by a US physician with 5 years of experience in tumor marking. The detailed clinical characteristics of these patients can be found in Table 1.

Table 1.

Demographic Statics of Patients in the Radiomics Dataset.

Characteristics	Training cohort			Testing cohort
Total images	118			51
	LNM−(n = 81)	LNM + (n = 37)	p	LNM−(n = 39)	LNM + (n = 12)	p
Age			0.36			0.67
Mean	51.32	52.51		53.20	52.67
Range	31–79	25–76		27–73	38–72
SD	9.83	10.23		10.26	10.89
Histological type			0.87			0.62
Squamous cell carcinoma	74	33		32	10
Adenocarcinoma	5	3		6	1
Adenosquamous cell carcinoma	2	1		1	1
Tumor stage			0.63			0.91
I	51	25		22	7
II	30	12		17	5

(1) p value is calculated from the univariate association test between subgroups. (2) Fisher's exact test and chi-square test were used for categorized variables. LNM: lymph node metastasis, -: negative, +: positive, SD: standard deviation

Style Transfer Model

The structure of style transfer model in this study is adapted from CycleGAN ^{20, 23}, which contains two generator-discriminator pairs (Gxy, Dx, Gyx, Dy) ²⁴. As shown in Figure 3, with a X-style image inputting to the generator G1, a Y-style image outputs. Similarly, with a Y-style image inputting to the generator Gyx, a X-style image outputs. D discriminates between the generated image and the target image, while G strives to generate images that are indistinguishable from the target.

Figure 3.

The structure of style transfer network contains two generator-discriminator pairs (Gxy, Dx, Gyx, Dy); the top right is a schematic representation of the underlying CycleGAN model framework with two distributions X,Y, generators Gxy, Gyx for mapping X to Y and Y to X, and two discriminators Dx, Dy to discriminate between the transformed image; the left side is the modified generators with the added attention module to the second convolution module, as well a spectral normalization was applied in each convolution module to improve the learning ability of the model.

Self-Attention (SE Model)

In the second and subsequent convolutional layers, we have incorporated the attention mechanism proposed by ²⁵ which allows us to superimpose all feature map channels. By using global mean pooling, we can obtain a value that has a global perceptual field. The resulting weight coefficient vectors correspond to the feature map and reflect the importance of each channel. Then, the weight coefficient vectors are multiplied with feature map reinforcing important features and suppressing unimportant.

Total Loss

CycleGAN's loss function consists of six components, which can be categorized into three major types:

Adversarial loss: this component aims to ensure that the generated images from the source domain resemble the target domain images and vice versa. It encourages the generator to produce realistic images that can fool the discriminator. The source domain loss $L_{G A N} (G_{x y}, D_{y})$ and target domain loss $L_{G A N} (G_{y x}, D_{x})$ are:

L_{G A N} (G_{x y}, D_{y}, X, Y) = E_{Y \sim p_{d a t a (Y)}} [l o g D_{y} (Y)] + E_{X \sim p_{d a t a (X)}} [l o g (1 - D_{y} (G_{x y} (X)))]

(1)

L_{G A N} (G_{y x}, D_{x}, Y, X) = E_{X \sim p_{d a t a (X)}} [l o g D_{x} (X)] + E_{Y \sim p_{d a t a (Y)}} [l o g (1 - D_{x} (G_{y x} (Y)))]

(2)

The adversarial loss helps in aligning the distributions of the source and target domains.

Cycle consistency loss: it ensures the sample transformed from one domain to another remains unchanged. This loss term is crucial for preserving content during style transfer. It maintains the integrity of the original sample by penalizing any discrepancies between the input and the reconstructed output after the cycle. The loss of cycle consistency is as follows:

L_{c y c} (G_{x y}, G_{y x}) = E_{x \sim p_{d a t a (X)}} [G_{y x} {(G_{x y} (x) - x)}_{1}] + E_{y \sim p_{d a t a (Y)}} [G_{x y} {(G_{y x} (y) - y)}_{1}]

(3)

The mix loss: we utilize a hybrid function of L1 and MS-SSIM ²⁶ to synthesize images with higher peak signal-to-noise ratio, better visual quality and more detailed content.This approach helps the model maintain stable gradients, making the generated images more perceptible to local structural changes without compromising solution stability. MS-SSIM is defined as:

M S - S S I M (p) = l_{M}^{α} (p) \cdot \prod_{j = 1}^{M} c s_{j}^{β_{j}} (p)

(4)

Where

l_{M}

and

c s_{j}

are the terms in SSIM (the full formula in supplemental material S1). For convenience, we set

α = β_{j} = 1

, for

j = {1, \dots, M} .

Thus the formula for our hybrid function is

L_{M i x} = α \cdot L_{M S - S S I M} + (1 - α) \cdot G_{σ_{G}^{M}} \cdot L_{ℓ_{1}}

(5)

Where G is the standard deviation of pixel p calculated using the Gaussian filter.

The Total Loss is as follows:

\begin{aligned} L_{t o t a l} & = (1 - φ) \cdot L_{M i x} + φ \cdot (L_{G A N} (G_{x y}, D_{y}, X, Y) \\ + L_{G A N} (G_{y x}, D_{x}, Y, X) + λ L_{c y c} (G_{x y}, G_{y x})) \end{aligned}

(6)

Where the

φ

is a constant and the

λ

controls the relative importance of the two objectives.

Training Details

In order to speed up convergence, prevent model collapse, and ensure stable model training, spectral normalization (SN) was added to the modules of CIL (Conv, InstanceNorm, Leaky Relu) and CTIR (Conv, Transpose, InstanceNorm, Relu) to satisfy 1-Lipschitz continuity ^{27, 28}.

During the model training process, we observed an imbalance issue between the generator and discriminator, which could lead to model instability or even collapse. To address this problem, we adopted the dual time-scale update proposed by Heusel et al ²⁹. In simple terms, this approach involves using different learning rates for the generator and discriminator. We set the learning rate for the generator to 0.001 and the learning rate for the discriminator to 0.004. This allows the discriminator to respond promptly and provide feedback to update the generator, resulting in making generated images exhibit more characteristics of the target domain.

In addition, a label smoothing strategy is applied that modifies the features label to 90% of their original values, in order to increase the training difficulty for the discriminator and balancing the generator and discriminator.

Phantom Study

Due to the real-time display and operator-dependent nature of ultrasound imaging, there is a high level of inter- and intra-observer variability, which makes the diagnostic results highly subjective. It is difficult to obtain paired images from the same patient using different scanners, making it challenging to assess the level of feature harmonization achieved by the style transfer model. To address this limitation and mitigate operator variability, Specifying the position and shape of calibration points in the ultrasound phantoms was employed. This allowed us to obtain paired ultrasound images from different scanners. This phantom methodology is also applied by ^30–35to evaluate GAN networks.

We considered the calibration points in the images as target regions and extracted radiomics features from them. By calculating and comparing the Pearson correlation coefficients of the calibration point features between the source domain images, generated images, and target domain images, we visualized the level of feature coordination. Additionally, peak signal-to-noise ratio (PSNR) ³⁶ was used for evaluating image quality, structural similarity index (SSIM) ³⁷ for assessing whether there were any changes in image structure, mutual information (MI) ³⁸ for measuring the shared information between two random variables and quantifying the similarity between two images. Fréchet Inception Distance (FID) ³⁹ is a metric commonly used to evaluate the quality and diversity of generated images, particularly in image generation tasks. It compares the distribution of generated images with that of real images in a specific feature space.

Predicting LNM After Harmonization

The validated style transfer models were applied to clinical images based on the report by Yi et al ¹². Four different scanners, namely ATL HDI 5000 (Philips), Voluson-E8 (GE Healthcare), Mylab classC (Esaote), and ACUSON S2000 (Siemens), were used to transfer the images to the HI VISION Preirus (HITACHI Ltd) style of ECC US images. Both CycleGAN and its improved versions (CycleGAN + SN, CycleGAN + SN + SE, and CycleGAN + SN + SE + Mix) were applied to the same set of clinical images. A fast image style transfer network model based on Fourier domain adaptation (FDA) was used for comparison. This network achieved image style transfer by swapping the high and low-dimensional features of two images ⁴⁰. For each model, the source domain images and generated images used the same mask and pyradiomics for feature extraction. The optimal features were selected using the Mann-Whitney U test and Least Absolute Shrinkage and Selection Operator (LASSO), and elastic net parameters were adjusted using ten-fold cross-validation ridge regression to avoid overfitting ⁴¹. The λ coefficient was adjusted to maximize the area under the receiver operating characteristic (ROC) curve (AUC). Based on the linear combination of selected features, radiomics scores were calculated, and predictive models were established before and after harmonization.

Statistical Analysis

PSNR, SSIM, FID and Pearson's correlation coefficient were calculated in Python (version 3.7.0). Statistical analyses were performed in R analysis platform (version 3.6.0) and OriginPro 2016. Selection of key features and logistic regression model building were done using the “glmnet” package. For continuous clinical variables, a two-sample t test was used. For categorical variables, Fisher's exact test and chi-square test were used t. For all tests, p < 0.05 was considered as statically significant.

Results

Phantom Data Results

As shown in Figure 4, the performance of the final model is compared with FDA, CycleGAN, and its improved models using ultrasound phantom data. Before and after harmonization, Figure 4a provides a comparison of image quality, structure, and feature domain between the original images and generated images with the PSNR, SSIM, FID, and MI for the original and generated images of 11.18 ± 0.69 versus 15.45 ± 0.55, 0.16 ± 0.006 versus 0.17 ± 0.006, 224.48 versus 206.18, and 6.85 ± 0.11 versus 4.78 ± 0.07, respectively. Figure 4b presented the Pearson correlation coefficients and heat maps between radiomics features extracted from the original and transferred images, in correlation with those from target images. The average correlation coefficients were 0.60 (95% confidence interval (CI), 0.53–0.65) and 0.81 (95% CI, 0.77-0.86) for original and generated images in correlation with the target images, respectively.

Figure 4.

Evaluation of image quality, structural similarity and radiomics reproducibility using US phantom images after style transfer: (a) a comparison of image quality and structure metrics between original and generated images; (b) Pearson correlation coefficients and heat maps depicting the radiomics features extracted from the target of original and generated images.

Clinical Data Results

The patient data in the final analysis was divided into a training set (n = 118) and a validation set (n = 51) in a 7:3 ratio. The training set and validation set consisted of 81 and 37 cases of confirmed LNM, and 39 and 12 cases of confirmed non-LNM, respectively. All style transfer models were tested, and the generated images and original images were evaluated using the same evaluation metrics as in the phantom study. Due to the unavailability of paired images from different scanners, only the MI and FID metrics were used for evaluation. A typical vision evaluation of the original and transferred images was shown in Figure 5. The image quality of the transferred images in comparison with their original ones were presented in Table 2 with model CycleGAN + SN + SE + Mix achieved a best FID and MI of 26.11 and 4.78 ± 0.07, respectively.

Figure 5.

Original images from different ultrasound devices and their transferred images, a1-4) original images from SIMENS_SONLINE, MYCLASS, GE-Volusion E8 and HDI5000; b1-4) corresponding generated images in the style of radiomics features of HITACHI-Preirus.

Table 2.

Evaluate the Performance of the Generated Images to Display the Style Conversion Model by Using our Proposed Evaluation Metrics.

Model	FID	MI（95%CI）
FDA	44.08	5.74 ± 0.10
CycleGAN	80.99	5.56 ± 0.14
CycleGAN + SN	70.74	5.37 ± 0.09
CycleGAN + SN + SE	56.63	5.12 ± 0.08
CycleGAN + SN + SE + Mix	26.11	4.78 ± 0.07

Note: (1)Bold indicates the highest value of the indicator in the table. FID: Fréchet Inception Distance, MI: Mutual information, SN: spectral normalization, SE: self-attention, Mix: hybrid loss function, FDA: Fourier domain adaptation, CycleGAN: Cycle Generative Adversarial Network

A total of 4, 15, 9, 9, and 5 radiomics features were screened out of 451 features from Model FDA, CycleGAN, CycleGAN + SN, CycleGAN + SN + SE, and CycleGAN + SN + SE + Mix, respectively, after the Mann–Whitney U test and the LASSO analysis. The detailed features and their corresponding coefficients, as well as the radiomics score calculation, were shown in supplemental material S1 file 2. The performance of LNM prediction radiomics models with features extracted from generated images of different style transfer models was shown in Figure 6a, with an AUC ranged from 0.73 (95% CI: 0.57-0.89) to 0.85 (95% CI: 0.74-0.96). Figure 6b showed the comparison of predictive accuracy for LNM in patients with ECC, presented using radiomics models constructed with both original images and style-transferred images. The results revealed an AUC of 0.78 (95% CI: 0.64-0.93) for original images and an AUC of 0.85 (95% CI: 0.74-0.96) for style-transferred images. Detailed performance and comparison among these models were presented in Table 3.

Figure 6.

Radiomics performance with transferred ultrasound images in the prediction of lymph node metastasis for patients with cervical cancer; (a) radiomics performance with generated images with different style transfer models; (b) radiomics performance comparison with original ultrasound images and transferred images.

Table 3.

Comparison of Prediction Model Performance Between Training Cohort and Validation Cohort.

	Style Transfer Model	Training cohort				Testing cohort
	AUC(95%CI)	ACC	SPE	SEN	AUC(95%CI)	ACC	SPE	SEN
FDA	0.82(0.74–0.89)	0.81	0.84	0.73	0.81(0.68–0.93)	0.74	0.67	0.89
CycleGAN	0.87(0.80–0.93)	0.83	0.85	0.77	0.75(0.61–0.90)	0.70	0.61	0.89
CycleGAN + SN	0.83(0.75–0.91)	0.81	0.84	0.73	0.73(0.57–0.89)	0.73	0.64	0.89
CycleGAN + SN + SE	0.79(0.71–0.88)	0.80	0.83	0.70	0.80(0.68–0.93)	0.77	0.70	0.89
CycleGAN + SN + SE + Mix	0.77(0.67–0.86)	0.75	0.76	0.70	0.85(0.74–0.96)	0.77	0.67	0.95

Notes: AUC: area under curve, ACC: accuracy, SPE: specificity, SEN: sensitivity, SN: spectral normalization, SE: self-attention, Mix: hybrid loss function, FDA: Fourier domain adaptation, CycleGAN: Cycle Generative Adversarial Network

Discussion

In this study, a CycleGAN-based style transfer network was adapted and trained to transfer US images from different devices to one specific device to improve image quality and homogeneity, so as to reduce the impact of different devices on radiomics for ECC patients. Radiomics model with generated US images achieved an AUC of 0.85 compared with 0.78 with original images in preoperative LNM prediction.

The phantom results demonstrated that the generated images after style transfer showed a better image quality and remained stable structures with a better SSIM and a higher PSNR. Pearson correlation analysis demonstrated that radiomics features extracted from the generated images are highly correlated with target images indicating a higher reproducibility in comparison with original images. The comparisons with recent research work on image harmonization are summarized in Table 4. The AUC of radiomics model with radiomics features extracted from US images acquired from multiple devices were 0.78 (95% CI, 0.64-0.93), which was higher than the reported AUC results of 0.66 (95% CI, 0.59-0.73) in the training cohort and higher than 0.61 (95% CI, 0.50-0.72) in the validation cohort in the study of Yi et al ¹².With proposed style transfer model, the radiomics model improved the AUC for LNM prediction to 0.85 (95% CI, 0.74-0.96) after transferring US images from other four devices to those of HI VISION Preirus (HITACHI Ltd).This was higher than the reported best achievable of AUC of 0.80 ± 0.17 with images from individual US device.

Table 4.

Recently Research of US Harmonization and Cervical Cancer Lymph Node Metastasis.

Reference	Data	Variation Across	Task	Result
Our method	Phantom, cervical cancer	Multiple vendors dataset	Auxiliary diagnosis, domain Adaptation	AUC of LNM prediction was improved from 0.78 to0.85, Radiomics Pearson correlation coefficient of 0.60 and 0.81 for original and generated images
Barkat et al ⁴²	Breast cancer	Multiple centers dataset	Segmentation	Dice scores of 0.91 and 0.70 for benign and malignancies
Usama et al ⁴³	Carotid Ultrasound	Dual vendors dataset	Domain Adaptation	Original and generated images exhibited image harmonization histogram correlations of 0.89 and 0.92, and noise reduction histogram correlations of 0.844 and 0.707, respectively
Chen et al ⁴⁴	CT	Single vendors dataset	Auxiliary diagnosis, Disease prediction	Applying the PCA algorithm to generate optimal feature vectors based on radiomics features from CT images for predicting lymph node metastasis in cervical cancer patients optimized the machine learning-based lymph node metastasis prediction model, leading to improved accuracy.
Wang et al ⁴⁵	MRI	Single vendors dataset	Auxiliary diagnosis, Disease prediction	MRI radiomics-based model exhibited favorable accuracy when used to predict LNM in patients with CC.
Yang et al ⁴⁶	PET/CT	Single vendors dataset	Auxiliary diagnosis, Disease prediction	Radiomics features from PET/CT have the potential to predict lymph node metastasis (LNM) and can serve as predictive factors for LNM.
Yang et al ⁴⁷	Cervical Ultrasound	Single vendors dataset	Auxiliary diagnosis, Disease prediction	Compared to the clinical model, both the US-radiomics model and the combined model show good performance in predicting preoperative lymph node metastasis in cervical cancer patients.
Haberl et al ²⁰	PET	Multiple centers dataset	Domain Adaptation	GANs are capable of performing image harmonization and increase reproducibility and predictive performance of radiomic features derived from different centers and canners.

Previous studies had shown that radiomics methods can non-invasively predict the preoperative lymph node status in cervical cancer patients ^44–47. However, these studies did not further investigate the reproducibility and stability of radiomics features. While Haberl et al ²⁰ considered the impact of different centers on PET radiomics, they did not analyze the influence of different scanners, acquisition protocols, and reconstruction settings on radiomics features. Yi et al demonstrated that the discriminative accuracy of US based radiomics could be of 17.8% difference with features extracted from different US devices ¹². One way to increase the producibility of radiomics features is to harmonize their statistical properties by normalization or batch-effect correction using the ComBat method ^{48, 49}. In the image domain, methods of standardization of image acquisition, post-processing of raw sensor-level image data, data augmentation techniques, and style transfer were usually applied to harmonize radiomics features ⁵⁰. GAN and neural style transfer (NST) techniques, or a combination of both, had been investigated intensively with CT, MRI and PET images to address the variability across multi-centric radiomic studies ¹⁰. Previously, Liu et al proposed a novel and general style transfer framework to remove the appearance shifts of US images to improve US image segmentation ⁵¹. Another study on CycleGAN shows that Pseudo anatomical images generated on breast US provides a more intuitive display, enhances tissue anatomy, and preserves tumor geometry; and can potentially improve diagnoses and clinical outcomes ⁴². Similarly, research on cardiac ultrasound has also demonstrated that CycleGAN has great improvement on image quality and image feature harmonization ⁴³. However, the review also mentions that DL is a ‘black box’ approach, the lack of interpretability of the models and the deep features generated are seen as a key limitation in clinical applications ⁵².

The combination of feature and image domain harmonization for US images based radiomics study is of great clinical value in the future studies. In this study, only US images of cervical cancer were tested for LNM prediction. US images for other cancer types are also potential in future studies. Our model can also be used in other diseases other than cervical cancer. Images from multiple centers were also needed to further validate the accuracy of the proposed style transfer models. In addition, style transfer networks can be applied not only in ultrasound but also in MRI and CT, etc As there are differences between devices and different scanning sequence parameters, this method can also be used for radiomics normalization.

Conclusion

The adapted CycleGAN network was feasible and accurate to convert US images acquired from different US devices into images of one specific device to improve the image quality and the performance of radiomics studies.

Supplemental Material

sj-docx-1-tct-10.1177_15330338241302237 - Supplemental material for Radiomics Harmonization in Ultrasound Images for Cervical Cancer Lymph Node Metastasis Prediction Using Cycle-GAN

Supplemental material, sj-docx-1-tct-10.1177_15330338241302237 for Radiomics Harmonization in Ultrasound Images for Cervical Cancer Lymph Node Metastasis Prediction Using Cycle-GAN by Zeshuo Zhao, Yuning Qin, Kai Shao, Yapeng Liu, Yangyang Zhang, Heng Li, Wenlong Li, Jiayi Xu, Jicheng Zhang, Boda Ning, Xianwen Yu, Xiance Jin and Juebin Jin in Technology in Cancer Research & Treatment

Footnotes

Abbreviations

Author's Contributions

Jue-bin Jin, Xian-ce Jin, Ze-shuo Zhao, and Ji-cheng Zhang made substantial contributions to the conception of the study. Jue-bin Jin, Xian-ce Jin, Ze-shuo Zhao, Ji-cheng Zhang, and Xian-wen Yu critically revised the manuscript for important intellectual content. Jue-bin Jin, Xian-ce Jin, and Ze-shuo Zhao agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Jue-bin Jin, Xian-ce Jin, Ze-shuo Zhao, Ji-cheng Zhang, Xian-wen Yu, Kai Shao, Yang-yang Zhang, Ya-peng Liu, Heng Li, and Wen-long Li analyzed the data. Jue-bin Jin, Xian-ce Jin, Ze-shuo Zhao, Ji-cheng Zhang, Xian-wen Yu, Kai Shao, Yang-yang Zhang, Ya-peng Liu, and Bo-da Ning interpreted the data. Ya-peng Liu, Yang-yang Zhang, and Kai Shao have the checking, and verification review of clinical data. Ji-cheng Zhang, Heng Li, Wen-long Li, Yu-ning Qin, and Jia-yi Xu acquired the data. Ze-shuo Zhao drafted the manuscript.

Data Availability Statement

Data is available upon request to the corresponding author.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethics Statement

The study was conducted in accordance with the Declaration of Helsinki and approved by the Research Ethics Committee (ECCR no.2019059) of the author's hospital. Written informed consent was waived for this retrospective study to maintain patient data confidentiality.

Funding

This research was supported partially by a key project of Zhejiang Natural Science Foundation [Z24A050009], a Key project of Zhejiang Provincial Health Science and Technology Program [WKJ-ZJ-2437], a Major project of Wenzhou Science and Technology Bureau [ZY2022016, ZY2020011], a project of Wenzhou Science and Technology Bureau (Y2023798), Zhejiang Engineering Research Center for innovation and application of Intelligent Radiotherapy Technology, Zhejiang-Hong Kong Precision Theranostics of Thoracic Tumors Joint Laboratory, and Wenzhou key Laboratory of basic science and translational research of radiation oncology.

ORCID iD

Xiance Jin

Supplemental Material

Supplemental material for this article is available online.

References

Noble

Boukerroui

. Ultrasound image segmentation: A survey. IEEE Trans Med Imaging 2006;25(8):987-1010. doi:https://doi.org/10.1109/TMI.2006.877092

Yeo

Romero

. Optical ultrasound simulation-based training in obstetric sonography. J Matern Fetal Neonatal Med 2022;35(13):2469-2484. doi:https://doi.org/10.1080/14767058.2020.1786519

Guo

Qin

Fei

. Ultrasound imaging technologies for breast cancer detection and management: A review. Ultrasound Med Biol 2018;44(1):37-70. doi:https://doi.org/10.1016/j.ultrasmedbio.2017.09.012

Wang

Zhang

Zhu

. Automatic diagnosis for thyroid nodules in ultrasound images by deep neural networks. Med Image Anal 2020;61:101665. doi:https://doi.org/10.1016/j.media.2020.101665

Guo

Qiao

, et al. Radiomics analysis on ultrasound for prediction of biologic behavior in breast invasive ductal carcinoma. Clin Breast Cancer 2018;18(3):e335-e344. doi:https://doi.org/10.1016/j.clbc.2017.08.002

Cobo

Bonet-Carne

Martínez-Terrón

, et al. Feasibility and reproducibility of fetal lung texture analysis by automatic quantitative ultrasound analysis and correlation with gestational age. Fetal Diagn Ther 2012;31(4):230-236. doi:https://doi.org/10.1159/000335349

Bonet-Carne

Palacio

Cobo

, et al. Quantitative ultrasound texture analysis of fetal lungs to predict neonatal respiratory morbidity. Ultrasound Obstet Gynecol 2015;45(4):427-433. doi:https://doi.org/10.1002/uog.13441

Jin

Zhang

, et al. Noninvasive prediction of lymph node status for patients with early-stage cervical cancer based on radiomics features from ultrasound images. Eur Radiol 2020;30(7):4117-4124. doi:https://doi.org/10.1007/s00330-020-06692-1

Gillies

Kinahan

Hricak

. Radiomics: Images are more than pictures, they are data. Radiology. 2016;278(2):563-577. doi:https://doi.org/10.1148/radiol.2015151169

10.

Mali

Abdalla

Woodruff

, et al. Making radiomics more reproducible across scanner and imaging protocol variations: A review of harmonization methods. J Pers Med 2021;11(9):842.

11.

Jin

Zhu

Teng

Xie

Jin

. The accuracy and radiomics feature effects of multiple U-net-based automatic segmentation models for transvaginal ultrasound images of cervical cancer. J Digit Imaging 2022;35(4):983-992. doi:https://doi.org/10.1007/510278-022-00620-2

12.

Lei

Zhang

, et al. The influence of different ultrasonic machines on radiomics models in prediction lymph node metastasis for patients with cervical cancer. Technol Cancer Res Treat 2022;21:15330338221118412. doi: https://doi.org/10.1177/15330338221118412

13.

Teng

Liang

, et al. The effects of automatic segmentations on preoperative lymph node Status prediction models with ultrasound radiomics for patients with early stage cervical cancer. Technol Cancer Res Treat 2022;21:15330338221099396. doi:https://doi.org/10.1177/15330338221099396

14.

Boellaard

Delgado-Bolton

Oyen

, et al. FDG PET/CT: EANM procedure guidelines for tumour imaging: Version 2.0. Eur J Nucl Med Mol Imaging 2015;42(2):328-354. doi:https://doi.org/10.1007/s00259-014-2961-x

15.

Lee

Steinmann

Ding

, et al. Radiomics feature robustness as measured using an MRI phantom. Sci Rep. 2021;11(1):3973.

16.

Masson

Da-Ano

Lucia

, et al. Statistical harmonization can improve the development of a multicenter CT-based radiomic model predictive of nonresponse to induction chemotherapy in laryngeal cancers. Med Phys. 2021;48(7):4099-4109.

17.

Ibrahim

Primakov

Beuque

, et al. Radiomics for precision medicine: Current challenges, future prospects, and the proposal of a new framework. Methods. 2021;188:20-29. doi:https://doi.org/10.1016/j.ymeth.2020.05.022

18.

Alessandrini

De Craene

Bernard

, et al. A pipeline for the generation of realistic 3D synthetic echocardiographic sequences: Methodology and open-access database. IEEE Trans Med Imaging 2015;34(7):1436-1451.

19.

Gao

. Neural style transfer improves 3d cardiovascular mr image segmentation on inconsistent data. Springer. 2019:128-136. doi: https://doi.org/10.1007/978-3-030-32245-8_15

20.

Haberl

, Spielvogel CP, Jiang Z, et al.

Multicenter PET image harmonization using generative adversarial networks.

Eur J Nucl Med Mol Imaging 2024;51:2532-2546. doi:https://doi.org/10.1007/s00259-024-06708-8

21.

Isola

Zhu

J-Y

Zhou

Efros

. Image-to-image translation with conditional adversarial networks. Paper presented at: the IEEE conference on computer vision and pattern recognition. 2017:1125-1134.

22.

Choe

Lee

K-H

, et al. Deep learning–based image conversion of CT reconstruction kernels improves radiomics reproducibility for pulmonary nodules or masses. Radiology. 2019;292(2):365-373.

23.

Kim

, Park S M, Lee S, et al.

A deep learning harmonization of multi-vendor MRI for robust intervertebral disc segmentation.

IEEE Access. 2024;12:19482-19499. doi: https://doi.org/10.1109/ACCESS.2024.3360272

24.

Zhu

J-Y

Park

Isola

Efros

. Unpaired image-to-image translation using cycle-consistent adversarial networks. Paper presented at: the IEEE international conference on computer vision (ICCV). 2017:2223-2232.

25.

Zhang

Goodfellow

Metaxas

Odena

. Self-attention generative adversarial networks. PMLR. Paper presented at: Machine Learning Research. 2019:7354-7363.

26.

Zhao

Gallo

Frosio

Kautz

. Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging. 2017;3(1):47-57. doi:https://doi.org/10.1109/TCI.2016.2644865

27.

Huang

Zhou

Guo

Wang

. A stability-enhanced CycleGAN for effective domain transformation of unpaired ultrasound images. Biomed Signal Proces. 2022;77:103831. doi:https://doi.org/10.1016/j.bspc.2022.103831.

28.

Miyato

Kataoka

Koyama

Yoshida

. Spectral normalization for generative adversarial networks. ArXiv. Published online 2018. arXiv:180205957. doi:https://doi.org/10.48550/arXiv.1802.05957

29.

Heusel

Ramsauer

Unterthiner

Nessler

Hochreiter

. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Paper presented at: Advances in neural information processing systems. 2017:30.

30.

Peng

Huang

Wang

, et al. A real-time medical ultrasound simulator based on a generative adversarial network model. In: 2019 IEEE international conference on image processing (ICIP). IEEE; 2019:4629-4633.

31.

Wang

Fang

, et al. High-resolution image reconstruction for portable ultrasound imaging devices. EURASIP Journal on Advances in Signal Processing. 2019;2019(56):1-12.

32.

Long

Tian

. Spatial and channel attention-based conditional wasserstein GAN for direct and rapid image reconstruction in ultrasound computed tomography. Biomed Eng Lett 2024;14(1):57-68.

33.

Khor

Ning

Zhang

, et al. Ultrasound speckle reduction using wavelet-based generative adversarial network. IEEE J Biomed Health Inform 2022;26(7):3080-3091.

34.

Jiang

Zhang

Luo

, et al. A generalized image quality improvement strategy of cone-beam CT using multiple spectral CT labels in Pix2pix GAN. Physics in Medicine & Biology. 2022;67(11):115003.

35.

Vukovic

Ruvinov

Antico

, et al. Automatic GAN-based MRI volume synthesis from US volumes: A proof of concept investigation. Sci Rep 2023;13(1):21716.

36.

Korhonen

You

. Peak signal-to-noise ratio revisited: Is simple beautiful? IEEE. Paper presented at: 2012 Fourth International Workshop on Quality of Multimedia Experience. 2012:37-38. doi: https://doi.org/10.1109/QoMEX.2012.6263880

37.

Wang

Bovik

Sheikh

Simoncelli

. Image quality assessment: From error visibility to structural similarity. IEEE Trans Image Process 2004;13(4):600-612.

38.

Kraskov

Stögbauer

Grassberger

. Estimating mutual information. Physical Review E. 2004;69(6):066138. doi:https://doi.org/10.1103/PhysRevE.69.066138

39.

Obukhov

Krasnyanskiy

. Quality assessment method for GAN based on modified metrics inception score and fréchet inception distance. Springer. Paper presented at: 4th Computational Methods in Systems and Software 2020. 2020:102-114.

40.

Yang

. Soatto S. Fda: Fourier domain adaptation for semantic segmentation. Paper presented at: the IEEE/CVF conference on computer vision and pattern recognition. 2020:4085-4095.

41.

Friedman

Hastie

Tibshirani

. Regularization paths for generalized linear models via coordinate descent. J Stat Softw 2010;33(1):1.

42.

Barkat

Freiman

Azhari

. Image translation of breast ultrasound to Pseudo anatomical display by CycleGAN. Bioengineering. 2023;10(3):388. https://doi.org/10.3390/bioengineering10030388

43.

Usama

Nyman

Naslund

, et al. A domain adaptation model for carotid ultrasound: image harmonization, noise reduction, and impact on cardiovascular risk markers[J]. ArXiv. Published online 2024. arXiv:240705163. doi:https://doi.org/10.48550/arXiv.2407.05163

44.

Chen

, Liu W, Thai T C, et al.

Developing a new radiomics-based CT image marker to detect lymph node metastasis among cervical cancer patients.

Comput Methods Programs Biomed. 2020;197:105759.

45.

Wang

Zhang

. A MRI radiomics-based model for prediction of pelvic lymph node metastasis in cervical cancer. World J Surg Oncol 2024;22(1):55.

46.

Yang

, Zhang W, Liu C, et al.

Predictive value and potential association of PET/CT radiomics on lymph node metastasis of cervical cancer.

Annals of Medicine and Surgery. 2024;86(2):805-810.

47.

Yang

, Wang Y, Zhang J, et al.

A novel ultrasound-based radiomics model for the preoperative prediction of lymph node metastasis in cervical cancer.

Ultrasound Med Biol 2024;50(12):1793-1799. doi:0.1016/j.ultrasmedbio.2024.07.013

48.

Chatterjee

Vallières

Dohan

, et al. Creating robust predictive radiomic models for data from independent institutions using normalization. IEEE Transactions on Radiation and Plasma Medical Sciences. 2019;3(2):210-215. doi:https://doi.org/10.1109/TRPMS.2019.2893860

49.

Orlhac

Boughdad

Philippe

, et al. A postreconstruction harmonization method for multicenter radiomic studies in PET. J Nucl Med 2018;59(8):1321-1328. doi:https://doi.org/10.2967/jnumed.117.199935

50.

Da-Ano

Visvikis

Hatt

. Harmonization strategies for multicenter radiomics investigations. Physics in Medicine & Biology. 2020;65(24):24TR02. doi:https://doi.org/10.1088/1361-6560/aba798

51.

Liu

Yang

Gao

, et al. Remove appearance shift for ultrasound image segmentation via fast and universal style transfer. IEEE. Paper presented at: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). 2020:1824-1828.

52.

Alzubaidi

Zhang

Humaidi

, et al. Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions. J Big Data 2021;8(53):1-74. doi:https://doi.org/10.1186/s40537-021-00444-8

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB