A novel deep learning technique for medical image analysis using improved optimizer

Abstract

Application of Convolutional neural network in spectrum of Medical image analysis are providing benchmark outputs which converges the interest of many researchers to explore it in depth. Latest preprocessing technique Real ESRGAN (Enhanced super resolution generative adversarial network) and GFPGAN (Generative facial prior GAN) are proving their efficacy in providing high resolution dataset. Objective: Optimizer plays a vital role in upgrading the functioning of CNN model. Different optimizers like Gradient descent, Stochastic Gradient descent, Adagrad, Adadelta and Adam etc. are used for classification and segmentation of Medical image but they suffer from slow processing due to their large memory requirement. Stochastic Gradient descent suffers from high variance and is computationally expensive. Dead neuron problem also proves to detrimental to the performance of most of the optimizers. A new optimization technique Gradient Centralization is providing the unparalleled result in terms of generalization and execution time. Method: Our paper explores the next factor which is the employment of new optimization technique, Gradient centralization (GC) to our integrated framework (Model with advanced preprocessing technique). Result and conclusion: Integrated Framework of Real ESRGAN and GFPGAN with Gradient centralization provides an optimal solution for deep learning models in terms of Execution time and Loss factor improvement.

Introduction

Integrated framework of deep learning model with advanced preprocessing technique like Real ESRGAN ¹ and GFPGAN ² gives significant elevation in terms of efficiency of these models. Efficiency shoots up to 5–10% via pipe lining of these preprocessing technique with base models. On a way to further optimize the deep learning models, usage of optimizer is explored. Optimizer plays an indispensable role in the functioning of CNN. They help in adjusting weights so as to minimize the incurred losses. Many optimizers have been employed in CNN like Gradient descent,^3–8 Stochastic Gradient descent,^9–14 Ada grad,^15–18 Ada delta^19–21 and Adam^22–25 etc. for classification and segmentation of Medical image. Gradient descent iteratively reduces a loss function by moving in the direction opposite to that of steepest ascent. As its uses entire training set for calculation which requires large amount of memory and hence slow down process. Stochastic gradient descent is a variant of gradient descent and update model parameter one by one which leads to high variance and make it computationally expensive. Stochastic gradient descent with momentum is an improved version which takes in account the previous update to fine tune the final update direction and hence leads to more stability but extra hyper parameter is added. So far learning rate has been constant but with the introduction of Ada grad (Adaptive Gradient descent) concept of adaptive learning rate emerges that is it uses different learning rate for each and every neuron and for each and every hidden layer based on different iteration but it too leads to dead neuron problem. RMS-prop^26–30 (Root mean square propagation) is a special version of Ada grad in which learning rate is exponential average of gradient instead of cumulative sum of squared gradient but it suffers from slow learning rate. Ada delta is an extension of Ada grad. Here it, does not set default learning rate and uses the ratio of running average of previous time steps to current gradient. It removes decaying rate problem but it is computationally expensive. Adam (Adaptive moment estimator) optimizer is one of the popular gradient descent optimization algorithm. It computes adaptive learning rate for each parameter and stores both the decaying average of past gradients and also the decaying average of past squared gradient. It has little memory requirement and is computationally efficient.

Aforementioned optimization techniques operate on activation or weight vectors and hence their adoption to pre-trained models pose a great challenge. So a new method, Gradient centralization^31–34 is used which acts on gradient of weight vectors and centralizes the gradient vector to have zero mean. Thus introduction of Gradient centralization accelerate the training process and enhances the generalization performance.

Our research work explores the functioning of GC (Gradient centralization) on our integrated framework of deep learning models. Next Section illustrates the related work, Proposed work describes the achieved result via integrated framework and proposed work which includes further optimization with novel Gradient Centralization optimizer. Experimental result and discussion tabulates the experimental result and comparison of loss curves and Future work and conclusion winds the research paper with Future work and conclusion.

Related work

Significant efforts are made in the direction of improving the efficiency of deep learning models. Application of voting based method to deep learning models helps in strengthening the correct classification of lung disease. ³⁵ Preprocessing techniques plays an important role in boosting efficacy of deep learning models. ³⁶ Other factor like Optimizer too plays an pertinent role in efficiency improvement of Deep learning algorithms as they decide the rate of convergence of algorithm towards an optimal value. Many Researchers are evolving the usage of new optimizers to explore their functioning in deep learning models. Zhang ³⁷ proposes method, normalized direction-preserving Adam (ND-Adam), which enables more precise control of the direction and step size for updating weight vectors and experimentally proves to improve in generalization performance of adam optimizer. Khaire and Dhanalakshmi ³⁸ brings out another improvement in momentum of adam optimizer by evaluating the gradient after applying the current velocity and called its as iAdam. It works efficiently for the high-dimensional dataset and converges rapidly. iAdam is simple to execute, computationally effective, well suited for high-dimensional datasets, converge smoothly and operate efectively even for higher levels of learning Usage of appropriate optimizer for particular input dataset greatly affects the efficiency of architecture used for classifying that dataset. Halgamuge et al. ³⁹ carried out comparison of six optimizer Ada grad], Ada delta, Rms prop, Adam, Nadam, and SGD (Stochastic gradient descent) and identify the best optimizer as Ada grad which estimate potential fire occurrences in given locations and gives precise predictions with less error and processes real-time data. Adagrad has highest prediction rate 43.93 min and Nd adam and has highest accuracy and lowest loss rate that is 98.86% and 0.03%. Kandel et al. ⁴⁰ improves the efficiency of its CNN based classifier by comparing six different first-order stochastic gradient-based optimizers and adaptative based optimizers emerges out as best for classifying histopathology images for low learning rate and produces an AUC of 94% and Ada grad proves to be low performer even on low learning rate. Yaqub et al. ⁴¹ perform a comparative analysis of 10 different gradient descent-based optimizers, Adaptive Gradient (Ada grad), Adaptive Delta (Ada Delta), Stochastic Gradient Descent (SGD), Adaptive Momentum (Adam), Cyclic Learning Rate (CLR), Adaptive Max Pooling (Ada max), Root Mean Square Propagation (RMS Prop), Nesterov Adaptive Momentum (Nadam), and Nesterov accelerated gradient (NAG) for CNN for brain tumor classification and segmentation and Adam optimizer helps in attaining the greatest efficiency of 99.2%. Chowdhury et al. ⁴² explores the CNN with one, two, three, four and five hidden layers with combination of three optimizers namely,Rms prop, Adam and SGD and finds CNN with four hidden layer using SGD optimizer having greatest testing accuracy of 91%. Bera and Shrivastava ⁴³ performs Hyper spectral image (HSI) classification with seven different optimizers SGD, Ada grad, Ada delta, Rms prop, Adam, Ada Max, and Nadam using Deep CNN model and establishes the superiority of Adam optimizer achieving average accuracy of 98% on every HSI datasets. Perin and Picek ⁴⁴ illustrates that hyper parameter optimizer plays indispensable role in in deep learning-based side-channel analysis. He experimentally shows that Adam and Rmsprop works well for shorter training phases as they easily overfitand strongly supports Ada grad for longer training phases. Poojary and Pai ⁴⁵ used CNN based model Resnet 50 and Inception v3 for kaggle cat versus dog classification and carried out comparative analysis for three optimizer namely Adam, SGD, RMS prop and RMS prop performed extremely well with training accuracy of 99%. Vani and Rao ⁴⁶ implemented the usage of seven optimizers namely Stochastic Gradient Descent (SGD), RmsProp, Adam, Ada max, Ada grad, Ada delta, and Nadam are implemented in CNN on Indian Pines Dataset Ada max excels with 99.58% accuracy. Yaqub et al. ⁴¹ carries out Brain tumor classification and segmentation using BraTS2015 data set for 10 optimizer namely namely Adaptive Gradient (Ada grad), Adaptive Delta (Ada Delta), Stochastic Gradient Descent (SGD), Adaptive Momentum (Adam), Cyclic Learning Rate (CLR), Adaptive Max Pooling (Ada max), Root Mean Square Propagation (RMS Prop), Nesterov Adaptive Momentum (Nadam), and Nesterov accelerated gradient (NAG) and proves Adam to be efficient with accuracy of 99.2%. Dubey et al. ⁴⁷ works out on step size taken for each parameter and proposed an Diffgrad optimization technique which takes larger step size for faster gradient changing parameter and lower step size for lower gradient changing parameter. This method outperforms optimizer technique like SGDM, Ada Grad, Ada Delta, RMS Prop, AMS Grad, and Adam. Taqi et al. ⁴⁸ proposes Alzheimer disease (AD) classification based by using four different optimizers Ada grad, ProximalAda grad, Adam, and RMS Prop and Rms prop works with 100% accuracy. Thavasimani and Srinath ⁴⁹ experimented with various optimizers such as ADAM, SGD, RMS prop, Ada delta, Ada max, Ada grad, Nadam to detect bot accounts with the help of deep learning model from CRESCI-2017 twitter dataset issued by Indiana University and establishes the highest accuracy of 98.90% with RMS prop. Elangovan and Nath ⁵⁰ bring out the role of optimizer in improving the performance of deep neural network for image classification problem and analyzes three standard first-order optimizers like stochastic gradient descent with momentum (SGDM), adaptive moment estimation (Adam), and root mean square propagation (RMS Prop) for detecting glaucoma employing architectures like Alex net,VGG-19 and Resnet 101. Adam optimizer shows better result in this.

Proposed work

Our proposed work is the continuation of our effort to optimize the deep learning models. Imbibing Advanced preprocessing technique like Real ESRGAN^51–59 and GFP Gan^2,60–68 with base deep learning model like NASNet^69–78 and DenseNet 201^79–88 produces an significant results in context of validation accuracy. Results achieved via this pipe lining is tabulated in Table 1 for Real ESRGAN and GFP Gan respectively.

Table 1.

Result of integrated model.

REAL ESRGAN	SKIN ACCURACY		LUNG ACCURACY		RETINA ACCURACY
Model used	Without preprocessing (%)	With preprocessing (%)	Without preprocessing (%)	With preprocessing (%)	Without preprocessing (%)	With preprocessing (%)
DenseNet 201	28	72	85	89	74	88
NASNet	35	63	77	78	73	89

GFPGAN	SKIN		LUNG		RETINA
Model used	Without preprocessing (%)	With preprocessing (%)	Without preprocessing (%)	With preprocessing (%)	Without preprocessing (%)	With preprocessing (%)
DenseNet 201	28	62	85	86	74	89
NASNet	35	73	77	78	73	84

For skin dataset massive improvement is achieved for validation accuracy. Retina dataset also responds well for integrated framework. Minimal enhancement is observed for lung dataset. But overall positive trends are observed using images preprocessed with Real ESRGAN and GFPGAN as input to our base models.

Our next step to further optimize the integrated deep learning models with the use of novel optimizer Gradient centralization technique.

Gradient centralization

Gradient centralization (GC) can enhance DNN(Deep neural network) ultimate generalisation performance while also accelerating the training process. As indicated in Figure 1 GC works by by centralizing the gradient vectors to have a mean of zero and thus directly manipulates gradients. Gradient Centralization further enhances the loss function and its gradient Lipschitzness, making the training process more effective and stable. Batch Normalization and Weight standardization technique also reduces the Lipschitzness of loss function but they operate on activation and weight vectors and were not able to adapt to pre trained models but Gradient Centralization takes in account the gradients vectors to achieve improvement. Z-score standardization can also be used to normalize the gradient but normalising gradient does not increase training’s stability. Instead, calculating the gradient vectors’ means and then centralizing the gradients so that their means are zero effectively helps in achieving the desired result. This is the underlying principle of Gradient centralization method. It can be easily embedded into gradient based current optimization algorithm like Adam, SGDM with one line of code and helps in increasing training process, improves generalization capability and better tune it to pre trained models.

Figure 1.

Illustration of GC operation on matrix/tensor of weights in fully connected layer (left) and convolutional (right). GC computes the column/slice mean of gradient matrix/tensor and centralizes each column/slice to have zero mean.

Formula of GC

Let Gradient of a weight vector obtained through back propagation for a Fully connected layer or Convolutional layer be $\nabla {\mathrm{w}}_{\mathrm{i}}\mathrm{L}\hspace{0.17em}\left(\mathrm{i}=1,2,\dots ,\mathrm{N}\right)$ and GC operator be denoted by Φ_GC then formulation of Gradient centralization is given by:

${\mathrm{\Phi }}_{\text{GC}}\hspace{0.17em}\left({\nabla }_{\text{wi}}\mathrm{L}\right)={\nabla }_{\text{wi}}\mathrm{L}\hspace{0.17em}-\hspace{0.17em}\mathrm{\mu }{\nabla }_{\text{wi}}\mathrm{L}$

(1)

where $\mathrm{\mu }{\nabla }_{\text{wi}}\mathrm{L}=1/\mathrm{M}\hspace{0.17em}\left({{\displaystyle \sum }}_{\mathrm{j}=1}^{\mathrm{M}}\nabla {\mathrm{W}}_{\mathrm{i},\mathrm{j}}\mathrm{L}\right)$ .

Mean value of each column of gradient matrix is subtracted from its each column value. In this way each gradient of loss function w.r.t to weight vector is transform so that its mean becomes zero. computation of GC is quite simple and efficient. Matrix formulation of equation (1) is also given by:

${\mathrm{\Phi }}_{\text{GC}}\hspace{0.17em}\left({\nabla }_{\mathrm{W}}\mathrm{L}\right)=\mathrm{P}{\nabla }_{\mathrm{W}}\mathrm{L}$

(2)

where $\mathrm{P}=\mathrm{I}\hspace{0.17em}-\hspace{0.17em}{\text{ee}}^{\mathrm{T}}$ and P is known as cool operator and I is the identity matrix of $\mathrm{M}*\mathrm{M}$ order and and e = 1/√M

$\left[\begin{array}{llllll}1& 0& 0& \cdots & \cdots & 0\\ 0& 1& 0& \cdots & \cdots & 0\\ 0& 0& 1& 0& \cdots & 0\\ 0& 0& 0& \cdots & \cdots & 1\end{array}\right]$

Experimental result and discussion

In our proposed research work, integrated deep model (model pipeline with advanced preprocessing technique) is further enhanced by employing an new optimizer Gradient Centralization. GC can be easily embedded into existing optimization algorithm like SGDM, Adam and RMS prop by adding a single line of code. We have embedded GC with Rms prop. DenseNet 201 and NASNet deep models are used as base models. Three datasets have been used to carry out the experiment: Lung, Retina and skin. Inculcating GC in our integrated model brings out improvement in three parameters of model which are Training accuracy, Training Loss and Execution time. Result of our experiment for GFPGAN integrated models are tabulated in Tables 2–4 for all three datasets.

Table 2.

Result for optimized GFPGAN for lung dataset.

Model used	Accuracy		Loss		Execution time (sec.)
	WITH out GCTF	With GCTF	Without GCTF	With GCTF	Without GCTF	With GCTF
DenseNet201	91.875	94.0625	0.214071	0.05539	1393.98	647.126
NASNet	86.9565	90.625	0.37491	0.11726	1042.77	638.959

Table 3.

Result for optimized GFPGAN for retina dataset.

Model used	Training accuracy-validation accuracy		Training loss-validation loss		Execution time (sec.)
	Without GCTF	With GCTF	Without GCTF	With GCTF	Without GCTF	With GCTF
DenseNet201	92.0792	99.0099	0.227322	0.0357533	1199.61	812.791
NASNet	90.625	95.3125	0.3010	0.101164	1654.65	842.769

Table 4.

Result for optimized GFPGAN forskin dataset.

Model used	Training accuracy-validation accuracy		Training loss-validation loss		Execution time (sec.)
	Without GCTF	With GCTF	Without GCTF	With GCTF	Without GCTF	With GCTF
DenseNet201	98.3031	100	0.04873	0.009683	776.049	505.114
NASNet	96.6102	99.661	0.09873	0.0304052	738.592	511.332

Enhancement in Accuracy has been observed for all three datasets but retina dataset responds remarkably better as compared to other datasets. Training loss has also been reduced which appends to our positive result. Improvement in execution time is the major break through achieved via this new optimizer algorithm. Significant reduction has been observed as clear from table. Our result supports the underlying principle of GC which states of accelerating the training process.

Experiments for another pre-preprocessing technique Real ESRGAN integrated deep learning model is carried out which also employed a new optimized method of Gradient Centralization and the results are tabulated in Tables 5–7 respectively for three datasets: Skin, Retina and Lung.

Table 5.

Result of optimized real ESRGAN for skin dataset.

Model used	Accuracy		Loss		Exection time (sec.)
	Without GCTF	With GCTF	Without GCTF	With GCTF	Without GCTF	With GCTF
DenseNet201	99.3548	100	0.027379	0.00645161	1613.32	1112.79
NASNet	98.38	99.3548	0.0562655	0.026688	1639.68	1113.87

Table 6.

Result of optimized real ESRGAN for retina dataset.

Model used	Accuracy		Loss		exection time (sec.)
	Without GCTF	With GCTF	Without GCTF	With GCTF	Without GCTF	With GCTF
DenseNet201	97.11	98.75	0.0714245	0.0235976	1427.4	1049.87
NASNet	94.23	95	0.17624	0.10362	1877.56	1015.65

Table 7.

Result of optimized real ESRGAN for lung dataset.

Model used	Accuracy		Loss		Exection time (sec.)
	Without GCTF	With GCTF	Without GCTF	With GCTF	Without GCTF	With GCTF
DenseNet201	95.315	93.4375	0.145507	0.0640958	1849.38	848.866
NASNet	96.3021	97.704	0.25616	0.08136	2483.02	819.201

Upward trends are observed for integrated DenseNet 201 in terms of accuracy for skin and retina dataset but slight down trend is noticed for lung dataset which shows an deviation from our expectation. NASNet exhibit an similar trend for all three datasets. Training loss comes out with positive result and significant drop is observed for three datasets. Execution time too shows substantial drop which further strengthen the concept of imbibing this new optimized technique in our integrated model.

Evaluation of loss curve on imbibing Gradient Centralization technique for Dense net model pipe lined with Real ESRGAN and GfpGan for all three datasets: Retina, Lung and Skin brings out an interesting predictions as indicated by Figures 2–4 respectively as shown below:

Figure 2.

(a)Loss curve for real ESRGAN pipe lined DenseNet model for retina dataset (b) loss curve for GFPGAN pipe lined DenseNet model for retina dataset.

Figure 3.

(a)Loss curve for real ESRGAN pipe lined DenseNet model for lung dataset (b) loss curve for GFPGAN pipe lined DenseNet model for lung dataset.

Figure 4.

(a) Loss curve for real ESRGAN pipe lined DenseNet model for skin dataset (b) loss curve for GFPGAN pipe lined DenseNet model for skin dataset.

Experiments shows that DenseNet model when pipe lined with Real ESRGAN preprocessing technique for skin dataset gives optimum results for losses incurred during training which is 0.006451 and is minimum as compared with other combinations of dataset and technique.

Loss curves for combination of Gradient Centralization with NASNet model pipelined with Real ESRGAN and GFPGAN technique are shown in Figures 5–7 below for all three datasets.

Figure 5.

(a) Loss curve for real ESRGAN pipe lined NASNet model for skin dataset (b) loss curve for GFPGAN pipe lined NASNet t model for skin dataset.

Figure 6.

(a) Loss curve for real ESRGAN pipe lined NASNet model for lung dataset (b) loss curve for GFPGAN pipe lined NASNet t model for lung dataset.

Figure 7.

(a) Loss curve for real ESRGAN pipe lined NASNet model for retina dataset (b) loss curve for GFPGAN pipe lined NASNet t model for retina dataset.

Smooth loss curves are observed for skin dataset when NASNet pipe lined with Real ESRGAN is employed with GC is used. Though downward trend is shown by all datasets with both the preprocessing techniques but integrated Real ESRGAN with GC shows better result than GFPGAN.

Future work and conclusions

Our research work explores the Gradient Centralization optimization on integrated deep architecture using NASNet and DenseNet as base models. We get motivating results for parameters like Training loss and execution time. Reduction in execution time is a pertinent outcome achieved in our research work. Though Training accuracy was slight down for DenseNet in lung database but others parameters shows, this inculcation of new optimization technique, a significant step towards optimizing the performance of deep learning architecture for Medical Image analysis. In future, various other permutations can be explored like using other deep learning model like Resnet, Mobile net etc.We have embedded Gradient Centralization with Rms prop optimizer to carry out our experiment. Other optimizer like Adam, SGDM can also be used to embed Gradient Centralization. We have used Real ESRGAN and GFPGAN preprocessing integrated model and taken Skin, Lung and retina datasets. Analysis can also be carried out by advanced preprocessing technique like Swin transformers which basically require more GPU. Wide Spectrum of Medical dataset for experimentation can also be increased to evaluate effect of this new integrated approach.