Sage Journals: Discover world-class research

Abstract

Application of Convolutional neural network in spectrum of Medical image analysis are providing benchmark outputs which converges the interest of many researchers to explore it in depth. Latest preprocessing technique Real ESRGAN (Enhanced super resolution generative adversarial network) and GFPGAN (Generative facial prior GAN) are proving their efficacy in providing high resolution dataset. Objective: Optimizer plays a vital role in upgrading the functioning of CNN model. Different optimizers like Gradient descent, Stochastic Gradient descent, Adagrad, Adadelta and Adam etc. are used for classification and segmentation of Medical image but they suffer from slow processing due to their large memory requirement. Stochastic Gradient descent suffers from high variance and is computationally expensive. Dead neuron problem also proves to detrimental to the performance of most of the optimizers. A new optimization technique Gradient Centralization is providing the unparalleled result in terms of generalization and execution time. Method: Our paper explores the next factor which is the employment of new optimization technique, Gradient centralization (GC) to our integrated framework (Model with advanced preprocessing technique). Result and conclusion: Integrated Framework of Real ESRGAN and GFPGAN with Gradient centralization provides an optimal solution for deep learning models in terms of Execution time and Loss factor improvement.

Keywords

gradient centralization medical image analysis real enhanced super resolution generative adversarial network generative facial prior GAN

Introduction

Integrated framework of deep learning model with advanced preprocessing technique like Real ESRGAN¹ and GFPGAN² gives significant elevation in terms of efficiency of these models. Efficiency shoots up to 5–10% via pipe lining of these preprocessing technique with base models. On a way to further optimize the deep learning models, usage of optimizer is explored. Optimizer plays an indispensable role in the functioning of CNN. They help in adjusting weights so as to minimize the incurred losses. Many optimizers have been employed in CNN like Gradient descent,^3–8 Stochastic Gradient descent,^9–14 Ada grad,^15–18 Ada delta^19–21 and Adam^22–25 etc. for classification and segmentation of Medical image. Gradient descent iteratively reduces a loss function by moving in the direction opposite to that of steepest ascent. As its uses entire training set for calculation which requires large amount of memory and hence slow down process. Stochastic gradient descent is a variant of gradient descent and update model parameter one by one which leads to high variance and make it computationally expensive. Stochastic gradient descent with momentum is an improved version which takes in account the previous update to fine tune the final update direction and hence leads to more stability but extra hyper parameter is added. So far learning rate has been constant but with the introduction of Ada grad (Adaptive Gradient descent) concept of adaptive learning rate emerges that is it uses different learning rate for each and every neuron and for each and every hidden layer based on different iteration but it too leads to dead neuron problem. RMS-prop^26–30 (Root mean square propagation) is a special version of Ada grad in which learning rate is exponential average of gradient instead of cumulative sum of squared gradient but it suffers from slow learning rate. Ada delta is an extension of Ada grad. Here it, does not set default learning rate and uses the ratio of running average of previous time steps to current gradient. It removes decaying rate problem but it is computationally expensive. Adam (Adaptive moment estimator) optimizer is one of the popular gradient descent optimization algorithm. It computes adaptive learning rate for each parameter and stores both the decaying average of past gradients and also the decaying average of past squared gradient. It has little memory requirement and is computationally efficient.

Aforementioned optimization techniques operate on activation or weight vectors and hence their adoption to pre-trained models pose a great challenge. So a new method, Gradient centralization^31–34 is used which acts on gradient of weight vectors and centralizes the gradient vector to have zero mean. Thus introduction of Gradient centralization accelerate the training process and enhances the generalization performance.

Our research work explores the functioning of GC (Gradient centralization) on our integrated framework of deep learning models. Next Section illustrates the related work, Proposed work describes the achieved result via integrated framework and proposed work which includes further optimization with novel Gradient Centralization optimizer. Experimental result and discussion tabulates the experimental result and comparison of loss curves and Future work and conclusion winds the research paper with Future work and conclusion.

Related work

Significant efforts are made in the direction of improving the efficiency of deep learning models. Application of voting based method to deep learning models helps in strengthening the correct classification of lung disease.³⁵ Preprocessing techniques plays an important role in boosting efficacy of deep learning models.³⁶ Other factor like Optimizer too plays an pertinent role in efficiency improvement of Deep learning algorithms as they decide the rate of convergence of algorithm towards an optimal value. Many Researchers are evolving the usage of new optimizers to explore their functioning in deep learning models. Zhang³⁷ proposes method, normalized direction-preserving Adam (ND-Adam), which enables more precise control of the direction and step size for updating weight vectors and experimentally proves to improve in generalization performance of adam optimizer. Khaire and Dhanalakshmi³⁸ brings out another improvement in momentum of adam optimizer by evaluating the gradient after applying the current velocity and called its as iAdam. It works efficiently for the high-dimensional dataset and converges rapidly. iAdam is simple to execute, computationally effective, well suited for high-dimensional datasets, converge smoothly and operate efectively even for higher levels of learning Usage of appropriate optimizer for particular input dataset greatly affects the efficiency of architecture used for classifying that dataset. Halgamuge et al.³⁹ carried out comparison of six optimizer Ada grad], Ada delta, Rms prop, Adam, Nadam, and SGD (Stochastic gradient descent) and identify the best optimizer as Ada grad which estimate potential fire occurrences in given locations and gives precise predictions with less error and processes real-time data. Adagrad has highest prediction rate 43.93 min and Nd adam and has highest accuracy and lowest loss rate that is 98.86% and 0.03%. Kandel et al.⁴⁰ improves the efficiency of its CNN based classifier by comparing six different first-order stochastic gradient-based optimizers and adaptative based optimizers emerges out as best for classifying histopathology images for low learning rate and produces an AUC of 94% and Ada grad proves to be low performer even on low learning rate. Yaqub et al.⁴¹ perform a comparative analysis of 10 different gradient descent-based optimizers, Adaptive Gradient (Ada grad), Adaptive Delta (Ada Delta), Stochastic Gradient Descent (SGD), Adaptive Momentum (Adam), Cyclic Learning Rate (CLR), Adaptive Max Pooling (Ada max), Root Mean Square Propagation (RMS Prop), Nesterov Adaptive Momentum (Nadam), and Nesterov accelerated gradient (NAG) for CNN for brain tumor classification and segmentation and Adam optimizer helps in attaining the greatest efficiency of 99.2%. Chowdhury et al.⁴² explores the CNN with one, two, three, four and five hidden layers with combination of three optimizers namely,Rms prop, Adam and SGD and finds CNN with four hidden layer using SGD optimizer having greatest testing accuracy of 91%. Bera and Shrivastava⁴³ performs Hyper spectral image (HSI) classification with seven different optimizers SGD, Ada grad, Ada delta, Rms prop, Adam, Ada Max, and Nadam using Deep CNN model and establishes the superiority of Adam optimizer achieving average accuracy of 98% on every HSI datasets. Perin and Picek⁴⁴ illustrates that hyper parameter optimizer plays indispensable role in in deep learning-based side-channel analysis. He experimentally shows that Adam and Rmsprop works well for shorter training phases as they easily overfitand strongly supports Ada grad for longer training phases. Poojary and Pai⁴⁵ used CNN based model Resnet 50 and Inception v3 for kaggle cat versus dog classification and carried out comparative analysis for three optimizer namely Adam, SGD, RMS prop and RMS prop performed extremely well with training accuracy of 99%. Vani and Rao⁴⁶ implemented the usage of seven optimizers namely Stochastic Gradient Descent (SGD), RmsProp, Adam, Ada max, Ada grad, Ada delta, and Nadam are implemented in CNN on Indian Pines Dataset Ada max excels with 99.58% accuracy. Yaqub et al.⁴¹ carries out Brain tumor classification and segmentation using BraTS2015 data set for 10 optimizer namely namely Adaptive Gradient (Ada grad), Adaptive Delta (Ada Delta), Stochastic Gradient Descent (SGD), Adaptive Momentum (Adam), Cyclic Learning Rate (CLR), Adaptive Max Pooling (Ada max), Root Mean Square Propagation (RMS Prop), Nesterov Adaptive Momentum (Nadam), and Nesterov accelerated gradient (NAG) and proves Adam to be efficient with accuracy of 99.2%. Dubey et al.⁴⁷ works out on step size taken for each parameter and proposed an Diffgrad optimization technique which takes larger step size for faster gradient changing parameter and lower step size for lower gradient changing parameter. This method outperforms optimizer technique like SGDM, Ada Grad, Ada Delta, RMS Prop, AMS Grad, and Adam. Taqi et al.⁴⁸ proposes Alzheimer disease (AD) classification based by using four different optimizers Ada grad, ProximalAda grad, Adam, and RMS Prop and Rms prop works with 100% accuracy. Thavasimani and Srinath⁴⁹ experimented with various optimizers such as ADAM, SGD, RMS prop, Ada delta, Ada max, Ada grad, Nadam to detect bot accounts with the help of deep learning model from CRESCI-2017 twitter dataset issued by Indiana University and establishes the highest accuracy of 98.90% with RMS prop. Elangovan and Nath⁵⁰ bring out the role of optimizer in improving the performance of deep neural network for image classification problem and analyzes three standard first-order optimizers like stochastic gradient descent with momentum (SGDM), adaptive moment estimation (Adam), and root mean square propagation (RMS Prop) for detecting glaucoma employing architectures like Alex net,VGG-19 and Resnet 101. Adam optimizer shows better result in this.

Proposed work

Our proposed work is the continuation of our effort to optimize the deep learning models. Imbibing Advanced preprocessing technique like Real ESRGAN^51–59 and GFP Gan^2,60–68 with base deep learning model like NASNet^69–78 and DenseNet 201^79–88 produces an significant results in context of validation accuracy. Results achieved via this pipe lining is tabulated in Table 1 for Real ESRGAN and GFP Gan respectively.

Table 1.

Result of integrated model.

REAL ESRGAN	SKIN ACCURACY		LUNG ACCURACY		RETINA ACCURACY
Model used	Without preprocessing (%)	With preprocessing (%)	Without preprocessing (%)	With preprocessing (%)	Without preprocessing (%)	With preprocessing (%)
DenseNet 201	28	72	85	89	74	88
NASNet	35	63	77	78	73	89

GFPGAN	SKIN		LUNG		RETINA
Model used	Without preprocessing (%)	With preprocessing (%)	Without preprocessing (%)	With preprocessing (%)	Without preprocessing (%)	With preprocessing (%)
DenseNet 201	28	62	85	86	74	89
NASNet	35	73	77	78	73	84

For skin dataset massive improvement is achieved for validation accuracy. Retina dataset also responds well for integrated framework. Minimal enhancement is observed for lung dataset. But overall positive trends are observed using images preprocessed with Real ESRGAN and GFPGAN as input to our base models.

Our next step to further optimize the integrated deep learning models with the use of novel optimizer Gradient centralization technique.

Gradient centralization

Gradient centralization (GC) can enhance DNN(Deep neural network) ultimate generalisation performance while also accelerating the training process. As indicated in Figure 1 GC works by by centralizing the gradient vectors to have a mean of zero and thus directly manipulates gradients. Gradient Centralization further enhances the loss function and its gradient Lipschitzness, making the training process more effective and stable. Batch Normalization and Weight standardization technique also reduces the Lipschitzness of loss function but they operate on activation and weight vectors and were not able to adapt to pre trained models but Gradient Centralization takes in account the gradients vectors to achieve improvement. Z-score standardization can also be used to normalize the gradient but normalising gradient does not increase training’s stability. Instead, calculating the gradient vectors’ means and then centralizing the gradients so that their means are zero effectively helps in achieving the desired result. This is the underlying principle of Gradient centralization method. It can be easily embedded into gradient based current optimization algorithm like Adam, SGDM with one line of code and helps in increasing training process, improves generalization capability and better tune it to pre trained models.

Figure 1.

Illustration of GC operation on matrix/tensor of weights in fully connected layer (left) and convolutional (right). GC computes the column/slice mean of gradient matrix/tensor and centralizes each column/slice to have zero mean.

Formula of GC

Let Gradient of a weight vector obtained through back propagation for a Fully connected layer or Convolutional layer be $\nabla w_{i} L (i = 1, 2, \dots, N)$ and GC operator be denoted by Φ_GC then formulation of Gradient centralization is given by:

Φ_{GC} (\nabla_{wi} L) = \nabla_{wi} L - µ \nabla_{wi} L

(1)where

µ \nabla_{wi} L = 1 / M (\sum_{j = 1}^{M} \nabla W_{i, j} L)

Mean value of each column of gradient matrix is subtracted from its each column value. In this way each gradient of loss function w.r.t to weight vector is transform so that its mean becomes zero. computation of GC is quite simple and efficient. Matrix formulation of equation (1) is also given by:

Φ_{GC} (\nabla_{W} L) = P \nabla_{W} L

(2)where

P = I - {ee}^{T}

and P is known as cool operator and I is the identity matrix of

M * M

order and and e = 1/√M

[\begin{array}{l} 1 & 0 & 0 & \dots & \dots & 0 \\ 0 & 1 & 0 & \dots & \dots & 0 \\ 0 & 0 & 1 & 0 & \dots & 0 \\ 0 & 0 & 0 & \dots & \dots & 1 \end{array}]

Experimental result and discussion

In our proposed research work, integrated deep model (model pipeline with advanced preprocessing technique) is further enhanced by employing an new optimizer Gradient Centralization. GC can be easily embedded into existing optimization algorithm like SGDM, Adam and RMS prop by adding a single line of code. We have embedded GC with Rms prop. DenseNet 201 and NASNet deep models are used as base models. Three datasets have been used to carry out the experiment: Lung, Retina and skin. Inculcating GC in our integrated model brings out improvement in three parameters of model which are Training accuracy, Training Loss and Execution time. Result of our experiment for GFPGAN integrated models are tabulated in Tables 2–4 for all three datasets.

Table 2.

Result for optimized GFPGAN for lung dataset.

Model used	Accuracy		Loss		Execution time (sec.)
Model used	WITH out GCTF	With GCTF	Without GCTF	With GCTF	Without GCTF	With GCTF
DenseNet201	91.875	94.0625	0.214071	0.05539	1393.98	647.126
NASNet	86.9565	90.625	0.37491	0.11726	1042.77	638.959

Table 3.

Result for optimized GFPGAN for retina dataset.

Model used	Training accuracy-validation accuracy		Training loss-validation loss		Execution time (sec.)
Model used	Without GCTF	With GCTF	Without GCTF	With GCTF	Without GCTF	With GCTF
DenseNet201	92.0792	99.0099	0.227322	0.0357533	1199.61	812.791
NASNet	90.625	95.3125	0.3010	0.101164	1654.65	842.769

Table 4.

Result for optimized GFPGAN forskin dataset.

Model used	Training accuracy-validation accuracy		Training loss-validation loss		Execution time (sec.)
Model used	Without GCTF	With GCTF	Without GCTF	With GCTF	Without GCTF	With GCTF
DenseNet201	98.3031	100	0.04873	0.009683	776.049	505.114
NASNet	96.6102	99.661	0.09873	0.0304052	738.592	511.332

Enhancement in Accuracy has been observed for all three datasets but retina dataset responds remarkably better as compared to other datasets. Training loss has also been reduced which appends to our positive result. Improvement in execution time is the major break through achieved via this new optimizer algorithm. Significant reduction has been observed as clear from table. Our result supports the underlying principle of GC which states of accelerating the training process.

Experiments for another pre-preprocessing technique Real ESRGAN integrated deep learning model is carried out which also employed a new optimized method of Gradient Centralization and the results are tabulated in Tables 5–7 respectively for three datasets: Skin, Retina and Lung.

Table 5.

Result of optimized real ESRGAN for skin dataset.

Model used	Accuracy		Loss		Exection time (sec.)
Model used	Without GCTF	With GCTF	Without GCTF	With GCTF	Without GCTF	With GCTF
DenseNet201	99.3548	100	0.027379	0.00645161	1613.32	1112.79
NASNet	98.38	99.3548	0.0562655	0.026688	1639.68	1113.87

Table 6.

Result of optimized real ESRGAN for retina dataset.

Model used	Accuracy		Loss		exection time (sec.)
Model used	Without GCTF	With GCTF	Without GCTF	With GCTF	Without GCTF	With GCTF
DenseNet201	97.11	98.75	0.0714245	0.0235976	1427.4	1049.87
NASNet	94.23	95	0.17624	0.10362	1877.56	1015.65

Table 7.

Result of optimized real ESRGAN for lung dataset.

Model used	Accuracy		Loss		Exection time (sec.)
Model used	Without GCTF	With GCTF	Without GCTF	With GCTF	Without GCTF	With GCTF
DenseNet201	95.315	93.4375	0.145507	0.0640958	1849.38	848.866
NASNet	96.3021	97.704	0.25616	0.08136	2483.02	819.201

Upward trends are observed for integrated DenseNet 201 in terms of accuracy for skin and retina dataset but slight down trend is noticed for lung dataset which shows an deviation from our expectation. NASNet exhibit an similar trend for all three datasets. Training loss comes out with positive result and significant drop is observed for three datasets. Execution time too shows substantial drop which further strengthen the concept of imbibing this new optimized technique in our integrated model.

Evaluation of loss curve on imbibing Gradient Centralization technique for Dense net model pipe lined with Real ESRGAN and GfpGan for all three datasets: Retina, Lung and Skin brings out an interesting predictions as indicated by Figures 2–4 respectively as shown below:

Figure 2.

(a)Loss curve for real ESRGAN pipe lined DenseNet model for retina dataset (b) loss curve for GFPGAN pipe lined DenseNet model for retina dataset.

Figure 3.

(a)Loss curve for real ESRGAN pipe lined DenseNet model for lung dataset (b) loss curve for GFPGAN pipe lined DenseNet model for lung dataset.

Figure 4.

(a) Loss curve for real ESRGAN pipe lined DenseNet model for skin dataset (b) loss curve for GFPGAN pipe lined DenseNet model for skin dataset.

Experiments shows that DenseNet model when pipe lined with Real ESRGAN preprocessing technique for skin dataset gives optimum results for losses incurred during training which is 0.006451 and is minimum as compared with other combinations of dataset and technique.

Loss curves for combination of Gradient Centralization with NASNet model pipelined with Real ESRGAN and GFPGAN technique are shown in Figures 5–7 below for all three datasets.

Figure 5.

(a) Loss curve for real ESRGAN pipe lined NASNet model for skin dataset (b) loss curve for GFPGAN pipe lined NASNet t model for skin dataset.

Figure 6.

(a) Loss curve for real ESRGAN pipe lined NASNet model for lung dataset (b) loss curve for GFPGAN pipe lined NASNet t model for lung dataset.

Figure 7.

(a) Loss curve for real ESRGAN pipe lined NASNet model for retina dataset (b) loss curve for GFPGAN pipe lined NASNet t model for retina dataset.

Smooth loss curves are observed for skin dataset when NASNet pipe lined with Real ESRGAN is employed with GC is used. Though downward trend is shown by all datasets with both the preprocessing techniques but integrated Real ESRGAN with GC shows better result than GFPGAN.

Future work and conclusions

Our research work explores the Gradient Centralization optimization on integrated deep architecture using NASNet and DenseNet as base models. We get motivating results for parameters like Training loss and execution time. Reduction in execution time is a pertinent outcome achieved in our research work. Though Training accuracy was slight down for DenseNet in lung database but others parameters shows, this inculcation of new optimization technique, a significant step towards optimizing the performance of deep learning architecture for Medical Image analysis. In future, various other permutations can be explored like using other deep learning model like Resnet, Mobile net etc.We have embedded Gradient Centralization with Rms prop optimizer to carry out our experiment. Other optimizer like Adam, SGDM can also be used to embed Gradient Centralization. We have used Real ESRGAN and GFPGAN preprocessing integrated model and taken Skin, Lung and retina datasets. Analysis can also be carried out by advanced preprocessing technique like Swin transformers which basically require more GPU. Wide Spectrum of Medical dataset for experimentation can also be increased to evaluate effect of this new integrated approach.

Footnotes

Acknowledgements

We want to thank Graphic era hill university for their efficient support regarding this study.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Vertika Agarwal

References

Wang

Xie

Dong

, et al. Real-esrgan: training real-world blind super-resolution with pure synthetic data. In: Proceedings of the IEEE/CVF international conference on computer vision, Montreal, BC, 11–17 October 2021, pp. 1905–1914.

Wang

Zhang

, et al. Towards real-world blind face restoration with generative facial prior. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Nashville, TN, 20–25 June 2021, pp. 9168–9178.

Bisong

. Optimization for machine learning: gradient descent. Building machine learning and deep learning models on google cloud platform: a comprehensive guide for beginners. 2019; 203–207.

Ruder

. An overview of gradient descent optimization algorithms. 2016. arXiv preprint arXiv:1609.04747.

Chandra

Xie

Ragan-Kelley

, et al. Gradient descent: the ultimate optimizer. Adv Neural Inf Process Syst 2022; 35: 8214–8225.

Andrychowicz

Denil

Gomez

, et al. Learning to learn by gradient descent by gradient descent. Adv Neural Inf Process Syst. 2016; 29.

Kaoudi

Quiané-Ruiz

Thirumuruganathan

, et al. A cost-based optimizer for gradient descent optimization. In: Proceedings of the 2017 ACM international conference on management of data, Chicago, IL, 9 May 2017, 977–992.

Jiang

. Learning gradient descent: better generalization and longer horizons. In: International conference on machine learning, Sydney, NSW, 17 July 2017. PMLR, pp. 2247–2255.

Gitman

Lang

Zhang

, et al. Understanding the role of momentum in stochastic gradient methods. Adv Neural Inf Process Syst. 2019; 32.

10.

Yang

. Modified convolutional neural network based on dropout and the stochastic gradient descent optimizer. Algorithms 2018; 11(3): 28.

11.

Bottou

. Stochastic gradient descent tricks. In: Neural networks: tricks of the trade. 2nd ed. Berlin, Germany: Springer Berlin Heidelberg, 2012, pp. 421–436.

12.

Klein

Pluim

Staring

, et al. Adaptive stochastic gradient descent optimisation for image registration. Int J Comput Vis 2009; 81: 227–239.

13.

Bottou

. Large-scale machine learning with stochastic gradient descent. In: Proceedings of COMPSTAT’2010: 19th international conference on computational statistics, Paris France, 22–27 August 2010, pp. 177–186. Physica-Verlag.

14.

Cui

Zhang

Tüske

, et al. Evolutionary stochastic gradient descent for optimization of deep neural networks. Adv Neural Inf Process Syst 2018; 31.

15.

Zhang

Lei

Zhao

. An improved adagrad gradient descent optimization algorithm. In: 2018 Chinese automation congress (CAC), Xi’an, China, 30 November 2018, pp. 2359–2362. IEEE.

16.

Desai

. Comparative analysis of optimizers in deep neural networks. International J of Innovative Sci and Res Techn 2020; 5(10): 959–962.

17.

Anil

Gupta

Koren

, et al. Memory efficient adaptive optimization. Adv Neural Inf Process Syst. 2019; 32.

18.

Lydia

Francis

. Adagrad—an optimizer for stochastic gradient descent. Int J Inf Comput Sci 2019; 6(5): 566–568.

19.

Gill

Sharma

Anand

, et al. Brain tumor detection using VGG19 model on adadelta and SGD optimizer. In: 2022 6th international conference on electronics, communication and aerospace technology, Coimbatore, India, 1–3 December 2022, pp. 1407–1412. IEEE.

20.

Zaheer

Shaziya

. A study of the optimization algorithms in deep learning. In: 2019 third international conference on inventive systems and control (ICISC), Coimbatore, India, 10–11 January 2019, pp. 536–539. IEEE.

21.

Uppal

Gupta

Juneja

, et al. Enhancing accuracy in brain stroke detection: multi-layer perceptron with Adadelta, RMSProp and AdaMax optimizers. Front Bioeng Biotechnol 2023; 11: 1257591.

22.

Bock

Goppold

Weiß

. An improvement of the convergence proof of the ADAM-optimizer. 2018. arXiv preprint arXiv:1804.10587.

23.

Khan

Cao

, et al. BAS-ADAM: an ADAM based approach to improve the performance of beetle antennae search optimizer. IEEE/CAA J Autom Sinica 2020; 7(2): 461–471.

24.

Mehta

Paunwala

Vaidya

. CNN based traffic sign classification using Adam optimizer. In: 2019 international conference on intelligent computing and control systems (ICCS), Madurai, India, 15–17 May 2019, pp. 1293–1298. IEEE.

25.

Melinte

Vladareanu

. Facial expressions recognition for human–robot interaction using deep convolutional neural networks with rectified adam optimizer. Sensors 2020; 20(8): 2393.

26.

Vijendra Babu

Karthikeyan

Kumar

, et al. Performance analysis of cost and accuracy for whale swarm and RMSprop optimizer. IOP Conf Ser Mater Sci Eng 2020; 993(1): 012080.

27.

Zou

Shen

Jie

, et al. A sufficient condition for convergences of adam and rmsprop. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Long Beach, CA, 15–20 June 2019, pp. 11127–11135.

28.

Gill

Anand

Gupta

. Categorization of chocolates using robust sequential model on RMS prop optimizer. In: 2023 world conference on communication & computing (WCONF), Raipur, India, 14–16 July 2023, pp. 1–4. IEEE.

29.

Mukkamala

Hein

. Variants of rmsprop and adagrad with logarithmic regret bounds. In: International conference on machine learning, Sydney, NSW, 17 July 2017, pp. 2545–2553. PMLR.

30.

Tiwari

Yadav

Misra

. Performance evaluation of optimizers in the classification of marble surface quality using CNN. In: Proceedings of third doctoral symposium on computational intelligence: DoSCI 2022, Singapore, 10 November 2022, pp. 181–191. Springer Nature Singapore.

31.

Yong

Huang

Hua

, et al. Gradient centralization: a new optimization technique for deep neural networks. In: Computer vision–ECCV 2020: 16th European conference, Glasgow, UK, 23–28 August 2020, pp. 635–652. Springer International Publishing.

32.

Fuhl

Kasneci

. Weight and gradient centralization in deep neural networks. 2020. arXiv preprint arXiv:2010.00866.

33.

Fuhl

Kasneci

. Weight and gradient centralization in deep neural networks. In: Artificial neural networks and machine learning–ICANN 2021: 30th international conference on artificial neural networks, Bratislava, Slovakia, 14–17 September 2021, pp. 227–239. Springer International Publishing.

34.

Zang

Foo

Bernadin

, et al. Facial emotion recognition using asymmetric pyramidal networks with gradient centralization. IEEE Access 2021; 9: 64487–64498.

35.

Agarwal

Lohani

Bist

, et al. Application of voting based approach on deep learning algorithm for lung disease classification. In: 2022 international conference on science and technology (ICOSTECH), Batam City, Indonesia, 3 February 2022, pp. 01–07. IEEE.

36.

Pal

Sudeep

. Preprocessing for image classification by convolutional neural networks. In: 2016 IEEE international conference on recent trends in electronics, information & communication technology (RTEICT), Bangalore, India, 20–21 May 2016, pp. 1778–1781. IEEE.

37.

Zhang

. Improved adam optimizer for deep neural networks. In: 2018 IEEE/ACM 26th international symposium on quality of service (IWQoS), Banff, AB, 4–6 June 2018, pp. 1–2. IEEE.

38.

Khaire

Dhanalakshmi

. High-dimensional microarray dataset classification using an improved adam optimizer (iAdam). J Ambient Intell Hum Comput 2020; 11(11): 5187–5204.

39.

Halgamuge

Daminda

Nirmalathas

. Best optimizer selection for predicting bushfire occurrences using deep learning. Nat Hazards 2020; 103(1): 845–860.

40.

Kandel

Castelli

Popovič

. Comparative study of first order optimizers for image classification using convolutional neural networks on histopathology images. J Imaging 2020; 6(9): 92.

41.

Yaqub

Feng

Zia

, et al. State-of-the-art CNN optimizer for brain tumor segmentation in magnetic resonance images. Brain Sci 2020; 10(7): 427.

42.

Chowdhury

Dasgupta

Nanda

. Analysis of various optimizer on CNN model in the application of pneumonia detection. In: 2021 3rd international conference on signal processing and communication (ICPSC), Coimbatore, India, 13–14 May 2021, pp. 417–421. IEEE.

43.

Bera

Shrivastava

. Analysis of various optimizers on deep convolutional neural network model in the application of hyperspectral remote sensing image classification. Int J Rem Sens 2020; 41(7): 2664–2683.

44.

Perin

Picek

. On the influence of optimizers in deep learning-based side-channel analysis. In: Selected areas in cryptography: 27th international conference, Halifax, NS, Canada, Virtual Event, 21–23 October 2020. Springer International Publishing, pp. 615–636. Revised Selected Papers 27 2021.

45.

Poojary

Pai

. Comparative study of model optimization techniques in fine-tuned CNN models. In: 2019 international conference on electrical and computing technologies and applications (ICECTA), Ras Al Khaimah, UAE, 19 November 2019, pp. 1–4. IEEE.

46.

Vani

Rao

. An experimental approach towards the performance assessment of various optimizers on convolutional neural network. In: 2019 3rd international conference on trends in electronics and informatics (ICOEI), 23 April 2019, pp. 331–336. IEEE.

47.

Dubey

Chakraborty

Roy

, et al. DiffGrad: an optimization method for convolutional neural networks. IEEE Transact Neural Networks Learn Syst 2020; 31(11): 4500–4511.

48.

Taqi

Awad

Al-Azzo

, et al. The impact of multi-optimizers and data augmentation on tensorflow convolutional neural network performance. In: 2018 IEEE conference on multimedia information processing and retrieval (MIPR), Miami, FL, 10–12 Apr 2018, pp. 140–145. IEEE.

49.

Thavasimani

Srinath

. Deep learning techniques: a case study on comparative analysis of various optimizers to detect bots from CRESCI-2017 dataset. International J of Adv Sci and Techn 2020; 29(04): 10040–10053.

50.

Elangovan

Nath

. Performance analysis of optimizers for glaucoma diagnosis from fundus images using transfer learning. In: Machine learning, deep learning and computational intelligence for wireless communication, Singapore, 2021, pp. 507–518. Springer.

51.

Wang

, et al. Esrgan: enhanced super-resolution generative adversarial networks. In: Proceedings of the European conference on computer vision (ECCV) workshops, Munich, Germany, 8–14 September 2018.

52.

Liu

Wang

, et al. RTSRGAN: real-time super-resolution generative adversarial networks. In: 2019 seventh international conference on advanced cloud and big data (CBD), Suzhou, China, 21 September 2019, pp. 321–326. IEEE.

53.

Zhu

Lei

Qin

, et al. Improved image super-resolution based on real-ESRGAN. IEEE Access 2023; 11: 45334–45348.

54.

Fritsche

Timofte

. Frequency separation for real-world super-resolution. In: 2019 IEEE/CVF international conference on computer vision workshop (ICCVW), Seoul, Korea, 27 October 2019, pp. 3599–3608. IEEE.

55.

Sreelakshmy

Kovoor

. Generative inpainting of high-resolution images: redefined with real-ESRGAN. Int J Artif Intell Tool 2022; 31(05): 2250035.

56.

Castillo

Escobar

Pérez

, et al. Generalized real-world super-resolution through adversarial robustness. In: Proceedings of the IEEE/CVF international conference on computer vision, Montreal, BC, 11–17 October 2021, pp. 1855–1865.

57.

Toan

. Super-resolution method for reconstructing street images from surveillance system based on real-ESRGAN. In: Proceedings of student computing research symposium (SCORES’22), New York, NY, 6 October 2022, 4.

58.

Çetin

ŞB

. Real-ESRGAN: a deep learning approach for general image restoration and its application to aerial images. Adv Rem Sensing 2023; 3(2): 90–99.

59.

Rashid

Shakibapour

Ebrahimi

. MR image resolution enhancement using real-ESRGAN.

60.

Kumar

Vatsa

. Influence of GFP GAN on melanoma classification. In: 2022 IEEE integrated STEM education conference (ISEC), Princeton, NJ, 26 March 2022, pp. 334–339. IEEE.

61.

Kumari

Dubey

Mishra

. A cascaded method for real face image restoration using GFP-GAN. International J of Innov Res in Techn and Manag 2022; 6(3): 9–105.

62.

Hua

. Person re-identification system integrating GFP-GAN and Yolov4 models. In: 2022 8th annual international conference on network and information systems for computers (ICNISC), Hangzhou, China, 16–19 September 2022, pp. 258–264. IEEE.

63.

Shi

Chen

, et al. GCFSR: a generative and controllable face super resolution method without facial and gan priors. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, New Orleans, LA, 2022, pp. 1889–1898.

64.

Exum

Facial image deblurring & recognition using GFP-GAN. In: Undergraduate research and creative inquiry

symposia, 25–28 April 2013, 319.

65.

Messaoud

Douaa

Souhila

, et al. Boosting cross-quality face verification using blind face restoration. In: 2023 international conference of the biometrics special interest group (BIOSIG), Darmstadt, Germany, 20–22 September 2023, pp. 1–6. IEEE.

66.

Bengherabi

Laib

Lasnami

, et al. Boosting cross-quality face verification using blind face restoration. 2023. arXiv preprint arXiv:2308.07967.

67.

Yuchao

, et al. “Vqfr: Blind face restoration with vector-quantized dictionary and parallel decoder.” European Conference on Computer Vision. Cham Switzerland: Springer Nature, 2022.

68.

Conmy

Mukherjee

Schönlieb

. StyleGAN-induced data-driven regularization for inverse problems. In: ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP), Singapore, 23 May 2022, pp. 3788–3792. IEEE.

69.

Martínez

Jacinto

. Performance evaluation of the NASNet convolutional network in the automatic identification of COVID-19. Int J Adv Sci Eng Inf Technol 2020; 10(2): 662–667.

70.

Bharati

Podder

Mondal

, et al. Optimized NASNet for diagnosis of COVID-19 from lung CT images. In: International conference on intelligent systems design and applications. Cham, Switzerland: Springer International Publishing, 12 December 2020, pp. 647–656.

71.

Falconí

Pérez

Aguilar

. Transfer learning in breast mammogram abnormalities classification with mobilenet and nasnet. In: 2019 international conference on systems, signals and image processing (IWSSIP), Osijek, Croatia, 5 June 2019, pp. 109–114. IEEE.

72.

Radhika

Devika

Aswathi

, et al. Performance analysis of NASNet on unconstrained ear recognition. In: Nature inspired computing for data science. Berlin, Germany: Springer Nature, 2020, 57–82.

73.

Saxen

Werner

Handrich

, et al. Face attribute detection with mobilenetv2 and nasnet-mobile. In: In: 2019 11th international symposium on image and signal processing and analysis (ISPA), Dubrovnik, Croatia, 23–25 September 2019, pp. 176–180. IEEE.

74.

Adedoja

Owolawi

Mapayi

. Deep learning based on nasnet for plant disease recognition using leave images. In: 2019 international conference on advances in big data, computing and data communication systems (icABCD), Winterton, South Africa, 5 August 2019, pp. 1–5. IEEE.

75.

Mehmood

Alshammari

Alanazi

, et al. Improved colorization and classification of intracranial tumor expanse in MRI images via hybrid scheme of Pix2Pix-cGANs and NASNet-large. J of King Saud Univ-Comp and Inform Sci 2022; 34(7): 4358–4374.

76.

Qin

Wang

. Nasnet: a neuron attention stage-by-stage net for single image deraining. 2019. arXiv preprint arXiv:1912.03151.

77.

Zhang

. Lung segmentation with NASNet-large-decoder net. 2023. arXiv preprint arXiv:2303.10315.

78.

Adedoja

Owolawi

Mapayi

, et al. Intelligent mobile plant disease diagnostic system using NASNet-mobile deep learning. IAENG Int J Comput Sci 2022; 49(1): 216–231.

79.

Wang

Zhang

. DenseNet-201-based deep neural network with composite learning factor and precomputation for multiple sclerosis classification. ACM Trans Multimed Comput Commun Appl 2020; 16(2s): 1–19.

80.

Han

Chen

, et al. A generic intelligent tomato classification system for practical applications using DenseNet-201 with transfer learning. Sci Rep 2021; 11(1): 15824.

81.

Sanghvi

Patel

Agarwal

, et al. A deep learning approach for classification of COVID and pneumonia using DenseNet-201. Int J Imag Syst Technol 2023; 33(1): 18–38.

82.

Salim

Saeed

Basurra

, et al. DenseNet-201 and xception pre-trained deep learning models for fruit recognition. Electronics 2023; 12(14): 3132.

83.

Zeng

Liu

, et al. Utilization of DenseNet201 for diagnosis of breast abnormality. Mach Vis Appl 2019; 30: 1135–1144.

84.

Jaiswal

Gianchandani

Singh

, et al. Classification of the COVID-19 infected patients using DenseNet201 based deep transfer learning. J Biomol Struct Dyn 2021; 39(15): 5682–5689.

85.

Zhao

Shuai

, et al. Dermoscopy image classification based on StyleGAN and DenseNet201. IEEE Access 2021; 9: 8659–8679.

86.

Adhinata

Rakhmadani

Wibowo

, et al. A deep learning using DenseNet201 to detect masked or non-masked face. Jurnal Informatika 2021; 9(1): 115–121.

87.

Khare

Havaldar

. Predicting the anterior slippage of vertebral lumbar spine using Densenet-201. Biomed Signal Process Control 2023; 86: 105115.

88.

Ovreiu

Paraschiv

Ovreiu

. Deep learning & digital fundus images: glaucoma detection using DenseNet. In: 2021 13th international conference on electronics, computers and artificial intelligence (ECAI), Pitesti, Romania, 1–3 July 2021, pp. 1–4. IEEE.

A novel deep learning technique for medical image analysis using improved optimizer

Abstract

Keywords

Introduction

Related work

Proposed work

Gradient centralization

Formula of GC

Experimental result and discussion

Future work and conclusions

Footnotes

Acknowledgements

Declaration of conflicting interests

Funding

ORCID iD

References