The activation functions are essential in neural networks because they are the ones responsible for gaining the abstract characteristics of the input through the use of nonlinear transformations. The paper presents a new activation function, which is referred to as Hybrid Discontinuous Activation Function (HDAF). It is continuously differentiable except at zero, unbounded above, bounded below, and non-monotonic. Our results demonstrate that HDAF outperforms ReLU on a number of challenging datasets and neural network models. In the beginning of the experiment, neural networks are trained and classified using benchmark data like Breast Cancer Wisconsin (Diagnostic) and Iris Flower datasets and HDAF achieved 92.98% and 95.56% accuracy, respectively. Secondly, experiments were conducted on VGG16 over the MNIST and CIFAR10 datasets. The HDAF obtained 99.21%, and 84.95% accuracy respectively. Statistical feature measurements demonstrate that HDAF has the best mean accuracy, lowest standard deviation, lowest Root Mean squared error, lowest variance, and lowest Mean squared error. The study indicated that HDAF has faster convergence compared to ReLU, making it a valuable factor in deep learning. The findings from the experiments indicate that HDAF can be a promising for ReLU, leading to better performance in neural network models.
LiHPanYZhaoJ, et al.Skin disease diagnosis with deep learning: a review. Neurocomputing2021; 464: 364–393.
2.
DubeySR. A decade survey of content based image retrieval using deep learning. IEEE Trans Circuits Syst Video Technol2021; 32: 2687–2704.
3.
ShaoFChenLShaoJ, et al.Deep learning for weakly-supervised object detection and localization: a survey. Neurocomputing2022; 496: 192–207.
4.
XiaXPanXLiN, et al.Gan-based anomaly detection: a review. Neurocomputing2022; 493: 497–535.
5.
Abdel-ZaherAMEldeibAM. Breast cancer classification using deep belief networks. Expert Syst Appl2016; 46: 139–144.
6.
RautelaKKumarDKumarV. A systematic review on breast cancer detection using deep learning techniques. Archiv Comput Methods Eng2022; 29: 4599–4629.
7.
GuoYFengFHaoX, et al.Jac-net: Joint learning with adaptive exploration and concise attention for unsupervised domain adaptive person re-identification. Neurocomputing2022; 483: 262–274.
8.
IslamMMKarrayFAlhajjR, et al.A review on deep learning techniques for the diagnosis of novel coronavirus (covid-19). IEEE Access2021; 9: 30551–30572.
9.
ZhangRWangLChengS, et al.MLP-based classification of COVID-19 and skin diseases. Expert Syst Appl2023; 228: 120389.
10.
BarzekarHYuZ. C-net: a reliable convolutional neural network for biomedical image classification. Expert Syst Appl2022; 187: 116003.
11.
KaliyarRK. Fake news detection using a deep neural network. In: 2018 4th International conference on computing communication and automation (ICCCA), 2018, pp.1–7.
12.
UnnikrishnanASowmyaVSomanK. Deep learning architectures for land cover classification using red and near-infrared satellite images. Multimed Tools Appl2019; 78: 18379–18394.
13.
KhalilRAJonesEBabarMI, et al.Speech emotion recognition using deep learning techniques: a review. IEEE Access2019; 7: 117327.
14.
MoYWuYYangX, et al.Review the state-of-the-art technologies of semantic segmentation based on deep learning. Neurocomputing2022; 493: 626–646.
15.
AinQTAliMRiazA, et al.Sentiment analysis using deep learning techniques: a review. Int J Adv Comput Sci Appl2017; 8 .
16.
NagarajuMChawlaP. Systematic review of deep learning techniques in plant disease detection. Int J Syst Assur Eng Manage2020; 11: 547–560.
17.
JiangWLuoJ. Graph neural network for traffic forecasting: a survey. Expert Syst Appl2022; 207: 117921.
18.
RosenblattF. The perceptron: a probabilistic model for information storage and organization in the brain. Psychol Rev1958; 65: 386.
19.
HochreiterSSchmidhuberJ. Long short-term memory. Neural Comput1997; 9: 1735–1780.
20.
BourlardHKampY. Auto-association by multilayer perceptrons and singular value decomposition. Biol Cybern1988; 59: 291–294.
21.
ElmanJL. Finding structure in time. Cogn Sci1990; 14: 179–211.
22.
BroomheadDSLoweD. Radial basis functions, multi-variable functional interpolation and adaptive networks. Technical report, Royal Signals and Radar Establishment Malvern (United Kingdom), 1988.
23.
LeCunYBottouLBengioY, et al.Gradient-based learning applied to document recognition. Proc IEEE1998; 86: 2278–2324.
24.
BengioYLamblinPPopoviciD, et al.Greedy layer-wise training of deep networks. Adv Neural Inf Process Syst2006; 19.
25.
Goodfellow IanJJeanP-AMehdiM, et al.Generative adversarial nets. In: Proceedings of the 27th international conference on neural information processing systems, Vol. 2, 2014, pp.2672–2680.
26.
DubeySRChakrabortySRoySK, et al.Diffgrad: an optimization method for convolutional neural networks. IEEE Trans Neural Netw Learn Syst2019; 31: 4500–4511.
27.
BashaSSDubeySRPulabaigariV, et al.Impact of fully connected layers on performance of convolutional neural networks for image classification. Neurocomputing2020; 378: 112–119.
28.
LiuJLiuYZhangQ. A weight initialization method based on neural network with asymmetric activation function. Neurocomputing2022; 483: 171–182.
29.
XuQZhangMGuZ, et al.Overfitting remedy by sparsifying regularization on fully-connected layers of CNNS. Neurocomputing2019; 328: 69–74.
30.
SrivastavaYMuraliVDubeySR. A performance evaluation of loss functions for deep face recognition. In: National conference on computer vision, pattern recognition, image processing, and graphics, 2019, pp.322–332. Springer.
31.
DuchWJankowskiN. Survey of neural transfer functions. Neural Comput Surv1999; 2: 163–212.
32.
NairVHintonGE. Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th international conference on international conference on machine learning, ICML’10, 2010, pp.807–814, Madison, WI, USA. Omnipress.
33.
GlorotXBordesABengioY. Deep sparse rectifier neural networks. In: Proceedings of the fourteenth international conference on artificial intelligence and statistics, 2011, pp.315–323. JMLR Workshop and Conference Proceedings.
34.
ClevertD-AUnterthinerTHochreiterS. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.072892015.
35.
MaasALHannunAYNgAY, et al.Rectifier nonlinearities improve neural network acoustic models. In: Proc. icml, Vol 30, 2013, p.3. Citeseer.
36.
HeKZhangXRenS, et al.Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision, 2015, pp.1026–1034.
37.
KlambauerGUnterthinerTMayrA, et al.Self-normalizing neural networks. Adv Neural Inf Process Syst2017; 30.
38.
XuBWangNChenT, et al.Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853, 2015.
39.
RawatURaiC. Advancing brain tumor detection with neurofusion: an innovative cnn-lstm model featuring a novel activation function. Serv Orient Comput Appl2024; 1–19.
40.
KalimHChugASinghAP. Modswish: a new activation function for neural network. Evol Intell2024b; 17: 2637–2647.
41.
KalimHChugASinghAP. Enhancement of neural network performance with the use of two novel activation functions: modexp and modexpm. Opt Memory Neural Netw2024a; 33: 286–301.
RamachandranPZophBLeQV. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
44.
DugasCBengioYBélisleF, et al.Incorporating second-order functional knowledge for better option pricing. Adv Neural Inf Process Syst2000; 13.
45.
ElliottDL. A better activation function for artificial neural networks. University of Maryland. Systems Research Center, 1993.
46.
Adriana MercioniMHolbanS. Telu: a new activation function for deep learning. In: 2020 International symposium on electronics and telecommunications (ISETC), 2020, pp. 1–4.
47.
VagerwalA. Deeper learning with colu activation. arXiv preprint arXiv:2112.12078, 2021.
48.
Mithra NoelMTrivediADuttaP, et al.Growing cosine unit: a novel oscillatory activation function that can speedup training and reduce parameters in convolutional neural networks. arXiv e-prints, 2021, pages arXiv–2108.
49.
LiuXDiX. Tanhexp: a smooth activation function with high convergence speed for lightweight neural networks. IET Comput Vis2021; 15: 136–150.
50.
WangXRenHWangA. SMISH: a novel activation function for deep learning methods. Electronics2022; 11: 540.
51.
NagSBhattacharyyaMMukherjeeA, et al.Serf: towards better training of deep neural networks using log-softplus error activation function. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2023, pp.5324–5333.
