Sage Journals: Discover world-class research

Abstract

Purpose

Automatic facial expression recognition systems have advanced significantly in recent years. However, recognizing facial expressions accurately from images remains challenging due to variations in facial characteristics and subtle differences between emotions. Identifying identical emotions may exhibit distinct facial characteristics due to inter-person variability, complicating emotion recognition models. Additionally, discerning between two different facial emotions with minimal variation presents a significant challenge. Despite these challenges, convolutional neural networks (CNNs) have made remarkable progress in advancing emotion recognition.

Method

Our proposed method aims to address these challenges by detecting facial emotions in two phases. Firstly, we utilize mL-mB fused (One-over-another) Local Binary Pattern (LBP) images with global and local contrast enhancement, along with fusion of global with local contrast enhanced images using Bernoulli's distribution and Shannon entropy. Subsequently, we employ a Customized Convolution Neural Network model (CCNN) to train and recognize facial expressions.

Results

Experimental findings indicate that our CCNN model achieves significantly improved accuracy when trained using the fusion of global with local contrast enhanced images using Bernoulli's distribution. Notably, the accuracy rates for the CK+, JAFFE, KDEF, FER 2013, and RAF-DB datasets using fused images with global and local contrast enhancement using Bernoulli's distribution are 99.6%, 91.8%, 90.2%, 76.2%, and 49.5%, respectively. These results underscore the effectiveness of our proposed method in facial expression recognition.

Keywords

engineering simulation mathematical modeling and fuzzy logic humanComputer interfaces and intelligent systems fuzzy decision making and decision support systems Index termsBernoulli's distribution image level fusion textural pattern Shannon entropy facial expression recognition

Get full access to this article

View all access options for this article.

References

Agrawal

Mittal

(2020). Using CNN for facial expression recognition: A study of the effects of kernel size and number of filters on accuracy. The Visual Computer, 36(2), 405–412. https://doi.org/10.1007/s00371-019-01630-9

Alizadeh

Fazel

(2017). Convolutional neural networks for facial expression recognition (pp. 1–8). Stanford Univ., Tech. Rep. https://doi.org/10.48550/arXiv.1704.06756

Almaev

T. R.

Valstar

M. F.

(2013). Local gabor binary patterns from three orthogonal planes for automatic facial expression recognition. In 2013 Humaine association conference on affective computing and intelligent interaction (pp. 356–361). IEEE. https://doi.org/10.1109/ACII.2013.65

Aouayeb

Hamidouche

Soladie

Kpalma

Seguier

(2021). Learning vision transformer with squeeze and excitation for facial expression recognition. arXiv preprint arXiv:2107.03107. https://doi.org/ 10.48550/arXiv.2107.03107

Arun

A. N.

Maheswaravenkatesh

Jayasankar

(2023). Facial micro emotion detection and classification using swarm intelligence based modified convolutional network. Expert Systems with Applications, 233, 120947. https://doi.org/10.1016/j.eswa.2023.120947

Ashir

A. M.

Eleyan

Akdemir

(2020). Facial expression recognition with dynamic cascaded classifier. Neural Computing and Applications, 32(10), 6295–6309. https://doi.org/10.1007/s00521-019-04138-4

Benitez-Quiroz

C. F.

Srinivasan

Martinez

A. M.

(2019). Discriminant functional learning of color features for the recognition of facial action units and their intensities. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(12), 2835–2845. https://doi.org/10.1109/TPAMI.2018.2868952

Chen

Chi

(2018). Facial expression recognition in video with multiple feature fusion. IEEE Transactions on Affective Computing, 9(1), 38–50. https://doi.org/10.1109/TAFFC.2016.2593719

Chen

Wen

Yang

Chen

Wang

(2024). Multi-geometry embedded transformer for facial expression recognition in videos. Expert Systems with Applications, 249, 123635. https://doi.org/10.1016/j.eswa.2024.123635

10.

Dey

Chowdhury

Ghosh

Kahali

(2023). T2-fuzzy multi-fused facial image fusion (T2FMIF): An efficient face recognition. Journal of Intelligent & Fuzzy System, 45(1), 743–761. https://doi.org/10.3233/JIFS-224288

11.

Ekundayo

O. S.

Viriri

(2021). Facial expression recognition: A review of trends and techniques. IEEE Access, 9, 136944–136973. https://doi.org/10.1109/ACCESS.2021.3113464

12.

Feng

Ren

(2018). Dynamic facial expression recognition based on two-stream-cnn with lbp-top. In 2018 5th IEEE International conference on cloud computing and intelligence systems (CCIS) (pp. 355–359). IEEE. https://doi.org/10.1109/CCIS.2018.8691380

13.

Ghosh

Dey

Kahali

(2022). Type-2 fuzzy blended improved DS evidence theory based decision fusion for face recognition. Applied Soft Computing, 125, 109179. https://doi.org/10.1016/j.asoc.2022.109179

14.

Goodfellow

I. J.

Erhan

Carrier

P. L.

Courville

Mirza

Hamner

Cukierski

Tang

Thaler

Lee

D.-H.

Zhou

Ramaiah

Feng

Wang

Athanasakis

Taylor

J. S.-

Milakov

Park

Bengio

(2013). Challenges in representation learning: A report on three machine learning contests. In Neural information processing. ICONIP. https://doi.org/10.1007/978-3-642-42051-1_16

15.

Guo

Wang

Zhang

(2023). DLSANet: Facial expression recognition with double-code LBP-layer spatial-attention network. IET Image Processing, 17(9), 2659–2672. https://doi.org/10.1049/ipr2.12817

16.

Guo

Zhang

(2010). A completed modeling of local binary pattern operator for texture classification. IEEE Transactions on Image Processing, 19(6), 1657–1663. https://doi.org/10.1109/TIP.2010.2044957

17.

Hafiane

Seetharaman

Zavidovique

(2007). Median binary pattern for textures classification. In International conference image analysis and recognition (pp. 387–398). Springer. https://doi.org/10.1007/978-3-540-74260-9_35

18.

Heikkilä

Pietikäinen

Schmid

(2009). Description of interest regions with local binary patterns. Pattern Recognition, 42(3), 425–436. https://doi.org/10.1016/j.patcog.2008.08.014

19.

Kaplan

Kaya

Kuncan

Minaz

M. R.

Ertunç

H. M.

(2020). An improved feature extraction method using texture analysis with LBP for bearing fault diagnosis. Applied Soft Computing, 87, 106019. https://doi.org/10.1016/j.asoc.2019.106019

20.

Kar

Banerjee

(2021). Intensity prediction of tropical cyclone using multilayer multi-block local binary pattern and tree-based classifiers over North Indian Ocean. Computers & Geosciences, 154, 104798. https://doi.org/10.1016/j.cageo.2021.104798

21.

Karnati

Seal

Yazidi

Krejcar

(2022). FLEPNet: Feature level ensemble parallel network for facial expression recognition. IEEE Transactions on Affective Computing, 13(4), 2058–2070. https://doi.org/10.1109/TAFFC.2022.3208309

22.

Kaya

Ertuğrul

ÖF

Tekin

(2015). Two novel local binary pattern descriptors for texture analysis. Applied Soft Computing, 34, 728–735. https://doi.org/10.1016/j.asoc.2015.06.009

23.

Kumar

Sagar

Punetha

(2023). A comparative study on facial expression recognition using local binary patterns, convolutional neural network and frequency neural network. Multimedia Tools and Applications, 82(16), 24369–24385. https://doi.org/10.1007/s11042-023-14753-y

24.

Deng

(2019). Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Transactions on Image Processing, 28(1), 356–370. https://doi.org/10.1109/TIP.2018.2868382

25.

Deng

(2017). Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2852–2861). https://doi.org/10.1109/CVPR.2017.277

26.

Liao

Zhu

Lei

Zhang

S. Z.

(2007). Learning multi-scale block local binary patterns for face recognition. In Advances in biometrics: International conference, ICB 2007, Seoul, Korea, August 27–29 (pp. 828–837). Springer. https://doi.org/10.1007/978-3-540-74549-5_87

27.

Liu

Fieguth

Guo

Wang

Pietikäinen

(2017). Local binary features for texture classification: Taxonomy and experimental study. Pattern Recognition, 62, 135–160. https://doi.org/10.1016/j.patcog.2016.08.032

28.

Lucey

Cohn

J. F.

Kanade

Saragih

Ambadar

Matthews

(2010). The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In 2010 IEEE computer society conference on computer vision and pattern recognition-workshops (pp. 94–101). IEEE. https://doi.org/10.1109/CVPRW.2010.5543262

29.

Lundqvist

Flykt

Öhman

(1998). Karolinska directed emotional faces. PsycTESTS Dataset, 91, 630. https://doi.org/10.1037/t27732-000

30.

Lyons

Kamachi

Gyoba

(1998). The Japanese female facial expression (JAFFE) dataset [data set]. Zenodo. https://doi.org/10.5281/zenodo.3451524

31.

Lyons

M. J.

Kamachi

Gyoba

(2020). Coding facial expressions with Gabor wavelets (IVC special issue), arXiv:2009.05938. https://doi.org/10.48550/arXiv.2009.05938

32.

Mehta

Egiazarian

(2016). Dominant rotated local binary patterns (DRLBP) for texture classification. Pattern Recognition Letters, 71, 16–22. https://doi.org/10.1016/j.patrec.2015.11.019

33.

Mohan

Seal

Krejcar

Yazidi

(2021). FER-net: Facial expression recognition using deep neural net. Neural Computing and Applications, 33(15), 9125–9136. https://doi.org/10.1007/s00521-020-05676-y

34.

Mukhopadhyay

Dey

Ghosh

Shaw

R. N.

(2022, October). Facial emotion recognition based on textural pattern and histogram of oriented gradient. In In international conference on advanced communication and intelligent systems (pp. 111–119). Springer Nature. https://doi.org/10.1007/978-3-031-25088-0_9

35.

Mukhopadhyay

Dey

Kahali

(2023). A deep-learning-based facial expression recognition method using textural features. Neural Computing and Applications, 35(9), 6499–6514. https://doi.org/10.1007/s00521-022-08005-7

36.

Nanni

Lumini

Brahnam

(2012). Survey on LBP based texture descriptors for image classification. Expert Systems with Applications, 39(3), 3634–3641. https://doi.org/10.1016/j.eswa.2011.09.054

37.

Nguyen

D. T.

Cho

S. R.

Park

K. R.

(2014). Human age estimation based on multi-level local binary pattern and regression method. In Future information technology: Future tech 2014 (pp. 433–438). Springer. https://doi.org/10.1007/978-3-642-55038-6_67

38.

Oğuz

Ertuğrul

ÖF

(2024). Emotion recognition by skeleton-based spatial and temporal analysis. Expert Systems with Applications, 238, 121981. https://doi.org/10.1016/j.eswa.2023.121981

39.

Ojala

Matti

Topi

(2002). Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(7), 971–987. https://doi.org/10.1109/TPAMI.2002.1017623

40.

Pietikäinen

Zhao

(2015). Two decades of local binary patterns: A survey. In Advances in independent component analysis and learning machines (pp. 175–210). Academic Press. https://doi.org/10.1016/B978-0-12-802806-3.00009-9

41.

Rashad

Alebiary

D. M.

Aldawsari

El-Sawy

A. A.

AbuEl-Atta

A. H.

(2024, Jul 1). CCNN-SVM: Automated model for emotion recognition based on custom convolutional neural networks with SVM. Information, 15(7), 384. https://doi.org/10.3390/info15070384

42.

Sarvakar

Senkamalavalli

Raghavendra

Kumar

J. S.

Manjunath

Jaiswal

(2023). Facial emotion recognition using convolutional neural networks. Materials Today: Proceedings, 80, 3560–3564. https://doi.org/10.1016/j.matpr.2021.07.297

43.

Shu

Song

Shi

Huang

X. J.

(2021). Multiple channels local binary pattern for color texture representation and classification. Signal Processing: Image Communication, 98, 116392. https://doi.org/10.1016/j.image.2021.116392

44.

Sing

J. K.

Dey

Ghosh

(2019). Confidence factor weighted Gaussian function induced parallel fuzzy rank-level fusion for inference and its application to face recognition. Information Fusion, 47, 60–71. https://doi.org/10.1016/j.inffus.2018.07.005

45.

Suguna

Anandhakumar

(2011). Multi-level local binary pattern analysis for texture characterization. In International conference on advances in computing and information technology (pp. 375–386). Springer. https://doi.org/10.1007/978-3-642-22555-0_39

46.

Sukhavasi

S. B.

Sukhavasi

S. B.

Elleithy

El-Sayed

Elleithy

(2022). A hybrid model for driver emotion detection using feature fusion approach. International Journal of Environmental Research and Public Health, 19(5), 3085. https://doi.org/10.3390/ijerph19053085

47.

Sun

Zhang

Bai

Liu

(2023). A discriminatively deep fusion approach with improved conditional GAN (im-cGAN) for facial expression recognition. Pattern Recognition, 135, 109157. https://doi.org/10.1016/j.patcog.2022.109157

48.

Verma

Mandal

Reddy

S. K.

Meedimale

Y. R.

Vipparthi

S. K.

(2023). Efficient neural architecture search for emotion recognition. Expert Systems with Applications, 224, 119957. https://doi.org/10.1016/j.eswa.2023.119957

49.

Wei

Yang

Luu

A. T.

Dong

(2024). Learning facial expression and body gesture visual information for video emotion recognition. Expert Systems with Applications, 237, 121419. https://doi.org/10.1016/j.eswa.2023.121419

50.

Zhang

Chu

Xiang

Liao

S. Z.

(2007). Face detection based on multi-block lbp representation. In Advances in biometrics: International conference, ICB 2007, Seoul, Korea, August 27–29 (pp. 11–18). Springer. https://doi.org/10.1007/978-3-540-74549-5_2

51.

Zhao

Pietikainen

(2007). Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE transactions on Pattern Analysis and Machine Intelligence, 29(6), 915–928. https://doi.org/10.1109/TPAMI.2007.1110

Synergizing Bernoulli and Entropy for Robust Multi-Scale LBP Image Fusion in Facial Emotion Recognition

Abstract

Purpose

Method

Results

Keywords

Get full access to this article

References