To solve the problem of limitations of current attention mechanisms in extracting key facial expression features and the problem of low accuracy of facial expression recognition due to insufficient consideration of feature information fusion in the receptive field by convolutional neural networks. In this paper, we propose a facial expression recognition network based on wide attention (WA) and a multi-scale fusion (MF) mechanism (wide attention and multi-scale fusion [WAMF]). WA by extracting the background information of facial expression images while focusing on texture information, thus achieving better feature extraction. The MF mechanism is added at the connection points of layers in ResNet, where features extracted from each upper layer are fused using different-sized convolutional kernels and input into the lower layer. Finally, a viewpoint-invariant Capsule Net is used as the classification network after receiving the feature maps. The proposed WAMF model was applied to two publicly available datasets, CK+ and Jaffe, achieving excellent recognition rates of 98.98% and 98.46%, respectively.
AvaniVSShailaSGVadivelA (2021) Interval graph of facial regions with common intersection salient points for identifying and classifying facial expression. Multimedia Tools and Applications80: 3367–3390.
2.
AyecheFAltiA (2021) HDG and HDGG: An extensible feature extraction descriptor for effective face and facial expressions recognition. Pattern Analysis and Applications24: 1095–1110.
3.
BystroffCThorssonVBakerD (2000) HMMSTR: A hidden Markov model for local sequence-structure correlations in proteins. Journal of Molecular Biology301(1): 173–190.
4.
CaoSYaoYAnG (2020) E2-capsule neural networks for facial expression recognition using AU-aware attention. IET Image Processing14(11): 2417–2424.
5.
ElArabyMEShamsMY (2021) Face retrieval system based on elastic web crawler over cloud computing. Multimedia Tools and Applications80: 11723–11738.
6.
FernandezPDMPeñaFAGRenTI, et al. (2019) FERAtt: Facial expression recognition with attention net. In: 2019 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp.837–846. IEEE. DOI: https://doi.org/10.1109/CVPRW.2019.00112.
7.
GeraDBalasubramanianS (2021) Landmark guidance independent spatio-channel attention and complementary context information based facial expression recognition. Pattern Recognition Letters145: 58–66.
8.
HeKZhangXRenS, et al. (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp.770–778. IEEE. DOI: https://doi.org/10.1109/CVPR.2016.90.
9.
HintonGESalakhutdinovRR (2006) Reducing the dimensionality of data with neural networks. Science313(5786): 504–507.
10.
HouQZhouDFengJ (2021) Coordinate attention for efficient mobile network design. In: 2021 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp.13708–13717. IEEE. DOI: https://doi.org/10.1109/CVPR46437.2021.01350.
11.
HuJShenLSunG (2018) Squeeze-and-excitation networks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp.7132–7141. IEEE. DOI: https://doi.org/10.1109/CVPR.2018.00745.
12.
JeongMKoBC (2018) Driver’s facial expression recognition in real-time for safe driving. Sensors18(12): 4270.
13.
JiangLHuangC (2022) Improving expression recognition by fusing attention mechanisms with residual networks. Computer Technology and Development32(5): 42–46 + 52.
14.
KhaliluzzamanMPervinSIslamMR, et al. (2019) Automatic facial expression recognition using shallow convolutional neural network. In: 2019 IEEE international conference on robotics, automation, artificial-intelligence and internet-of-things (RAAICON), pp.98–103. IEEE. DOI: https://doi.org/10.1109/RAAICON48939.2019.42.
15.
LiBMehtaSAnejaD, et al. (2019) A facial affect analysis system for autism spectrum disorder. In: 2019 IEEE international conference on image processing (ICIP), pp.4549–4553. IEEE. DOI: https://doi.org/10.1109/ICIP.2019.8803604.
16.
LiJJinKZhouD, et al. (2020a) Attention mechanism-based CNN for facial expression recognition. Neurocomputing411: 340–350.
17.
LiKJinYAkramMW, et al. (2020b) Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy. The Visual Computer36: 391–404.
18.
LiuSLongYXuG, et al. (2021) An optimized Capsule-LSTM model for facial expression recognition with video sequences. Available at: https://arxiv.org/abs/2106.07564 (accessed 27 May 2021).
19.
LiuXLiCDaiC, et al. (2022) Nonnegative tensor factorization based on low-rank subspace for facial expression recognition. Mobile Networks and Applications27: 58–69.
20.
LopesATAguiarESouzaA, et al. (2017) Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order. Pattern Recognition61: 610–628.
21.
LuceyPCohnJFKanadeT, et al. (2010) The extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE.
22.
LyonsMJAkematsuSKamachiM, et al. (1998) Coding facial expressions with Gabor wavelets. 3rd IEEE International Conference on Automatic Face and Gesture Recognition: 200–205. IEEE.
23.
MohanKSealAKrejcarO, et al. (2021) Facial expression recognition using local gravitational force descriptor-based deep convolution neural networks. IEEE Transactions on Instrumentation and Measurement70: 1–12.
24.
PanticMRothkrantzLJM (2000) Expert system for automatic analysis of facial expressions. Image and Vision Computing18(11): 881–905.
25.
QianZMuJTianF, et al. (2022) Facial expression recognition based on strong attention mechanism and residual network. Multimedia Tools and Applications82: 14287–14306.
26.
RaoTLiJWangX, et al. (2021) Facial expression recognition with multiscale graph convolutional networks. IEEE MultiMedia28(2): 11–19.
27.
RubelASAhsan ChowdhuryAKabirMH (2019) Facial expression recognition using adaptive robust local complete pattern. In: 2019 IEEE international conference on image processing (ICIP), pp.41–45. IEEE. DOI: https://doi.org/10.1109/ICIP.2019.8802911.
28.
SadeghiHRaieA (2019) Human vision inspired feature extraction for facial expression recognition. Multimedia Tools and Applications78: 30335–30353.
29.
SarhanSNasrAAShamsMY (2020) Multipose face recognition-based combined adaptive deep learning vector quantization. Computational Intelligence and Neuroscience2020(1): 8821868.
30.
SuykensJAKVandewalleJ (1999) Least squares support vector machine classifiers. Neural Processing Letters9: 293–300.
31.
TarekZShohiebSMElhadyAM, et al. (2023) Eye detection-based deep belief neural networks and speeded-up robust feature algorithm. Computer Systems Science and Engineering45(3): 3195–3213.
32.
VinayAGuptaABharadwajA, et al. (2018) Unconstrained face recognition using Bayesian classification. Procedia Computer Science143: 519–527.
33.
WangWFuYSunQ, et al. (2020) Learning to augment expressions for few-shot fine-grained facial expression recognition. Available at: https://arxiv.org/abs/2001.06144 (accessed January 2020).
34.
WangZZengFLiuS, et al. (2021) OAENEt: Oriented attention ensemble for accurate facial expression recognition. Pattern Recognition112: 107694.
XiaoZTongHQuR, et al. (2023) Capmatch: Semi-supervised contrastive transformer capsule with feature-based knowledge distillation for human activity recognition. IEEE Transactions on Neural Networks and Learning Systems: 1–15. DOI: https://doi.org/10.1109/TNNLS.2023.3344294.
37.
XiaoZXingHQuR, et al. (2024a) Densely knowledge-aware network for multivariate time series classification. IEEE Transactions on Systems, Man, and Cybernetics: Systems54(4): 2192–2204.
38.
XiaoZXingHZhaoB, et al. (2024b) Deep contrastive representation learning with self-distillation. IEEE Transactions on Emerging Topics in Computational Intelligence8(1): 3–15.
39.
XiaoZXuXXingH, et al. (2024c) DTCM: Deep transformer capsule mutual distillation for multivariate time series classification. IEEE Transactions on Cognitive and Developmental Systems16(4): 1445–1461.
40.
YangHCiftciUYinL (2018) Facial expression recognition by de-expression residue learning. In: 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp.2168–2177. IEEE. DOI: https://doi.org/10.1109/CVPR.2018.00231.
41.
ZhangHHuangBTianG (2020) Facial expression recognition based on deep convolution long short-term memory networks of double-channel weighted mixture. Pattern Recognition Letters131: 128–134.
42.
ZhangPKongWTengJ (2022) Face expression recognition based on multi-scale feature attention mechanism. Computer Engineering and Applications58(1): 182–189.
43.
ZouWZhangDLeeD (2022) A new multi-feature fusion based convolutional neural network for facial expression recognition. Applied Intelligence52: 2918–2929.
