Facial expression recognition plays an essential role in affective computing, mental illness diagnosis and rehabilitation. Therefore, facial expression recognition has attracted more and more attention over the years.
OBJECTIVE:
The goal of this paper was to improve the accuracy of the Electroencephalogram (EEG)-based facial expression recognition.
METHODS:
In this paper, we proposed a fusion facial expression recognition method based on EEG and facial landmark localization. The EEG signal processing and facial landmark localization are the two key parts. The raw EEG signals is preprocessed by discrete wavelet transform (DWT). The energy feature vector is composed of energy features of the reconstructed signal. For facial landmark localization, images of the subjects’ facial expression are processed by facial landmark localization, and the facial features are calculated by landmarks of essence. In this research, we fused the energy feature vector and facial feature vector, and classified the fusion feature vector with the support vector machine (SVM).
RESULTS:
From the experiments, we found that the accuracy of facial expression recognition was increased 4.16% by fusion method (86.94 4.35%) than EEG-based facial expression recognition (82.78 5.78%).
CONCLUSION:
The proposed method obtain a higher accuracy and a stronger generalization capability.
LotteFBougrainLCichockiA, et al. A review of classification algorithms for EEG-based brain-computer interfaces: A 10-year update. Journal of Neural Engineering2018; 15(3).
2.
WolpawJRBirbaumerNHeetderksWJ, et al. Brain-computer interface technology: A review of the first international meeting. IEEE Transactions on Rehabilitation Engineering2000; 8(2): 164-173.
3.
WolpawJRBirbaumerNMcfarlandDJ, et al. Brain-computer interfaces for communication and control. Clinical Neurophysiology2002; 113(6): 767-791.
4.
YinEZhouZJiangJ, et al. A speedy hybrid BCI spelling approach combining P300 and SSVEP. IEEE Transactions on Biomedical Engineering2014; 61(2): 473-483.
5.
ChaiRFLingSHHunterGP, et al. Brain-computer interface classifier for wheelchair commands using neural network with fuzzy particle swarm optimization. IEEE Journal of Biomedical and Health Informatics2014; 18(5): 1614-1624.
6.
GaoQDouLBelkacemAN, et al. Noninvasive electroencephalogram based control of a robotic arm for writing task using hybrid BCI system. Biomed Research International2017; 2017: 1-8.
7.
GaoQZhaoXWYuX, et al. Controlling of smart home system based on brain-computer interface. Technology and Health Care2018; 26(5): 769-783.
8.
ByrnePJ. Importance of facial expression in facial nerve rehabilitation. Current Opinion in Otolaryngology & Head & Neck Surgery2004; 12(4): 332-335.
9.
LiMXuHPLiuXW, et al. Emotion recognition from multichannel EEG signals using K-nearest neighbor classification. Technology and Health Care2018; 26: S509-S519.
10.
LiuXWangQSLiuD, et al. Human emotion classification based on multiple physiological signals by wearable system. Technology and Health Care2018; 26: S459-469.
11.
PetrantonakisPCHadjileontiadisLJ. Emotion recognition from EEG using higher order crossings. IEEE Transactions on Information Technology in Biomedicine2010; 14(2): 186-197.
12.
HashemianMPourghassemH. Decision-level fusion-based structure of autism diagnosis using interpretation of EEG signals related to facial expression modes. Neurophysiology2017; 49(1): 59-71.
13.
MorecraftRJStilwell-MorecraftKSRossingWR. The motor cortex and facial expression: new insights from neuroscience. Neurologist2004; 10(5): 235-249.
14.
RicciardiLBaggioPRicciardiD, et al. Rehabilitation of hypomimia in Parkinson’s disease: a feasibility study of two different approaches. Neurological Sciences2016; 37(3): 431-436.
15.
ChinZYAngKKGuanC. Multiclass voluntary facial expression classification based on Filter Bank Common Spatial Pattern. 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society; 2008 August 20–25; Vancouver, BC, Canada. USA: IEEE Engineering in Medicine and Biology Society, 2008; 2008: pp. 1005-1008.
16.
LinJSJiangZY. An EEG-based BCI system to facial action recognition. Wireless Personal Communications2016; 94(3): 1-15.
17.
FariaBMReisPLauN. Cerebral Palsy EEG Signals Classification: Facial Expressions and Thoughts for Driving an Intelligent Wheelchair. 2012 IEEE 12th International Conference on Data Mining Workshops (ICDM 2012); 2012 December 10–13; Brussels, Belgium. USA: IEEE Computer Society, 2012: pp. 33-40.
18.
HegerDPutzeFSchultzT. Online Recognition of Facial Actions for Natural EEG-Based BCI Applications, Affective Computing and Intelligent Interaction. Proceedings 4th International Conference (ACII 2011); 2011 October 9–12; Memphis, TN, USA. Germany: Springer-Verlag, 2011: pp. 436-446.
19.
CaiBXiaoSJiangL, et al. A rapid face recognition BCI system using single-trial ERP. 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER 2013); 2013 November 6–8; San Diego, California, USA. IEEE, 2013: pp. 89-92.
20.
TothJArvanehM. Facial expression classification using EEG and gyroscope signals. 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society; 2017 July 11–15; Jeju Island, Korea. USA: IEEE Engineering in Medicine and Biology Society, 2017; 2017: pp. 1018-1021.
21.
LiuCXHamJPostmaE, et al. Representing affective facial expressions for robots and embodied conversational agents by facial landmarks. International Journal of Social Robotics2013; 5(4): 619-626.
22.
LinJRXiaoLYWuT. Face recognition for video surveillance with aligned facial landmarks learning. Technology and Health Care2018; 26; S169-S178.
23.
GhimireDLeeJ. Geometric feature-based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensors2013; 13(6): 7714-7734.
24.
HappySLRoutrayA. Automatic facial expression recognition using features of salient facial patches. IEEE Transactions on Affective Computing2015; 6(1): 1-12.
25.
ZhaoXDellandreaEChenL, et al. Accurate landmarking of three-dimensional facial data in the presence of facial expressions and occlusions using a three-dimensional statistical facial feature model. IEEE Transactions on Systems Man and Cybernetics Part B Cybernetics2011; 41(5): 1417.
26.
SubasiA. EEG signal classification using wavelet feature extraction and a mixture of expert model. Expert Systems With Applications2007; 32(4): 1084-1093.
27.
KingDE. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research2009; 10: 1755-1758.
28.
BurgesCJC. A tutorial on Support Vector Machines for pattern recognition. Data Mining and Knowledge Discovery1998; 2(2): 121-167.
29.
LutsJOjedaFVan de PlasR, et al. A tutorial on support vector machine-based methods for classification problems in chemometrics. Analytica Chimica Acta2010; 665(2): 129-145.
30.
ChangCCLinCJ. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology2012; 2: 1-27.
