Identification of micro expressions in a video sequence by Euclidean distance of the facial contours

Abstract

This paper presents an automatic facial micro-expression recognition system (FMER) from video sequence. Identification and classification are performed on basic expressions: happy, surprise, fear, disgust, sadness, anger, and neutral states. The system integrates three main steps. The first step consists in face detection and tracking over three consecutive frames. In the second step, the facial contour extraction is performed on each frame to build Euclidean distance maps. The last task corresponds to the classification which is achieved with two methods; the SVM and using convolutional neural networks. Experimental evaluation of the proposed system for facial micro-expression identification is performed on the well-known databases (Chon and Kanade and CASME II), with six and seven facial expressions for each classification method.

Keywords

Facial expression identification micro-expression Euclidian distance convolutional neural network Alexnet Support vector machines (SVM)

Get full access to this article

View all access options for this article.

References

M.B.

Abdulrazzaq,

M.R.

Mahmood,

S.R.M.

Zeebaree,

M.H.

Abdulwahab,

Zebari and

Sallow, An analytical appraisal for supervised classifiers performance on facial expression recognition based on relief-F feature selection, Journal of Physics: Conference Series (2021). doi:10.1088/1742-6596/1804/1/012055.

I.P.

Adegun and

H.B.

Vadapalli, Facial micro-expression recognition: A machine learning approach, Scientific African 8 (2020), e00465. doi:10.1016/j.sciaf.2020.e00465.

Amine,

Rziza and

Aboutajdine, SVM-based face recognition using genetic search for frequency-feature subset selection, Image and Signal Processing Lecture Notes in Computer Science (2008), 321–328. doi:10.1007/978-3-540-69905-7_37.

Canny, A computational approach to edge detection, in: IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. PAMI-8, no. 6, 1986, pp. 679–698. doi:10.1109/TPAMI.1986.4767851.

H.I.

Dino and

M.B.

Abdulrazzaq, A comparison of four classification algorithms for facial expression recognition, Polytechnic Journal 10 (2020), 74–80. doi:10.25156/ptj.v10n1y2020.

Ekman, Universals and cultural differences in facial expressions of emotion, in: Nebraska Symposium on Motivation,

Cole, ed., University of Nebraska Press, Lincoln, 1971, pp. 207–283.

Ghimire,

Z.M.

Yaseen,

Farooque,

R.C.

Deo,

Zhang and

Tao, Streamflow prediction using an integrated methodology based on convolutional neural network and long short-term memory networks, Sci Rep 11 (2021), 17497. doi:10.1038/s41598-021-96751-4.

Jing,

Wang,

See and

Wenbin, Micro-expression recognition based on 3D flow convolutional neural network, in: Pattern Analysis and Applications, 2018. doi:10.1007/s10044-018-0757-5.

Kherchaoui and

Houacine, Facial expression identification system with Euclidean distance of facial edges, in: 6th International Conference of Soft Computing and Pattern Recognition (SoCPaR), 2014, pp. 6–10. doi:10.1109/SOCPAR.2014.7007973.

10.

Kok,

Jahmunah,

S.L.

Oh,

Zhou,

Gururajan,

Tao,

Hao Cheong,

Gururajan,

Molinari and

U.R.

Acharya, Automated prediction of sepsis using temporal convolutional network, Computers in Biology and Medicine 127 (2020), 103957, ISSN 0010-4825. doi:10.1016/j.compbiomed.2020.103957.

11.

Krizhevsky,

Sutskever and

G.E.

Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in Neural Information Processing Systems, 2012, pp. 1097–1105.

12.

T.T.Q.

Le,

T.-K.

Tran and

Rege, Dynamic image for micro-expression recognition on region-based framework, in: IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), 2020, pp. 75–81. doi:10.1109/IRI49571.2020.00019.

13.

Li,

Pfister,

Huang,

Zhao and

Pietikäinen, in: A Spontaneous Micro-Expression Database: Inducement, Collection and Baseline, 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 2013, pp. 1–6. doi:10.1109/FG.2013.6553717.

14.

Lucey,

J.F.

Cohn,

Kanade,

Saragih,

Ambadar and

Matthews, The extended cohn-kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression, in: IEEE Computer Society Conference on Computer Vision and Pattern Recognition – Workshops, 2010, pp. 94–101. doi:10.1109/CVPRW.2010.5543262.

15.

Osuna,

Freund and

Girosi, Training support vector machines: An application to face detection, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (1997), 130–136. doi:10.1109/CVPR.1997.609310.

16.

Schölkopf and

Smola, Support Vector Machines and Kernel Algorithms, 2002.

17.

Tjondronegoro,

Tao,

Sasongko and

C.H.

Lau, Multi-modal summarization of key events and top players in sports tournament videos, IEEE Workshop on Applications of Computer Vision (WACV) (2011), 471–478. doi:10.1109/WACV.2011.5711541.

18.

V.N.

Vapnik, Statistical Learning Theory, Wiley, New York, 1998.

19.

Viola and

M.J.

Jones, Rapid object detection using a boosted cascade of simple features, in: Proceedings of the IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, Vol. 1, 2001, pp. 511–518.

20.

Viola and

M.J.

Jones, Robust real time object detection, International Journal of Computer Vision 57 (2004), 137–154. doi:10.1023/B:VISI.0000013087.49260.fb.

21.

Wang,

See,

Oh,

Phan,

Rahulamathavan,

Chong Ling,

Tan and

Li, Effective Recognition of Facial Micro-Expressions with Video Motion Magnification, Multimedia Tools and Applications (2017). doi:10.1007/s11042-016-4079-6.

22.

W.J.

Yan,

Li,

S.J.

Wang,

Zhao,

Y.J.

Liu,

Y.H.

Chen and

Fu, CASME II, An improved spontaneous micro-expression database and the baseline evaluation, PLoS One 9(1) (2014). doi:10.1371/journal.pone.0086041.

23.

Zhao and

Pietikainen, Dynamic texture recognition using local binary patterns with an application to facial expressions, IEEE Trans Pattern Anal Mach Intell 29(6) (2007), 915–928. doi:10.1109/TPAMI.2007.1110.