Myoelectric human computer interaction using CNN-LSTM neural network for dynamic hand gesture recognition

Abstract

Human-computer interaction(HCI) has broad range of applications. One particular application domain is rehabilitation devices. Several bioelectric signals can potentially be used in HCI systems in general and rehabilitation devices in particular. Surface ElectroMyoGraphic(sEMG) signal is one of the more important bioelectric signals in this context. The sEMG signal is formed by muscle activation although the details are rather complex. Applications of sEMG are referred is commonly referred to as myoelectric control since the dominant use of this signal is to activate a device even if (as the term control may imply) feedback is not always used in the process. With the development of deep neural networks, various deep learning architectures are used for sEMG-based gesture recognition with many researchers having reported good performance. Nevertheless, challenges remain in accurately recognizing sEMG patterns generated by gestures produced by hand or the upper arm. For instance one of the difficulties in hand gesture recognition is the influence of limb positions. Several papers have shown that the accuracy of gesture classification decreases when the limb position changes even if the gesture remains the same. Prior work by our team has shown that dynamic gesture recognition is in principle more reliable in detecting human intent, which is often the underlying idea of gesture recognition. In this paper, a Convolutional Neural Network (CNN) with Long Short-Term Memory or LSTM (CNN-LSTM) is proposed to classify five common dynamic gestures. Each dynamic gesture would be performed in five different limb positions as well. The trained neural network model is then used to enable a human subject to control a 6 DoF (Degree of Freedom) robotic arm with 1 DoF gripper. The results show a high level of accurate performance achieved with the proposed approach. In particular, the overall accuracy of the dynamic gesture recognition is 84.2%. The accuracies vary across subjects but remain at approximately 90%for some subjects.

Keywords

Human-computer interaction sEMG signal neural network gesture recognition

Get full access to this article

View all access options for this article.

References

Wang

and Wang

, Deep convolutional neural network for decoding emg for human computer interaction, in: 2020 11th IEEE Annual Ubiquitous Computing, Electronics Mobile Communication Conference (UEMCON), 2020, pp. 0554–0557. doi: 10.1109/UEMCON51285.2020.9298064.

Sun

, Xu

, Li

, Xu

, Kong

, Jiang

, Tao

and Chen

, Intelligent human computer interaction based on non redundant emg signal, Alexandria Engineering Journal 59(3) (2020) 1149–1157, a Special Section on: Computational Methods in Eneering and Artificial Intelligence. doi:https://doi.org/10.1016/j.aej.2020.01.015.

Sharmila

, Hybrid control approaches for hands-free high level human–computer interface-a review, Journal of Medical Engineering & Technology 45(1) (2021), 6–13. pMID: 33191811. arXiv: https://doi.org/10.1080/03091902.2020.1838642, doi:10.1080/03091902.2020.1838642. URL https://doi.org/10.1080/03091902.2020.1838642.

Subba

, Chingtham

T.S.

A survey: Emg signal-based controller for human–computer interaction, in: H.K.D. Sarma, S. Borah and N. Dutta (Eds.), Advances in Communication, Cloud, and Big Data, Springer Singapore, Singapore, 2019, pp. 117–125.

Yasen

M, J.S.

. A systematic review on hand gesture recognition techniques, Peer J Computer Science 5 (2019), e218. doi:https://doi.org/10.7717/peerj-cs.218

Shanmuganathan

V, Y.H.K.M.e.a.

, R-cnn and wavelet feature extraction for hand gesture recognition with emg signals, Neural Comput & Applic 32 (2020), 16723–16736. doi:https://doi.org/10.1007/355s00521-020-05349-w

Ameur

, Ben Khalifa

and Bouhlel

M.S.

. Leapgesturedb: A public leap motion database applied for dynamic hand gesture recognition in surgical procedures, in: V.E. Balas, L.C. Jain, M.M. Balas and S.N. Shahbazova, (Eds.), Soft Computing Applications, Springer International Publishing, Cham, 2021, pp. 125–138.

Shi

W.-T.

, Lyu

Z.-J.

, Tang

S.-T.

, Chia

T.-L.

and Yang

C.-Y.

, A bionic hand controlled by hand gesture recognition based on surface emg signals: A preliminary study, Biocybernetics and Biomedical Engineering 38(1) (2018), 126–135. doi:https://doi.org/10.1016/j.bbe.2017.11.001.

Song

, Han

, Lin

, Yan

, Luo

, Liao

, Zhang

, Wang

, Xie

, Wang

, Chen

and Bai

, Design of a flexible wearable smart semg recorder integrated gradient boosting decision tree based hand gesture recognition, IEEE Transactions on Biomedical Circuits and Systems 13(6) (2019), 1563–1574. doi:10.1109/TBCAS.2019.2953998.

10.

Grigorescu

, Trasnea

, Cocias

and Macesanu

, A survey of deep learning techniques for autonomous driving, Journal of Field Robotics 37(3) (2020), 362–386. doi:10.1002/rob.21918.

11.

Mozaffari

, Al-Jarrah

O.Y.

, Dianati

, Jennings

and Mouzakitis

Deep learning-based vehicle behavior prediction for autonomous driving applications: A review, IEEE Transactions on Intelligent Transportation Systems (2020), 1–15. doi:10.1109/TITS.2020.3012034.

12.

Sharmin

, Rahut

S.K.

and Huq

M.R.

, Bengali spoken digit classification:Adeep learning approach using convolutional neural network, Procedia Computer Science 171 (2020), 1381–1388. Third International Conference on Computing and Network Communications (CoCoNet’19). doi:https://doi.org/10.1016/j.procs.2020.04.148.

13.

Santhanavijayan

, Naresh Kumar

, Deepak

, A semantic-aware strategy for automatic speech recognition incorporating deep learning models, in: S.C. Satapathy, V. Bhateja, B. Janakiramaiah and Y.-W. Chen (Eds.), Intelligent System Design, Springer Singapore, Singapore, 2021,pp. 247–254.

14.

Ozdemir

M.A.

, Kisa

D.H.

, Guren

, Onan

and Akan

. Emg based hand gesture recognition using deep learning, in: 2020 Medical Technologies Congress (TIPTEKNO), 2020, pp. 1–4. doi:10.1109/TIPTEKNO50054.2020.9299264.

15.

Xie

, Meng

, Li

and Harland

, Gesture recognition from bio-signals using hybrid deep neural networks, in: 2020 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA), 2020, pp. 493–499. doi:10.1109/ICAICA50127.2020.9182510.

16.

Mukhopadhyay

A.K.

and Samui

, An experimental study on upper limb 395 position invariant emg signal classification based on deep neural network, Biomedical Signal Processing and Control 55 (2020), 101669. doi:https://doi.org/10.1016/j.bspc.2019.101669.

17.

Campbell

, Phinyomark

and Scheme

, Current trends and confounding factors in myoelectric control: Limb position and contraction intensity, Sensors 20(6). doi: 10.3390/s20061613

18.

Mitra

and Acharya

, Gesture recognition: A survey, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37(3) (2007), 311–324. doi:10.1109/TSMCC.2007.893280.

19.

Shin

Myoelectric human computer interaction using reliable temporal sequence-based myoelectric classification for dynamic hand gestures, Doctoral dissertation, Texas A & M University (2016). URL https://hdl.handle.net/1969.1/174271.

20.

Naranjo-Torres

, Mora

, Hernández-García

, Barrientos

R.J.

, Fredes

and Valenzuela

, A review of convolutional neural network applied to fruit image processing, Applied Sciences 10(10). doi:10.3390/app10103443.

21.

Huang

, He

, Tan

, Jiang

, Li

and Yu

, Jointly network image processing: Multi-task image semantic segmentation of indoor scene based on cnn, IET Image Processing 14(15) (2020), 3689–3697. arXiv:https://ietresearch.onlinelibrary.wiley.com/doi/pdf/10.1049/iet-ipr.2020.0088, doi:https://doi.org/10.1049/iet-ipr.2020.0088.

22.

Huang

and Chen

, Surface emg decoding for hand gestures based on spectrogram and cnn-lstm, in: 2019 2nd China Symposium on Cognitive 420 Computing and Hybrid Intelligence (CCHI), 2019, pp. 123–126. doi:10.1109/CCHI.2019.8901936

23.

Santurkar

, Tsipras

, Ilyas

and Madry

How does batch normalization help optimization? (2019). arXiv:1805.11604.

24.

Srivastava

, Hinton

, Krizhevsky

, Sutskever

and Salakhutdinov

, Dropout: A simple way to prevent neural networks from overfitting, Journal of Machine Learning Research 15(56) (2014), 1929–1958.

25.

, Si

, Hu

and Zhang

, A review of recurrent neural networks: LSTM cells and network architectures, Neural Computation 31(7) (2019), 1235–1270. arXiv:https://direct.mit.edu/neco/article-pdf/31/7/1235/1053200/neco_a_01199.pdf, doi:10.1162/neco_a_01199.

26.

Hochreiter

and Schmidhuber

, Long short-term memory, Neural Computation 9(8) (1997), 1735–1780. doi:10.1162/neco.1997.9.8.1735.

27.

Chimmula

V.K.R.

and Zhang

, Time series forecasting of covid-19 trans-mission in canada using lstm networks, Chaos, Solitons Fractals 135 (2020), 109864. doi:https://doi.org/10.1016/j.chaos.2020.109864.

28.

Yang

, Zhou

, Ai

, Bin

, Hanjalic

, Shen

H.T.

and Ji

, Video captioning by adversarial lstm, IEEE Transactions on Image Processing 27(11) (2018), 5600–5611. doi:10.1109/TIP.2018.2855422.

29.

Huang

, Tan

, Lin

and Tian

, A lstm-based bidirectional translation model for optimizing rare words and terminologies, in: 2018 International Conference on Artificial Intelligence and Big Data (ICAIBD), 2018, pp. 185–189. doi:10.1109/ICAIBD.2018.8396191.10.1109/ICAIBD.2018.8396191

30.

Wang

, Wu

, Dey

, Fong

and Ashour

A.S.

, Deep back propagation–long short-term memory network based upper-limb semg signal classification for automated rehabilitation, Biocybernetics and Biomedical Engineering 40(3) (2020), 987–1001. doi:https://doi.org/10.1016/j.bbe.2020.05.003

31.

Cai

and Zhu

, A hybrid CNN-LSTM network for hand gesture recognition with surface EMG signals , in: X. Jiang and H. Fujita (Eds.), Thirteenth International Conference on Digital Image Processing (ICDIP 2021), Vol. 11878, International Society for Optics and Photonics, SPIE, 2021, pp. 20–28. doi:10.1117/12.2601074

32.

Islam

M.Z.

, Islam

M.M.

and Asraf

, A combined deep cnn-lstm network for the detection of novel coronavirus (covid-19) using x-ray images, Informatics in Medicine Unlocked 20 (2020), 100412. doi:https://doi.org/10.1016/j.imu.2020.100412.