Seq2seq model for human action recognition based on skeleton and two-layer bidirectional LSTM

Abstract

Human action recognition (HAR) plays an important role in social interaction in various fields. This study proposes a light-weight skeleton and two-layer bidirectional LSTM-based Seq2Seq model (SB2_Seq2Seq) for HAR to trade off recognition accuracy, users’ privacy and computer resource usage. An experiment was conducted to compare the proposed SB2_Seq2Seq with other skeleton-based Seq2Seq models and non-skeleton RGB video frame-based LSTM, CNN and seq2seq models. The UCF50 dataset was used for model evaluation, where 60%, 20% and 20% for model training, validation and testing, respectively. The experimental results show that the proposed model achieves 93.54% accuracy with 0.0214 Mean Square Error (MSE), suggesting that the proposed model outperforms all the other models. Besides, it also shows that the proposed model achieves state-of-the-art accuracy compared with state-of-the-arts methods in literature.

Keywords

Human action recognition (HAR)skeleton Seq2Seq Convolutional Neural Network (CNN)Long Short-Term Memory (LSTM)deep leaning tradeoff privacy UCF50

Get full access to this article

View all access options for this article.

References

J.K.

Aggarwal and

M.S.

Ryoo, Human activity analysis: A review, ACM Comput. Surv 43(3) (2011), 1–43. doi:10.1145/1922649.1922653.

Cao,

Simon,

S.E.

Wei and

Sheikh, Realtime multi-person 2d pose estimation using part affinity fields, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2017, pp. 7291–7299.

Y.P.

Chen and

W.G.

Qiu, Review of human behavior recognition algorithms based on vision, Comput. Appl. Res. 36(7) (2019), 1–10.

Dasari,

Zhang,

Yu,

Huang and

Gao, Human action recognition using hybrid deep evolving neural networks, in: 2022 International Joint Conference on Neural Networks (IJCNN), IEEE, 2022, pp. 1–8.

Dasari and

C.W.

Chen, Mpeg cdvs feature trajectories for action recognition in videos, in: Proceedings of IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), IEEE, 2018, pp. 301–304.

D.D.

Dawn and

S.H.

Shaikh, A comprehensive survey of human action recognition with spatio-temporal interest point (STIP) detector, Vis. Comput. 32(3) (2016), 289–306. doi:10.1007/s00371-015-1066-2.

Donahue,

L.A.

Hendricks,

Guadarrama,

Rohrbach,

Venugopalan,

Darrell and

Saenko, Long term recurrent convolutional networks for visual recognition and description, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2015, pp. 2625–2634.

J.L.

Elman, Finding structure in time, Cognitive Sci 14(2) (1990), 179–211. doi:10.1207/s15516709cog1402_1.

J.C.V.

Gemert,

Jain,

Gati and

C.G.M.

Snoek, APT: Action localization proposals from dense trajectories, in: Proceedings of the British Machine Vision Conference, 2015, pp. 7–10.

10.

Han,

Reily,

Hoff and

Zhang, Space-time representation of people based on 3D skeletal data: A review, Computer Vision and Image Understanding 158 (2017), 85–105. doi:10.1016/j.cviu.2017.01.011.

11.

Hochreiter and

Schmidhuber, Long short-term memory, Neural Comput. 9(8) (1997), 1735–1780. doi:10.1162/neco.1997.9.8.1735.

12.

Hussain,

Ullah and

S.W.

Baik, Vision transformer and deep sequence learning for human activity recognition in surveillance videos, Computational Intelligence and Neuroscience 2022 (2022).

13.

Iqbal,

Milan and

Gall, PoseTrack: Joint multi-person pose estimation and tracking, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4654–4663.

14.

S.W.

Ji,

Xu,

Yang and

Yu, 3D convolutional neural networks for human action recognition, IEEE T. Pattern Anal 35(1) (2013), 221–231. doi:10.1109/TPAMI.2012.59.

15.

M.I.

Jordan, Serial order: A parallel distributed processing approach, in: Proceedings of the Advances in Psychology, 1997, pp. 471–495.

16.

Klaser,

Marszalek and

Schmid, A spatio-temporal descriptor based on 3D-gradients, in: Proceedings of the British Machine Vision Conference, 2008, pp. 1–10.

17.

Krizhevsky,

Sutskever and

G.E.

Hinton, ImageNet classification with deep convolutional neural networks, Commun. ACM 60(6) (2017), 84–90. doi:10.1145/3065386.

18.

Laptev,

Marszalek,

Schmid and

Rozenfeld, Learning realistic human actions from movies, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2008, pp. 1–8.

19.

Lee,

Kim,

Kang and

Lee, Ensemble deep learning for skeleton-based action recognition using temporal sliding lstm networks, in: Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 2017, pp. 1012–1020.

20.

Leyva,

Sanchez and

C.T.

Li, Compact and low-complexity binary feature descriptor and Fisher vectors for video analytics, IEEE Transactions on Image Processing 28(12) (2019), 6169–6184. doi:10.1109/TIP.2019.2922826.

21.

Li,

Wang,

Hou and

Li, Skeleton-based action recognition using LSTM and CNN, in: 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), IEEE, 2017, pp. 585–590.

22.

Li and

Wang, Human activity recognition based on residual network and BiLSTM, Sensors 22(2) (2022), 635. doi:10.3390/s22020635.

23.

Liu,

Shahroudy,

Xu and

Wang, Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition, Lect. Notes. Comput. Sc. (2016), 816–833. doi:10.1007/978-3-319-46487-9_50.

24.

Mehmood,

Chen,

M.A.

Akbar and

A.A.

Alsanad, Human action recognition of spatiotemporal parameters for skeleton sequences using MTLN feature learning framework, Electronics 10(21) (2021), 2708. doi:10.3390/electronics10212708.

25.

G.T.

Papadopoulos,

Axenopoulos and

Daras, Real-time skeleton-tracking-based human action recognition using kinect data, in: Proceedings of the International Conference on Multimedia Modeling, 2014, pp. 473–483. doi:10.1007/978-3-319-04114-8_40.

26.

S.N.

Paul and

Y.J.

Singh, Survey on video analysis of human walking motion, Int. J. Signal Process. Image Process. Pattern Recognition 7 (2014), 99–122.

27.

K.K.

Reddy and

Shah, Recognizing 50 human action categories of web videos, Machine vision and applications 24(5) (2013), 971–981. doi:10.1007/s00138-012-0450-4.

28.

Scovanner,

Ali and

Shah, A 3-dimensional sift descriptor and its application to action recognition, in: Proceedings of the 15th International Conference on Multimedia, 2007, pp. 357–360. doi:10.1145/1291233.1291311.

29.

Shi,

Zhang,

Cheng and

Lu, Skeleton-based action recognition with directed graph neural networks, in: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

30.

Simonyan and

Zisserman, Two-stream convolutional networks for action recognition in videos, in: Proceedings of the Advances in Neural Information Processing Systems, 2014, pp. 568–576.

31.

Solmaz,

S.M.

Assari and

Shah, Classifying web videos using a global video descriptor, Machine vision and applications 24(7) (2013), 1473–1485. doi:10.1007/s00138-012-0449-x.

32.

Tang,

Tian,

Lu,

Li and

Zhou, Deep progressive reinforcement learning for skeleton-based action recognition, in: Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 5323–5332.

33.

Tran,

Bourdev,

Fergus,

Torresani and

Paluri, Learning spatiotemporal features with 3d convolutional networks, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015, pp. 4489–4497.

34.

Ullah,

Ahmad,

Muhammad,

Sajjad and

S.W.

Baik, Action recognition in video sequences using deep bi-directional LSTM with CNN features, IEEE access 6 (2017), 1155–1166. doi:10.1109/ACCESS.2017.2778011.

35.

Ullah,

Muhammad,

Ding,

Palade,

I.U.

Haq and

S.W.

Baik, Efficient activity recognition using lightweight CNN and DS-GRU network for surveillance applications, Applied Soft Computing 103 (2021), 107102. doi:10.1016/j.asoc.2021.107102.

36.

M.S.

Uzzaman,

Debnath,

M.A.

Uddin,

M.M.

Islam,

M.A.

Talukder and

Parvez, LRCN based human activity recognition from video data, SSRN Electronic Journal (2022), preprint. doi:10.2139/ssrn.4173741.

37.

Vrskova,

Hudec,

Kamencay and

Sykora, Human activity classification using the 3DCNN architecture, Applied Sciences 12(2) (2022), 931. doi:10.3390/app12020931.

38.

Wang,

Klaser,

Schmid and

C.L.

Liu, Dense trajectories and motion boundary descriptors for action recognition, Int. J. Comput. Vis. 103(1) (2013), 60–79. doi:10.1007/s11263-012-0594-8.

39.

Wang,

Li,

Wu,

Li,

Snoussi and

Wu, ResLNet: Deep residual LSTM network with longer input for action recognition, Frontiers of Computer Science 16(6) (2022), 1–9.

40.

Xiao,

Xu,

Xing,

Song,

Wang and

Zhao, A federated learning system with enhanced feature extraction for human activity recognition, Knowledge-Based Systems 229 (2021), 10733. doi:10.1016/j.knosys.2021.107359.

41.

S.K.

Yadav,

Tiwari,

H.M.

Pandey and

S.A.

Akbar, Skeleton-based human activity recognition using ConvLSTM and guided feature learning, Soft Computing 26(2) (2022), 877–890. doi:10.1007/s00500-021-06238-7.

42.

Yilmaz and

Shah, Actions sketch: A novel action representation, in: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2005, pp. 984–990.

43.

Zhang,

Li,

Tai,

Wen and

Jiang, Motion video recognition in speeded-up robust features tracking, Electronics 11(18) (2022), 2959. doi:10.3390/electronics11182959.

44.

Zhang,

C.P.

Lim and

Yu, Intelligent human action recognition using an ensemble model of evolving deep networks with swarm-based optimization, Knowledge-Based Systems 220 (2021), 106918. doi:10.1016/j.knosys.2021.106918.

45.

P.F.

Zhang,

K.Z.

He,

Z.Z.

OuYang and

J.Y.

Zhang, Multifunctional intelligent outdoor mobile robot testbed-THMR-V, Robot 24(02) (2002), 97–101.

46.

Zhang and

Li, A deep reinforcement learning based human behavior prediction approach in smart home environments, in: Proceedings of the 2019 International Conference on Robots & Intelligent System (ICRIS), 2019.

47.

Z.F.

Zhang,

Z.M.

Lv,

C.Q.

Gan and

Q.Y.

Zhu, Human action recognition using convolutional LSTM and fully-connected LSTM with different attentions, Neurocomputing 410 (2020), 304–316. doi:10.1016/j.neucom.2020.06.032.

48.

Zhe,

Simon,

S.E.

Wei and

Sheikh, Realtime multi-person 2D pose estimation using part affinity fields, in: Proceedings of the IEEE 2017 Conference on Computer Vision and Pattern Recognition (CVPR), 2017.

49.

Zhu,

Chen and

Brown, A sequence-to-sequence model-based deep learning approach for recognizing activity of daily living for senior care, J. Biomed. Inform 84 (2018), 148–158. doi:10.1016/j.jbi.2018.07.006.

50.

Zhu,

Vial and

Lu, Tornado: A spatio-temporal convolutional regression network for video action proposal, in: Proceedings of the IEEE International Conference on Computer Vision, 2017.

51.

Zhu,

Lan,

Xing,

Zeng,

Li,

Shen and

Xie, Co-occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 30, 2016.