Though deep learning networks have proven ability to perform video analytics in complex environments, there is an increased attention towards the development of compact networks which would facilitate edge processing and the result of which have yielded high performance compressed deep learning networks such as, MobileNet, PWCNet and BindsNet. In the work proposed herein, a dual network configuration is used for human action recognition, wherein, the MobileNet captures the spatial appearance of the action sequences and the PWCNet is used to extract the motion vectors. A novel Spiking Neural Network (SNN) based configuration is used as the classifier and the SNN implementation is based on BindsNet. The proposed configuration is experimentally validated on challenging datasets, viz., HMDB51 and UCF101. The experimental results demonstrate that the proposed work is superior to the state-of-the-art techniques and comparable in few cases.
LaptevI., On space-time interest points, Int J Comput Vis64(2-3) (2005), 107–123.
2.
BerlinS.J., JohnM., Human interaction recognition through deep learning network, In Proceedings of the IEEE international Carnahan conference on Security Technology (ICCST), Orlando, FL, USA, (2016), pp. 1–4.
3.
AslanM.F., DurduA., SabanciK., Human action recognition with bag of visual words using different machine learning methods and hyperparameter optimization, Neural Comput Appl (2019), 1–13.
4.
ScovannerP., AliS., ShahM., A 3-dimensional sift descriptor and its application to action recognition, In Proceedings of the 15th ACM international conference on Multimedia, ACM, Augsburg, Bavaria, Germany (2007), 357–360.
5.
KlaserA., MarszałekM., SchmidC., A spatio-temporal descriptor based on 3d-gradients, In Proceedings of the 19th British Machine Vision Conference (BMVC2008), Leeds, UK, (2008), pp. 1–10.
6.
El-GhaishH., HusseinM.E., ShoukryA., OnaiR., Human Action Recognition Based on Integrating Body Pose, Part Shape, and Motion, IEEE Access6 (2018), 49040–49055.
7.
WangH., SchmidC., Action recognition with improved trajectories, In Proceedings of the IEEE International Conference on Computer Vision (2013), 3551–3558.
8.
AnF.P., Human Action Recognition Algorithm Based on Adaptive Initialization of Deep Learning Model Parameters and Support Vector Machine, IEEE Access6 (2018), 59405–59421.
9.
BouwmansT., Garcia-GarciaB., Background Subtraction in Real Applications: Challenges, Current Models and Future Directions, arXiv preprint arXiv:1901.03577 (2019).
10.
EllisC., MasoodS.Z., TappenM.F., LaViolaJ.J., SukthankarR., Exploring the trade-off between accuracy and observational latency in action recognition, Int J Comput Vis101(3) (2013), 420–436.
11.
LiuY., WuQ., TangL., ShiH., Gaze-assisted multi-stream deep neural network for action recognition, IEEE Access5 (2017), 19432–19441.
12.
GaoY., XiangX., XiongN., HuangB., LeeH.J., AlrifaiR., JiangX., FangZ., Human action monitoring for healthcare based on deep learning, IEEE Access6 (2018), 52277–52285.
13.
LadjailiaA., BouchrikaI., MerouaniH.F., HarratiN., MahfoufZ., Human activity recognition via optical flow: decomposing activities into basic actions, Neural Comput Appl (2019), 1–14.
14.
TranD., BourdevL., FergusR., TorresaniL., PaluriM., Learning spatiotemporal features with 3d convolutional networks, In Proceedings of the IEEE international conference on Computer Vision (ICCV), Santiago, Chile, (2015), pp. 4489–4497.
15.
JiS., XuW., YangM., YuK., 3D convolutional neural networks for human action recognition, IEEE Trans Pattern Anal Mach Intell35(1) (2013), 221–231.
16.
YaoG., LeiT., ZhongJ., JiangP., Learning multi-temporal-scale deep information for action recognition, Appl Intell49(6) (2019), 2017–2029.
17.
ZhangL., WuX., LuoD., Human activity recognition with HMM-DNN model, In Proceedings of the IEEE international conference on Cognitive Informatics & Cognitive Computing (ICCI* CC), Beijing, China, (2015), 192–197.
18.
UllahA., MuhammadK., Del SerJ.,
BaikS.W. and
AlbuquerqueV., Activity recognition using temporal optical flow convolutional features and multi-layer LSTM, IEEE Trans Ind Electron (2018).
19.
ArifogluD., BouchachiaA., Activity recognition and abnormal behaviour detection with recurrent neural networks, Procedia Comput Sci110 (2017), 86–93.
20.
SzegedyC., LiuW., JiaY., SermanetP., ReedS., AnguelovD., ErhanD., VanhouckeV., RabinovichA., Going deeper with convolutions, In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, (2015), pp. 1–9.
21.
WangL., XuY., ChengJ., XiaH., YinJ., WuJ., Human action recognition by learning spatio-temporal features with deep neural networks, IEEE Access6 (2018), 17913–17922.
22.
TongM., LiM., BaiH., MaL., ZhaoM., DKD–DAD: a novel framework with discriminative kinematic descriptor and deep attention-pooled descriptor for action recognition, Neural Comput Appl (2019), 1–18.
23.
HanY., ZhangP., ZhuoT., HuangW., ZhangY., Going deeper with two-stream ConvNets for action recognition in video surveillance, Pattern Recogn Lett107 (2018), 83–90.
24.
JiaY., ShelhamerE., DonahueJ., KarayevS., LongJ., GirshickR., GuadarramaS., DarrellT., Caffe:Convolutional architecture for fast feature embedding, In Proceedings of the 22nd ACM international conference on Multimedia, ACM, NewYork, USA, (2014), pp. 675–678.
25.
HeK., ZhangX., RenS., SunJ., Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, (2015), pp. 1026–1034.
26.
NikoueiS.Y., ChenY., SongS., XuR., ChoiB.Y., FaughnanT.R., Real-time human detection as an edge service enabled by a lightweight cnn, In Proceedings of the 2018 IEEE International Conference on Edge Computing (EDGE), San Francisco, CA, USA, (2018), pp. 125–129.
27.
IandolaF.N., HanS., MoskewiczM.W., AshrafK., DallyW.J., KeutzerK., SqueezeNet: AlexNet-level accuracy with 50× fewer parameters and <0.5 MB model size. arXiv preprint arXiv:1602.07360, (2016).
28.
SunD., YangX., LiuM.Y., KautzJ., PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume, In Proceedings of the IEEE/CVF international conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, (2018), pp. 8934–8943.
29.
HazanH., SaundersD.J., KhanH., SanghaviD.T., SiegelmannH.T., KozmaR., BindsNET: A machine learning-oriented spiking neural networks library in Python, Front Neuroinform12, (2018), 89.
30.
IlgE., MayerN., SaikiaT., KeuperM., DosovitskiyA., BroxT., Flownet 2.0: Evolution of optical flow estimation with deep networks, In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, (2017), pp. 2462–2470.
31.
BengioY., LeeD.H., BornscheinJ., MesnardT., LinZ., Towards biologically plausible deep learning, arXiv preprint arXiv:1502.04156, (2015).
MaM., MarturiN., LiY., LeonardisA., StolkinR., Region-sequence based six-stream CNN features for general and fine-grained human action recognition in videos, Pattern Recognit76, (2018), 506–521.
34.
LiQ., QiuZ., YaoT., MeiT., RuiY., LuoJ., Action recognition by learning deep multi-granular spatio-temporal video representation, In Proceedings of the ACM Int Conf on Multimedia Retrieval (ICMR), New York, USA, (2016), 159–166.
35.
SunL., JiaK., YeungD.Y., ShiB.E., Human action recognition using factorized spatio-temporal convolutional networks, In Proceedings of the IEEE international conference on Computer Vision (ICCV), Santiago, Chile, (2015), 4597–4605.
36.
BilenH., FernandoB., GavvesE., VedaldiA., Action recognition with dynamic image networks, IEEE Trans Pattern Anal Mach Intell40(12), (2017), 2799–2813.
37.
MajdM., SafabakhshR., A motion-aware ConvLSTM network for action recognition, Appl Intell49, (2019), 2515–2521.
38.
WangL., XiongY., WangZ., QiaoY., LinD., TangX., Van GoolL., Temporal segment networks: Towards good practices for deep action recognition, In Proceedings of the European Conference on Computer Vision(ECCV), Springer, Cham, (2016), pp. 20–36.
39.
VarolG., LaptevI., SchmidC., Long-term temporal convolutions for action recognition, IEEE Trans Pattern Anal Mach Intell40(6), (2017), 1510–1517.
40.
ZareA., MoghaddamH.A., SharifiA., Video spatiotemporal mapping for human action recognition by convolutional neural network, Pattern Anal Appl, (2019), 1–15.
41.
JhuangH., SerreT., WolfL., PoggioT., A biologically inspired system for action recognition, In Proceedings of the international conference on Computer Vision (ICCV), (2007).
42.
DhobleK., NuntalidN., IndiveriG., KasabovN., Online spatio-temporal pattern recognition with evolving spiking neural networks utilising address event representation, rank order, and temporal spike learning, In Proceedings of the IEEE international joint conference on Neural Networks (IJCNN), Brisbane, QLD, Australia, (2012), pp. 1–7.
43.
MengY., JinY., YinJ., Modeling activity-dependent plasticity in BCM spiking neural networks with application to human behavior recognition, IEEE Trans on Neural Netw22(12) (2011), 1952–1966.
44.
LiuH., ShuN., TangQ., ZhangW., Computational model based on neural network of visual cortex for human action recognition, IEEE Trans Neural Netw Learn Syst29(5) (2018), 1427–1440.
45.
HuangG., LiuZ., Van Der MaatenL. and
WeinbergerK.Q., Densely connected convolutional networks, In Proceedings of the IEEE international conference on Computer Vision and Pattern Recognition (CVPR), (2017), pp. 4700–4708.
DiehlP.U., CookM., Unsupervised learning of digit recognition using spike-timing-dependent plasticity, Frontiers in Comput Neurosci9 (2015), 99.
48.
LeeJ.H., DelbruckT., PfeifferM., Training deep spiking neural networks using backpropagation, Front Neurosci10 (2016), 508.
49.
SoomroK., ZamirA.R., ShahM., UCF101: A dataset of 101 human actions classes from videos in the wild, arXiv preprint arXiv:1212.0402, (2012).
50.
KuehneH., JhuangH., GarroteE., PoggioT., SerreT., HMDB: a large video database for human motion recognition, In Proceedings of the IEEE international conference on Computer Vision, Barcelona, Spain, (2011), 2556–2563.
51.
DalalN., TriggsB., Histograms of oriented gradients for human detection, In Proceedings of the IEEE Computer Society conference on Computer Vision and Pattern Recognition (CVPR), San Diego, CA, USA, (2005).
52.
ZhangH., XinM., WangS., YangY., ZhangL., WangH., End-to-end temporal attention extraction and human action recognition, Mach Vision Appl29(7) (2018), 1127–1142.
