A working human abnormal operation recognition approach based on the deep multi-instance sorting model

Abstract

Modeling abnormal behavior detection in the intelligent video monitoring system for recognition or detection of special event has attracted significant research interest in recent years. In order to achieve more effective recognition of abnormal behavior detection of the intelligent video surveillance system, this paper proposes a working human abnormal operation recognition approach based on deep multi-instance sorting model. Firstly, the uncut long video is sparse to obtain normal and abnormal behavior video segments, and the RGB and optical flow features in the segment are extracted by the deep convolution network. Video feature vectors are obtained by the consensus function and feature extractor. Then, multi-instance sorting learning is used to assign abnormality scores between 0 and 1 to feature vectors. When the abnormal values of abnormal packets are higher than those of normal packets, the abnormality score is returned and the high abnormality score is determined as an abnormal behavior. The experiments on the open THUMOS14 data set and our own XAGCWD data set using CUDA GPU accelerated computing to demonstrate that our approach improves the recognition accuracy about 10.8% and high accuracy of abnormal detection. The main purpose of this research is to apply the model proposed in this paper to the intelligent workshop behavior monitoring system to effectively realize the safety management of workshop personnel.

Keywords

Intelligent video surveillance abnormal behavior detection feature extractor convolutional neural network

Get full access to this article

View all access options for this article.

References

Rude DJ, Adams S and Beling PA. Task recognition from joint tracking data in an operational manufacturing cell. J Intell Manuf 2015; 1–15.

Zhang

. Industrial 4.0 and intelligent manufacturing. Mech Des Manuf Eng 2014; 43: 1–5.

Chaudhary

Khan

Bhatnagar

. Multiple abnormal activity detection in videos. Procedia Comput Sci 2018; 125: 336–345.

Sultani W, Chen C and Shah M. Real-world anomaly detection in surveillance videos. In: The IEEE conference on computer vision and pattern recognition (CVPR), 2018, pp.6479–6488. New York: IEEE.

Zhao B, Li FF and Xing EP. Online detection of unusual events in videos via dynamic sparse coding. In: IEEE conference on computer vision and pattern recognition, 2011, pp.3313–3320. Washington DC: IEEE Computer Society.

Zhang B, Wang L, Wang Z, et al. Real-time action recognition with enhanced motion vector CNNs. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), 2016, pp.2718–2726. Washington DC: IEEE Computer Society.

Karpathy A, Toderici G, Shetty S, et al. Large-scale video classification with Convolutional Neural Networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp.1725–1732. New York: IEEE.

Wang

Liu

Wang

, et al. Deep learning-based human motion recognition for predictive context-aware human-robot collaboration. CIRP Ann 2018; 67: 17–20.

Yang

, et al. 3D convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Machine Intell 2013; 35: 221–231.

10.

Ding

, et al. Workflow recognition method based on 3D convolutional neural networks. Comput Integr Manuf Syst 2018; 24: 1747–1757.

11.

Simonyan K and Zisserman A. Two-stream convolutional networks for action recognition in videos. In: Advances in neural information processing systems. Montreal: NIPS Press, 2014, pp.568–576.

12.

Li CK, Wang PC, Wang S, et al. Skeleton-based action recognition using LSTM and CNN. In: Proceedings of IEEE international conference on multimedia & expo, 2017, pp.585–590. New York: IEEE.

13.

Wang L, Xiong Y, Wang Z, et al. Temporal segment networks: towards good practices for deep action recognition. In: Proceedings of the European conference on computer vision. Cham: Springer, 2016, pp.20–36.

14.

Karpathy A, Toderici G, Shetty S, et al. Large-scale video classification with convolutional neural networks. In: 2014 IEEE conference on computer vision and pattern recognition (CVPR), 2014. New York: IEEE.

15.

Hasan M, Choi J, Neumann J, et al. Learning temporal regularity in video sequences. In: The IEEE conference on computer vision and pattern recognition (CVPR), 2016, pp.733–742. New York: IEEE.

16.

Li C, Han Z, Ye Q, et al. Abnormal behavior detection via sparse reconstruction analysis of trajectory. In: International conference on image & graphics, 2011, pp.807–810. New York: IEEE.

17.

18.

Andrews

Tsochantaridis

Hofmann

. Support vector machines for multiple-instance learning. Adv Neural Inform Process Syst 2002; 15: 561–568.

19.

Idrees H, Zamir AR, Jiang YG, et al. The THUMOS challenge on action recognition for videos “in the wild”. Comput Vision Image Understand 2017; 155: 1–23.

20.

Simonyan K and Zisserman A. Very deep convolutional networks for large-scale image recognition. Comput Sci 2014; 1409–1556.

21.

Peng X, Wang L, Wang X, et al. Bag of visual words and fusion methods for action recognition: comprehensive study and good practice. Comput Vision Image Understand 2016; 150: 109–125.

22.

Sanin A, Sanderson C, Harandi MT, et al. Spatio-temporal covariance descriptors for action and gesture recognition. In: Proceedings of IEEE workshop on applications of computer vision, 2013, pp.103–110. New York: IEEE.