The scene complexity and derived social risks of cluster behavior are far greater than individual behavior. Cluster behavior recognition has emerged as a key research area in both computer vision and public safety. Methods in this domain have seen continuous advancements, with steady improvements in recognition speed and accuracy. This paper provides a comprehensive review of cluster behavior recognition technology as documented in recent literature. Based on feature sources, recognition methods are categorized into manually designed approaches and machine learning-based models. Manually designed approaches include methods focused on overall cluster characteristics and those analyzing individual behaviors within groups. Machine learning models are further subdivided into non-interaction modeling, global member interaction modeling, and key individual interaction modeling. Key technologies in machine learning for cluster behavior recognition are also examined, with emphasis on feature extraction techniques like Long Short-Term Memory Networks, 3D Convolutional Neural Networks, and Inflated 3D Convolutional Neural Networks, as well as interaction modeling approaches using Graph Convolutional Networks and Graph Attention Networks. Finally, this paper discusses the problems that need to be solved urgently and prospects the future development trend of cluster behavior recognition.
DengCKangXZhuZ, et al.Behavior recognition based on category subspace in crowded videos. IEEE Access2020; 8: 222599–222610.
2.
QiaoSWangLGaoZ. Group behavior recognition based on deep hierarchical network. Neural Comput Appl2020; 32(10): 5389–5398.
3.
SchneiderAKruegerEVollenwyderB, et al.Understanding the relations between crowd density, safety perception and risk-taking behavior on train station platforms: A case study from Switzerland. Transp Res Interdiscip Perspect2021; 10: 100390.
4.
WieczorekGTahirSBUDAkhterI, et al.Vehicle detection and recognition approach in multi-scale traffic monitoring system via graph-based data optimization. Sensors2023; 23(3): 1731.
5.
WangYBizuBPraveenaY. Deep learning based smart monitoring of indoor stadium video surveillance. J Mult-Valued Log S2021; 36: 151–167.
6.
PangC. Simulation of student classroom behavior recognition based on cluster analysis and random forest algorithm. J Intell Fuzzy Syst2021; 40(2): 2421–2431.
7.
SelvakumarS. An effective framework of human abnormal behaviour recognition and tracking using multiscale dilated assisted residual attention network. Expert Syst Appl2024; 247: 123264.
8.
RezaeeKRezakhaniSMKhosraviMR, et al.A survey on deep learning-based real-time crowd anomaly detection for secure distributed video surveillance. Personal Ubiquitous Comput2024; 28(1): 135–151.
9.
LiYYangGSuZ, et al.Human activity recognition based on multi environment sensor data. Inf Fusion2023; 91: 47–63.
10.
QiMWangYQinJ, et al.stagNet: An attentive semantic RNN for group activity and individual action recognition. IEEE T Circ. Syst Vid2020; 30(2): 549–565.
11.
ZhouYNiBYanS, et al.Recognizing pair-activities by causality analysis. ACM T Intel Syst Tec2011; 2(1): 1–20.
12.
NiBYanSKassimA. Recognizing human group activities with localized causalities. In: IEEE conference on computer vision and pattern recognition, Miami, FL, USA, 20–25 June 2009, pp. 1470–1477.
13.
ZhangYGeWChangMC, et al.Group context learning for event recognition. In: IEEE workshop on the applications of computer vision, Breckenridge, CO, 09–11 January 2012, pp. 249–255.
14.
ChengZQinLHuangQ, et al.Group activity recognition by Gaussian processes estimation. In: 20th international conference on pattern recognition, Istanbul, Turkey, 23–26 August 2010, pp. 3228–3231.
15.
ZhangCYangXLinW, et al.Recognizing human group behaviors with multi-group causalities. In: IEEE/WIC/ACM International conferences on web intelligence and intelligent agent technology, Macau, China, 04–07 December 2012, pp. 44–48.
16.
Azorin-LopezJSaval-CalvoMFuster-GuilloA, et al.Group activity description and recognition based on trajectory analysis and neural networks. In: International joint conference on neural networks, Vancouver, BC, 24–29 July 2016, pp. 1585–1592.
17.
KimYJChoNGLeeSW. Group activity recognition with group interaction zone. In: 22nd international conference on pattern recognition, Stockholm, Sweden, 24–28 August 2014, pp. 3517–3521.
18.
DalalNTriggsB. Histograms of oriented gradients for human detection. In: IEEE computer society conference on computer vision and pattern recognition, San Diego, CA, USA, pp. 20886-25893June 2005.
19.
LoweDG. Distinctive image features from scale-invariant keypoints. Int J Comput Vis2004; 60(2): 91–110.
20.
ChoiWShahidKSavareseS. What are they doing? collective activity classification using spatio-temporal relationship among people. In: IEEE 12th international conference on computer vision workshops, Kyoto, Japan, 27 September 2009–04, pp. 1282–1289.
21.
ChoiWShahidKSavareseS. Learning context for collective activity recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Colorado Springs, CO, USA, 20–25 June 2011, pp. 3273–3280.
22.
LanTWangYMoriG, et al.Retrieving actions in group contexts. In: European conference on computer vision. Heraklion, Crete, Greece, 5-11 September 2010, 2010, pp. 181–194.
23.
KanekoTShimosakaMOdashimaS, et al.Viewpoint invariant collective activity recognition with relative action context. In: European conference on computer vision. Florence, Italy,7-13 October 2012, 2012, pp. 253–262.
24.
AmerMRTodorovicS. A chains model for localizing participants of group activities in videos. In: International conference on computer vision, Barcelona, Spain, 06–13 November 2011, pp. 786–793.
25.
LanTWangYYangW, et al.Discriminative latent models for recognizing contextual group activities. IEEE T Pattern Anal2012; 34(8): 1549–1562.
26.
AmerMRXieDZhaoM, et al.Cost-sensitive top-down/bottom-up inference for multiscaleactivity recognition. In: European conference on computer vision. Florence, Italy, 7-13 October 2012, 2012, pp. 187–200.
27.
LanTSigalLMoriG. Social roles in hierarchical models for human activity recognition. In: IEEE conference on computer vision and pattern recognition, Providence, RI, USA, 16–21 June 2012, pp. 1354–1361.
28.
AmerMRLeiPTodorovicS. HiRF: Hierarchical random field for collective activity recognition in videos. In: European conference on computer vision. Zurich, Switzerland, 6-12 September 2014, 2014, pp. 572–585.
29.
IbrahimMSMuralidharanSDengZ, et al.A hierarchical deep temporal model for group activity recognition. In: IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 27–30 June 2016, pp. 1971–1980.
30.
BagautdinovTAlahiAFleuretF, et al.Social scene understanding: End-to-end multi-person action localization and collective activity recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition,Honolulu, HI, USA, 21-26July 2017, 2017, pp. 3425–3434.
31.
GammulleHDenmanSSridharanS, et al.Multi-level sequence GAN for group activity recognition. In: Asian conference on computer vision. Perth, WA, Australia, 2-6 December 2018, 2018, pp. 331–346.
32.
WuLYangZHeJ, et al.Ontology-based global and collective motion patterns for event classification in basketball videos. IEEE T Circ Syst Vid2020; 30(7): 2178–2190.
33.
ShuXZhangLSunY, et al.Host–parasite: graph LSTM-in-LSTM for group activity recognition. IEEE T Neur Net Lear2021; 32(2): 663–674.
34.
WangCHuXMengW, et al.Research on group behavior recognition method based on multi-stream architecture and long short-term memory network. Chin J Electron2020; 48(4): 800–804.
35.
DengZVahdatAHuH, et al.Structure inference machines: Recurrent neural networks for analyzing relations in group activity recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 27–30 June 2016, pp. 4772–4781.
36.
XuDFuHWuL, et al.Group activity recognition by using effective multiple modality relation representation with temporal-spatial attention. IEEE Access2020; 8: 65689–65698.
37.
HuGCuiBHeY, et al.Progressive relation learning for group activity recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Seattle, WA, USA, 13–19 June 2020, pp. 980–989.
38.
LiuBQingLWangZ, et al.Group activity recognition based on partitioned attention mechanism and interactive position relationship. J Comput Appl2022; 42(7): 2052–2057.
39.
LuLLuYYuR, et al.GAIM: Graph attention interaction model for collective activity recognition. IEEE Trans Multimed2020; 22(2): 524–539.
40.
WangCXueH. Group activity recognition based on GFU and HierarchicaI LSTM. Chin J Electron2020; 48(8): 1465–1471.
41.
ZhangPTangYHuJF, et al.Fast collective activity recognition under weak supervision. IEEE Trans Image Process2020; 29: 29–43.
42.
KuangZTieX. Improved actor relation graph based group activity recognition. In International Conference on Smart Multimedia arseille, France, 25-27 August 2022,2020. DOI: 10.48550/arXiv.2010.12968.
43.
DuanYWangJ. Learning key actors and their interactions for group activity recognition. In: Chinese conference on pattern recognition and computer vision. Berlin: Springer, 2021, pp. 53–65.
44.
TsaiTYLinYYJengSK, et al.End-to-End key-player-based GroupActivity recognition network applied to basketball offensive tactic identification inLimited data scenarios. IEEE Access2021; 9: 104395–104404.
45.
JiSXuWYangM, et al.3D convolutional neural networks for human action recognition. IEEE T Pattern Anal2012; 35(1): 221–231.
46.
CarreiraJZissermanA. Quo Vadis, action recognition? a new model and the kinetics dataset. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, HI, USA, 21–26 July 2017, pp. 6299–6308.
47.
FengYShanSLiuY, et al.DRGCN: Deep relation GCN for group activity recognition. In: 27th international conference on neural information processing. Bangkok: Springer, 2020, pp. 361–368.
48.
VelikoviPCucurullGCasanovaA, et al.Graph attention networks. In: International Conference on Learning Representations (ICLR), Vancouver, BC, Canada, 30 April - 3May 2018, pp. 1–12.
