Application of Spatial Temporal Graph Convolutional Networks in Human Abnormal Behavior Recognition

doi:10.3778/j.issn.1002-8331.2107-0441

Abstract

Abstract: Under the monitoring scenario, due to the shortage of monitoring resources, abnormal pedestrian behavior is prone to miss detection. In view of this problem, a method to identify abnormal human behavior in video surveillance scene is proposed to assist monitoring personnel to find abnormal behavior in time. The human skeleton is extracted from the pedestrians in the image using OpenPose. The graph attention mechanism based on the graph attention network （GAT） is integrated to solve the problem that the graph convolution network has a single way to aggregate the features of joint points. On the basis of the improved graph convolutional network, the spatial temporal graph convolutional networks（ST-GCN） is used to identify the abnormal behavior of pedestrians joint points information. The experimental results show that the recognition algorithm proposed in this paper has an accuracy of 85.48% for the defined behavior, and can accurately identify the abnormal behavior of pedestrians in surveillance video.

Key words: OpenPose algorithm, spatial temporal graph convolutional networks（ST-GCN）, graph attention mechanism, , behavior recognition

摘要： 在监控场景下，由于监控资源短缺，行人异常行为容易发生漏检。针对该问题，提出了一种视频监控场景下的人体异常行为识别的方法，辅助监控人员及时发现异常。使用OpenPose对图像中行人进行人体骨架提取。针对图卷积网络对关节点特征聚合方式单一的问题，融合了基于图注意力网络（graph attention network，GAT）的图注意力机制。在改进后的图卷积网络的基础上，利用时空图卷积神经网络（spatial temporal graph convolutional networks，ST-GCN），对行人关节点信息进行异常行为识别。实验结果表明，提出的识别算法对定义的行为识别准确率达85.48%，能够准确地识别监控视频中行人的异常行为。

关键词: OpenPose算法, 时空图卷积网络, 图注意力机制, 行为识别

ZHANG Weilan, QI Hua, LI Sheng. Application of Spatial Temporal Graph Convolutional Networks in Human Abnormal Behavior Recognition[J]. Computer Engineering and Applications, 2022, 58(12): 122-131.

张蔚澜, 齐华, 李胜. 时空图卷积网络在人体异常行为识别中的应用[J]. 计算机工程与应用, 2022, 58(12): 122-131.

References

[1] 裴利沈，刘少博，赵雪专.人体行为识别研究综述[J].计算机科学与探索，2022，16（2）：305-322.
PEI L S，LIU S B，ZHAO X Z.Review of human behavior recognition research[J].Journal of Frontiers of Computer Science and Technology，2022，16（2）：305-322.
[2] 李扬志，袁家政，刘宏哲.基于时空注意力图卷积网络模型的人体骨架动作识别算法[J].计算机应用，2021，41（7）：1915-1921.
LI Y Z，YUAN J Z，LIU H Z.Human skeleton-based action recognition algorithm based on spatio-temporal attention graph convolutional network model[J].Journal of Computer Applications，2021，41（7）：1915-1921.
[3] SIMONYAN K，ZISSERMAN A.Two-stream convolu-tional networks for action recognition in videos[C]//Advances in Neural Information Processing Systems.Montreal：MIT Press，2014：568-576.
[4] SUN L，JIA K，YEUNG D Y，et al.Human action recog-nition using factorized spatio-temporal convolutional networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：4597-4605.
[5] WANG L，QIAO Y，TANG X.Action recognition with tra-jectory-pooled deep-convolutional descriptors[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Boston：IEEE，2015：4305-4314．
[6] WANG L，XIONG Y，WANG Z，et al.Temporal segment networks：Towards good practices for deep action recog-nition[C]//European Conference on Computer Vision（ECCV），2016：20-36.
[7] ZHAO Y，XIONG Y，WANG L，et al.Temporal action detection with structured segment networks[C]//Proceedings of IEEE International Conference on Computer Vision.Venice：IEEE，2017：2914-2923.
[8] LIU T，DONG X，WANG Y，et al.Double-layer conditional random fields model for human action recognition[J].Signal Processing：Image Communication，2020，80：115672.
[9] JI S，XU W，YANG M，et al.3D convolutional neural networks for human action recognition[J].IEEE Transactions on Pattern Analysis And Machine Intelligence，2012，35（1）：221-231.
[10] TRAN D，BOURDEV L，FERGUS R，et al.Learning spatiotemporal features with 3d convolutional networks[C]//Proceedings of the IEEE International Conference on Computer Vision.Santiago，Chile：IEEE，2015：4489-4497.
[11] JI X，WANG C，LI Y，et al.Hidden Markov model-based human action recognition using mixed features[J].Journal of Computational Information Systems，2013，9：3659-3666.
[12] ZHANG S，LIU X，XIAO J.On geometric features for skeleton-based action recognition using multilayer LSTM networks[C]//Proceedings of 2017 IEEE Winter Conference on Applications of Computer Vision.California：IEEE，2017：148-157.
[13] LI C，ZHONG Q，XIE D，et al.Skeleton-based action recognition with convolutional neural networks[C]//Proceedings of 2017 IEEE International Conference on Multimedia & Expo Workshops.HongKong：IEEE，2017：597-600.
[14] KE Q，BENNAMOUN M，AN S，et al.A new representation of skeleton sequences for 3d action recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Hawaii：IEEE，2017：3288-3297.
[15] 曹毅，刘晨，盛永健，等.基于三维图卷积与注意力增强的行为识别模型[J].电子与信息学报，2021，43（7）：2071-2078.
CAO Y，LIU C，SHENG Y J，et al.Action recognition model based on 3d graph convolution and attention enhanced[J].Journal of Electronics & Information Technology，2021，43（7）：2071-2078.
[16] YAN S J，XIONG Y J，LIN D H.Spatial temporal graph convolutional networks for skeleton-based action[C]//Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence.Louisiana：AAAI，2018：7444-7452.
[17] KAY W，CARRERIA J，SIMONYAN K，et al.The kinetics human action video dataset[J].arXiv：1705.06950，2017.
[18] LIU J，SHAHROUDY A，PEREZ M，et al.NTU RGB+D 120：a large scale benchmark for 3D human activity understanding[C]//IEEE Transactions on Pattern Analysis and Machine Intelligence（TPAMI），2020，42（10）：2684-2701.
[19] CAO Z，HIDALGO G，SIMON T，et al.OpenPose：realtime multi-person 2D pose estimation using part affinity fields[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Hawaii：IEEE，2017：7291-7299.
[20] 陈皓，肖利雪，李广，等.基于人体关节点数据的攻击性行为识别[J].计算机应用，2019，39（8）：2235-2241.
CHEN H，XIAO L X，LI G，et al.Aggressive behavior recognition based on human joint point data[J].Journal of Computer Applications，2019，39（8）：2235-2241.
[21] 李梦荷，许宏吉，石磊鑫，等.基于骨骼关键点检测的多人行为识别[J].计算机科学，2021，48（4）：138-143.
LI M H，XU H J，SHI L X，et al.Multi-person activity recognition based on bone keypoints detection[J].Computer Science，2021，48（4）：138-143.
[22] VELICKOVIC P，CUCURULL G，CASANOVA A，et al.Graph attention networks[J].arXiv：1710.10903，2017.