Human Skeleton Action Recognition Model Integrated Internal and External Dependences

doi:10.3778/j.issn.1002-8331.2207-0198

Abstract

Abstract: Human action recognition based on the dynamic skeleton graph is a hot research field in computer vision. Traditional recognition methods are mostly based on the local natural physical connection in the human skeleton（internal dependence）. However, many implicit non-local joint connections such as interaction between hands and feet is also important for human motion recognition. In this paper, the concept of external dependence is proposed to represent this implicit non-physical connection, a skeleton graph can be processed through the mechanism of internal dependence and external dependence, and the spatial graph convolutional calculation can be completed by integrating internal and external dependences. Further, by adding an appropriate time convolution module, a spatial temporal graph convolutional networks integrated internal and external dependencies（IED-STGCN） is designed in this paper. Experiments show that IED-STGCN has higher recognition accuracies than ones of the existing spatial temporal graph convolutional networks（ST-GCN） models. For example, on the Kinetics dataset, it is about 2.5?percentage points higher; on the X-Sub dataset, 3.4?percentage points higher; and on X-View dataset, 3.8?percentage points higher. The main techniques in this study include temporal convolution（TC）, Internal dependencies graph convolution（IGC）, and external dependencies convolution（EGC）. In order to verify their effectiveness, related technical ablation experiments are completed.

Key words: action recognition, spatial temporal graph convolution networks, internal dependence, external dependence

摘要： 基于动态骨架图的人体动作识别是计算机视觉领域中的一个研究热点。传统识别方法大多是建立在人体骨架的局部自然物理连接上（内在依赖）。然而，许多隐含的非局部的关节连接有时对于人体动作识别是不可忽略的，如手脚的互动等。引入外在依赖概念来表示这种隐式的非物理连接，并通过内、外依赖机制来处理骨架图，完成内、外依赖的空间图卷积融合。通过设计合适的时间卷积模块，进一步构建融合内外依赖的的时空图卷积网络（IED-STGCN）。实验表明，IED-STGCN在Kinetics数据集上的识别精度比现有的时空图卷积网络（ST-GCN）提升了2.5个百分点，在X-Sub和X-View两个数据集上的识别精度分别比现有的ST-GCN模型提升了3.4和3.8个百分点。该研究的主要技术有时间卷积（TC）、内在依赖图卷积（IGC）以及外在依赖图卷积（EGC）等，通过消融实验说明了这些技术的有效性。

关键词: 动作识别, 时空图卷积网络, 内在依赖, 外在依赖

MAO Guojun, WANG Yijin. Human Skeleton Action Recognition Model Integrated Internal and External Dependences[J]. Computer Engineering and Applications, 2023, 59(21): 132-140.

毛国君, 王一锦. 融合内外依赖的人体骨架动作识别模型[J]. 计算机工程与应用, 2023, 59(21): 132-140.

References

[1] NWAKANMA C I，ISLAM F B，MARESKA M P，et al.Iot-based vibration sensor data collection and emergency detection classification using long short term memory（LSTM）[C]//2021 International Conference on Artificial Intelligence in Information and Communication（ICAIIC），2021：273-278.
[2] XIE Y Z，TANG Z H，SONG A G.Motion simulation and human-computer interaction system for lunar exploration[J].Applied Sciences，2022，12（5）：2312.
[3] 朱相华，智敏，殷雁君.基于2D CNN和Transformer的人体动作识别[J].电子测量技术，2022，45（15）：123-129.
ZHU X H，ZHI M，YIN Y J.Human action recognition based on 2D CNN and Transformer[J].Electronic Measurement Technology，2022，45（15）：123-129.
[4] 杨睿.基于Kinect深度信息获取的职业技能训练识别方法研究[J].电子设计工程，2022，30（13）：64-68.
YANG R.Research on recognition method of vocational skill training based on Kinect depth information acquisition[J].Electronic Design Engineering，2022，30（13）：64-68.
[5] CHEN Y X，ZHANG Z Q，YUAN C F，et al.Channel-wise topology refinement graph convolution for skeleton-based action recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：13359-13368.
[6] XIE Y L，ZHANG Y，REN F.Temporal enhanced graph convolution network for skeleton based action recognition[J].IET Computer Vision，2022，16（3）：266-279.
[7] 张芷蒙，彭璟，江小平，等.基于高斯噪声与ST-GCN的人体骨架动作识别[J].中南民族大学学报（自然科学版），2022，41（4）：459-466.
ZHANG Z M，PENG J，JIANG X P，et al.Skeleton-based action recognition based on Gaussian noise and spatial temporal graph convolutional networks[J].Journal of South-central Minzu University（Natural Science Edition），2022，41（4）：459-466.
[8] ZHANG J X，YE G X，TU Z G，et al.A spatial attentive and temporal dilated（SATD） GCN for skeleton based action recognition[J].CAAI Transactions on Intelligence Technology，2022，7（1）：46-55.
[9] WANG J，LIU Z C，WU Y，et al.Mining actionlet ensemble for action recognition with depth cameras[C]//2012 IEEE Conference on Computer Vision and Pattern Recognition，2012：1290-1297.
[10] VEMULAPALLI R，ARRATE F，CHELLAPPA R.Human action recognition by representing 3d skeletons as points in a lie group[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2014：588-595.
[11] FERNANDO B，GAVVES E，ORAMAS J M，et al.Modeling video evolution for action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：5378-5387.
[12] DU Y，WANG W，WANG L.Hierarchical recurrent neural network for skeleton based action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：1110-1118.
[13] SHAHROUDY A，LIU J，NG T T，et al.Ntu rgb+ d：a large scale dataset for 3d human activity analysis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：1010-1019.
[14] LIU J，SHAHROUDY A，XU D，et al.Spatio-temporal lstm with trust gates for 3d human action recognition[C]//European Conference on Computer Vision，2016：816-833.
[15] TU J H，LIU M Y，LIU H.Skeleton-based human action recognition using spatial temporal 3D convolutional neural networks[C]//2018 IEEE International Conference on Multimedia and Expo（ICME），2018：1-6.
[16] SOO KIM T，REITER A.Interpretable 3d human action analysis with temporal convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops，2017：20-28.
[17] KE Q，BENNAMOUN M，AN S，et al.A new representation of skeleton sequences for 3d action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：3288-3297.
[18] ZHOU P，WU Z Q，WEN G Q，et al.Multi-scale graph classification with shared graph neural network[J].World Wide Web，2023，26：949-966.
[19] ZENG D Y，ZHOU L，LIU W L，et al.A simple graph neural network via layer sniffer[C]//2022 IEEE International Conference on Acoustics，Speech and Signal Processing（ICASSP），2022：5687-5691.
[20] PU S L，WU Y F，SUN X，et al.Hyperspectral image classification with localized graph convolutional filtering[J].Remote Sensing，2021，13（3）：526.
[21] WELLING M，KIPF T N.Semi-supervised classification with graph convolutional networks[C]//International Conference on Learning Representations（ICLR 2017），2016.
[22] YAN S J，XIONG Y J，LIN D H.Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Thirty-Second AAAI Conference on Artificial Intelligence，2018.
[23] LI M，CHEN S，CHEN X，et al.Actional-structural graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：3595-3603.
[24] SHI L，ZHANG Y，CHENG J，et al.Two-stream adaptive graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：12026-12035.
[25] LIU Z，ZHANG H，CHEN Z，et al.Disentangling and unifying graph convolutions for skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：143-152.
[26] DUAN H，ZHAO Y，CHEN K，et al.Revisiting skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：2969-2978.
[27] ZHANG Y H，MENG Y，ZHAO C，et al.Integrating label propagation with graph convolutional networks for recommendation[J].Neural Computing and Applications，2022，34（10）：8211-8225.
[28] CARREIRA J，ZISSERMAN A.Quo vadis，action recognition? a new model and the kinetics dataset[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：6299-6308.
[29] CAO Z，SIMON T，WEI S E，et al.Realtime multi-person 2d pose estimation using part affinity fields[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：7291-7299.
[30] SONG S J，LAN C L，XING J L，et al.An end-to-end spatio-temporal attention model for human action recognition from skeleton data[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2017.
[31] ZHANG P F，LAN C L，XING J L.View adaptive recurrent neural networks for high performance human action recognition from skeleton data[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2117-2126.
[32] LIU M Y，LIU H，CHEN C.Enhanced skeleton visualization for view invariant human action recognition[J].Pattern Recognition，2017，68：346-362.
[33] TANG Y S，TIAN Y，LU J W，et al.Deep progressive reinforcement learning for skeleton-based action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：5323-5332.