GCN与CNN融合的特征细化骨骼行为识别方法

doi:10.3778/j.issn.1002-8331.2408-0007

摘要/Abstract

摘要： 图卷积网络（graph convolutional network，GCN）因有效保留骨骼特征能力的优势，在当前主流研究中得到广泛应用并取得显著效果。然而，固定时序卷积核的大小使得时间卷积过程中感受野受限，并且图卷积过程中对骨骼特征信息提取不充分、跨尺度特征细化、多层语义特征连接问题还需要进一步解决。针对这些问题，设计了一个融合网络，既用到了GCN能够保留骨骼特征的优势，又用到了卷积神经网络（convolutional neural network，CNN）较强的空间特征提取能力。在该网络中多分支时序增强卷积（multi-branch temporal enhanced convolution，MTE Conv）设置了不同的分支以及时序增强获得更为丰富的跨尺度细化特征。图顶点增强模块（graph vertex enhanced module，GVEM）作为GCN与CNN之间进行多层次语义特征连接，使得图骨骼特征能够更好地映射到CNN中进行时空特征的提取。在NTU-RGB+D 60的X-view与NTU-RGB+D 120的X-set两大基准上取得了97.63%和91.16%的准确率，表明所提出的方法具有优越的性能。

关键词: 行为识别, 骨骼数据, 特征融合, 图卷积神经网络（GCN）

Abstract: Graph convolutional networks (GCNs) have been widely applied and have achieved significant results in current mainstream research due to their ability to effectively capture skeleton features. However, the size of the fixed temporal convolution kernel limits the receptive field in the temporal convolution process, and the problems of insufficient extraction of bone feature information, cross-scale feature refinement, and multi-layer semantic feature connection in the graph convolution process need to be further solved. Aiming at these problems, a fusion network is designed, which utilizes the advantage of GCN in retaining skeleton features and the strong ability of convolutional neural network (CNN) in extracting spatial features. In the network, the multi-branch temporal enhanced convolution (MTE Conv) is set up with different branches and temporal enhancement to obtain more diverse fine-grained features at different scales. The graph vertex enhanced module (GVEM) serves as a multi-level semantic feature connection between GCN and CNN, enabling the graph skeleton features to be better mapped to CNN for spatial-temporal feature extraction. The accuracy of 97.63% and 91.16% is achieved on the X-view of NTU-RGB+D 60 and X-set of NTU-RGB+D 120, indicating that the proposed method has superior performance.

Key words: action recognition, skeletal data, feature fusion, graph convolutional network (GCN)

陈星启, 宋涛, 邹洋杨. GCN与CNN融合的特征细化骨骼行为识别方法[J]. 计算机工程与应用, 2025, 61(22): 226-234.

CHEN Xingqi, SONG Tao, ZOU Yangyang. Feature Refinement Skeletal Action Recognition Method Based on GCN and CNN Fusion[J]. Computer Engineering and Applications, 2025, 61(22): 226-234.

参考文献

[1] GAMMULLE H, AHMEDT-ARISTIZABAL D, DENMAN S, et al. Continuous human action recognition for human-machine interaction: a review[J]. ACM Computing Surveys, 2023, 55(13S): 1-38.
[2] NIU Z H, LU K, XUE J, et al. From methods to applications: a review of deep 3D human motion capture[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34(11): 11340-11359.
[3] DUAN H D, ZHAO Y, CHEN K, et al. Revisiting skeleton-based action recognition[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 2959-2968.
[4] CAI D Q, KANG Y, YAO A B, et al. Ske2Grid: skeleton-to-grid representation learning for action recognition[C]//Proceedings of the 40th International Conference on Machine Learning, 2023: 3431-3441.
[5] SHI L, ZHANG Y F, CHENG J, et al. Two-stream adaptive graph convolutional networks for skeleton-based action recog-nition[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 12018-12027.
[6] ZHANG Y, SUN Z H, DAI M, et al. Cross-scale spatiotemporal refinement learning for skeleton-based action recognition[J]. IEEE Signal Processing Letters, 2024, 31: 441-445.
[7] TU Z G, ZHANG J X, LI H Y, et al. Joint-bone fusion graph convolutional network for semi-supervised skeleton action recognition[J]. IEEE Transactions on Multimedia, 2023, 25: 1819-1831.
[8] SONG Y F, ZHANG Z, SHAN C F, et al. Constructing stronger and faster baselines for skeleton-based action recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(2): 1474-1488.
[9] HU H S, FANG Y, HAN M, et al. Multi-scale adaptive graph convolution network for skeleton-based action recognition[J]. IEEE Access, 2024, 12: 16868-16880.
[10] 王彩玲, 闫晶晶, 张智栋. 基于多模态数据的人体行为识别方法研究综述[J]. 计算机工程与应用, 2024, 60(9): 1-18.
WANG C L, YAN J J, ZHANG Z D. Review on human action recognition methods based on multimodal data[J]. Computer Engineering and Applications, 2024, 60(9): 1-18.
[11] 吴浩原, 熊辛, 闵卫东, 等. 基于多级特征融合和时域扩展的行为识别方法[J]. 计算机工程与应用, 2023, 59(7): 134-142.
WU H Y, XIONG X, MIN W D, et al. Action recognition method based on multi-level feature fusion and temporal extension[J]. Computer Engineering and Applications, 2023, 59(7): 134-142.
[12] YAN S J, XIONG Y J, LIN D H. Spatial temporal graph convolutional networks for skeleton-based action recognition[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32(1): 7444-7452.
[13] SHAO Y M, MAO L T, YE L X, et al. H2GCN: a hybrid hyper-graph convolution network for skeleton-based action recognition[J]. Journal of King Saud University-Computer and Information Sciences, 2024, 36(5): 102072.
[14] CHEN Y X, ZHANG Z Q, YUAN C F, et al. Channel-wise topology refinement graph convolution for skeleton-based action recognition[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2022: 13339-13348.
[15] HUANG J H, WANG Z M, PENG J, et al. Feature reconstruction graph convolutional network for skeleton-based action recognition[J]. Engineering Applications of Artificial Intelligence, 2023, 126: 106855.
[16] ZHAO Z F, CHEN Z W, LI J N, et al. Glimpse and zoom: spatio-temporal focused dynamic network for skeleton-based action recognition[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34(7): 5616-5629.
[17] CHEN J R, KAO S H, HE H, et al. Run, don’t walk: chasing higher FLOPS for faster neural networks[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2023: 12021-12031.
[18] SHI L, ZHANG Y F, CHENG J, et al. Skeleton-based action recognition with multi-stream adaptive graph convolutional networks[J]. IEEE Transactions on Image Processing, 2020, 29: 9532-9545.
[19] SONG Y F, ZHANG Z, SHAN C F, et al. Stronger, faster and more explainable: a graph convolutional baseline for skeleton-based action recognition[C]//Proceedings of the 28th ACM International Conference on Multimedia. New York: ACM, 2020: 1625-1633.
[20] SHAHROUDY A, LIU J, NG T T, et al. NTU RGB+D: a large scale dataset for 3D human activity analysis[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 1010-1019.
[21] LIU J, SHAHROUDY A, PEREZ M, et al. NTU RGB D 120: a large-scale benchmark for 3D human activity understanding[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(10): 2684-2701.
[22] HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 7132-7141.
[23] WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of the 15th European Conference on Computer Vision. Cham: Springer, 2018: 3-19.
[24] WANG Q L, WU B G, ZHU P F, et al. ECA-net: efficient channel attention for deep convolutional neural networks[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 11531-11539.
[25] QIN Z Q, ZHANG P Y, WU F, et al. FcaNet: frequency channel attention networks[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2022: 763-772.
[26] SALMAN H, PARKS C, SWAN M, et al. OrthoNets: orthogonal channel attention networks[C]//Proceedings of the 2023 IEEE International Conference on Big Data. Piscataway: IEEE, 2024: 829-837.
[27] CHI H G, HA M H, CHI S, et al. InfoGCN: representation learning for human skeleton-based action recognition[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 20154-20164.
[28] QIU H L, HOU B. Multi-grained clip focus for skeleton-based action recognition[J]. Pattern Recognition, 2024, 148: 110188.
[29] YE F F, PU S L, ZHONG Q Y, et al. Dynamic GCN: context-enriched topology learning for skeleton-based action recognition[C]//Proceedings of the 28th ACM International Conference on Multimedia. New York: ACM, 2020: 55-63.
[30] LIU Z Y, ZHANG H W, CHEN Z H, et al. Disentangling and unifying graph convolutions for skeleton-based action recognition[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 140-149.
[31] WANG J, LUO Y M. SA-GCN: structure-aware graph convolutional networks for crowd pose estimation[J]. The Journal of Supercomputing, 2023, 79(9): 10046-10062.
[32] TIAN H Y, MA X, LI X, et al. Skeleton-based action recognition with select-assemble-normalize graph convolutional networks[J]. IEEE Transactions on Multimedia, 2023, 25: 8527-8538.