3D Human Pose Estimation Incorporating Dual-Channel Joint Constraints

doi:10.3778/j.issn.1002-8331.2308-0087

Abstract

Abstract: To tackle the issue of significant estimation errors in the distal joints of 3D human pose estimation, this paper presents a method called joint-constrained spatio-temporal graph network (JC-STGNet) that incorporates dual-channel joint constraints. Firstly, it introduces an adaptive weight module to separate different joint feature transformations, thereby enhancing the flexibility of feature representation. Moreover, the paper designs two novel convolutional kernels that leverage prior structural knowledge of the human body to capture more informative joint features. Lastly, it introduces an intra-joint constraint module to replace features in the distal joints with higher degrees of freedom, thereby improving the accuracy of the model’s estimation for these joints. Experimental results on the Human3.6M dataset demonstrate that JC-STGNet achieves a 3.1?mm and 12.4?mm improvement in accuracy, as measured by the MPJPE and P-MPJPE metrics, respectively, compared to the baseline spatio-temporal graph network. Furthermore, this proposed method reduces the parameter count by 22.27% compared to networks with similar accuracy.

Key words: human pose estimation, spatio-temporal graph convolutional network, human topology structure

摘要： 针对三维人体姿态估计中末端关节存在较大估计误差的问题，以时空图卷积网络为基础，提出融合双通道关节约束的三维人体姿态估计方法（JC-STGNet）。提出自适应权重模块，解耦不同关节特征变换，提高特征表达灵活性；结合人体先验结构知识，设计两个新颖的卷积核，获取更丰富的关节特征；最后，引入关节内部约束模块，对自由度较高的末端关节进行特征替换，提高模型对末端关节的估计准确度。在Human3.6M数据集上的实验表明，与基线时空图卷积网络相比，JC-STGNet在MPJPE和P-MPJPE两个指标下，准确度分别提升了3.1?mm和12.4?mm；与同准确度的网络相比，本文的参数量减少了22.27%。

关键词: 人体姿态估计, 时空图卷积网络, 人体拓扑结构

WEI Na, JIAO Minghai. 3D Human Pose Estimation Incorporating Dual-Channel Joint Constraints[J]. Computer Engineering and Applications, 2024, 60(23): 146-154.

卫娜, 焦明海. 融合双通道关节约束的三维人体姿态估计[J]. 计算机工程与应用, 2024, 60(23): 146-154.

References

[1] 何坚, 郭泽龙, 刘乐园, 等. 基于滑动窗口和卷积神经网络的可穿戴人体活动识别技术[J]. 电子与信息学报, 2022, 44(1): 168-177.
HE J, GUO Z L, LIU L Y, et al. Human activity recognition technology based on sliding window and convolutional neural network[J]. Journal of Electronic Information, 2022, 44(1): 168-177.
[2] 李杰. 结合注意力和纹理特征增强的行人再识别[J]. 计算机科学与探索, 2022, 16(3): 661-668.
LI J. Attention and texture feature enhancement for person re-identification[J]. Journal of Frontiers of Computer Science & Technology, 2022, 16(3): 661-668.
[3] 王发明, 李建微, 陈思喜. 三维人体姿态估计研究综述[J]. 计算机工程与应用, 2021, 57(10): 26-38.
WANG F M, LI J W, CHEN S X. Overview of research on 3D human pose estimation[J]. Computer Engineering Applications, 2021, 57(10): 26-38.
[4] MARTINEZ J, HOSSAIN R, ROMERO J, et al. A simple yet effective baseline for 3D human pose estimation[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 2640-2649.
[5] 王仕宸, 黄凯, 陈志刚, 等. 深度学习的三维人体姿态估计综述[J]. 计算机科学与探索, 2023, 17(1): 74-87.
WAND S C, HE K, CHEN Z G, et al. Survey on 3D human pose estimation of deep learning[J]. Journal of Frontiers of Computer Science & Technology, 2023, 17(1): 74-87.
[6] JAHANGIRI E, YUILLE A L. Generating multiple diverse hypotheses for human 3D pose consistent with 2D joint detections[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops, 2017: 805-814.
[7] ZOU Z, TANG W. Modulated graph convolutional network for 3D human pose estimation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 11477-11487.
[8] LI W, LIU H, GUO T, et al. GraphMLP: a graph MLP-like architecture for 3D human pose estimation[J]. arXiv:2206. 06420, 2022.
[9] LIU X, LI P, NI D, et al. LightPose: a lightweight and efficient model with transformer for human pose estimation[C]//Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing, 2022: 2674-2678.
[10] ZHENG C, ZHU S, MENDIETA M, et al. 3D human pose estimation with spatial and temporal transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 11656-11665.
[11] LI W, LIU H, TANG H, et al. MHFormer: multi-hypothesis transformer for 3D human pose estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 13147-13156.
[12] DABRAL R, MUNDHADA A, KUSUPATI U, et al. Learning 3D human pose from structure and motion[C]//Proceedings of the European Conference on Computer Vision, 2018: 668-683.
[13] WANDT B, ROSENHAHN B. RepNet: weakly supervised training of an adversarial reprojection network for 3D human pose estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 7782-7791.
[14] WANG L, CHEN Y, GUO Z, et al. Generalizing monocular 3D human pose estimation in the wild[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2019.
[15] HOSSAIN M R I, LITTLE J J. Exploiting temporal information for 3D human pose estimation[C]//Proceedings of the European Conference on Computer Vision, 2018: 68-84.
[16] PAVLLO D, FEICHTENHOFER C, GRANGIER D, et al. 3D human pose estimation in video with temporal convolutions and semi-supervised training[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 7753-7762.
[17] CAI Y, GE L, LIU J, et al. Exploiting spatial-temporal relationships for 3D pose estimation via graph convolutional networks[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 2272-2281.
[18] WANG J, YAN S, XIONG Y, et al. Motion guided 3D pose estimation from videos[C]//Proceedings of the European Conference on Computer Vision, 2020: 764-780.
[19] ZHAO L, PENG X, TIAN Y, et al. Semantic graph convolutional networks for 3D human pose regression[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 3425-3435.
[20] VELICKOVIC P, CUCURULL G, CASANOVA A, et al. Graph attention networks[J]. arXiv:1710.10903, 2017.
[21] LIU J, ROJAS J, LI Y, et al. A graph attention spatio-temporal convolutional network for 3D human pose estimation in video[C]//Proceedings of the 2021 IEEE International Conference on Robotics and Automation, 2021: 3374-3380.
[22] LIU K, DING R, ZOU Z, et al. A comprehensive study of weight sharing in graph networks for 3D human pose estimation[C]//Proceedings of the 16th European Conference on Computer Vision, 2020: 318-334.
[23] CHANG L, QIN F, LI A. A novel backtracking scheme for attitude determination-based initial alignment[J]. IEEE Transactions on Automation Science and Engineering, 2014, 12(1): 384-390.
[24] CAI J, LIU H, DING R, et al. HTNet: human topology aware network for 3D human pose estimation[C]//Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2023: 1-5.
[25] LEE K, LEE I, LEE S. Propagating LSTM: 3D pose estimation based on joint interdependency[C]// Proceedings of the European Conference on Computer Vision, 2018: 119-135.
[26] LIN J, LEE G H. Trajectory space factorization for deep video-based 3D human pose estimation[J]. arXiv:1908. 08289, 2019.
[27] LIU R, SHEN J, WANG H, et al. Attention mechanism exploits temporal contexts: real-time 3D human pose reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 5064-5073.