Hand Pose Estimation Based on Multi-Feature Enhancement

doi:10.3778/j.issn.1002-8331.2210-0089

Abstract

Abstract: Hand pose estimation is one of the important research directions of computer vision, which plays an important role in human-computer interaction, virtual reality, robot control and other application fields. At present, hand pose estimation has the problem of single feature representation method. This paper proposes a feature construction method of hand key point connection relationship and a key point feature aggregation enhancement method based on hand motion semantic relationship to improve the hand feature representation and information sharing ability. Aiming at the occlusion problem in hand target detection and image segmentation, a hand contour feature extraction method is designed to improve the preprocessing effect. Based on the proposed multi-feature representation and enhancement method, a depth learning neural network model based on full convolution structure is constructed to avoid the problem of spatial information loss caused by direct regression calculation of 3D pose information, thus effectively improving the accuracy of 3D hand pose estimation. Compared with the SOTA model on DO, ED, RHD datasets, it has achieved a competitive effect, and the average AUC result has reached 93.3%, indicating that the proposed method also has good universality.

Key words: 3D hand pose estimation, feature enhancement, convolutional neural network, full convolutional neural network

摘要： 手部姿态估计是计算机视觉研究的重要方向之一，在人机交互、虚拟现实、机器人控制等应用领域发挥着重要作用。针对目前手部姿态估计存在特征表示方法单一性的问题，提出手部关键点连接关系特征构建方法与基于手部运动语义关系的关键点特征聚合增强方法，提高手部特征表达与信息共享能力；针对手部目标检测与图像分割等预处理方法中存在的遮挡性问题，设计手部轮廓特征提取方法，提高预处理效果；基于所提出的多特征表示与增强方法，构建了一个基于全卷积结构的深度学习神经网络模型，避免直接回归计算3D姿态信息导致的空间信息丢失问题，从而有效提高了3D手部姿态估计精度。在DO、ED、RHD等多个数据集上与SOTA模型相比，均取得了竞争性的效果，且平均AUC结果达到了93.3%，说明所提出的方法也具有较好普适性。

关键词: 3D手部姿态估计, 特征增强, 卷积神经网络, 全卷积神经网络

FENG Xinxin, GAO Shu. Hand Pose Estimation Based on Multi-Feature Enhancement[J]. Computer Engineering and Applications, 2024, 60(6): 207-213.

奉鑫鑫, 高曙. 基于多特征增强的手部姿态估计方法[J]. 计算机工程与应用, 2024, 60(6): 207-213.

References

[1] ZHANG F, DAI G, PENG X. A survey on human-computer interaction in virtual reality[J]. Scientia Sinica Informationis, 2016, 46(12): 1711-1736.
[2] YU H C, YANG X D, ZHANG Y W, et al. A review on the recognition of mid-air gestures[J]. Science & Technology Review, 2017: 35(16): 64-73.
[3] VULETIC T. Systematic literature review of hand gestures used in human computer interaction interfaces[J]. International Journal of Human-Computer Studies, 2019,129: 74-94.
[4] MIN Y, HAO A, CHAI X, et al. Visual alignment constraint for continuous sign language recognition[C]//Computer Vision and Pattern Recognition, 2021: 11542-11551.
[5] WU X, FINNEGAN D, O'NEILL E, et al. Handmap: robust hand pose estimation via intermediate dense guidance map supervision[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 237-253.
[6] FANG L, LIU X, LIU L, et al. JGR-P2O: joint graph reasoning based pixel-to-offset prediction network for 3D hand pose estimation from a single depth image[C]//Computer Vision and Pattern Recognition, 2020: 120-137.
[7] IQBAL U, MOLCHANOV P, BREUEL T M, et al. Hand pose estimation via latent 2.5D heatmap regression[C]//Computer Vision and Pattern Recognition, 2018: 118-134.
[8] GE L, REN Z, LI Y, et al. 3D hand shape and pose estimation from a single RGB image[C]//Computer Vision and Pattern Recognition, 2019: 10833-10842.
[9] ZHOU Y, HABERMANN M, XU W, et al. Monocular real-time hand shape and motion capture using multi-modal data[C]//Computer Vision and Pattern Recognition, 2020: 5346-5355.
[10] YANG L L, LI S L, LEE D, et al. Aligning latent spaces for 3D hand pose estimation[C]//2019IEEE/CVF International Conference on Computer Vision, 2019: 2335-2343.
[11] 马佳铭. 基于图卷积的手部姿态估计与形状重建研究[D]. 大连:大连理工大学, 2021.
MA J M. Research on hand pose estimation and shape reconstruction based on graph convolutional network[D]. Dalian: Dalian University of Technology, 2021.
[12] DOOSTI B. Hand pose estimation: a survey[J]. arXiv:1903.01013, 2019.
[13] MEHTA D, SRIDHAR S, SOTNYCHENKO O, et al. VNect: real-time 3D human pose estimation with a single RGB camera[C]//International Conference on Computer Graphics and Interactive Techniques, 2017: 1-14.
[14] 李一博. 面向手物交互场景的手部检测与姿态估计[D]. 大连: 大连理工大学, 2021.
LI Y B. Hand detection and pose estimation in hand-object interaction scene[D]. Dalian: Dalian University of Technology, 2021.
[15] ZIMMERMANN C, CEYLAN D, YANG J, et al. FreiHAND: a dataset for markerless capture of hand pose and shape from single RGB images[C]//Computer Vision and Pattern Recognition, 2019: 813-822.
[16] ZIMMERMANN C, BROX T. Learning to estimate 3D hand pose from single rgb images[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 4903-4911.
[17] CHEN X, LIU Y, MA C, et al. Camera-space hand mesh recovery via semantic aggregation and adaptive 2D-1D registration[C]//Computer Vision and Pattern Recognition, 2021: 13274-13283.
[18] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Computer Vision and Pattern Recognition, 2015: 770-778.
[19] SIMON T, JOO H, MATTHEWS I, et al. Hand keypoint detection in single images using multiview bootstrapping[C]//Computer Vision and Pattern Recognition, 2017: 1145-1153.
[20] ZHANG J, JIAO J, CHEN M, et al. 3D hand pose tracking and estimation using stereo matching[J]. arXiv:1610.07214, 2016.
[21] MUELLER F, BERNARD F, SOTNYCHENKO O, et al. Ganerated hands for real-time 3D hand tracking from monocular RGB[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 49-59.
[22] SRIDHAR S, MUELLER F, ZOLLH?FER M, et al. Real-time joint tracking of a hand manipulating an object from RGB-D input[C]//European Conference on Computer Vision, 2016: 294-310.
[23] MUELLER F, MEHTA D, SOTNYCHENKO O, et al. Real-time hand tracking under occlusion from an egocentric RGB-D sensor[C]//Computer Vision and Pattern Recognition, 2017: 1154-1163.
[24] BOUKHAYMA A, BEM R D, TORR P H. 3D hand shape and pose from images in the wild[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 10843-10852.
[25] BAEK S, KIM K I, KIM T K. Pushing the envelope for rgb-based dense 3D hand pose estimation via neural rendering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 1067-1076.
[26] ZHANG X, LI Q, MO H, et al. End-to-end hand mesh recovery from a monocular rgb image[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 2354-2364.
[27] THEODORIDIS T, CHATZIS T, SOLACHIDIS V, et al. Cross-modal variational alignment of latent spaces[C]//Computer Vision and Pattern Recognition, 2020: 960-961.
[28] 刘家祥. 基于深度学习的三维手势估计方法[D]. 成都：电子科技大学, 2022.
LIU J X. Research on 3D gesture estimation methods based on depth learning[D]. Chengdu:University of Electronic Science and Technology of China, 2022.
[29] CHENG W, PARK J H, KO J H. HandFoldingNet: a 3D hand pose estimation network using multiscale-feature guided folding of a 2D hand skeleton[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 11260-11269.