基于深度学习的二维单人姿态估计综述

doi:10.3778/j.issn.1002-8331.2403-0152

摘要/Abstract

摘要： 人体姿态估计是计算机视觉领域的一项关键技术，它通过检测人体关键点以识别人体姿态。随着深度学习的快速发展，其已成为人体姿态估计的主流技术并取得了显著进展。围绕单人姿态估计问题，从数据预处理、网络架构设计、监督学习方法以及后处理技术四个维度对基于深度学习的单人姿态估计研究进行回顾，同时探讨关键点表征的新方式及Transformer模型在该领域的应用，此外还介绍了常用的数据集和性能估计指标，深入讨论当前单人姿态估计领域的挑战和发展方向。

关键词: 二维人体姿态估计, 单人姿态估计, 深度学习, 关键点检测, 计算机视觉

Abstract: Human pose estimation is a key technology in the field of computer vision that identifies human postures by detecting body keypoints. With the rapid advancement of deep learning, it has become the dominant approach in human pose estimation, achieving significant progress. This paper reviews single-person pose estimation research based on deep learning, examining the issue from four perspectives: data preprocessing, network architecture design, supervised learning methods, and post-processing techniques. It also explores new representations of keypoints and the application of Transformer models in this area. Additionally, the paper introduces common datasets and performance metrics, and delves into the current challenges and future directions in the field of single-person pose estimation.

Key words: two-dimensional human pose estimation, single-person pose estimation, deep learning, keypoint detection, computer vision

中图分类号:

','1');return false;" target="_blank">

苏妍妍, 邱志良, 李帼, 陆声链, 陈明. 基于深度学习的二维单人姿态估计综述[J]. 计算机工程与应用, 2024, 60(21): 18-37.

SU Yanyan, QIU Zhiliang, LI Guo, LU Shenglian, CHEN Ming. Review on Deep Learning-Based 2D Single-Person Pose Estimation[J]. Computer Engineering and Applications, 2024, 60(21): 18-37.

参考文献

[1] VATS A, ANASTASIU D C. Key point-based driver activity recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022: 3274-3281.
[2] 李聪林, 王琪冰, 陆佳炜, 等. 基于数字孪生的电梯乘客异常行为建模与识别方法[J]. 计算机工程与应用, 2023, 59(19): 274-284.
LI C L, WANG Q B, LU J W, et al. Modeling and recognition method of elevator passenger abnormal behavior based on digital twin[J]. Computer Engineering and Applications, 2023, 59(19): 274-284.
[3] LI X, CHEN Z, ZHONG Z, et al. Human-machine collaboration method based on key nodes of human posture[C]//Proceedings of the 2022 IEEE Asia-Pacific Conference on Image Processing, Electronics and Computers (IPEC), 2022: 140-146.
[4] SONG Y F, ZHANG P, LIU L B. Human-machine interaction system with vision-based gesture recognition[J]. Computer Science, 2019, 46(S2): 570-574.
[5] SHAO M Y, VAGG T, SEIBOLD M, et al. Towards a low-cost monitor-based augmented reality training platform for at-home ultrasound skill development[J]. Journal of Imaging, 2022, 8(11): 305.
[6] BASIRATZADEH S, LEMAIRE E D, BADDOUR N. A novel augmented reality mobile-based application for biomechanical measurement[J]. BioMed, 2022, 2(2): 255-269.
[7] 张宇, 温光照, 米思娅, 等. 基于深度学习的二维人体姿态估计综述[J]. 软件学报, 2022, 33(11): 4173-4191.
ZHANG Y, WEN G Z, MI S Y, et al. Overview on 2D human pose estimation based on deep learning[J]. Journal of Software, 2022, 33(11): 4173-4191.
[8] HUANG J, ZHU Z, GUO F, et al. The devil is in the details: delving into unbiased data processing for human pose estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020: 5700-5709.
[9] 王仕宸, 黄凯, 陈志刚, 等. 深度学习的三维人体姿态估计综述[J]. 计算机科学与探索, 2023, 17(1): 74-87.
WANG S C, HUANG K, CHEN Z G, et al. Survey on 3D human pose estimation of deep learning[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(1): 74-87.
[10] YANG Y, RAMANAN D. Articulated human detection with flexible mixtures of parts[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 35(12): 2878-2890.
[11] MUNEA T L, JEMBRE Y Z, WELDEGEBRIEL H T, et al. The progress of human pose estimation: a survey and taxonomy of models applied in 2D human pose estimation[J]. IEEE Access, 2020, 8: 133330-133348.
[12] CHEN Y, TIAN Y, HE M. Monocular human pose estimation: a survey of deep learning-based methods[J]. Computer Vision and Image Understanding, 2020, 192: 102897.
[13] SAMKARI E, ARIF M, ALGHAMDI M, et al. Human pose estimation using deep learning: a systematic literature review[J]. Machine Learning and Knowledge Extraction, 2023, 5(4): 1612-1659.
[14] DUBEY S, DIXIT M. A comprehensive survey on human pose estimation approaches[J]. Multimedia Systems, 2023, 29(1): 167-195.
[15] ZHENG C, WU W, CHEN C, et al. Deep learning-based human pose estimation: a survey[J]. ACM Computing Surveys, 2023, 56(1): 1-37.
[16] ZHANG F, ZHU X, WANG C. A comprehensive survey on single-person pose estimation in social robotics[J]. International Journal of Social Robotics, 2022, 14(9): 1995-2008.
[17] LI Y, YANG S, LIU P, et al. SimCC: a simple coordinate classification perspective for human pose estimation[C]//Proceedings of the European Conference on Computer Vision, 2022: 89-106.
[18] GU K, YANG L, YAO A. Removing the bias of integral pose regression[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 11067-11076.
[19] JOHNSON S, EVERINGHAM M. Clustered pose and nonlinear appearance models for human pose estimation[C]//Proceedings of British Machine Vision Conference, 2010: 1-11.
[20] SAPP B, TASKAR B. Modec: multimodal decomposable models for human pose estimation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Portland, OR, USA, 2013: 3674-3681.
[21] ANDRILUKA M, PISHCHULIN L, GEHLER P, et al. 2D human pose estimation: new benchmark and state of the art analysis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 2014: 3686-3693.
[22] LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft coco: common objects in context[C]//Proceedings of the 13th European Conference on Computer Vision (ECCV 2014), Zurich, Switzerland, 2014: 740-755.
[23] WU J, ZHENG H, ZHAO B, et al. AI challenger: a large-scale dataset for going deeper in image understanding[J]. arXiv:1711.06475, 2017.
[24] ZHAO L, WANG N, GONG C, et al. Estimating human pose efficiently by parallel pyramid networks[J]. IEEE Transactions on Image Processing, 2021, 30: 6785-6800.
[25] TOMPSON J, GOROSHIN R, JAIN A, et al. Efficient object localization using convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 2015: 648-656.
[26] CHENG B, XIAO B, WANG J, et al. HigherHRNet: scale-aware representation learning for bottom-up human pose estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020: 5386-5395.
[27] PENG X, TANG Z, YANG F, et al. Jointly optimize data augmentation and network training: adversarial data augmentation in human pose estimation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 2018: 2226-2234.
[28] NEWELL A, YANG K, DENG J. Stacked hourglass networks for human pose estimation[C]//Proceedings of the 14th European Conference on Computer Vision, Amsterdam, The Netherlands, 2016: 483-499.
[29] XIAO B, WU H, WEI Y. Simple baselines for human pose estimation and tracking[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 466-481.
[30] SUN K, XIAO B, LIU D, et al. Deep high-resolution representation learning for human pose estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019: 5693-5703.
[31] CHEN Y, SHEN C, WEI X S, et al. Adversarial posenet: a structure-aware convolutional network for human pose estimation[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 1212-1221.
[32] HUANG J, ZHU Z, HUANG G, et al. AID: pushing the performance boundary of human pose estimation with information dropping augmentation[J]. arXiv:2008.07139, 2020.
[33] WANG D, XIE W, CAI Y, et al. Adaptive data augmentation network for human pose estimation[J]. Digital Signal Processing, 2022, 129: 103681.
[34] BIN Y, CAO X, CHEN X, et al. Adversarial semantic data augmentation for human pose estimation[C]//Proceedings of the 16th European Conference on Computer Vision, Glasgow, UK, 2020: 606-622.
[35] JIANG W, JIN S, LIU W, et al. Posetrans: a simple yet effective pose transformation augmentation for human pose estimation[C]//Proceedings of the European Conference on Computer Vision, 2022: 643-659.
[36] ELHARROUSS O, AKBARI Y, ALMAADEED N, et al. Backbones-review: feature extraction networks for deep learning and deep reinforcement learning approaches[J]. arXiv:2206.08016, 2022.
[37] NGUYEN T D, KRESOVIC M. A survey of top-down approaches for human pose estimation[J]. arXiv:2202.02656, 2022.
[38] TOMPSON J J, JAIN A, LECUN Y, et al. Joint training of a convolutional network and a graphical model for human pose estimation[C]//Advances in Neural Information Processing Systems, 2014: 1799-1807.
[39] CHEN X, YUILLE A L. Articulated pose estimation by a graphical model with image dependent pairwise relations[C]//Advances in Neural Information Processing Systems, 2014: 1736-1744.
[40] BIN Y, CHEN Z M, WEI X S, et al. Structure-aware human pose estimation with graph convolutional networks[J]. Pattern Recognition, 2020, 106: 107410.
[41] 马金林, 崔琦磊, 马自萍, 等. 预加权调制密集图卷积网络三维人体姿态估计[J]. 计算机科学与探索, 2024, 18(4): 963-977.
MA J L, CUI Q L, MA Z P , et al. Pre-weighted modulated dense graph convolutional networks for 3D human pose estimation[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(4): 963-977.
[42] SUN X, SHANG J, LIANG S, et al. Compositional human pose regression[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 2602-2611.
[43] TANG W, YU P, WU Y. Deeply learned compositional models for human pose estimation[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 190-206.
[44] TOSHEV A, SZEGEDY C. DeepPose: human pose estimation via deep neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Columbus, OH, USA, 2014: 1653-1660.
[45] CHEN H, FENG R, WU S, et al. 2D Human pose estimation: a survey[J]. Multimedia Systems, 2023, 29(5): 3115-3138.
[46] CARREIRA J, AGRAWAL P, FRAGKIADAKI K, et al. Human pose estimation with iterative error feedback[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016: 4733-4742.
[47] WEI S E, RAMAKRISHNA V, KANADE T, et al. Convolutional pose machines[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 4724-4732.
[48] SU Z, YE M, ZHANG G, et al. Cascade feature aggregation for human pose estimation[J]. arXiv:1902.07837, 2019.
[49] BELAGIANNIS V, ZISSERMAN A. Recurrent human pose estimation[C]//Proceedings of the 12th IEEE International Conference on Automatic Face & Gesture Recognition, 2017: 468-475.
[50] TANG W, WU Y. Does learning specific features for related parts help human pose estimation?[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019: 1107-1116.
[51] HWANG J, PARK S, KWAK N. Athlete pose estimation by a global-local network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops(CVPR), Honolulu, HI, USA, 2017: 58-65.
[52] HUA G, LI L, LIU S. Multipath affinage stacked—hourglass networks for human pose estimation[J]. Frontiers of Computer Science, 2020, 14: 1-12.
[53] KE L, CHANG M C, QI H, et al. Multi-scale structure-aware network for human pose estimation[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 713-728.
[54] YANG W, LI S, OUYANG W, et al. Learning feature pyramids for human pose estimation[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 1281-1290.
[55] TIAN Y, HU W, JIANG H, et al. Densely connected attentional pyramid residual network for human pose estimation[J]. Neurocomputing, 2019, 347: 13-23.
[56] LI S, LIU Z Q, CHAN A B. Heterogeneous multi-task learning for human pose estimation with deep convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 2014: 482-489.
[57] FAN X, ZHENG K, LIN Y, et al. Combining local appearance and holistic view: dual-source deep neural networks for human pose estimation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 1347-1355.
[58] SHAMSAFAR F, EBRAHIMNEZHAD H. Uniting holistic and part-based attitudes for accurate and robust deep human pose estimation[J]. Journal of Ambient Intelligence and Humanized Computing, 2021, 12: 2339-2353.
[59] CHOU C J, CHIEN J T, CHEN H T. Self adversarial training for human pose estimation[C]//Proceedings of the 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2018: 17-30.
[60] SHAMSOLMOALI P, ZAREAPOOR M, ZHOU H, et al. Amil: adversarial multi-instance learning for human pose estimation[J]. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2020, 16: 1-23.
[61] LI Y, ZHANG S, WANG Z, et al. TokenPose: learning keypoint tokens for human pose estimation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 11313-11322.
[62] LI K, WANG S, ZHANG X, et al. Pose recognition with cascade transformers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 1944-1953.
[63] YANG S, QUAN Z, NIE M, et al. TransPose: keypoint localization via transformer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 11802-11812.
[64] YUAN Y, FU R, HUANG L, et al. HRFormer: high-resolution vision transformer for dense predict[C]//Advances in Neural Information Processing Systems, 2021: 7281-7293.
[65] XU Y, ZHANG J, ZHANG Q, et al. ViTPose: simple vision transformer baselines for human pose estimation[C]//Advances in Neural Information Processing Systems, 2022: 38571-38584.
[66] LIU H, LIU F, FAN X, et al. Polarized self-attention: towards high-quality pixel-wise regression[J]. arXiv:2107. 00782, 2021.
[67] YU C, XIAO B, GAO C, et al. Lite-HRNet: a lightweight high-resolution network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 10440-10450.
[68] 林远强, 郜辉, 王鹏, 等. 引入级联通道注意力的轻量化人体姿态估计[J]. 计算机工程与应用, 2024, 60(13): 219-227.
LIN Y Q, GAO H, WANG P, et al. Lightweight human pose estimation with cascaded channel attention[J]. Computer Engineering and Applications, 2024, 60(13): 219-227.
[69] 傅裕, 高树辉. 改进YOLOv8s-Pose多人姿态估计轻量化模型研究[J/OL]. 计算机科学与探索(2024-05-07)[2024-07-01]. https://link.cnki.net/urlid/11.5602.TP.20240507. 1148.002.
FU Y, GAO S H. Research on the lightweight model of multi-person pose estimation based on the improved YOLOv8s-Pose[J]. Journal of Frontiers of Computer Science and Technology(2024?05?07)[2024?07?01].https://link.cnki.net/urlid/11.5602.TP.20240507.1148.002.
[70] ZHANG F, ZHU X, YE M. Fast human pose estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019: 3517-3526.
[71] NIE X, FENG J, ZUO Y, et al. Human pose estimation with parsing induced learner[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 2100-2108.
[72] JIANG T, LU P, ZHANG L, et al. RTMPose: real-time multi-person pose estimation based on mmpose[J]. arXiv:2303. 07399, 2023.
[73] YANG Z, ZENG A, YUAN C, et al. Effective whole-body pose estimation with two-stages distillation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023: 4210-4220.
[74] LIFSHITZ I, FETAYA E, ULLMAN S. Human pose estimation using deep consensus voting[C]//Proceedings of the 14th European Conference on Computer Vision, Amsterdam, The Netherlands, 2016: 246-260.
[75] GENG Z, WANG C, WEI Y, et al. Human pose as co-mpositional tokens[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 2023: 660-671.
[76] NIBALI A, HE Z, MORGAN S, et al. Numerical coordinate regression with convolutional neural networks[J]. arXiv: 1801.07372, 2018.
[77] QU H, XU L, CAI Y, et al. Heatmap distribution matching for human pose estimation[C]//Advances in Neural Information Processing Systems, 2022: 24327-24339.
[78] LI J, CHEN T, SHI R, et al. Localization with sampling-argmax[C]//Advances in Neural Information Processing Systems, 2021: 27236-27248.
[79] LI J, BIAN S, ZENG A, et al. Human pose regression with residual log-likelihood estimation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 11025-11034.
[80] PAPANDREOU G, ZHU T, KANAZAWA N, et al. Towards accurate multi-person pose estimation in the wild[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017: 4903-4911.
[81] ZHANG F, ZHU X, DAI H, et al. Distribution-aware coordinate representation for human pose estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 7093-7102.
[82] LUVIZON D C, TABIA H, PICARD D. Human pose regression by combining indirect part detection and contextual information[J]. Computers & Graphics, 2019, 85: 15-22.
[83] GU K, CHEN R, YAO A. On the calibration of human pose estimation[J]. arXiv:2311.17105, 2023.
[84] GAI D, FENG R, MIN W, et al. Spatiotemporal learning transformer for video-based human pose estimation[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023: 33(9): 4564-4576 .
[85] ZHANG S H, LI R, DONG X, et al. Pose2seg: detection free human instance segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019: 889-898.
[86] 李佳宁, 王东凯, 张史梁. 基于深度学习的二维人体姿态估计: 现状及展望[J]. 计算机学报, 2024, 47(1): 231-250.
LI J N, WANG D K, ZHANG S L. Deep-learning-based 2D human pose estimation: present and future[J]. Chinese Journal of Computers, 2024, 47(1): 231-250.
[87] 蔡哲栋, 应娜, 郭春生, 等. YOLOv3剪枝模型的多人姿态估计[J]. 中国图象图形学报, 2021, 26(4): 837-846.
CAI Z D, YING N, GUO C S, et al. Research on multiperson pose estimation combined with YOLOv3 pruning model[J]. Journal of lmage and Graphics, 2021, 26(4): 837-846.
[88] KYROLLOS D G, FULLER A, GREENWOOD K, et al. Under the cover infant pose estimation using multimodal data[J]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 1-12.
[89] DAI Y, LIN Y T, LIN X P, et al. Sloper4D: a scene-aware dataset for global 4d human pose estimation in urban environments[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 2023: 682-692.
[90] DELMAS G, WEINZAEPFEL P, MORENO-NOGUER F, et al. Posefix: correcting 3D human poses with natural language[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023: 15018-15028.
[91] VERMA A, TIWARI V, LOVANSHI M, et al. A human body part semantic segmentation enabled parsing for human pose estimation[C]//Proceedings of the 5th International Conference on Image, Video and Signal Processing, 2023: 43-50.
[92] GRECO A, VENTO B. PAR Contest 2023: pedestrian attributes recognition with multi-task learning[C]//Proceedings of the International Conference on Computer Analysis of Images and Patterns, 2023: 3-12.
[93] GU A, DAO T. Mamba: Linear-time sequence modeling with selective state spaces[J]. arXiv:2312.00752, 2023.
[94] 杨旭升, 吴江宇, 胡佛, 等. 基于渐进高斯滤波融合的多视角人体姿态估计[J]. 自动化学报, 2024, 50(3): 607-616.
YANG X S, WU J Y, HU F, et al. Multi-view human pose estimation based on progressive gaussian filtering fusion[J]. Acta Automatica Sinica, 2024, 50(3): 607-616.