Summary of Dynamic Gesture Recognition Based on Vision

doi:10.3778/j.issn.1002-8331.2105-0314

Abstract

Abstract:

Gestures have played a very important role in human communication since ancient times, and the visual dynamic gesture identification technology is to use new technologies such as computer vision and IOT（Internet of Things） perception, and 3D visual sensors, allowing the machine to understand human gestures, thus making humanity and machine more good communication, because of far-reaching research significance for human-machine interaction. The sensor techniques used in dynamic gesture identification are introduced, and the technical parameters of the related sensors are compared. By tracking the dynamic gesture recognition technology of vision at home and abroad, the processing process of dynamic gesture recognition is first stated：gesture detection and segmentation, gesture tracking, gesture classification. By comparing the methods involved in each process, it can be seen that deep learning has strong fault tolerance, robustness, high parallelism, anti-interference, etc., which has achieved great achievements above the traditional learning algorithm in the field of gesture identification. Finally, the challenges currently encountering and the future possible development of dynamic gesture identification are analyzed.

Key words: visual dynamic gesture identification, gesture detection and segmentation, gesture tracking, gesture classification, gesture recognition

摘要：

手势自古以来在人类交流方面扮演着非常重要的角色，而基于视觉的动态手势识别技术是利用计算机视觉、物联网感知等新兴技术和3D视觉传感器等新型设备让机器能够理解人类的手势，从而让人类能和机器更好地交流，因此对于人机交互等领域的研究很有意义。介绍了动态手势识别中所用到的传感器技术，并比较了相关传感器的技术参数。通过追踪近年来国内外关于视觉的动态手势识别技术，陈述了动态手势识别的处理流程：手势检测与分割、手势追踪、手势分类。通过对比各流程所涉及的方法，可以发现深度学习具有较强的容错性、高度并行性、抗干扰性等一系列优点，在手势识别领域取得了远高于传统学习算法的成就。最后分析了动态手势识别目前遇到的挑战和未来可能的发展方向。

关键词: 视觉动态手势识别, 手势检测与分割, 手势追踪, 手势分类, 手势识别

XIE Yinggang, WANG Quan. Summary of Dynamic Gesture Recognition Based on Vision[J]. Computer Engineering and Applications, 2021, 57(22): 68-77.

解迎刚，王全. 基于视觉的动态手势识别研究综述[J]. 计算机工程与应用, 2021, 57(22): 68-77.

References

[1] 林清宇.基于Kinect的手势检测与追踪研究[D].南京：南京邮电大学，2020.
LIN Q Y.The research of hand detection and tracking using Kinect[D].Nanjing：Nanjing University of Posts and Telecommunications，2020.
[2] 司潇.基于LeapMotion的采煤场景协同手势控制虚拟仿真研究与实现[D].青岛：山东科技大学，2020.
SI X.Research and implementation of virtual simulation of cooperative gesture control of coal mining scene based on LeapMotion[D].Qingdao：Shandong University of Science and Technology，2020.
[3] CHEN L，WEI H，FERRYMAN J.A survey of human motion analysis using depth imagery[J].Pattern Recognition Letters，2013，34（15）：1995-2006.
[4] PANDURANGA H T，MANI C.Dynamic hand gesture recognition system：a short survey[C]//2018 International Conference on Inventive Research in Computing Applications，2018：689-694.
[5] 李国城.基于Kinect视觉功能对体感手势目标识别[J].科技创新与应用，2021，11（14）：38-41.
LI G C.Identification of sports governance based on Kinect vision[J].Technology Innovation and Application，2021，11（14）：38-41.
[6] MAZHAR O，RAMDANI S，NAVARRO B，et al.Towards real-time physical human-robot interaction using skeleton information and hand gestures[C]//2018 IEEE/RSJ International Conference on Intelligent Robots and Systems，2018：1-6.
[7] LAI Y，CHEN W，LI Y，et al.3D pointing gesture recognition for human-robot interaction[C]//2016 Chinese Control and Decision Conference，2016：4959-4964.
[8] WANG C，LIU Z，CHAN S C.Superpixel-based hand gesture recognition with Kinect depth camera[J].IEEE Transactions on Multimedia，2015，17（1）：29-39.
[9] SAINI R，KUMAR P，ROY P P，et al.A novel framework of continuous human-activity recognition using Kinect[J].Neurocomputing，2018，311：99-111.
[10] REN Z，YUAN J S，MENG J J，et al.Robust part-based hand gesture recognition using Kinect sensor[J].IEEE Transactions on Multimedia，2013，15（5）：1110-1120.
[11] YING S，LI C，LI G，et al.Gesture recognition based on Kinect and SEMG signal fusion[J].Mobile Networks and Applications，2018，23（4）：797-805.
[12] GUNA J，JAKUS G，POGANIK M，et al.An analysis of the precision and reliability of the leap motion sensor and its suitability for static and dynamic tracking[J].Sensors，2014，14（2）：3702-3720.
[13] REGENBRECHT H，COLLINS J，HOERMANN S.A leap-supported，hybrid AR interface approach[C]//25th Australian Computer-Human Interaction Conference：Augmentation，Application，Innovation，Collaboration，2013：281-284.
[14] WEICHERT F，BACHMANN D，RUDAK B，et al.Analysis of the accuracy and robustness of the leap motion controller[J].Sensors，2013，13（5）：6380-6393.
[15] VANDERSTEEGEN M，REUSEN W，BEECK K V，et al.Low-latency hand gesture recognition with a low resolution thermal imager[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2020：440-449.
[16] CHU X，LIU J，SHIMAMOTO S.A sensor-based hand gesture recognition system for Japanese sign language[C]//2021 IEEE 3rd Global Conference on Life Sciences and Technologies，2021：311-312.
[17] KAPUSCINSKI T，OSZUST M，WYSOCKI M.Recognition of signed dynamic expressions observed by TOF camera[C]//2013 Signal Processing：Algorithms，Architectures，Arrangements，and Applications，2013：291-296.
[18] THURAU C，HLAVAC V.Pose primitive based human action recognition in videos or still images[C]//2008 IEEE Conference on Computer Vision and Pattern Recognition，2008.
[19] LAN X，MA A J，YUEN P C.Multi-cue visual tracking using robust feature-level fusion based on joint sparse representation[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition，2014.
[20] LIU H，WANG L.Gesture recognition for human-robot collaboration：a review[J].International Journal of Industrial Ergonomics，2018，68：355-367.
[21] HAN J，SHAO L，XU D，et al.Enhanced computer vision with Microsoft Kinect sensor：a review[J].IEEE Transactions on Cybernetics，2013，43（5）：1318-1334.
[22] FENG J，SHEN W，GAO Y，et al.View-point-independent hand gesture recognition with Kinect[J].Signal Image & Video Processing，2014，8（S1）：163-172.
[23] 胡宗承，周亚同，史宝军，等.结合注意力机制和特征融合的静态手势识别[J/OL].计算机工程（2021-05-14）[2021-05-20].https：//doi.org/10.19678/j.issn.1000-3428.0060912.
HU Z C，ZHOU Y T，SHI B J，et al.Static gesture recognition based on attention mechanism and feature fusion[J/OL].Computer Engineering（2021-05-14）[2021-05-20].https：//doi.org/10.19678/j.issn.1000-3428.0060912.
[24] 岳昊恩，袁亮，吕凯.结合帧差法与Mean Shift的抗遮挡跟踪算法[J].现代电子技术，2019，42（12）：180-182.
YUE H E，YUAN L，LU K.Anti-occlusion tracking algorithm combining frame difference method and mean shift[J].Modern Electronic Technique，2019，42（12）：180-182.
[25] BOUWMANS T，GARCIA-GARCIA B.Background subtraction in real applications：challenges，current models and future directions[J].Computer Science Review，2020，35：100204.
[26] RANJAN A，ROMERO J，BLACK M.Learning human opticalflow[EB/OL].（2018-07-22）[2019-12-28].https：//arxiv.org/abs/1806.05666.
[27] GUO Y H，SENGüR A，AKBULUT Y，et al.An effective color image segmentation approach using neutrosophic adaptive mean shift clustering[J].Meaurement，2018，119：28-40.
[28] ALLEN J G，XU R，JIN J S.Tracking using CamShift algorithm and multiple quantized feature spaces[C]//2003 Pan-Sydney Area Workshop on Visual Information Processing，2003.
[29] HANEBECK U D.Template matching using fast normalized cross correlation[J].Proceeding of SPIE on Optical Pattern Recognition XII，2001，4387：95-102.
[30] NGUYEN H T，SMEULDERS A.Fast occluded object tracking by a robust appearance filter[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2004，26（8）：1099-1104.
[31] SHAUL ORON D L，AHARON B，AVIDAN S.Locally orderless tracking[J].International Journal of Computer Vision，2015，111（2）：213-228.
[32] 孙玉宝，肖亮，韦志辉，等.图像稀疏表示的结构自适应子空间匹配追踪算法研究[J].计算机学报，2012，35（8）：1751-1758.
SUN Y B，XIAO L，WEI Z H，et al.A structure-adaptive matching pursuit subspace search algorithm for effective image sparse representation[J].Chinese Journal of Computers，2012，35（8）：1751-1758.
[33] ROSS D A，LIM J，LIN R S.Incremental learning for robust visual tracking[J].International Journal of Computer Vision，2008，77：125-141.
[34] MAGGIO E，CAVALLARO A.Tracking by sampling trackers[C]//2011 International Conference on Computer Vision，2011：1195-1202.
[35] 李东洁，李东阁，杨柳.卷积神经网络在动态手势跟踪中的应用[J].计算机科学与探索，2020，14（5）：841-847.
LI D J，LI D G，Yang L.Application of convolutional neural network in dynamic gesture tracking[J].Journal of Frontiers of Computer Science and Technology，2020，14（5）：841-847.
[36] YUAN D，ZHANG X，LIU J，et al.A multiple feature fused model for visual object tracking via correlation filters[J].Multimedia Tools and Applications，2019，78（19）：27271-27290.
[37] DANELLJAN M，KHAN F S，FELSBERG M，et al.Adaptive color attributes for real-time visual tracking[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition，2014：1090-1097.
[38] KHAN B，KHAN A K，RAJA G，et al.Implementation of modified mean-shift tracking algorithm for occlusion handling[J].Life Science Journal，2013，10（11S）：337-342.
[39] 李妍妍，田瑞娟，张弦弦.一种基于帧差法结合Kalman滤波的运动目标跟踪方法[J].兵工自动化，2019，38（4）：29-32.
LI Y Y，TIAN R J，ZHANG X X.Moving object tracking method based on frame difference and Kalman filter[J].Ordnance Industry Automation，2019，38（4）：29-32.
[40] QIAN X Y，HAN L，ZHANG Y L，et al.An object tracking method using deep learning and adaptive particle filter for night fusion image[C]//2017 International Conference on Progress in Informatics and Computing，2017：138-142.
[41] ZHAO H Q，XIANG K，CAO S X，et al.Robust visual tracking via CAMShift and structural local sparse appearance model[J].Journal of Visual Communication and Image Representation，2016，34：176-186.
[42] 王彩红，沈燕飞，王毅，等.基于时空上下文的手势跟踪与识别[J].计算机工程与应用，2016，52（9）：202-207.
WANG C H，SHEN Y F，WANG Y，et al.Gesture tracking and recognition based on spatio-temporal context[J].Computer Engineering and Applications，2016，52（9）：202-207.
[43] 齐静，徐坤，丁希仑.机器人视觉手势交互技术研究进展[J].机器人，2017，39（4）：565-584.
QI J，XU K，DING X L.Vision-based hand gesture recognition for human-robot interaction：a review[J].Robot，2017，39（4）：565-584.
[44] PISHARADY P K，VADAKKEPAT P，POH L A.Attention based segmentation and recognition algorithm for hand postures against complex backgrounds[M]//Computational intelligence in multi-feature visual pattern recognition.Singapore：Springer，2014：107-131.
[45] LECUN Y，BENGIO Y，HINTON G.Deep learning[J].Nature，2015，521（7553）：436-444.
[46] SHIN S，KIM W Y.Skeleton-based dynamic hand gesture recognition using a part-based GRU-RNN for gesture-based interface[J].IEEE Access，2020，8：50236-50243.
[47] QIANG B，ZHAI Y J，ZHOU M L，et al.Squeeze-net and fusion network-based accurate fast fully convolutional network for hand detection and gesture recognition[J].IEEE Access，2021，9：77661-77674.
[48] ALNAIM N，ABBOD M，ALBAR A.Hand gesture recognition using convolutional neural network for people who have experienced a stroke[C]//2019 3rd International Symposium on Multidisciplinary Studies and Innovative Technologies，2019：1-6.
[49] CHUNG H，CHUNG Y，TSAI W.An efficient hand gesture recognition system based on deep CNN[C]//2019 IEEE International Conference on Industrial Technology，Melbourne，2019：853-858.
[50] BAO P，MAQUEDA A I，DEL-BLANCO C R，et al.Tiny hand gesture recognition without localization via a deep convolutional network[J].IEEE Transactions on Consumer Electronics，2017，63（3）：251-257.
[51] LI G F，TANG H，SUN Y，et al.Hand gesture recognition based on convolution neural network[J].Cluster Computing，2019，22（2）：2719-2729.
[52] LIN H，HSU M，CHEN W.Human hand gesture recognition using a convolution neural network[C]//2014 IEEE International Conference on Automation Science and Engineering，2014：1038-1043.
[53] JOHN V，BOYALI A，MITA S，et al.Deep learning-based fast hand gesture recognition using representative frames[C]//2016 International Conference on Digital Image Computing：Techniques and Applications，2016：1-8.
[54] WU X Y.A hand gesture recognition algorithm based on DC-CNN[J].Multimedia Tools and Applications，2020，79：9193-9205.
[55] 崔虎，黄仁婧，陈青梅，等.基于异步多时域特征的动态手势识别方法[J/OL].计算机工程与应用（2021-05-28）[2021-07-23].http：//kns.cnki.net/kcms/detail/11.2127.TP.20210528.
1355.010.html.
CUI H，HUANG R J，CHEN Q M，et al.Dynamic gesture recognition method based on asynchronous multi-time domain features[J/OL].Computer Engineering and Applications（2021-05-28）[2021-07-23].http：//kns.cnki.net/kcms/detail/11.2127.TP.20210528.1355.010.html.
[56] 王粉花，张强，黄超，等.融合双流三维卷积和注意力机制的动态手势识别[J].电子与信息学报，2021，43（5）：1389-1396.
WANG J H，ZHANG Q，HUANG C，et al.Dynamic gesture recognition combining two-stream 3D convolution with attention mechanisms[J].Journal of Electronics & Information Technology，2021，43（5）：1389-1396.
[57] NGUYEN X S，BRUN L.A neural network based on SPD Manifold learning for skeleton-based hand gesture recognition[C]//2019 IEEE Conference on Computer Vision and Pattern Recognition，Long Beach，2019：12036-12045.
[58] ZENG J H，SUN Y R，WANG F.A natural hand gesture system for intelligent human-computer interaction and medical assistance[C]//2012 Third Global Congress on Intelligent Systems，2012：382-385.
[59] PANSARE J R，GAWANDE S H，INGLE M.Realtime static hand gesture recognition for American sign language（ASL） in complex background[J].Journal of Signal and Information Processing，2012（3）：364-367.
[60] KARABASI M，BHATTI Z，SHAH A.A model for real-time recognition and textual representation of Malaysian sign language through image processing[C]//International Conference on Advanced Computer Science Applications & Technologies，2014：95-200.
[61] KULKARNI V S，DR S.Appearance based recognition of American sign language using gesture segmentation[J].International Journal on Computer Science & Engineering，2010，1（3）：560-565.
[62] SUNUWAR J，BORAH S，PRADHAN R.Gesture recognition approaches and its applicability：a study[C]//2020 4th International Conference on Electronics，Communication and Aerospace Technology，2020：1458-1463.
[63] TAYLOR J，LUFF B，TOPALIAN A，et al.Efficient and precise interactive hand tracking through joint，continuous optimization of pose and correspondences[J].ACM Transactions on Graphics，2016，35（4）：143.
[64] DESAI S，DESAI A.Human computer interaction through hand gestures for home automation using Microsoft Kinect[C]//International Conference on Communication and Networks：Advances in Intelligent Systems and Computing.Singapore：Springer，2017，508：19-29.
[65] STARNER T，WEAVER J，PENTLAND A.Real-time American sign language recognition using desk and wearable computer based video[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，1998，20（12）：1371-1375.