Dynamic Gesture Recognition Method Based on Asynchronous Multi-Time Domain Features

doi:10.3778/j.issn.1002-8331.2103-0532

Abstract

Abstract: Accurate and timely gesture recognition is of great significance in augmented reality technology. Aiming at the spatio-temporal features that characterize complex gesture sequences, a gesture recognition method based on asynchronous multi-time domain features is proposed. It extracts the short-term spatiotemporal features of the video sequences with different temporal resolution through a lightweight three-dimensional convolutional network, and learns long-term spatio-temporal features through an improved convolutional long short-term memory network. The temporal and spatial features of different steps are fused into asynchronous multi-time domain features to classify and recognize gestures. Compared with other state-of-the-art approaches, the experimental results show that the proposed method has a high dynamic gesture recognition accuracy.

Key words: augmented reality, dynamic gesture recognition, spatiotemporal feature, asynchronous feature, fusion prediction

摘要： 准确及时地手势识别在增强现实技术中具有重要的意义。针对表征复杂手势序列的时空特征，提出了一种基于异步多时域时空特征的手势识别方法。该方法通过轻量级三维卷积网络提取视频序列的不同时间步态的短期时空特征，通过改进的卷积长短期记忆网络学习长期时空特征，将不同步态的时空特征融合为异步多时域特征，以此来对手势进行分类识别。通过与其他主流方法进行比较，实验结果证明了提出的方法具有较高的动态手势识别率。

关键词: 增强现实, 动态手势识别, 时空特征, 异步特征, 融合预测

CUI Hu, HUANG Renjing, CHEN Qingmei, HUANG Chuhua. Dynamic Gesture Recognition Method Based on Asynchronous Multi-Time Domain Features[J]. Computer Engineering and Applications, 2022, 58(21): 163-171.

崔虎, 黄仁婧, 陈青梅, 黄初华. 基于异步多时域特征的动态手势识别方法[J]. 计算机工程与应用, 2022, 58(21): 163-171.

References

[1] ASADI-AGHBOLAGHI M，CLAPéS A，BELLANTONIO M，et al.A survey on deep learning based approaches for action and gesture recognition in image sequences[C]//2017 IEEE International Conference on Automatic Face & Gesture Recognition（FG 2017），Washington，DC，USA，2017：476-483.
[2] 孙玉，袁贞明，孙晓燕.基于Leap Motion的动态手势识别[J].计算机工程与应用，2019，55（13）：151-157.
SUN Y，YUAN Z M，SUN X Y.Dynamic gesture recognition based on leap motion[J].Computer Engineering and Applications，2019，55（13）：151-157.
[3] 谭立行，鲁嘉淇，张笑楠，等.基于轻量级OpenPose改进的幻影机手势交互系统[J].计算机工程与应用，2021，57（16）：159-166.
TAN L X，LU J Q，ZHANG X N，et al.Improved ghostmachine gesture interaction system based on lightweight OpenPose[J].Computer Engineering and Applications，2021，57（16）：159-166.
[4] SIMONYAN K，ZISSERMAN A.Two-stream convolutional networks for action recognition in videos[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems（NLPS），Cambridge，USA，2014：568-576.
[5] TRAN A，CHEONG L F.Two-stream flow-guided convolutional attention networks for action recognition[C]// 2017 IEEE International Conference on Computer Vision Workshop（ICCVW），Venice，Italy，2017：3110-3119.
[6] LIU J，FURUSAWA K，TATEYAMA T，et al.An improved hand gesture recognition with two-stage convolution neural networks using a hand color image and its pseudo-depth image[C]//2019 IEEE International Conference on Image Processing（ICIP），2019：375-379.
[7] K?PüKLü O，GUNDUZ A，KOSE N，et al.Real-time hand gesture detection and classification using convolutional neural networks[C]//2019 14th IEEE International Conference on Automatic Face & Gesture Recognition（FG 2019），Lille，France，2019：1-8.
[8] HARA K，KATAOKA H，SATOH Y.Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and ImageNet?[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），Salt Lake City，UT，USA，2018：6546-6555.
[9] MIN Y，ZHANG Y，CHAI X，et al.An efficient PointLSTM for point clouds based gesture recognition[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），Seattle，WA，USA，2020：5760-5769.
[10] SHI X，CHEN Z，WANG H，et al.Convolutional LSTM network：a machine learning approach for precipitation nowcasting[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems（NIPS），Cambridge，MA，USA，2015：802-810.
[11] LIU Z，HU H，ZHANG J.Spatiotemporal fusion networks for video action recognition[J].Neural Processing Letters，2019，50（1）：1877-1890.
[12] 余思泉，曹江涛，李平，等.基于空间金字塔特征包的手势识别算法[J].智能系统学报，2015，10（3）：429-435.
YU S Q，CAO J T，LI P，et al.Hand gesture recognition based on the spatial pyramid bag of features[J].CAAI Transactions on Intelligent Systems，2015，10（3）：429-435.
[13] 胡敦利，柯浩然，张维.基于Kinect和ROS的骨骼轨迹人体姿态识别研究[J].高技术通讯，2020，30（2）：177-184.
HU D L，KE H R，ZHANG W.Research on human body attitude recognition based on Kinect and ROS[J].High Technology Letters，2020，30（2）：177-184.
[14] 杨世强，罗晓宇，李小莉，等.基于DBN-HMM的人体动作识别[J].计算机工程与应用，2019，55（15）：169-176.
YANG S Q，LUO X Y，LI X L，et al.Human action recognition based on DBN-HMM[J].Computer Engineering and Applications，2019，55（15）：169-176.
[15] VAROL G，LAPTEV I，SCHMID C.Long-term temporal convolutions for action recognition[J].IEEE Transactions on Pattern Analysis & Machine Intelligence，2018，40（6）：1510-1517.
[16] SARKAR A，GEPPERTH A，HANDMANN U，et al.Dynamic hand gesture recognition for mobile systems using deep LSTM[C]//International Conference on Intelligent Human Computer Interaction，Evry，France，2017：19-31.
[17] ZHU G，ZHANG L，MEI L，et al.Large-scale isolated gesture recognition using pyramidal 3D convolutional networks[C]//2016 23rd International Conference on Pattern Recognition（ICPR），Cancun，Mexico，2016：19-24.
[18] LIU W，ANGUELOV D，ERHAN D，et al.SSD：single shot MultiBox detector[C]//European Conference on Computer Vision（ECCV），Amsterdam，The Netherlands，2016：21-37.
[19] FEICHTENHOFER C，FAN H，MALIK J，et al.SlowFast networks for video recognition[C]//2019 IEEE/CVF International Conference on Computer Vision（ICCV），Seoul，Korea（South），2019：6201-6210.
[20] CHEN Y，KALANTIDIS Y，LI J，et al.Multi-fiber networksfor video recognition[C]//15th European Conference on Computer Vision（ECCV），Munich，Germany，2018：364-380.
[21] TRAN D，BOURDEV L，FERGUS R，et al.Learning spatiotemporal features with 3D convolutional networks[C]//2015 IEEE International Conference on Computer Vision（ICCV），Santiago，Chile，2015：4489-4497.
[22] TRAN D，WANG H，TORRESANI L，et al.A closer look at spatiotemporal convolutions for action recognition[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），Salt Lake City，UT，USA，2018：6450-6459.
[23] RUSSAKOVSKY O，DENG J，SU H，et al.ImageNet large scale visual recognition challenge[J].International Journal of Computer Vision，2015，115（3）：211-252.
[24] HE K，ZHANG X，REN S，et al.Delving deep into rectifiers：surpassing human-level performance on ImageNet classification[C]//2015 IEEE International Conference on Computer Vision（ICCV），Santiago，Chile，2015：1026-1034.
[25] 程子均，马六章，张翼翔.基于LSTM-FC的瓦斯浓度时空分布预测[J].计算机工程与应用，2020，56（16）：258-264.
CHEN Z J，MA L Z，ZHANG Y X.Prediction of spatiotemporal distribution of gas concentration based on LSTM-FC model[J].Computer Engineering and Applications，2020，56（16）：258-264.
[26] MATERZYNSKA J，BERGER G，BAX I，et al.The Jester dataset：a large-scale video dataset of human gestures[C]//2019 IEEE/CVF International Conference on Computer Vision Workshop（ICCVW），Seoul，Korea（South），2019：2874-2882.
[27] LIU L，SHAO L.Learning discriminative representations from RGB-D video data[C]//International Joint Conference on Artificial Intelligence（IJCAI），Beijing，China，2013：1493-1500.
[28] BALDISSERA F B，VARGAS F L.A light implementation of a 3D convolutional network for online gesture recognition[J].IEEE Latin America Transactions，2020，18（2）：319-326.
[29] LI Y，XU L，SHU W，et al.AutoGesNet：auto gesture recognition network based on neural architecture search[C]//2020 12th International Conference on Advanced Computational Intelligence（ICACI），Dali，China，2020：57-262.
[30] NISHIDA N，NAKAYAMA H.Multimodal gesture recognition using multi-stream recurrent neural network[C]// Pacific-Rim Symposium on Image and Video Technology（PSIVT），Auckland，New Zealand，2015：682-694.
[31] 马正文，蔡坚勇，刘磊，等.基于RGB-D视频的多模态手势识别[J].计算机系统应用，2018，27（12）：234-239.
MA Z W，CAI J Y，LIU L，et al.Multimodal gesture recognition based on RGB-D video[J].Computer Systems & Applications，2018，27（12）：234-239.