Multimodal Gesture Recognition Algorithm Based on Shallow 3D Dense Networks

doi:10.3778/j.issn.1002-8331.1806-0271

Computer Engineering and Applications ›› 2019, Vol. 55 ›› Issue (19): 166-172.DOI: 10.3778/j.issn.1002-8331.1806-0271

Previous Articles Next Articles

Multimodal Gesture Recognition Algorithm Based on Shallow 3D Dense Networks

DENG Zhifang, YUAN Jiazheng, LIU Hongzhe, YUAN Chunfeng, ZHANG Hongyuan

1.Beijing Key Laboratory of Information Service Engineering, Beijing Union University, Beijing 100101, China
2.Office of Academic Research, Beijing Open University, Beijing 100081, China
3.National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China

Online:2019-10-01 Published:2019-09-30

基于浅三维稠密网的多模态手势识别算法

邓智方，袁家政，刘宏哲，原春锋，张宏源

1.北京联合大学北京市信息服务工程重点实验室，北京 100101
2.北京开放大学科学研究处，北京 100081
3.中国科学院自动化研究所模式识别重点实验室，北京 100190

Abstract

Abstract: Gesture recognition aims at understanding dynamic gestures of the human body, and is one of the most important ways of human-computer interaction. A multimodal gesture recognition method based on a shallow 3D dense network is proposed by extending the two-dimensional dense network into a 3D dense network and adding the Inception structure, which is named Spatial Temporal 3D（ST3D） dense network. The proposed method is evaluated on the Charlearn LAP large-scale Isolated Gesture Dataset（IsoGD）and achieves the best results. Experimental results show that the proposed method can effectively learn short, mid and long term spatiotemporal features of gestures in video samples.

Key words: Spatial Temporal 3D（ST3D） dense network, Inception structure, multimodal, gesture recognition

摘要： 手势识别旨在理解人体的动态手势，是人机交互领域极其重要的交互方式之一。该方法通过将二维稠密网扩展为三维稠密网，并加入Inception结构，提出了一种基于浅三维稠密网的多模态手势识别方法，将其命名为Spatial Temporal 3D（ST3D） dense network。所提出的方法在手势识别公开数据集大规模离散手势数据集（IsoGD）上进行了评估，并取得了目前最好效果。实验证明，所提方法能够有效地学习到视频样本中手势的短期、中期以及长期时空特征。

关键词: ST3D方法, Inception结构, 多模态, 手势识别

DENG Zhifang, YUAN Jiazheng, LIU Hongzhe, YUAN Chunfeng, ZHANG Hongyuan. Multimodal Gesture Recognition Algorithm Based on Shallow 3D Dense Networks[J]. Computer Engineering and Applications, 2019, 55(19): 166-172.

邓智方，袁家政，刘宏哲，原春锋，张宏源. 基于浅三维稠密网的多模态手势识别算法[J]. 计算机工程与应用, 2019, 55(19): 166-172.

[1]	LU Lixia, ZOU Junzhong, GUO Yucheng, ZHANG Jian, WANG Bei. Prediction of Knee Injury Based on Multimodal Fusion [J]. Computer Engineering and Applications, 2021, 57(9): 225-232.
[2]	HAN Wenjing, LUO Xiaoshu, YANG Rixing. Research on Compound Gesture Recognition Method [J]. Computer Engineering and Applications, 2021, 57(4): 108-113.
[3]	WANG Chuanyu, LI Weixiang, CHEN Zhenhuan. Reserch of Multi-modal Emotion Recognition Based on Voice and Video Images [J]. Computer Engineering and Applications, 2021, 57(23): 163-170.
[4]	XIE Yinggang, WANG Quan. Summary of Dynamic Gesture Recognition Based on Vision [J]. Computer Engineering and Applications, 2021, 57(22): 68-77.
[5]	REN Zeyu, WANG Zhenchao, KE Zunwang, LI Zhe, Wushour·Silamu. Survey of Multimodal Data Fusion [J]. Computer Engineering and Applications, 2021, 57(18): 49-64.
[6]	TAN Lixing, LU Jiaqi, ZHANG Xiaonan, LIU Yuhong, ZHANG Rongfen. Improved Ghost Machine Gesture Interaction System Based on Lightweight OpenPose [J]. Computer Engineering and Applications, 2021, 57(16): 159-166.
[7]	HUANG Hongzhan, MENG Zuqiang. Bidirectional Attention Mechanism Based Multimodal Sentiment Classification Method [J]. Computer Engineering and Applications, 2021, 57(11): 119-127.
[8]	LI Song, ZHOU Yatong, CHI Yue, HE Jingfei, ZHANG Shili. Application of Gaussian Process Mixture Model on Network Traffic Prediction [J]. Computer Engineering and Applications, 2020, 56(5): 186-193.
[9]	TIAN Yuan, LI Fangdi. Research Review on Human Body Gesture Recognition Based on Depth Data [J]. Computer Engineering and Applications, 2020, 56(4): 1-8.
[10]	SUN Yingying, JIA Zhentang, ZHU Haoyu. Survey of Multimodal Deep Learning [J]. Computer Engineering and Applications, 2020, 56(21): 1-10.
[11]	ZHANG Junhao, WU Fei, ZHU Hai. WiFi Gesture Recognition Method Based on GD-Kmeans and Fresnel Theory [J]. Computer Engineering and Applications, 2020, 56(19): 126-131.
[12]	ZHANG Longjiao, ZENG Xiaoqin. Research on Gesture Recognition of sEMG Based on Deep Neural Network [J]. Computer Engineering and Applications, 2019, 55(23): 113-119.
[13]	XIAO Ziya, LIU Sheng, HAN Feifei, YU Jianfang. Crow Search Algorithm Based on Directing of Sine Cosine Algorithm [J]. Computer Engineering and Applications, 2019, 55(21): 52-59.
[14]	TONG Jingran, MAO Li, SUN Jun. Multimodal Pedestrian Detection Algorithm Based on Fusion Feature Pyramids [J]. Computer Engineering and Applications, 2019, 55(19): 214-222.
[15]	SUN Yu1，2, YUAN Zhenming1，2, SUN Xiaoyan1，2. Dynamic Gesture Recognition Based on Leap Motion [J]. Computer Engineering and Applications, 2019, 55(13): 151-157.

Multimodal Gesture Recognition Algorithm Based on Shallow 3D Dense Networks

基于浅三维稠密网的多模态手势识别算法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics