Gesture Recognition of Traffic Police Based on Spatio-Temporal Feature Fusion

doi:10.3778/j.issn.1002-8331.2212-0110

Abstract

Abstract: In recent years, with the development of human pose estimation technology, gesture recognition technology based on skeleton key points comes into being. This paper proposes a GCPM-AGRU model for gesture recognition of traffic police. In order to locate the key points of human body more accurately, the convolution pose machine (CPM) is improved. Firstly, the idea of residuals, channel split and channel shuffle are added to the feature extraction module, so that the designed feature extraction module can better extract image features. In addition, the parallel multi-branch Inception4d structure is added in the first stage of CPM, which makes the CPM network have the idea of multi-scale feature fusion, and effectively improves the problem of human key point location. Secondly, a GRU based on attention mechanism is proposed, which allocates different weights to each frame to achieve different degrees of attention to each frame, so as to obtain better time information. Finally, it combines the spatio-temporal feature information to carry out traffic police gesture recognition. The accuracy of traffic police gesture recognition reaches 93.7%, which is 2.95 percentage points higher than before the improvement of network.

Key words: gesture recognition, human body key points, convolution attitude machine, GRU, spatio-temporal feature information

摘要： 近年来，随着人体姿态估计技术的发展，基于骨架关键点的手势识别技术应运而生。提出了一个GCPM-AGRU模型进行交通警察手势识别。为了更准确地定位人体关键点，对卷积姿态机（CPM）进行改进。在特征提取模块中加入残差思想、通道拆分和通道重组，设计后的特征提取模块更好提取图片特征；在CPM第一阶段加入并行多分支Inception4d结构，使CPM网络具有多尺度特征融合思想，有效改进对人体关键点定位的问题；提出基于注意力机制的GRU，通过为每帧分配不同权重来达到对每帧不同程度的关注，从而更好获取时间信息；结合时空特征信息进行交通警察手势识别。交通警察手势识别的准确度达到了93.7%，相比网络改进之前提高了2.95个百分点。

关键词: 手势识别, 人体关键点, 卷积姿态机, GRU, 时空特征信息

DU Bing, ZHAO Ji. Gesture Recognition of Traffic Police Based on Spatio-Temporal Feature Fusion[J]. Computer Engineering and Applications, 2024, 60(8): 250-257.

杜兵, 赵骥. 基于时空特征融合的交通警察手势识别[J]. 计算机工程与应用, 2024, 60(8): 250-257.

References

[1] TRAN D, BOURDEV L, FERGUS R, et al. Learning spatiotemporal features with 3D convolutional networks[C]//Proceedings of the IEEE International Conference on Computer Vision, 2015: 4489-4497.
[2] 韩文静, 罗晓曙, 杨日星. 一种复合型手势识别方法研究[J]. 计算机工程与应用, 2021, 57(4): 108-113.
HAN W J, LUO X S, YANG R X. Research on a compound gesture recognition method[J]. Computer Engineering and Applications, 2021, 57(4): 108-113.
[3] SHI L, ZHANG Y, CHENG J, et al. Skeleton-based action recognition with directed graph neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 7912-7921.
[4] 谭立行, 鲁嘉淇, 张笑楠, 等. 基于轻量级OpenPose改进的幻影机手势交互系统[J]. 计算机工程与应用, 2021, 57(16): 159-166.
TAN L X, LU J Q, ZHANG X N, et al. Improved ghost machine gesture interaction system based on lightweight OpenPose[J]. Computer Engineering and Applications, 2021, 57(16): 159-166.
[5] CAI X, GUO T, WU X, et al. Gesture recognition method based on wireless data glove with sensors[J]. Sensor Letters, 2015, 13(2): 134-137.
[6] ZHANG T, LIN H, JU Z, et al. Hand gesture recognition in complex background based on convolutional pose machine and fuzzy Gaussian mixture models[J]. International Journal of Fuzzy Systems, 2020, 22(4): 1330-1341.
[7] HE J, ZHANG C, HE X, et al. Visual recognition of traffic police gestures with convolutional pose machine and handcrafted features[J]. Neurocomputing, 2020, 390: 248-259.
[8] KIM Y, MYUNG H. Gesture recognition with a skeleton-based keyframe selection module[J]. arXiv:2112.01736, 2021.
[9] 刘亮, 蒲浩洋. 基于 LSTM 的多维度特征手势实时识别[J]. 计算机科学, 2021, 48(8): 328-333.
LIU L, PU H Y. Real-time gesture recognition of multi-dimensional feature based on LSTM[J]. Computer Science, 2021, 48(8): 328-333.
[10] 侯莹莹, 李建军. 基于时空特征融合的动态手势识别[J]. 内蒙古科技大学学报, 2022, 41(2): 187-193.
HOU Y Y, LI J J. Dynamic gesture recognition based on spatio-temporal feature fusion[J]. Journal of Inner Mongolia University of Science and Technology, 2022, 41(2): 187-193.
[11] 崔虎, 黄仁婧, 陈青梅, 等. 基于异步多时域特征的动态手势识别方法[J]. 计算机工程与应用, 2022, 58(21): 163-171.
CUI H, HUANG R J, CHEN Q M, et al. Dynamic gesture recognition based on asynchronous multi time domain features[J]. Computer Engineering and Applications, 2022, 58(21): 163-171.
[12] LIU J, WANG Y, XIANG S, et al. HAN: an efficient hierarchical self-attention network for skeleton-based gesture recognition[J]. arXiv:2106.13391, 2021.
[13] SI C, CHEN W, WANG W, et al. An attention enhanced graph convolutional LSTM network for skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 1227-1236.
[14] 胡宗承, 周亚同, 史宝军, 等. 结合注意力机制与特征融合的静态手势识别算法[J]. 计算机工程, 2022, 48(4): 240-246.
HU Z C, ZHOU Y T, SHI B J, et al. Static gesture recognition algorithm combining attention mechanism and feature fusion[J]. Computer Engineering, 2022, 48(4): 240-246.
[15] SZEGEDY C, LIU W, JIA Y, et al. Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 1-9.
[16] YAO K, COHN T, VYLOMOVA K, et al. Depth-gated recurrent neural networks[J]. arXiv:1508.03790, 2015.
[17] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
[18] PIGOU L, VAN DEN OORD A, DIELEMAN S, et al. Beyond temporal pooling: recurrence and temporal convolutions for gesture recognition in video[J]. International Journal of Computer Vision, 2018, 126(2): 430-439.
[19] 郭宗鑫. 基于深度学习的交警手势识别研究[D]. 银川: 宁夏大学, 2021.
GUO Z X. Traffic police gesture recognition based on deep learning[D]. Yinchuan: Ningxia University, 2021.
[20] HUANG G, YANG J, JING S. Moving trajectory based traffic police gesture recognition via time series classification[C]//Proceedings of the 2021 17th International Conference on Computational Intelligence and Security, 2021: 318-322.
[21] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 4700-4708.