Real-Time Hand Detection Method Based on Lightweight Network

doi:10.3778/j.issn.1002-8331.2204-0084

Abstract

Abstract: Aiming at the hand detection method based on deep learning, the network model has complex structure, slow detection speed and serious memory consumption, which is difficult to meet the needs of real-time human-computer interaction at the mobile terminal, a lightweight network model is proposed to realize the real-time detection of hand. The model is based on SSD. Firstly, the lightweight network MobileNetV2 is used as the backbone feature extraction network of the model to reduce the amount of parameters and computational complexity of the model. Then, aiming at the problem of insufficient feature extraction, a multi-scale convolution and feature fusion module is proposed and applied to three prediction feature layers to increase the adaptability of the network to different scale features by connecting convolution kernels of different sizes. Finally, the candidate box acquisition method is improved. The [K]-means++ clustering algorithm is used to adaptively generate the candidate box suitable for the hand, and the hand is accurately located to improve the detection accuracy of the model. In order to verify the effectiveness of the proposed method, relevant experiments are carried out on two public data sets, Ego Hands and Oxford Hand. The experimental results show that the accuracy of this method on the two data sets is 96.56% and 73.56% respectively, the detection speed is 45.4 FPS and 41.2 FPS, and the memory occupied by the model is only 19.5 MB. Finally, the algorithm is deployed on the mobile terminal for testing, and the results show that this method can provide a lightweight method for hand detection.

Key words: lightweight model, hand detection, SSD, feature enhancement, mobile terminal

摘要： 针对基于深度学习的手部检测方法网络模型结构复杂，检测速度慢，内存消耗严重，难以满足移动端进行实时人机交互的需求，提出一种轻量化网络模型实现对手部的实时检测。该模型以SSD为基础结构，将轻量化网络MobileNetV2作为模型主干特征提取网络，减小模型的参数量和计算复杂度；针对特征提取不充分问题，提出多尺度卷积和特征融合模块，并应用在其中三个预测特征层，通过连接不同尺寸的卷积核，增加网络对不同尺度特征的适应性；改进候选框获取方式，采用[K]-means++聚类算法自适应生成适合手部的候选框，对手部进行准确定位来提高模型的检测精度。为验证所提方法的有效性，在Ego Hands、Oxford Hand两个公开数据集上进行相关实验。实验结果显示，该方法在两个数据集上的准确率分别达96.56%和73.56%，检测速度达45.4?FPS和41.2?FPS，且模型所占内存仅为19.5?MB。最后将算法部署在移动端进行测试，结果表明该方法能够为手部检测提供一个轻量化的方法。

关键词: 轻量化模型, 手部检测, SSD, 特征增强, 移动终端

JIN Fangrui, WANG Yangping, YONG Jiu. Real-Time Hand Detection Method Based on Lightweight Network[J]. Computer Engineering and Applications, 2023, 59(14): 192-200.

靳芳蕊, 王阳萍, 雍玖. 基于轻量化网络的实时手部检测方法[J]. 计算机工程与应用, 2023, 59(14): 192-200.

References

[1] MITTAL A，ZISSERMAN A，TORR P H S.Hand detection using multiple proposals[C]//Proceedings of BMVC，2011.
[2] CHEN D，LI G，SUN Y，et al.Fusion hand gesture segmentation and extraction based on CMOS sensor and 3D sensor[J].International Journal of Wireless and Mobile Computing，2017，12（3）：305-312.
[3] REN S，HE K，GIRSHICK R，et al.Faster R-CNN：towards real-time object detection with region proposal networks[C]//Advances in Neural Information Processing Systems，2015：91-99.
[4] ZHU X，LYU S，WANG X，et al.TPH-YOLOV5：improved YOLOV5 based on transformer prediction head for object detection on drone-captured scenarios[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：2778-2788.
[5] LIU W，ANGUELOV D，ERHAN D，et al.SSD：single shot multibox detector[C]//European Conference on Computer Vision.Cham：Springer，2016：21-37.
[6] LE T H N，ZHENG Y，ZHU C，et al.Multiple scale Faster-RCNN approach to driver’s cell-phone usage and hands on steering wheel detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops，2016：46-53.
[7] ROY K，MOHANTY A，SAHAY R R.Deep learning based hand detection in cluttered environment using skin segmentation[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops，2017：640-649.
[8] 姚炜铭，王晓华，吴楠.基于改进SSD模型的缝纫手部图像检测方法[J].激光与光电子学展，2020，57（18）：189-196.
YAO W M，WANG X H，WU N.Sewing hand image detection method based on improved SSD model[J].Progress in Laser and Optoelectronics，2020，57（18）：189-196.
[9] 程志宇，徐国庆，许犇，等.复杂背景下无锚框手势检测网络的构建[J].计算机工程与设计，2021，42（6）：1742-1748.
CHENG Z Y，XU G Q，XU B，et al.Construction of anchor free gesture detection network in complex background[J].Computer Engineering and Design，2021，42（6）：1742-1748.
[10] SANDLER M，HOWARD A，ZHU M，et al.Mobilenetv2：inverted residuals and linear bottlenecks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：4510-4520.
[11] 范晶晶，薛皓玮，吴欣鸿，等.引入重影特征映射和通道注意力机制的手势识别算法[J].计算机辅助设计与图形学学报，2022，34（3）：403-414.
FAN J J，XUE H W，WU X H，et al.Gesture recognition algorithm with ghosting feature mapping and channel attention mechanism[J].Journal of Computer-Aided Design and Computer Graphics，2022，34（3）：403-414.
[12] 张馨月，降爱莲.融合特征增强和自注意力的SSD小目标检测算法[J].计算机工程与应用，2022，58（5）：247-255.
ZHANG X Y，JIANG A L.SSD small target detection algorithm combining feature enhancement and self attention[J].Computer Engineering and Applications，2022，58（5）：247-255.
[13] SARMA D，BHUYAN M K.Hand detection by two-level segmentation with double-tracking and gesture recognition using deep-features[J].Sensing and Imaging，2022，23（1）：1-29.
[14] MUJAHID A，AWAN M J，YASIN A，et al.Real-time hand gesture recognition based on deep learning YOLOv3 model[J].Applied Sciences，2021，11（9）：4164.
[15] MAZHAR O，NAVARRO B，RAMDANI S，et al.A real-time human-robot interaction framework with robust background invariant hand gesture detection[J].Robotics and Computer-Integrated Manufacturing，2019，60：34-48.
[16] LIU C，SZIRáNYI T.Real-time human detection and gesture recognition for on-board UAV rescue[J].Sensors，2021，21（6）：2180.
[17] BENITEZ-GARCIA G，OLIVARES-MERCADO J，SANCHEZ-PEREZ G，et al.IPN hand：a video dataset and benchmark for real-time continuous hand gesture recognition[C]//International Conference on Pattern Recognition（ICPR），2021：4340-4347.