Fingertip Detection Model Based on UGF-Net

doi:10.3778/j.issn.1002-8331.2010-0235

Abstract

Abstract: In the field of human-computer interaction, accurate fingertip detection has a great impact on the richness and flexibility of interaction. However, due to the small size of fingertip, accurate and robust fingertip detection is still a challenging task. In order to improve the accuracy of fingertip detection, this paper proposes a fingertip detection model based on depth convolution neural network UGF-Net（unified-gesture-and-fingertip- network）. The model can be used for fingertip detection and gesture recognition at the same time. The YOLO algorithm is used to extract the gesture area, and the FCNN output visual Gaussian heat map is used to realize fingertip detection. Finally, the effectiveness and robustness of the proposed fingertip detection model are verified by experiments. The model is tested on the SCUT-Ego-Gesture data set. The results show that the accuracy of fingertip detection can reach 99.8%, and the average frame rate of real-time video image is 34.5 frame/s, which meets the requirements of real-time.

Key words: convolutional neural network, deep learning, fingertip detection, gesture recognition, target detection

摘要： 在人机交互领域，精确的人手指尖检测对交互的丰富度、灵活度有很大影响。然而，由于指尖的尺寸较小，精确、鲁棒的指尖检测目前仍然是一项颇具挑战性的任务。为了提升指尖检测的准确率与实时性，提出一种基于深度卷积神经网络的指尖检测模型UGF-Net（unified-gesture-and-fingertip-network）。该模型可以同时进行指尖检测与手势识别，利用YOLO算法来提取手势区域，通过FCNN输出可视化高斯热图来实现指尖检测。通过实验验证该指尖检测模型的有效性和鲁棒性，在SCUT-Ego-Gesture数据集上对模型进行了测试，结果表明，指尖检测的准确率可达到99.8%，且实时视频图像的平均帧率达到34.5?frame/s，满足实时性的要求。

关键词: 卷积神经网络, 深度学习, 指尖检测, 手势识别, 目标检测

LIU Jia, BIAN Fangzhou, CHEN Dapeng, LI Weibin. Fingertip Detection Model Based on UGF-Net[J]. Computer Engineering and Applications, 2022, 58(5): 225-231.

刘佳, 卞方舟, 陈大鹏, 李为斌. 基于UGF-Net的指尖检测模型[J]. 计算机工程与应用, 2022, 58(5): 225-231.

References

[1] 高晨，张亚军.基于Kinect深度图像的指尖检测与手势识别[J].计算机系统应用，2017，26（4）：192-197.
GAO Chen，ZHANG Yajun.Fingertip detection and gesture recognition based on Kinect depth image[J].Computer System Applications，2017，26（4）：192-197.
[2] 王艳，徐诗艺，谌海云.基于特征距离加权的手势识别[J].计算机科学，2017（z1）：220-223.
WANG Yan，XU Shiyi，CHEN Haiyun.Gesture recognition based on weighted feature distance[J].Computer Science，2017（z1）：220-223.
[3] 杨洋，郑紫微，孙兹昂，等.基于肤色分割与深度学习的手势识别[J].数据通信，2019（5）：8.
YANG Yang，ZHENG Ziwei，SUN Zi’ang，et al.Gesture recognition based on skin color segmentation and deep learning[J].Data Communication，2019（5）：8.
[4] WANG J，QIAN J，YING R，et al.Hand motion recognition based on a 3D fingertip detection fusion method[C]//2017 International Conference on Computational Science and Computational Intelligence（CSCI），2017：510-515.
[5] 孟浩，尹维考，李洪进，等.基于深度信息的指尖检测-跟踪-监督算法[J].仪器仪表学报，2019，40（6）：171-180.
MENg Hao，YIN Weikao，LI Hongjin，et al.Fingertip detection tracking supervision algorithm based on depth information[J].Acta Instrumenta Sinica，2019，40（6）：171-180.
[6] 贠卫国，史其琦，王民.基于深度卷积神经网络的多特征融合的手势识别[J].液晶与显示，2019，34（4）：417-422.
HU Weiguo，SHI Qiqi，WANG Min.Gesture recognition based on multi feature fusion based on deep convolution neural network[J].Liquid Crystal and Display，2019，34（4）：417-422.
[7] HUANG Y，LIU X，JIN L，et al.Deepfinger：a cascade convolutional neuron network approach to finger key point detection in egocentric vision with mobile camera[C]//2015 IEEE International Conference on Systems，Man，and Cybernetics，2015：2944-2949.
[8] NGUYEN H D，KIM Y C，KIM S H，et al.A method for fingertips detection using RGB-D image and convolution neural network[C]//2017 13th International Conference on Natural Computation，Fuzzy Systems and Knowledge Discovery（ICNC-FSKD），2017：783-785.
[9] WU W，LI C，CHENG Z，et al.YOLSE：egocentric fingertip detection from single RGB images[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops，2017：623-630.
[10] REDMON J，DIVVALA S，GIRSHICK R，et al.You only look once：unified，real-time object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：779-788.
[11] 张富凯，杨峰，李策，等.基于YOLOv3的快速车辆检测方法[J].计算机工程与应用，2019，55（2）：12-20.
ZHANG Fukai，YANG Feng，LI Ce，et al.Rapid vehicle detection method based on YOLOV3[J].Computer Engineering and Applications，2019，55（2）：12-20.
[12] 徐诚极，王晓峰，杨亚东.Attention-YOLO：引入注意力机制的YOLO检测算法[J].计算机工程与应用，2019，55（6）：13-23.
XU Chengji，WANG Xiaofeng，YANG Yadong.Attention YOLO：YOLO detection algorithm that introduces attention mechanism[J].Computer Engineering and Applications，2019，55（6）：13-23.
[13] LIU X，HUANG Y，ZHANG X，et al.Fingertip in the eye：an attention-based method for real-time hand tracking and fingertip detection in egocentric videos[C]//Chinese Conference on Pattern Recognition，2016：145-154.
[14] 潘秋羽，王伟，王明明，等.基于卷积特征建模的目标检测方法[J].计算机应用研究，2021（3）：928-931.
PAN Qiuyu，WANG Wei，WANG Mingming，et al.Object detection based on convolutional feature modeling[J].Application Research of Computers，2021（3）：928-931.
[15] 郑晓旭.第一视角下的手势识别方法研究与交互系统设计[D].广州：华南理工大学，2018.
ZHENG Xiaoxu.Research on gesture recognition method and design of interactive system from the first perspective[D].Guangzhou：South China University of Technology，2018.
[16] PFISTER T，CHARLES J，ZISSERMAN A.Flowing convnets for human pose estimation in videos[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：1913-1921.
[17] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409.1556，2014.
[18] 陆嘉慧，张树美，赵俊莉.基于深度学习的面部表情识别研究[J].计算机应用研究，2020，37（4）：966-972.
LU Jiahui，ZHANG Shumei，ZHAO Junli.Facial expression recognition based on deep learning[J].Application Research of Computers，2020，37（4）：966-972.