Abstract:

Human-Computer Interaction（HCI） is mostly realized by keyboard and mouse at present. The interactive algorithm based on deep learning gesture recognition is relatively low in accuracy, real-time performance and system stability. This paper proposes a novel ghost machine gesture interaction system improved on lightweight OpenPose. Through lightweight OpenPose, human hand is modeled into 21 key points. Then on the basis of MobileNetV1, Part Affinity Fields（PAF） method is applied to detect key points of human hand and draw a simple skeleton diagram. To further improve HCI’s real-time performance, Ghost Module is used to reduce the dimension of convolution layer, and the same recognition effect is obtained with fewer hardware resources. Finally, the verification environment is set up, and the pattern matching is carried out according to the drawn skeleton diagram of human hand. The interactive control instructions are generated based on the matching recognition results and then are transmitted to the Arduino UNO platform through Bluetooth communication to control the car to realize interactive response. After the initial training, the system achieves an accuracy rate of 58.7% on COCO2017 verification set, which maintains the recognition effect of the original OpenPose network and lightweight OpenPose network, and it realizes the speed of 32-36 frames per second and high gesture recognition rate on the household PC.

Key words: human-computer interaction, gesture recognition, deep learning, key point detection, ghost module

摘要：

目前人机交互方式多以键盘鼠标为主，而基于深度学习手势识别的交互方式算法准确率不高，且实时性和系统稳定性均有待提升。提出一种新颖的针对轻量级OpenPose进行改进的幻影机手势交互系统。采用轻量级OpenPose将人手简化建模为21个关键点，以MobileNetV1作为基础模型，应用部分亲和域（Part Affinity Fields，PAF）方法实现人手关键点的检测并画出简化骨骼图。为进一步提升人机交互系统的实时性，采用幻影模块（Ghost Module）对卷积层进行降维，用更少的硬件资源取得同样的识别效果。最后搭建验证环境，根据画出的人手骨骼图进行模式匹配，根据匹配识别结果生成交互控制指令，经由蓝牙通讯将指令传送至Arduino UNO平台控制小车实现交互响应。经过初步训练后，该系统在COCO2017验证集上能实现58.7%的准确率，保持了原始OpenPose网络和轻量OpenPose网络的人手关键点识别效果，在家用PC机上可实现每秒32~36帧的识别速率和较高的手势识别率。

关键词: 人机交互, 手势识别, 深度学习, 关键点检测, 幻影模块