Research Progress of Surface Electromyography Hand Motion Recognition

doi:10.3778/j.issn.1002-8331.2305-0269

Abstract

Abstract: Surface electromyography (sEMG) is a non-invasive method of measuring muscle activity, which contains rich information related to human motion and can be used for hand motion recognition. Hand motion recognition based on sEMG refers to the classification and recognition of hand motions by analyzing the sEMG signals of the hand muscles. Driven by the development of neural networks, sEMG has made great progress in the field of hand motion recognition. However, sEMG is faced with defects such as high noise and poor stability, which cannot be efficiently utilised, bringing great difficulties in acquiring high-precision hand movement recognition models and has hindered the translation and application of research results. This paper summarizes the research progress of sEMG hand motion recognition methods in detail. Firstly, public EMG datasets commonly used in the field of action recognition are introduced, and the self-test EMG set acquisition process is described. Then the existing sEMG hand motion recognition models are classified into three categories according to the different research methods: hand motion recognition based on machine learning, hand motion recognition based on deep learning and hand motion recognition based on hybrid network structure, and the related models are summarised and analyzed respectively, and suggestions are made for the shortcomings. Finally, the problems to be solved and the future development direction of hand action recognition research are prospected.

Key words: surface electromyography (sEMG), hand motion recognition, artificial neural network, algorithm model

摘要： 表面肌电信号（surface electromyography，sEMG）是一种测量肌肉活动的非侵入式方法，蕴含着关联人体运动的丰富信息，可用于手部动作识别。基于sEMG手部动作识别是指通过分析手部肌肉的sEMG信号，实现对手部动作的分类和识别。在神经网络发展的推动下，sEMG在手部动作识别领域取得了显著进展，但sEMG面临着噪声大、稳定性差等缺陷，难以有效利用，给高精度手部动作识别模型的获取带来了巨大困难，阻碍了研究成果的转化应用。详细归纳了sEMG手部动作识别方法的研究进展；介绍了常用于动作识别领域的公开肌电数据集，并介绍了自测肌电数据集采集流程；根据研究方法不同将现有的sEMG手部动作识别模型分为基于机器学习的手部动作识别、基于深度学习的手部动作识别和基于混合网络结构的手部动作识别三类，分别对相关模型进行总结分析，对不足之处提出建议；最后对手部动作识别研究需要解决的问题和未来发展方向进行了展望。

关键词: 表面肌电信号（sEMG）, 手部动作识别, 人工神经网络, 算法模型

LI Zhenjiang, WEI Dejian, FENG Yanyan, YU Fengfan, MA Yifan. Research Progress of Surface Electromyography Hand Motion Recognition[J]. Computer Engineering and Applications, 2024, 60(3): 29-43.

李振江, 魏德健, 冯妍妍, 于丰帆, 马一凡. 表面肌电手部动作识别的研究进展[J]. 计算机工程与应用, 2024, 60(3): 29-43.

References

[1] LI W, SHI P, YU H. Gesture recognition using surface electromyography and deep learning for prostheses hand: state-of-the-art, challenges, and future[J]. Frontiers in Neuroscience, 2021, 15: 621885.
[2] CHEN J, BI S, ZHANG G, et al. High-density surface EMG-based gesture recognition using a 3D convolutional neural network[J]. Sensors, 2020, 20(4): 1201.
[3] QI J, JIANG G, LI G, et al. Intelligent human-computer interaction based on surface EMG gesture recognition[J]. IEEE Access, 2019, 7: 61378-61387.
[4] PARK K H, LEE S W. Movement intention decoding based on deep learning for multiuser myoelectric interfaces[C]// Proceedings of the 2016 4th International Winter Conference on Brain-Computer Interface (BCI), 2016.
[5] XIONG D, ZHANG D, ZHAO X, et al. Deep learning for EMG-based human-machine interaction: a review[J]. IEEE/CAA Journal of Automatica Sinica, 2021, 8(3): 512-533.
[6] MEATTINI R, BENATTI S, SCARCIA U, et al. An sEMG-based human-robot interface for robotic hands using machine learning and synergies[J]. IEEE Transactions on Components, Packaging and Manufacturing Technology, 2018, 8(7): 1149-1158.
[7] AL HARRACH M, BOUDAOUD S, CARRIOU V, et al. Multi-muscle force estimation using data fusion and HD?sEMG: an experimental study[C]//Proceedings of the Fourth International Conference on Advances in Biomedical Engineering (ICABME), 2017: 1-4.
[8] 李自由, 赵新刚, 张弼, 等. 基于表面肌电的意图识别方法在非理想条件下的研究进展[J]. 自动化学报, 2021, 47(5): 955-969.
LI Z Y, ZHANG X G, ZHANG B, et al. Review of sEMG-based motion intent recognition methods in non-ideal conditions[J]. Acta Automatica Sinica, 2021, 47(5): 955-969.
[9] 王逸桐. 基于表面肌电信号的手势动作识别技术的应用研究[D]. 延吉：延边大学, 2021.
WANG Y T. Application research on gesture recognition technology based on surface electromyography[D]. Yanji：Yanbian University, 2021.
[10] DIRACO G, RESCIO G, SICILIANO P, et al. Review on human action recognition in smart living: sensing technology, multimodality, real-time processing, interoperability, and resource-constrained processing[J]. Sensors, 2023, 23(11): 5281.
[11] SOHN R, SON J, HWANG H, et al. Steady-state visual evoked potential (SSVEP)-based rehabilitation training system with functional electrical stimulation[J]. Journal of Biomedical Engineering Research, 2010, 31(5): 359-364.
[12] KONAR A, SAHA S, KONAR A, et al. EEG-gesture based artificial limb movement for rehabilitative applications[J]. Gesture Recognition: Principles, Techniques and Applications, 2018, 724: 243-268.
[13] MU D, LI F, YU L, et al. Study on exercise muscle fatigue based on sEMG and ECG data fusion and temporal convolutional network[J]. Plos One, 2022, 17(12): e0276921.
[14] FRIGO C, CRENNA P. Multichannel SEMG in clinical gait analysis: a review and state-of-the-art[J]. Clinical Biomechanics, 2009, 24(3): 236-245.
[15] SAGAYAM K M, HEMANTH D J. Hand posture and gesture recognition techniques for virtual reality applications: a survey[J]. Virtual Reality, 2017, 21: 91-107.
[16] JOSELLI M, CLUA E. Grmobile: a framework for touch and accelerometer gesture recognition for mobile games[C]//2009 VIII Brazilian Symposium on Games and Digital Entertainment, 2009: 141-150.
[17] RAHIM M A, SHIN J, ISLAM M R. Hand gesture recognition-based non-touch character writing system on a virtual keyboard[J]. Multimedia Tools and Applications, 2020, 79: 11813-11836.
[18] LI F, FEI J. Gesture recognition algorithm based on image information fusion in virtual reality[J]. Personal and Ubiquitous Computing, 2019, 23: 487-497.
[19] 赵新刚, 杨毕, 吴成东, 等. 基于表面肌电信号的人体运动意图识别方法研究[C]//2009全国虚拟仪器大会论文集 (二), 2009.
ZHAO X G, YANG B, WU C D, et al. Research on the recognition method of human motion intention based on surface electromyography signal[C]//Proceedings of the 2009 National Conference on Virtual Instrumentation (II), 2009.
[20] CAMPBELL E, PHINYOMARK A, SCHEME E. Deep cross-user models reduce the training burden in myoelectric control[J]. Frontiers in Neuroscience, 2021, 15: 657958.
[21] LUU D K, NGUYEN A T, JIANG M, et al. Deep learning-based approaches for decoding motor intent from peripheral nerve signals[J]. Frontiers in Neuroscience, 2021, 15: 667907.
[22] ZHAI X, JELFS B, CHAN R H, et al. Self-recalibrating surface EMG pattern recognition for neuroprosthesis control based on convolutional neural network[J]. Frontiers in Neuroscience, 2017, 11: 379.
[23] SULTANA A, AHMED F, ALAM M S. A systematic review on surface electromyography-based classification system for identifying hand and finger movements[J]. Healthcare Analytics, 2022: 100126.
[24] ATZORI M, GIJSBERTS A, KUZBORSKIJ I, et al. Characterization of a benchmark database for myoelectric movement classification[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2014, 23(1): 73-83.
[25] 付梦龙. 基于表面肌电信号的实时在线肌肉疲劳估计方法研究[D]. 合肥：中国科学技术大学, 2020.
FU M L. Research on real-time online muscle fatigue estimation method based on surface electromyography[D]. Hefei：University of Science and Technology of China, 2020.
[26] ORTIZ-CATALAN M, BRANEMARK R, HAKANSSON B. BioPatRec: a modular research platform for the control of artificial limbs based on pattern recognition algorithms[J]. Source Code Biol Med, 2013, 8: 11.
[27] FATIMAH B, SINGH P, SINGHAL A, et al. Hand movement recognition from sEMG signals using Fourier decomposition method[J]. Biocybernetics and Biomedical Engineering, 2021, 41(2): 690-703.
[28] AMMA C, KRINGS T, B?ER J, et al. Advancing muscle-computer interfaces with high-density electromyography[C]//Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 2015: 929-938.
[29] GENG W, DU Y, JIN W, et al. Gesture recognition by instantaneous surface EMG images[J]. Scientific Reports, 2016, 6: 36571.
[30] DAI Q, LI X, GENG W, et al. CAPG-MYO: a muscle-computer interface supporting user-defined gesture recognition[C]//Proceedings of the 9th International Conference on Computer and Communications Management, 2021: 52-58.
[31] DU Y, JIN W, WEI W, et al. Surface EMG-based inter-session gesture recognition enhanced by deep domain adaptation[J]. Sensors, 2017, 17(3): 458.
[32] 丁其川, 熊安斌, 赵新刚, 等. 基于表面肌电的运动意图识别方法研究及应用综述[J]. 自动化学报, 2016, 42(1): 13-25.
DING Q C, XIONG A B, ZHAO X G, et al. Research and application review of motion intention recognition method based on surface electromyography[J]. Acta Automatica Sinica, 2016, 42(1): 13-25.
[33] KANG S, KIM H, PARK C, et al. sEMG-based hand gesture recognition using binarized neural network[J]. Sensors, 2023, 23(3): 1436.
[34] 董泽萍, 仇大伟, 刘静. 表面肌电人体下肢动作识别预测进展[J]. 计算机工程与应用, 2023, 59(14): 39-50.
DONG Z P, QIU D W, LIU J. Progress in human lower extremity motion recognition and prediction based on surface electromyography[J]. Computer Engineering and Applications, 2023, 59(14): 39-50.
[35] CHOWDHURY R H, REAZ M B, ALI M A B M, et al. Surface electromyography signal processing and classification techniques[J]. Sensors, 2013, 13(9): 12431-12466.
[36] XUE J, SUN Z, DUAN F, et al. Underwater sEMG-based recognition of hand gestures using tensor decomposition[J]. Pattern Recognition Letters, 2023, 165: 39-46.
[37] ZHANG Z, HE C, YANG K. A novel surface electromyographic signal-based hand gesture prediction using a recurrent neural network[J]. Sensors (Basel), 2020, 20(14): 3994.
[38] TEPE C, DEMIR M C. Real-time classification of EMG myo armband data using support vector machine[J]. IRBM, 2022, 43(4): 300-308.
[39] BAHADOR A, YOUSEFI M, MARASHI M, et al. High accurate lightweight deep learning method for gesture recognition based on surface electromyography[J]. Computer Methods and Programs in Biomedicine, 2020, 195: 105643.
[40] PHINYOMARK A, PHUKPATTARANONT P, LIMSAKUL C. Feature reduction and selection for EMG signal classification[J]. Expert Systems with Applications, 2012, 39(8): 7420-7431.
[41] UR REHMAN M Z, GILANI S O, WARIS A, et al. A novel approach for classification of hand movements using surface EMG signals[C]//2017 IEEE International Symposium on Signal Processing and Information Technology (ISSPIT), 2017: 265-269.
[42] ATZORI M, GIJSBERTS A, CASTELLINI C, et al. Electromyography data for non-invasive naturally-controlled robotic hand prostheses[J]. Scientific Data, 2014, 1(1): 1-13.
[43] NARAYAN Y. SEMG signal classification using KNN classifier with FD and TFD features[J]. Materials Today: Proceedings, 2021, 37: 3219-3225.
[44] AL-TIMEMY A H, BUGMANN G, ESCUDERO J, et al. Classification of finger movements for the dexterous hand prosthesis control with surface electromyography[J]. IEEE Journal of Biomedical and Health Informatics, 2013, 17(3): 608-618.
[45] NOEMI GOZZI L M, MERCORIO F, PEDROCCHI A. XAI for myo-controlled prosthesis: explaining EMG data for hand gesture classification[J]. Knowledge-Based Systems, 2022, 240.
[46] SHAO J, NIU Y, XUE C, et al. Single-channel SEMG using wavelet deep belief networks for upper limb motion recognition[J]. International Journal of Industrial Ergonomics, 2020, 76: 1-10.
[47] DUAN F, REN X, YANG Y. A gesture recognition system based on time domain features and linear discriminant analysis[J]. IEEE Transactions on Cognitive and Developmental Systems, 2021, 13(1): 200-208.
[48] NAIK G R, ACHARYYA A, NGUYEN H T. Classification of finger extension and flexion of EMG and Cyberglove data with modified ICA weight matrix[C]//Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2014: 3829-3832.
[49] BENALCAZAR M E, MOTOCHE C, ZEA J A, et al. Real-time hand gesture recognition using the Myo armband and muscle activity detection[C]//2017 IEEE Second Ecuador Technical Chapters Meeting (ETCM), 2017: 1-6.
[50] BERGIL E, ORAL C, ERGUL E U. Efficient hand movement detection using k-means clustering and k-nearest neighbor algorithms[J]. Journal of Medical and Biological Engineering, 2020, 41(1): 11-24.
[51] CHEN X, WANG Z J. Pattern recognition of number gestures based on a wireless surface EMG system[J]. Biomedical Signal Processing and Control, 2013, 8(2): 184-192.
[52] MALEKI A N A A. Artificial neural network classifier in comparison with LDA and LS-SVM classifiers to recognize 52 hand postures and movements[C]//2014 4th International Conference on Computer and Knowledge Engineering (ICCKE), 2014.
[53] 蔡芝蔚, 吴淑燕, 宋俊锋. 基于 SVM 和组合优化模型的手势识别[J]. 系统仿真学报, 2016, 28(8): 1812-1817.
CAI Z W, WU S Y, SONG J F. Study on hand gesture recognition and portfolio optimization model based on SVM[J]. Journal of System Simulation, 2016, 28(8): 1812-1817.
[54] 朱旭, 刘静, 董泽萍, 等. 基于表面肌电图手势动作意图识别的系统综述[J]. 中国康复理论与实践, 2022, 28(9): 1032-1038.
ZHU X, LIU J, DONG Z P, et al. Gesture action intent recognition based on surface electromyography:a systematic review[J]. Chinese Rehabilitation Theory and Practice, 2022, 28(9): 1032-1038.
[55] DING Z, YANG C, TIAN Z, et al. sEMG-based gesture recognition with convolution neural networks[J]. Sustainability, 2018, 10(6): 1865.
[56] CHEN L, FU J, WU Y, et al. Hand gesture recognition using compact CNN via surface electromyography signals[J]. Sensors, 2020, 20(3): 672.
[57] C?Té-ALLARD U, FALL C L, CAMPEAU-LECOURS A, et al. Transfer learning for sEMG hand gestures recognition using convolutional neural networks[C]//2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2017: 1663-1668.
[58] TAM S, BOUKADOUM M, CAMPEAU-LECOURS A, et al. Intuitive real-time control strategy for high-density myoelectric hand prosthesis using deep and transfer learning[J]. Scientific Reports, 2021, 11(1): 11275.
[59] QI S, WU X, CHEN W H, et al. sEMG-based recognition of composite motion with convolutional neural network[J]. Sensors and Actuators A: Physical, 2020, 311:1-11.
[60] SAMADANI A. Gated recurrent neural networks for EMG-based hand gesture classification. A comparative study[C]//2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2018: 1-4.
[61] SIM?O M, NETO P, GIBARU O. EMG-based online classification of gestures with recurrent neural networks[J]. Pattern Recognition Letters, 2019, 128: 45-51.
[62] WANG C, GUO W, ZHANG H, et al. sEMG-based continuous estimation of grasp movements by long-short term memory network[J]. Biomedical Signal Processing and Control, 2020, 59:101774.
[63] TSINGANOS B C P, CORNELIS J, JANSEN B, et al. Improved gesture recognition based on sEMG signals and TCN[C]//2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019: 1169-1173.
[64] BETTHAUSER J L, KRALL J T, BANNOWSKY S G, et al. Stable responsive EMG sequence prediction and adaptive reinforcement with temporal convolutional networks[J]. IEEE Transactions on Biomedical Engineering, 2020, 67(6): 1707-1717.
[65] BETTHAUSER J L, KRALL J T, KALIKI R R, et al. Stable electromyographic sequence prediction during movement transitions using temporal convolutional networks[C]//2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), 2019:1046-1049.
[66] ZANGHIERI M, CONTI F, BURRELLO A，et al. Temporal variability analysis in sEMG hand grasp recognition using temporal convolutional networks[C]//2020 2nd IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), 2020: 228-232.
[67] ZANGHIERI M, BENATTI S, BURRELLO A, et al. Robust real-time embedded EMG recognition framework using temporal convolutional networks on a multicore IoT processor[J]. IEEE Transactions on Biomedical Circuits and Systems, 2020, 14(2): 244-256.
[68] COTE-ALLARD U, CAMPBELL E, PHINYOMARK A, et al. Interpreting deep learning features for myoelectric control: a comparison with handcrafted features[J]. Frontiers in Bioengineering and Biotechnology, 2020, 8: 158.
[69] CHEN H, TONG R, CHEN M, et al. A hybrid CNN-SVM classifier for hand gesture recognition with surface emg signals[C]//2018 International Conference on Machine Learning and Cybernetics (ICMLC), 2018: 619-624.
[70] HU Y, WONG Y, WEI W, et al. A novel attention-based hybrid CNN-RNN architecture for sEMG-based gesture recognition[J]. PLoS One, 2018, 13(10): e0206049.
[71] FARINA D, JIANG N, REHBAUM H, et al. The extraction of neural information from the surface EMG for the control of upper-limb prostheses: emerging avenues and challenges[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2014, 22(4): 797-809.
[72] OVUR S E, ZHOU X, QI W, et al. A novel autonomous learning framework to enhance sEMG-based hand gesture recognition using depth information[J]. Biomedical Signal Processing and Control, 2021, 66: 102444.
[73] RAHIMIAN E, ZABIHI S, ASIF A, et al. TEMGNet: deep transformer-based decoding of upperlimb sEMG for hand gestures recognition[J]. arXiv:2109.12379, 2021.
[74] MCINTOSH J, MARZO A, FRASER M, et al. Echoflex: hand gesture recognition using ultrasound imaging[C]//Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 2017: 1923-1934.
[75] YANG X, SUN X, ZHOU D, et al. Towards wearable a-mode ultrasound sensing for real-time finger motion recognition[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2018, 26(6): 1199-1208.