基于迁移学习和表面肌电信号的上肢动作识别

doi:10.3778/j.issn.1002-8331.2307-0046

摘要/Abstract

摘要： 准确识别脑卒中患者上肢运动意图是实现高效康复训练的关键步骤。为了提高基于表面肌电信号（surface electromyography，sEMG）的上肢动作识别精度，提出了一种结合预训练模型和支持向量机（support vector machine，SVM）的肌电动作识别方法。该方法充分考虑通道之间的关联性，将预处理后的时域信号通过短时傅里叶变换（short-time Fourier transform，STFT）转换为对应频谱图，并将所有通道的频谱图沿竖直方向拼接。利用两种微调的预训练模型VGG16和Resnet50对肌电图像提取特征，分别考虑三种上肢动作识别方案：仅使用微调的预训练模型进行识别、单个微调预训练模型提取特征后使用SVM进行识别、两个微调预训练模型提取特征拼接后使用SVM进行识别。实验结果表明，所提出的方法在采集的受试者肌电信号数据集上均达到90%以上的识别精度，可有效区分不同的上肢动作。

关键词: 上肢动作识别, 表面肌电信号（sEMG）, 短时傅里叶变换（STFT）, 预训练模型, 支持向量机（SVM）

Abstract: Accurate recognition of upper limb action intention in stroke patients is a key step towards efficient rehabilitation training. In order to improve the accuracy of upper limb action recognition based on surface electromyography (sEMG), a method is proposed that combines pre-trained models and support vector machine (SVM) classification. This method fully considers the correlation between channels and converts the preprocessed time-domain signal into corresponding spectrograms through short time Fourier transform (STFT), and concatenates the spectrograms of all channels in the vertical direction. Two fine-tuning pre-training models, VGG16 and Resnet50, are used to extract features from the EMG images. Three upper limb action recognition schemes are considered separately：using only fine-tuning pre-trained models for recognition, a single fine-tuning pre-trained model extracts features and uses SVM for recognition, and two fine-tuning pre-trained models extract feature concatenation and use SVM for recognition. The experimental results show that the proposed method achieves a recognition accuracy of over 90% on the collected subject EMG signal dataset, which can effectively differentiate between different upper limb action.

Key words: upper limb action recognition, surface electromyography (sEMG), short-time fourier transform (STFT), pre-trained models, support vector machine (SVM)

张恒玮, 徐林森, 陈根, 汪志焕, 眭翔. 基于迁移学习和表面肌电信号的上肢动作识别[J]. 计算机工程与应用, 2024, 60(20): 124-132.

ZHANG Hengwei, XU Linsen, CHEN Gen, WANG Zhihuan, SUI Xiang. Upper Limb Action Recognition Based on Transfer Learning and sEMG[J]. Computer Engineering and Applications, 2024, 60(20): 124-132.

参考文献

[1] SHENG B, ZHANG Y, MENG W, et al. Bilateral robots for upper-limb stroke rehabilitation: state of the art and future prospects[J]. Medical Engineering & Physics, 2016, 38(7): 587-606.
[2] HALL P, KROLL T, HICKEY J, et al. Patient and public involvement in stroke research:a scoping review protocol[J]. HRB Open Research, 2022, 4: 118.
[3] QASSIM H M, HASAN W Z W. A review on upper limb rehabilitation robots[J]. Applied Sciences, 2020, 10(19): 6976.
[4] CHOCKALINGAM M,VASANTHAN L T, BALASUBRAMANIAN S, et al. Experiences of patients who had a stroke and rehabilitation professionals with upper limb rehabilitation robots: a qualitative systematic review protocol[J]. BMJ Open, 2022, 12(9):065177.
[5] OZDEMIR M A, KISA D H, GUREN O, et al. Hand gesture classification using time-frequency images and transfer learning based on CNN[J]. Biomedical Signal Processing and Control, 2022, 77: 103787.
[6] ZHANG H, SU M. Hand gesture recognition of double-channel EMG signals based on sample entropy and PSO-SVM[J]. Journal of Physics: Conference Series, 2020, 1631: 012001.
[7] WANG B,WANG C,WANG L, et al. Recognition of sEMG hand actions based on cloud adaptive quantum chaos ions motion algorithm optimized SVM[J]. Journal of Mechanics in Medicine & Biology, 2019, 19(6): 1950047.
[8] GUO B, MA Y, YANG J, et al. Lw-CNN-based myoelectric signal recognition and real-time control of robotic arm for upper-limb rehabilitation[J]. Computational Intelligence and Neuroscience, 2020(1): 8846021.
[9] JAIME E L, LEO K C, OLIVER R, et al. Muscle-specific high-density electromyography arrays for hand gesture classification[J].IEEE Transactions on Biomedical Engineering, 2022, 69(5): 1758-1766.
[10] TUNCER T, DOGAN S, SUBASI A. Surface EMG signal classification using ternary pattern and discrete wavelet transform based feature extraction for hand movement recognition[J]. Biomedical Signal Processing and Control, 2020, 58: 101872.
[11] ZHANG X F, LI X, DAI J T, et al. The design of a hemiplegic upper limb rehabilitation training system based on surface EMG signals[J]. Journal of Advanced Mechanical Design, Systems, and Manufacturing, 2018, 12(1): 0031.
[12] 江茜, 李沿宏, 邹可, 等. 肌电信号多通道相关性特征手势识别方法[J]. 计算机工程与应用, 2023, 59(7): 102-109.
JIANG Q, LI Y H, ZHOU K, et al. Myoelectric gesture recognition based on multi-channel correlation feature[J]. Computer Engineering and Applications, 2023, 59(7): 102-109.
[13] CHENG S, WU L, ZHANG S, et al. A model for lumbar EMG signal recognition based on stacking integration learning[J]. IEEE Sensors Journal, 2023, 23(4): 3766-3775.
[14] 胡少康，张道辉，赵新刚，等. 基于特征工程与级联森林的中风患者手部运动肌电识别方法[J]. 机器人, 2021, 43(5): 526-538.
HU S K, ZHANG D H, ZHAO X G, et al. An sEMG-based hand motion recognition method for stroke patients with feature engineering and cascade forest[J]. Robot, 2021, 43(5): 526-538.
[15] ZHANG J, LING C, LI S. EMG signals based human action recognition via deep belief networks[J]. IFAC-Papers OnLine, 2019, 52(19): 271-276.
[16] SRAVANI C, BAJAJ V, TARAN S, et al. Flexible analytic wavelet transform based features for physical action identification using sEMG signals[J]. IRBM, 2020, 41(1): 18-22.
[17] PANCHOLI S, JOSHI A M. Advanced energy kernel-based feature extraction scheme for improved EMG-PR-based prosthesis control against force variation[J]. IEEE Transactions on Cybernetics, 2022, 52(5): 3819-3828.
[18] MANFREDO A, MATTEO C, HENNING M. Deep learning with convolutional neural networks applied to electromyography data: a resource for the classification of movements for prosthetic hands[J]. Front Neurorobot, 2016, 10: 9.
[19] SOROUSHMOJDEHI R, JAVADZADEH S, PEDROCCHI A, et al.Transfer learning in hand movement intention detection based on surface electromyography signals[J]. Frontiers in Neuroscience, 2022, 16: 977328.
[20] GENG W, DU Y, JIN W, et al. Gesture recognition by instantaneous surface EMG images[J]. Scientific Reports, 2016, 6(1): 36571.
[21] XU L, ZHANG K, YANG G, et al. Gesture recognition using dual-stream CNN based on fusion of sEMG energy kernel phase portrait and IMU amplitude image[J]. Biomedical Signal Processing and Control, 2022, 73: 103364.
[22] DU Y, JIN W, WEI W, et al. Surface EMG-based inter-session gesture recognition enhanced by deep domain adaptation[J]. Sensors, 2017, 17(3): 458.
[23] WANG W, YOU W, WANG Z, et al. Feature fusion-based improved capsule network for sEMG signal recognition[J].Computational Intelligence and Neuroscience, 2022(1): 7603319.
[24] KIM J M, CHOI G H, KIM M G, et al. User recognition system based on spectrogram image conversion using EMG signals[J]. Computers, Materials & Continua, 2022, 72(1): 1213-1227.
[25] C?Té-ALLARD U, FRAN?OIS N, FALL C L. A convolutional neural network for robotic arm guidance using sEMG based frequency-features[C]//Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2016: 2464-2470.
[26] C?Té-ALLARD U, FALL C L, DROUIN A, et al. Deep learning for electromyographic hand gesture signal classification using transfer learning[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2019, 27(4): 760-771.
[27] DEMIR F, BAJAJ V, INCE M C, et al. Surface EMG signals and deep transfer learning-based physical action classification[J]. Neural Computing & Applications, 2019, 31(12): 8455-8462.
[28] CHEN L, FU J, WU Y, et al. Hand gesture recognition using compact CNN via surface electromyography signals[J]. Sensors (Basel), 2020, 20(3): 672.
[29] VIMAL S, HAROLD R Y, MOHAMMAD S K, et al. R-CNN and wavelet feature extraction for hand gesture recognition with EMG signals[J]. Neural Computing and Applications, 2020, 32(21): 16723-16736.
[30] OH D C, JO Y U. Classification of hand gestures based on multi-channel EMG by scale average wavelet transform and convolutional neural network[J]. International Journal of Control, Automation and Systems, 2021, 19(3): 1443-1450.
[31] WANG Y, WU Q, DEY N, et al. Deep back propagation-long short-term memory network based upper-limb sEMG signal classification for automated rehabilitation[J].Biocybernetics and Biomedical Engineering, 2020, 40(3): 987-1001.
[32] WANG J. A wavelet denoising method based on the improved threshold function[C]//Proceedings of the 2014 International Conference on Wavelet Analysis and Pattern Recognition, 2014: 70-74.
[33] HUDGINS B, PARKER P, SCOTT R N. A new strategy for multifunction myoelectric control[J].IEEE Transactions on Biomedical Engineering,1993,40(1): 82-94.
[34] IMAN M, ARABNIA H R, RASHEED K. A review of deep transfer learning and recent advancements[J]. Technologies, 2023, 11(2): 40.