多自监督学习任务结合图神经网络的EEG情感识别

doi:10.3778/j.issn.1002-8331.2501-0094

摘要/Abstract

摘要： 针对脑电信号（electroencephalogram，EEG）情感识别中因标签缺失导致模型泛化性不足以及单任务自监督学习易过拟合的问题，提出了一种基于EEG频域特征的图神经网络模型，采用自监督多任务学习框架进行表征学习和情感识别。提取EEG数据的微分熵特征并构建图结构表示，通过通道掩蔽、频率掩蔽、空间拼图和频率拼图四种自监督任务进行多任务学习。结合切比雪夫图神经网络提取深层特征，通道掩蔽和频率掩蔽任务通过重建模块计算损失，空间拼图和频率拼图任务通过分类模块计算损失。训练完成后，冻结特征提取器参数并用于下游情感识别任务。在SEED和DEAP数据集上的实验结果显示，依赖被试的情感分类准确率在SEED数据集上达到89.87%（三分类），在DEAP数据集上，唤醒度和效价维度的两分类准确率分别为88.03%和89.70%；而独立被试的准确率分别在SEED数据集上为72.03%，在DEAP数据集上为65.38%和61.29%。这些结果表明，所提方法有效提升了分类性能，缓解了过拟合问题，且优于现有方法。

关键词: 脑电信号, 情感识别, 图神经网络, 自监督学习, 多任务

Abstract: To address the issues of the insufficient model generalization due to label scarcity and the overfitting tendency of single-task self-supervised learning in electroencephalogram (EEG) emotion recognition, this paper proposes a graph neural network model based on EEG frequency-domain features, within a self-supervised multi-task learning framework to enhance representation learning and emotion recognition. Differential entropy features are extracted from EEG data and represented as graph structures. Multi-task learning is performed through four self-supervised tasks： channel masking, frequency masking, spatial jigsaw, and frequency jigsaw. A Chebyshev graph neural network is utilized to extract deep-level features, where the channel masking and frequency masking tasks compute losses via a reconstruction module, while the spatial jigsaw and frequency jigsaw tasks compute losses via a classification module. After training, the feature extractor parameters are frozen and applied to downstream emotion recognition tasks. Experimental results on the SEED and DEAP datasets demonstrate that the subject-dependent emotion classification accuracy reaches 89.87% (three-class) on the SEED dataset, while on the DEAP dataset, the binary classification accuracies for the arousal and valence dimensions are 88.03% and 89.70%, respectively. For subject-independent scenarios, the accuracies are 72.03% on the SEED dataset and 65.38% and 61.29% on the DEAP dataset. These results indicate that the proposed method effectively improves classification performance, mitigates overfitting, and outperforms existing approaches.

Key words: electroencephalogram, emotion recognition, graph neural network, self-supervised learning, multi-task

陈景霞, 李小池, 王倩, 张鹏伟. 多自监督学习任务结合图神经网络的EEG情感识别[J]. 计算机工程与应用, 2025, 61(22): 205-214.

CHEN Jingxia, LI Xiaochi, WANG Qian, ZHANG Pengwei. EEG Emotion Recognition with Multi-Self-Supervised Learning Tasks Combined with Graph Neural Networks[J]. Computer Engineering and Applications, 2025, 61(22): 205-214.

参考文献

[1] JAFARI M, SHOEIBI A, KHODATARS M, et al. Emotion recognition in EEG signals using deep learning methods: a review[J]. Computers in Biology and Medicine, 2023, 165: 107450.
[2] 刘颖, 袁莉, 祖铄迪, 等. 基于多模态生理数据的情感识别综述[J]. 电子科技大学学报, 2024, 53(5): 720-731.
LIU Y, YUAN L, ZU S D, et al. Emotion recognition based on multimodal physiological data: a survey[J]. Journal of University of Electronic Science and Technology of China, 2024, 53(5): 720-731.
[3] NAWAZ R, CHEAH K H, NISAR H, et al. Comparison of different feature extraction methods for EEG-based emotion recognition[J]. Biocybernetics and Biomedical Engineering, 2020, 40(3): 910-926.
[4] WANG J, WANG M. Review of the emotional feature extraction and classification using EEG signals[J]. Cognitive Robotics, 2021, 1: 29-40.
[5] LIU H, LOU T Y, ZHANG Y Z, et al. EEG-based multimodal emotion recognition: a machine learning perspective[J]. IEEE Transactions on Instrumentation and Measurement, 2024, 73: 4003729.
[6] PRABOWO D W, NUGROHO H A, SETIAWAN N A, et al. A systematic literature review of emotion recognition using EEG signals[J]. Cognitive Systems Research, 2023, 82: 101152.
[7] LIU H R, ZHANG Y, LI Y J, et al. Review on emotion recognition based on electroencephalography[J]. Frontiers in Computational Neuroscience, 2021, 15: 758212.
[8] 雪雯, 陈景霞, 胡凯蕾, 等. 基于EEG和面部视频的多模态连续情感识别[J]. 陕西科技大学学报, 2024, 42(1): 169-176.
XUE W, CHEN J X, HU K L, et al. Multimodal continuous emotion recognition based on EEG and facial video[J]. Journal of Shaanxi University of Science & Technology, 2024, 42(1): 169-176.
[9] LI Y, ZHENG W M, WANG L, et al. From regional to global brain: a novel hierarchical spatial-temporal neural network model for EEG emotion recognition[J]. IEEE Transactions on Affective Computing, 2022, 13(2): 568-578.
[10] 陈景霞, 胡修文, 唐喆喆, 等. 基于卷积联合适应网络的脑电信号情感识别[J]. 数据采集与处理, 2022, 37(4): 814-824.
CHEN J X, HU X W, TANG Z Z, et al. EEG emotion recognition based on convolutional joint adaptation network[J]. Journal of Data Acquisition and Processing, 2022, 37(4): 814-824.
[11] QIU X K, WANG S L, WANG R Q, et al. A multi-head residual connection GCN for EEG emotion recognition[J]. Computers in Biology and Medicine, 2023, 163: 107126.
[12] GAO Y, FU X L, OUYANG T X, et al. EEG-GCN: spatio-temporal and self-adaptive graph convolutional networks for single and multi-view EEG-based emotion recognition[J]. IEEE Signal Processing Letters, 2022, 29: 1574-1578.
[13] TZENG E, HOFFMAN J, SAENKO K, et al. Adversarial discriminative domain adaptation[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 2962-2971.
[14] 陈景霞, 唐喆喆, 林文涛, 等. 用于脑电数据增强和情绪识别的自注意力GAN[J]. 计算机工程与应用, 2023, 59(5): 160-168.
CHEN J X, TANG Z Z, LIN W T, et al. Self-attention GAN for EEG data augmentation and emotion recognition[J]. Computer Engineering and Applications, 2023, 59(5): 160-168.
[15] LI Y, CHEN J, LI F, et al. GMSS: graph-based multi-task self-supervised learning for EEG emotion recognition[J]. IEEE Transactions on Affective Computing, 2023, 14(3): 2512-2525.
[16] LI G Q, CHEN N, NIU Y X, et al. MSLTE: multiple self-supervised learning tasks for enhancing EEG emotion recognition[J]. Journal of Neural Engineering, 2024, 21(2): 024003.
[17] CIPOLLA R, GAL Y, KENDALL A. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 7482-7491.
[18] DEFFERRARD M, BRESSON X, VANDERGHEYNST P. Convolutional neural networks on graphs with fast localized spectral filtering[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016: 3844-3852.
[19] ZHENG W L, LU B L. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks[J]. IEEE Transactions on Auto-nomous Mental Development, 2015, 7(3): 162-175.
[20] KOELSTRA S, MUHL C, SOLEYMANI M, et al. DEAP: a database for emotion analysis; using physiological signals[J]. IEEE Transactions on Affective Computing, 2012, 3(1): 18-31.
[21] CORTES C, VAPNIK V. Support-vector networks[J]. Machine Learning, 1995, 20(3): 273-297.