Self-Attention GAN for EEG Data Augmentation and Emotion Recognition

doi:10.3778/j.issn.1002-8331.2207-0372

Abstract

Abstract: To solve the problem of data scarcity in emotion recognition based on electroencephalogram（EEG） and the resulting low accuracy of emotion classification, a conditional Wasserstein generative adversarial network（SA-cWGAN） with self attention mechanism is proposed. The self-attention module is used to learn long-term context related global features from training data through, and Wasserstein distance and Lipschitz constraint of gradient penalty are used to optimize the loss function of the network, so as to generate high-quality EEG data to enhance the original training set. A large number of comparative emotion classification are carried out on DEAP and SEED datasets, in which the proposed SA-cWGAN method is used to generate differential entropy（DE） and power spectral density（PSD） features that are close to the distribution of the input EEG training data, to augment the original EEG training dataset. SVM classifier is then used to classify the augmented features into different emotion categories. The experimental results show that in arousal and valence dimensions of DEAP dataset, the accuracy of the augmented DE and PSD features is 16.63, 17.55?percentage points and 6.48, 8.34?percentage points higher than that of the original DE and PSD features, respectively. On the SEED dataset, the accuracy of the three classifications has been improved by 4.64 and 5.18?percentage points respectively. It proves that the features generated by the proposed method have good robustness. The accuracy and stability of EEG emotion recognition can be effectively improved by augmenting original training dataset with the features generated by introducing self-attention mechanism to GAN network.

Key words: electroencephalogram（EEG）, emotion recognition, data augmentation, generative adversarial network（GAN）, self-attention, conditional Wasserstein

摘要： 针对脑电信号（electroencephalogram，EEG）情绪识别中数据稀缺及由此导致的情感分类精度不高的问题，提出了一个引入自注意力机制的条件Wasserstein生成对抗网络（SA-cWGAN），通过自注意力模块从训练数据学习长时上下文相关的全局特征，采用Wasserstein距离和梯度惩罚的Lipschitz约束对网络的损失函数进行优化，进而生成高质量的EEG数据对原有训练集进行增强。所提方法分别在DEAP和SEED数据集上进行了大量的二分类和三分类对比实验，生成了与EEG训练数据分布接近的微分熵（DE）和功率谱密度（PSD）特征，以此来增强EEG训练数据集，采用SVM分类器对增强后的EEG特征进行情绪分类。实验结果表明，在DEAP数据集上的唤醒度和效价维度下，增强后的DE、PSD特征较原有DE、PSD特征二分类准确率分别提高了16.63、17.55个百分点和6.48、8.34个百分点；在SEED数据集下，三分类准确率分别提高了4.64、5.18个百分点，证明所提方法生成的特征具有良好的鲁棒性，也表明通过对GAN网络引入自注意力机制生成的特征增强原有训练数据集能够有效提高EEG情绪识别的准确率和稳定性。

关键词: 脑电信号（EEG）, 情绪识别, 数据增强, 生成对抗网络（GAN）, 自注意力, 条件Wasserstein

CHEN Jingxia, TANG Zhezhe, LIN Wentao, HU Kailei, XIE Jia. Self-Attention GAN for EEG Data Augmentation and Emotion Recognition[J]. Computer Engineering and Applications, 2023, 59(5): 160-168.

陈景霞, 唐喆喆, 林文涛, 胡凯蕾, 谢佳. 用于脑电数据增强和情绪识别的自注意力GAN[J]. 计算机工程与应用, 2023, 59(5): 160-168.

References

[1] KAUR S，SHARMA R.Emotion AI：integrating emotional intelligence with artificial intelligence in the digital workplace[M].Innovations in information and communication technologies.[S.l.]：Springer，2021：337-343.
[2] 权学良，曾志刚，蒋建华，等.基于生理信号的情感计算研究综述[J].自动化学报，2021，47（8）：1769-1784.
QUAN X L，ZENG Z G，JIANG J H，et al.Physiological signals based affective computing：a systematic review[J].Acta Automatica Sinica，2021，47（8）：1769-1784.
[3] PENG Y，QIN F W，KONG W Z，et al.GFIL：a unified framework for the importance analysis of features，frequency bands，and channels in EEG-based emotion recognition[J].IEEE Transaction on Cognitive and Develop-mental Systems，2022，14（3）：935-947.
[4] CRAIK A，HE Y，CONTRERAS-VIDAL J L.Deep learning for electroencephalogram（EEG） classification tasks：a review[J].Journal of Neural Engineering，2019：16031001.
[5] KOELSTRA S，MUHL C，SOLEYMANI M，et al.DEAP：a database for emotion analysis；using physiological signals[J].IEEE Transactions on Affective Computing，2012：18-31.
[6] ZHENG W L，LU B L.Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks[J].IEEE Transactions on Autonomous Mental Development，2015：162-175.
[7] DUAN R N，ZHU J Y，LU B L.Differential entropy feature for EEG-based emotion classification[C]//International IEEE/EMBS Conference on Neural Engineering（NER），2013：81-84.
[8] KATSIGIANNIS S，RAMZAN N.DREAMER：a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices[J].IEEE Journal of Biomedical and Health Informatics，2018：98-107.
[9] SOLEYMANI M，LICHTENAUER J，PUN T，et al.A multimodal database for affect recognition and implicit tagging[J].IEEE Transactions on Affective Computing，2012，3（1）：42-55.
[10] ZHANG Y D，DONG Z，CHEN X，et al.Image based fruit category classification by 13-layer deep convolutional neural network and data augmentation[J].Multimedia Tools and Application，2019（78）：3613-3632.
[11] DIELEMAN S，WILLETT K W，DAMBRE J.Rotation-invariant convolutional neural networks for galaxy morphology prediction[J].Monthly Notices of the Royal Astronomical Society，2015，450（2）：1441-1459.
[12] LOTTE F.Signal processing approaches to minimize or suppress calibration time in oscillatory activity-based brain-computer interfaces[J].Proceedings of the IEEE，2015，103（6）：871-890.
[13] WANG F，ZHONG S H，PENG J，et al.Data augmentation for EEG-based emotion recognition with deep convolutional neural networks[C]//International Conference on Multimedia Modeling，2018：82-93.
[14] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems（NIPS），2014：2672-2680.
[15] HARTMANN K G，SCHIRRMEISTER R T，BALL T.EEG-GAN：generative adversarial networks for electroencephalograhic（EEG） brain signals[J].arXiv：1806.01875，2018.
[16] ZHANG Q，LIU Y.Improving brain computer interface performance by data augmentation with conditional deep convolutional generative adversarial networks[J].arXiv：1806.07108，2018.
[17] LUO Y，LU B L.EEG data augmentation for emotion recognition using a conditional Wasserstein GAN[C]//2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society（EMBC），2018：234-238.
[18] BHAT S，HORTAL E.GAN-based data augmentation for improving the classification of EEG signals[C]//PETRA 2021：The 14th PErvasive Technologies Related to Assistive Environments Conference，2021：453-458.
[19] ARJOVSKY M，CHINTALA S，BOTTOU L.Wasserstein generative adversarial networks[C]//Proceedings of the 34th International Conference on Machine Learning，2017：214-223.
[20] GULRAJANI I，AHMED F，ARJOVSKY M，et al.Improved training of Wasserstein GANs[J].Neural Information Processing Systems，2017：5769-5779.
[21] MIRZA M，OSINDERO S.Conditional generative adversarial nets[J].arXiv：1411.1784，2014.
[22] ZHANG H，GOODFELLOW I，METAXAS D.Self-attention generative adversarial networks[C]//Proceedings of the 36th International Conference on Machine Learning，2019：7354-7363.
[23] 陈景霞，王丽艳，贾小云，等.基于深度卷积神经网络的脑电信号情感识别[J].计算机工程与应用，2019，55（18）：109-116.
CHEN J X，WANG L Y，JIA X Y，et al.EEG-based emotion recognition using deep convolutional neural network[J].Computer Engineering and Applications，2019，55（18）：109-116.
[24] SHI L C，JIAO Y Y，LU B L.Differential entropy feature for EEG-based vigilance estimation[C]//2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society（EMBC），2013：6627-6630.
[25] 陈景霞，郝为，张鹏伟，等.基于混合神经网络的脑电时空特征情感分类[J].软件学报，2021，32（12）：3869-3883.
CHEN J X，HAO W，ZHANG P W，et al.Emotion classification of spatiotemporal EEG features using hybrid neural networks[J].Journal of Software，2021，32（12）：3869-3883.