Research on Emotion Recognition Based on EEG Time-Frequency-Spatial Multi-Domain Feature Fusion

doi:10.3778/j.issn.1002-8331.2109-0083

Abstract

Abstract: The traditional emotion recognition based on electroencephalogram（EEG） mainly adopted a single EEG feature extraction approach. In order to make full use of the rich information contained in EEG, a new method of EEG emotion recognition based on multi-domain feature fusion is proposed. This paper extracts EEG features in time-domain, frequency-domain and space-domain, and fuses the three domain features as the input of the emotion recognition model. Firstly, the power spectral density of the three frequency bands of alpha, beta and gamma of the EEG signal in different time windows are calculated, and combined with the spatial information of the EEG electrode, the EEG images are formed. Then, the convolutional neural network（CNN） and bidirectional long short-term memory network（BLSTM） are used to construct the CNN-BLSTM model for emotion recognition, and the features of time, frequency and space domains are learned respectively. The method is verified in the SEED dataset. The results show that the method can effectively improve the accuracy of recognition, and the average recognition accuracy is 96.25%.

Key words: electroencephalogram（EEG）, feature extraction, CNN-BLSTM, emotion recognition

摘要： 传统基于脑电信号（electroencephalogram，EEG）的情感识别主要采用单一的脑电特征提取方法，为了充分利用EEG中蕴含的丰富信息，提出一种多域特征融合的脑电情感识别新方法。提取了EEG的时域、频域和空域特征，将三域特征进行融合作为情感识别模型的输入。首先计算不同时间窗EEG信号的alpha、beta、gamma三个频段功率谱密度，并结合脑电电极空间信息构成EEG图片，然后利用卷积神经网络（convolutional neural network，CNN）与双向长短期记忆网络（bidirectional long short-term memory network，BLSTM）构建CNN-BLSTM情感识别模型，分别对时、频、空三域特征进行学习。在SEED数据集对该方法进行验证，结果表明该方法能有效提高情感识别精度，平均识别准确率达96.25%。

关键词: 脑电信号（EEG）, 特征融合, CNN-BLSTM, 情感识别

WANG Lu, LIANG Mingjing, SHI Huiyu, WEN Xin, CAO Rui. Research on Emotion Recognition Based on EEG Time-Frequency-Spatial Multi-Domain Feature Fusion[J]. Computer Engineering and Applications, 2023, 59(4): 191-196.

王璐, 梁明晶, 石慧宇, 温昕, 曹锐. 基于脑电时频空多域特征融合的情感识别研究[J]. 计算机工程与应用, 2023, 59(4): 191-196.

References

[1] YIN Z，ZHAO M，WANG Y，et al.Recognition of emotions using multimodal physiological signals and an ensemble deep learning model[J].Computer Methods & Programs in Biomedicine，2017，140（12）：93-110.
[2] 贲晛烨，杨明强，张鹏，等.微表情自动识别综述[J].计算机辅助设计与图形学学报，2014，26（9）：1385-1395.
BEN J Y，YANG M Q，ZHANG P，et al.A survey of automatic micro expression recognition[J].Journal of Computer Aided Design and Graphics，2014，26（9）：1385-1395.
[3] 孙颖，马江河，张雪英.结合非线性全局特征和谱特征的脑电情感识别[J].计算机工程与应用，2018，54（17）：116-121.
SUN Y，MA J H，ZHANG X Y.EEG emotion recognition based on nonlinear global feature and spectral feature[J].Computer Engineering and Applications，2018，54（17）：116-121.
[4] JENKE R，PEER A，BUSS M.Feature extraction and selection for emotion recognition from EEG[J].IEEE Transactions on Affective Computing，2014，5（3）：327-339.
[5] 陈景霞，王丽艳，贾小云，等.基于深度卷积神经网络的脑电信号情感识别[J].计算机工程与应用，2019，55（18）：103-110.
CHEN J X，WANG L Y，JIA X Y，et al.Emotion recognition of EEG signals based on deep convolutional neural network[J].Computer Engineering and Applications，2019，55（18）：103-110.
[6] PETRANTONAKIS P C，HADJILEONTIADIS L J.Emotion recognition from EEG using higher order crossings[J].IEEE Transactions on Information Technology in Biomedicine，2009，14（2）：186-197.
[7] CRAIK A，HE Y，CONTRERAS-VIDAL J L.Deep learning for electroencephalogram（EEG） classification tasks：a review[J].Journal of Neural Engineering，2019，16（3）：031001.
[8] LIU S，TONG J，MENG J，et al.Study on an effective cross-stimulus emotion recognition model using EEGs based on feature selection and support vector machine[J].International Journal of Machine Learning and Cybernetics，2018，9（5）：721-726.
[9] 闫梦梦，吕钊，孙文慧.基于共同空间模式的情感脑电信号的空域特征提取[J].图学学报，2020，41（3）：424-429.
YAN M M，LU Z，SUN W H.Spatial feature extraction of emotional EEG signals based on common spatial pattern[J].Journal of Graphics，2020，41（3）：424-429.
[10] YANG G Y，WU Q，FU Y，et al.Continuous convolutional neural network with 3D input for EEG-based emotion recognition[C]//International Conference on Neural Information Processing.Cham：Springer，2018：433-443.
[11] 阚威，李云.基于LSTM的脑电情绪识别模型[J].南京大学学报（自然科学版），2019，55（1）：110-116.
KAN W，LI Y.EEG emotion recognition model based on LSTM[J].Journal of Nanjing University（Natural Science Edition），2019，55（1）：110-116.
[12] ZHENG W L，LU B L.Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks[J].IEEE Transactions on Auto-
nomous Mental Development，2015，7（3）：162-175.
[13] LI Y，HUANG J，ZHOU H，et al.Human emotion recognition with electroencephalographic multidimensional features by hybrid deep neural networks[J].Applied Sciences，2017，7（10）：1060.
[14] 韩小娟.脑电情感识别中多特征融合与分类研究[D].重庆：重庆邮电大学，2020.
HAN X J.Research on multi-feature fusion and classification in EEG emotion recognition[D].Chongqing：Chong-
qing University of Posts and Telecommunications，2020.
[15] DUAN R N，ZHU J Y，LU B L.Differential entropy feature for EEG-based emotion classification[C]//6th International IEEE/EMBS Conference on Neural Engineering，2013：81-84.
[16] ALAM R，ZHAO H，GOODWIN A，et al.Differences in power spectral densities and phase quantities due to processing of EEG signals[J].Sensors，2020，20（21）：6285.
[17] YYUCELBAS C，YYUCELBAS ?，?ZSEN S，et al.Detection of sleep spindles in sleep EEG by using the PSD methods[J].India Journal of Science and Technology，2016，9（25）：1-7.
[18] ALARCAO S M，FONSECA M J.Emotions recognition using EEG signals：a survey[J].IEEE Transactions on Affective Computing，2017，10（3）：374-393.
[19] CAO R，SHI H Y，WANG X，et al.Hemispheric asymmetry of functional brain networks under different emotions using EEG Data[J].Entropy，2020，22（9）：939.
[20] SRIVASTAVA N，HINTON G，KRIZHEVSKY A，et al.Dropout：a simple way to prevent neural networks from overfitting[J].The Journal of Machine Learning Research，2014，15（1）：1929-1958.
[21] 刘然，刘宇，顾进广.基于自适应学习率优化的AdaNet改进[J].计算机应用，2020，40（10）：2804-2810.
LIU R，LIU Y，GU J G.Improvement of AdaNet based on adaptive learning rate optimization[J].Journal of Computer Applications，2020，40（10）：2804-2810.
[22] LI X，SONG D，ZHANG P，et al.Exploring EEG features in cross-subject emotion recognition[J].Frontiers in Neuroscience，2018，12：162.
[23] CIMTAY Y，EKMEKCIOGLU E.Investigating the use of pretrained convolutional neural network on cross-subject and cross-dataset EEG emotion recognition[J].Sensors，2020，20（7）：2034.
[24] 魏琛，陈兰岚，张傲.基于集成卷积神经网络的脑电情感识别[J].华东理工大学学报（自然科学版），2019，45（4）：614-622.
WEI C，CHEN L L，ZHANG A.EEG emotion recognition based on ensemble CNN[J].Journal of East China University of Science and Technology（Natural Science Edition），2019，45（4）：614-622.
[25] ZHENG W L，GUO H T，LU B L.Revealing critical channels and frequency bands for emotion recognition from EEG with deep belief network[C]//7th International IEEE/EMBS Conference on Neural Engineering.Piscataway：IEEE，2015：154-157.