Emotion Recognition of EEG Based on Ensemble CapsNet

doi:10.3778/j.issn.1002-8331.2010-0263

Abstract

Abstract: In this paper, an emotion classification model based on ensemble CapsNet is proposed in order to obtain the characteristics of multiband information of EEG signals and the spatial information of channel distribution. Firstly, the wavelet packet transform（WPT） is employed to decompose the pre-processed EEG signals into sub-frequency bands（Theta,Alpha, Beta, and Gamma） and then wavelet coefficient energy values are computed as the features in the four frequence bands. Secondly, the features are spliced to form a multiband feature matrix which are trained through the CapsNet and then generate multiple capsule models. Finally, the ensemble CapsNet model is constructed by the ensemble method. The experimental results show that high frequency features have more obvious effect on emotion recognition than those on low frequency bands, and the feature in Gamma band has the best emotional representation among all the independent frequency bands. Furthermore, the emotional feature integrating multiband information has a higher identification accuracy than that of each single band. The proposed method can achieve an average accuracy of 95.11% and 92.43% for two-category and four-category identification tasks respectively, which presents a certain improvement compared with similar research.

Key words: emotion recognition, EEG, multiband feature matrix, CapsNet, ensemble learning

摘要： 为了充分提取脑电信号多频带的时频信息和保留导联空间分布的位置信息，提出了一种基于集成胶囊网络的情绪识别模型。对预处理过的脑电信号进行小波包特征提取，并将Theta、Alpha、Beta、Gamma四个频带的小波系数能量值填充于根据导联空间分布映射的稀疏矩阵中，拼接构成多频带特征矩阵，通过胶囊网络对特征数据进行训练，对不同频带的胶囊网络构建集成学习模型。实验结果表明，Gamma频带比低频带对情绪识别的敏感度更高，融合了多频带和空间信息的特征能够显著提升情绪识别的精度，最终集成胶囊网络将脑电情绪分为二类和四类的识别精度可以达到95.11%和92.43%，相比同类研究有一定的提升。

关键词: 情绪识别, 脑电信号, 多频带特征矩阵, 胶囊网络（CapsNet）, 集成学习

CHEN Qin, CHEN Lanlan, JIANG Runqiang. Emotion Recognition of EEG Based on Ensemble CapsNet[J]. Computer Engineering and Applications, 2022, 58(8): 175-184.

谌鈫, 陈兰岚, 江润强. 集成胶囊网络的脑电情绪识别[J]. 计算机工程与应用, 2022, 58(8): 175-184.

References

[1] PICARD R W，HEALEY J.Affective computing[J].Personal Technologies，1997，1（4）：231-240.
[2] 卢官明，袁亮，杨文娟.基于长短期记忆和卷积神经网络的语音情感识别[J].南京邮电大学学报（自然科学版），2018，38（5）：63-69.
LU G M，YUAN L，YANG W J.Speech emotion recognition based on long short-term memory and convolutional neural networks[J].Journal of Nanjing University of Posts and Telecommunications（Natural Science），2018，38（5）：63-69.
[3] MAO Q R，PAN X Y，ZHAN Y Z.Using Kinect for real-time emotion recognition via facial expression[J].Frontiers of Information Technology Electronic Engineering，2015，16（4）：272-283.
[4] 姜明星，胡敏，王晓华，等.视频序列中表情和姿态的双模态情感识别[J].激光与光电子学进展，2018，55（7）：167-174.
JIANG M X，HU M，WANG X H，et al.Dual-modal emotion recognition based on facial expression and body posture in video sequences[J].Laser & Optoelectronics Progress，2018，55（7）：167-174.
[5] 张雪英，王薇蓉，孙颖，等.面向脑电情感识别的改进多分类RVM模型研究[J].计算机工程与应用，2019，55（9）：112-117.
ZHANG X Y，WANG W R，SUN Y，et al.Research om improved multi-classification RVM for emotional recognition of EEG signal[J].Computer Engineering and Applications，2019，55（9）：112-117.
[6] PRASHANT L，ARUN K S.An approach to EEG based emotion recognition and classification using kernel density estimation[J].Procedia Computer Science，2015，48：574-581.
[7] AHMET M，AYDIN A.Emotion recognition from EEG signals by using multivariate empirical mode decomposition[J].Pattern Analysis and Applications，2018，21（1）：18-89.
[8] ZHANG J H，CHEN M，ZHAO S K.ReliefF-based EEG sensor selection methods for emotion recognition[J].Sensors（Basel，Switzerland），2016，16（10）：264-267.
[9] 项伟，吴迎年.基于改进共空间模式与深度信念网络的脑电信号识别算法研究[J].中国科学：信息科学，2018，48（7）：919-931.
XIANG W，WU Y N.Research on EEG recognition based on improved-common spatial patterns and deep-belief network algorithm[J].Scientia Sinica Informationis，2018，48（7）：919-931.
[10] ZAHRA A，ALIN M N.A self-organized recurrent neural network for estimating the effective connectivity and its application to EEG data[J].Computers in Biology and Medicine，2019，110：93-107.
[11] MAMMONE N，IERACITANO C，MORABITO F C，et al.A deep CNN approach to decode motor preparation of upper limbs from time-frequency maps of EEG signals at source level[J].Neural Networks，2020，124：357-372.
[12] GOPAL C J，RATNA S，ANUPAM A.A 1D-CNN-spectrogram based approach for seizure detection from EEG signal[J].Procedia Computer Science，2020，167：403-412.
[13] MICHIELLI N，ACHARYA U R，MOLINARI F.Cascaded LSTM recurrent neural network for automated sleep stage classification using single-channel EEG signals[J].Computers in Biology and Medicine，2019，106：71-81.
[14] ZHENG W Z，LU B L.Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks[J].IEEE Transactions on Autonomous Mental Development，2015，7（3）：162-175.
[15] TRIPATHI S，ACHARYA S，SHARMA R D.Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset[C]//Twenty-Ninth IAAI Conference，2017：4746-4752.
[16] KWON Y H，SHIN S B，KIM S D.Electroencephalography based fusion two-dimensional （2D）-convolution neural networks（CNN） model for emotion recognition system[J].Sensors（Basel，Switzerland），2018，18（5）：383-390.
[17] 李幼军，黄佳进，王海渊，等.基于SAE和LSTM RNN的多模态生理信号融合和情感识别研究[J].通信学报，2017，38（12）：109-120.
LI Y J，HUANG J J，WANG H Y，et al.Study of emotion recognition based on fusion multi-modal bio-signal with SAE and LSTM recurrent neural network[J].Journal on Communications，2017，38（12）：109-120.
[18] 蔡冬丽，钟清华，朱永升，等.三维输入卷积神经网络脑电信号情感识别[J].计算机工程与应用，2021，57（5）：161-167.
CAI D L，ZHONG Q H，ZHU Y S，et al.EEG emotion recognition using convolutional neural network with 3D input[J].Computer Engineering and Applications，2021，57（5）：161-167.
[19] CHEN G，WANG W，WANG Z.Two-dimensional discrete feature based spatial attention CapsNet for sEMG signal recognition[J].Applied Intelligence，2020，50（10）：3503-3520.
[20] KOELSTRA S，MUHL C，SOLEYMANI M，et al.A database for emotion analysis using physiological signals[J].IEEE Transactions on Affective Computing，2012，3（1）：81-89.
[21] ZHONG Y，ZHAO M Y，WANG Y X，et al.Recognition of emotions using multimodal physiological signals and an ensemble deep learning model[J].Computer Methods and Programs in Biomedicine，2017，140：93-110.
[22] 魏琛，陈兰岚，张傲.基于集成卷积神经网络的脑电情感识别[J].华东理工大学学报（自然科学版），2019，45（4）：614-622.
WEI C，CHEN L L，ZHANG A.Emotion recognition of EEG based on ensemble convolutional neural networks[J].Journal of East China University of Science and Technology（Natural Science），2019，45（4）：614-622.
[23] 苏建新.基于脑电信号的情绪识别研究[D].南京：南京邮电大学，2015.
SU J X.Emotion recognition research based on EEG signalD].Nanjing：Nanjing University of Posts and Telecommunications，2015.
[24] LI X，SONG D W，ZHANG P，et al.Emotion recognition from multi-channel EEG data through convolutional recurrent neural network[C]//Proceedings of the 2016 IEEE International Conference on Bioinformatics and Bio-Medicine（BIBM），Shenzhen，China，2016：352-359.
[25] LIU W，ZHENG L Z，LU B L.Multimodal emotion recognition using multimodal deep learning[C]//International Conference on Neural Information Processing.Japan：Springer，2016：521-529.
[26] SONG T F，ZHENG W M，SONG P.EEG emotion recognition using dynamical graph convolutional neural networks[J].IEEE Transactions on Affective Computing，2020，11（3）：532-541.
[27] VAL-CALVO M，áLVAREZ-SáNCHEZ J R，DíAZ-MORCILLO A，et al.On the use of lateralization for lightweight and accurate methodology for EEG real time emotion estimation using Gaussian-process classifier[C]//International Work-Conference on the Interplay Between Natural and Artificial Computation，2019：191-201.
[28] DIVYA G，GYANENDRA K.Emotion recognition in valence-arousal space from multi-channel EEG data and wavelet based deep learning framework[J].Procedia Computer Science，2020，171：857-867.
[29] 陈田，陈占刚，袁晓辉，等.基于脑电信号瞬时能量的情感识别方法[J].计算机工程，2019，45（4）：196-204.
CHEN T，CHEN Z G，YUAN X H，et al.Emotion recognition method based on instantaneous energy of electroencephalography[J].Computer Engineering，2019，45（4）：196-204.