EEG-fNIRS Emotion Recognition Combined with Multi-View Feature Fusion and Cross Attention Graph Convolution

doi:10.3778/j.issn.1002-8331.2409-0399

Abstract

Abstract: In order to obtain the complementary time-frequence-space information of EEG and fNIRS, a multi-view feature fusion and cross-attention map convolution (MF-MSDG-CAFF) method is proposed for EEG-fNIRS emotion recognition. Firstly, the different view features and spatial connections of each channel signal are extracted from EEG and fNIRS data induced by emotional video, and the graph structure data is constructed. Then, the convolution of static and dynamic graphs is introduced in parallel to integrate the connection information between different modal channels and capture the spatial interaction characteristics of brain cognition. Finally, the cross-attention network is used for feature fusion to improve the accuracy of emotion recognition. The results show that compared with single-view EEG, the proposed multi-view EEG method has higher recognition accuracy. Compared with the single mode recognition results of EEG and fNIRS alone, the recognition rate of the proposed fusion model is improved by 1.04 and 23.72 percentage points. Compared with the current EEG-fNIRS fusion methods, the recognition rate of the proposed fusion model is improved by 1.56~15.48 percentage points.

Key words: multi-view EEG, multimodal fusion, static and dynamic graph convolutional neural networks, cross attention, emotion recognition

摘要： 为了有效学习脑电（EEG）和功能近红外（fNIRS）信号的情感认知时-频-空域信息，提出一种多视图脑电的多路静动态图卷积交叉注意网络（MF-MSDG-CAFF）方法用于EEG-fNIRS情感识别。通过对情感视频诱发的EEG和fNIRS数据提取各通道信号的不同视图特征及其空间连接关系，构建图结构数据；并行引入静、动态图卷积，捕获不同模态通道间的连接信息和交互特性；通过交叉注意力网络进行特征融合，从而提高情感识别的准确率；结果表明，与单视图EEG相比，提出的多视图EEG方法拥有较高的识别准确率；与仅EEG和仅fNIRS单模态识别结果相比，提出的融合模型的识别率提升1.04和23.72个百分点；与当前常用的EEG-fNIRS融合方法相比，提出的融合模型的识别率提升1.56~15.48个百分点。

关键词: 多视图EEG, 多模态融合, 静动态图卷积神经网络, 交叉注意力, 情感识别

TAO Chenxi, ZHANG Xueying, CHEN Guijun. EEG-fNIRS Emotion Recognition Combined with Multi-View Feature Fusion and Cross Attention Graph Convolution[J]. Computer Engineering and Applications, 2025, 61(24): 187-196.

陶晨曦, 张雪英, 陈桂军. 结合多视图特征融合和交叉注意力图卷积的EEG-fNIRS情感识别[J]. 计算机工程与应用, 2025, 61(24): 187-196.

References

[1] 闫超, 张雪英, 张静, 等. 结合注意力机制和特征融合1DCNN的脑电情感识别[J]. 计算机工程与应用, 2023, 59(13): 171-177.
YAN C, ZHANG X Y, ZHANG J, et al. EEG emotion recognition combined with attention mechanism and feature fusion 1DCNN[J]. Computer Engineering and Applications, 2023, 59(13): 171-177.
[2] ALARC?O S M, FONSECA M J. Emotions recognition using EEG signals: a survey[J]. IEEE Transactions on Affective Computing, 2019, 10(3): 374-393.
[3] LI R H, YANG D L, FANG F, et al. Concurrent fNIRS and EEG for brain function investigation: a systematic, methodology-focused review[J]. Sensors, 2022, 22(15): 5865.
[4] BLANCO R, KOBA C, CRIMI A. Investigating the interaction between EEG and fNIRS: a multimodal network analysis of brain connectivity[J]. Journal of Computational Science, 2024, 82: 102416.
[5] SAVRAN A, CIFTCI K, CHANEL G, et al. Emotion detection in the loop from brain signals and facial images[C]//Proceedings of the NTERFACE 2006 Workshop, 2006.
[6] RABBANI M H R, ISLAM S M R. Deep learning networks based decision fusion model of EEG and fNIRS for classification of cognitive tasks[J]. Cognitive Neurodynamics, 2024, 18(4): 1489-1506.
[7] NOUR M, ?ZTüRK ?, POLAT K. A novel classification framework using multiple bandwidth method with optimized CNN for brain computer interfaces with EEG-fNIRS signals[J]. Neural Computing and Applications, 2021, 33(22): 15815-15829.
[8] 刘晋瑞, 宋婷, 舒智林, 等. 一种面向运动解码的EEG-fNIRS时频特征融合与协同分类方法[J]. 仪器仪表学报, 2022, 43(7): 165-173.
LIU J R, SONG T, SHU Z L, et al. A time-frequency feature fusion and collaborative classification method for motion decoding with EEG-fNIRS signals[J]. Chinese Journal of Scientific Instrument, 2022, 43(7): 165-173.
[9] WU W W, CHEN D M, LI Q P. A two-stage multi-modal multi-label emotion recognition decision system based on GCN[J]. International Journal of Decision Support System Technology, 2024, 16(1): 1-17.
[10] AN J P, CAI Q, SUN X L, et al. Attention-based cross-frequency graph convolutional network for driver fatigue estimation[J]. Cognitive Neurodynamics, 2024, 18(5): 3181-3194.
[11] CHEN J F, YU K W, WANG F, et al. Temporal convolutional network-enhanced real-time implicit emotion recognition with an innovative wearable fNIRS-EEG dual-modal system[J]. Electronics, 2024, 13(7): 1310.
[12] 王璐, 梁明晶, 石慧宇, 等. 基于脑电时频空多域特征融合的情感识别研究[J]. 计算机工程与应用, 2023, 59(4): 191-196.
WANG L, LIANG M J, SHI H Y, et al. Research on emotion recognition based on EEG time-frequency-spatial multi-domain feature fusion[J]. Computer Engineering and Applications, 2023, 59(4): 191-196.
[13] LI Y, LIU Y, GUO Y Z, et al. Spatio-temporal-spectral hierarchical graph convolutional network with semisupervised active learning for patient-specific seizure prediction[J]. IEEE Transactions on Cybernetics, 2022, 52(11): 12189-12204.
[14] SCHOLKMANN F, KLEISER S, METZ A J, et al. A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology[J]. NeuroImage, 2014, 85: 6-27.
[15] ZHANG G H, YU M J, LIU Y J, et al. SparseDGCNN: recognizing emotion from multichannel EEG signals[J]. IEEE Transactions on Affective Computing, 2023, 14(1): 537-548.
[16] DUAN R N, ZHU J Y, LU B L. Differential entropy feature for EEG-based emotion classification[C]//Proceedings of the 2013 6th International IEEE/EMBS Conference on Neural Engineering. Piscataway: IEEE, 2014: 81-84.
[17] BAO G C, YANG K, TONG L, et al. Linking multi-layer dynamical GCN with style-based recalibration CNN for EEG-based emotion recognition[J]. Frontiers in Neurorobotics, 2022, 16: 834952.
[18] 赵卿, 张雪英, 陈桂军, 等. 基于模态注意力图卷积特征融合的EEG和fNIRS情感识别[J]. 浙江大学学报(工学版), 2023, 57(10): 1987-1997.
ZHAO Q, ZHANG X Y, CHEN G J, et al. EEG and fNIRS emotion recognition based on modality attention graph convolution feature fusion[J]. Journal of Zhejiang University (Engineering Science), 2023, 57(10): 1987-1997.
[19] CHEN G J, LIU Y, ZHANG X Y. EEG fNIRS-based emotion recognition using graph convolution and capsule attention network[J]. Brain Sciences, 2024, 14(8): 820.
[20] ZHAO W, JIANG X L, ZHANG B C, et al. CTNet: a convolutional transformer network for EEG-based motor imagery classification[J]. Scientific Reports, 2024, 14: 20237.
[21] 谌鈫, 陈兰岚, 江润强. 集成胶囊网络的脑电情绪识别[J]. 计算机工程与应用, 2022, 58(8): 175-184.
CHEN Q, CHEN L L, JIANG R Q. Emotion recognition of EEG based on ensemble CapsNet[J]. Computer Engineering and Applications, 2022, 58(8): 175-184.