基于双流特征交叉融合Efficient Transformer的人脸表情识别

doi:10.3778/j.issn.1002-8331.2404-0349

摘要/Abstract

摘要： 面部表情识别在人机交互等现实应用中得到了越来越多的重视。为解决传统方法中由于类间相似性和类内差异引起的识别准确率低等问题，提出了一种双流特征交叉融合Efficient Transformer识别人脸表情的方法。使用IResNet50和MobileFaceNet分别提取人脸表情的图像和关键点的多尺度特征，同时采用通道注意力机制来增强关键特征并减少参数量；引入了交叉融合高效多头自注意力机制（cross fusion efficient multi-head self-attention，CFEMSA），对相同尺度的双流特征进行交叉融合，以突出面部显著特征；最后采用特征金字塔结构对不同尺度的交叉融合结果进行多尺度融合，以提高识别的准确性。提出的方法在RAF-DB、AffecNet-7和AffecNet-8数据集上的识别准确率分别为91.82%、67.46%和63.65%，实验结果证明该方法有效缓解了类间相似性和类内差异所引起的识别准确率低的问题。

关键词: 面部表情识别, Efficient Transformer, 交叉融合, 多尺度特征, 特征融合

Abstract: Facial expression recognition receives increasing attention in various practical applications such as human-computer interaction. To address the issues of low recognition accuracy caused by inter-class similarity and intra-class discrepancy in traditional methods, a dual-stream feature cross-fusion Efficient Transformer method for facial expression recognition is proposed. IResNet50 and MobileFaceNet are used to extract multi-scale image features and facial landmark features of facial expressions, while utilizing a channel attention mechanism to enhance key features and reduce the number of parameters. The cross fusion efficient multi-head self-attention (CFEMSA) mechanism is used to cross-fuse dual-stream features of the same scale to highlight salient facial features. Finally, a feature pyramid structure is used to perform multi-scale fusion of the cross-fusion results at different scales to enhance recognition accuracy. The proposed method achieves recognition accuracies of 91.82%, 67.46%, and 63.65% on the RAF-DB, AffecNet-7, and AffecNet-8 datasets, respectively. Experimental results show that the method effectively alleviates the problem of low recognition accuracy caused by inter-class similarity and intra-class discrepancy.

Key words: facial expression recognition, Efficient Transformer, cross fusion, , multiscale features, feature fusion

党宏社, 孟饶辰, 高宛蓉. 基于双流特征交叉融合Efficient Transformer的人脸表情识别[J]. 计算机工程与应用, 2025, 61(15): 251-257.

DANG Hongshe, MENG Raochen, GAO Wanrong. Facial Expression Recognition Based on Dual-Stream Feature Cross-Fusion Efficient Transformer[J]. Computer Engineering and Applications, 2025, 61(15): 251-257.

参考文献

[1] MEHRABIAN A. Communication without words[M]//Communication theory. London: Routledge, 2017: 193-200.
[2] HOSSAIN S, UMER S, ROUT R K, et al. Fine-grained image analysis for facial expression recognition using deep convolutional neural networks with bilinear pooling[J]. Applied Soft Computing, 2023, 134: 109997.
[3] JIANG M, YIN S L. Facial expression recognition based on convolutional block attention module and multi-feature fusion[J]. International Journal of Computational Vision and Robotics, 2023, 13(1): 21.
[4] LIU S, HUANG S C, FU W N, et al. A descriptive human visual cognitive strategy using graph neural network for facial expression recognition[J]. International Journal of Machine Learning and Cybernetics, 2024, 15(1): 19-35.
[5] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: Transformers for image recognition at scale[J]. arXiv:2010.11929, 2020.
[6] MA F, SUN B, LI S. Robust facial expression recognition with convolutional visual transformers[J]. arXiv:2103.16854, 2021.
[7] WANG K, PENG X J, YANG J F, et al. Region attention networks for pose and occlusion robust facial expression recognition[J]. IEEE Transactions on Image Processing, 2020.
[8] CHEN S K, WANG J F, CHEN Y D, et al. Label distribution learning on auxiliary label space graphs for facial expression recognition[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 13981-13990.
[9] FARD A P, MAHOOR M H. Ad-corre: adaptive correlation-based loss for facial expression recognition in the wild[J]. IEEE Access, 2022, 10: 26756-26768.
[10] XUE F L, WANG Q C, GUO G D. TransFER: learning relation-aware facial expression representations with transformers[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 3581-3590.
[11] 罗思诗, 李茂军, 陈满. 多尺度融合注意力机制的人脸表情识别网络[J]. 计算机工程与应用, 2023, 59(1): 199-206.
LUO S S, LI M J, CHEN M. Multi-scale integrated attention mechanism for facial expression recognition network[J]. Computer Engineering and Applications, 2023, 59(1): 199-206.
[12] 程卫月, 张雪琴, 林克正, 等. 融合全局与局部特征的深度卷积神经网络算法[J]. 计算机科学与探索, 2022, 16(5): 1146-1154.
CHENG W Y, ZHANG X Q, LIN K Z, et al. Deep convolutional neural network algorithm fusing global and local features[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(5): 1146-1154.
[13] HASANI B, MAHOOR M H. Facial expression recognition using enhanced deep 3D convolutional neural networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway: IEEE, 2017: 2278-2288.
[14] QIU Y H, WAN Y. Facial expression recognition based on landmarks[C]//Proceedings of the 2019 IEEE 4th Advanced Information Technology, Electronic and Automation Control Conference. Piscataway: IEEE, 2019: 1356-1360.
[15] ZHENG C, MENDIETA M, CHEN C. POSTER: a pyramid cross-fusion transformer network for facial expression recognition[C]//Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision Workshops. Piscataway: IEEE, 2023: 3138-3147.
[16] ZHANG Q, YANG Y B. Rest: an efficient transformer for visual recognition[C]//Advances in Neural Information Processing Systems, 2021: 15475-15485.
[17] DENG J K, GUO J, XUE N N, et al. ArcFace: additive angular margin loss for deep face recognition[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 4685-4694.
[18] LI S, DENG W H, DU J P. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 2584-2593.
[19] MOLLAHOSSEINI A, HASANI B, MAHOOR M H. AffectNet: a database for facial expression, valence, and arousal computing in the wild[J]. IEEE Transactions on Affective Computing, 2019, 10(1): 18-31.
[20] WANG K, PENG X J, YANG J F, et al. Suppressing uncertainties for large-scale facial expression recognition[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 6896-6905.
[21] SHE J H, HU Y B, SHI H L, et al. Dive into ambiguity: latent distribution mining and pairwise uncertainty estimation for facial expression recognition[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 6244-6253.
[22] ZHAO Z, LIU Q, ZHOU F. Robust lightweight facial expression recognition network with label distribution training[C]//Proceedings of the 35th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2021: 3510-3519.
[23] ZHANG Y, WANG C, LING X, et al. Learn from all: erasing attention consistency for noisy label facial expression recognition[C]//Proceedings of the 17th European Conference on Computer Vision. Berlin: Springer, 2022: 418-434.
[24] WEN Z, LIN W, WANG T, et al. Distract your attention: multi-head cross attention network for facial expression recognition[J]. Biomimetics, 2023, 8(2): 199.
[25] LIU C, HIROTA K, DAI Y. Patch attention convolutional vision transformer for facial expression recognition with occlusion[J]. Information Sciences, 2023, 619: 781-794.