结合三维交互注意力与语义聚合的表情识别

doi:10.3778/j.issn.1002-8331.2210-0398

摘要/Abstract

摘要： 针对传统卷积网络难以有效整合不同阶段人脸面部表情的特征、存在特征表征瓶颈以及无法高效利用上下文语义等问题，提出了一种结合三维交互注意力与语义聚合的面部表情识别方法。在秩扩展（ReXNet）网络的基础上对其进行优化，在消除表征瓶颈的情况下，融合上下文特征，使其更适配表情识别任务。为捕获判别性人脸表情细粒度特征，结合非本地块与跨维度信息交互理论构建了三维交互注意力。为充分利用表情的浅中层底层特征与高层语义特征，设计了语义聚合模块，将多级全局上下文特征与高级语义信息进行聚合，达到同一类别的表情语义相互增益、增强类内一致性的目的。实验表明，该方法在公开数据集RAF-DB、FERPlus和AffectNet-8上的准确率分别为88.89%、89.53%与62.22%，展现了该方法的先进性。

关键词: 人脸表情识别, 表征瓶颈, 三维交互注意力, 上下文语义

Abstract: A facial expression recognition method combining 3D augmented attention and semantic aggregation is proposed to address the problems that traditional convolutional networks are difficult to effectively integrate features of facial expressions of faces at different stages, have feature expression bottlenecks and cannot efficiently utilize contextual semantics. Firstly, it is optimized on the basis of rank expansion (ReXNet) network to fuse contextual features while eliminating expression bottlenecks to make it more suitable for expression recognition tasks. Secondly, to capture discriminative face expression fine-grained features, 3D augmented attention is constructed by combining non-local blocks with cross-dimensional information interaction theory. Finally, in order to fully utilize the shallow and mid-level underlying features and high-level semantic features of expressions, a semantic aggregation module is designed to aggregate multi-level global contextual features with high-level semantic information to achieve mutual semantic gain of expressions of the same class and enhance intra-class consistency. Experiments show that the accuracy of the method is 88.89%, 89.53% and 62.22% on the publicly available datasets RAF-DB, FERPlus and AffectNet-8, respectively, demonstrating the advancedness of the method.

Key words: facial expression recognition, expression bottlenecks, 3D interactive attention, contextual semantics

王广宇, 罗晓曙, 徐照兴, 丰芳宇, 许江杰. 结合三维交互注意力与语义聚合的表情识别[J]. 计算机工程与应用, 2024, 60(6): 238-248.

WANG Guangyu, LUO Xiaoshu, XU Zhaoxing, FENG Fangyu, XU Jiangjie. Expression Recognition Combining 3D Interactive Attention and Semantic Aggregation[J]. Computer Engineering and Applications, 2024, 60(6): 238-248.

参考文献

[1] TIAN Y, KANADE T, COHN J F. Recognizing action units for facial expression analysis[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(2): 19.
[2] ZHANG F, ZHANG T, MAO Q, et al. Joint pose and expression modeling for facial expression recognition[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 3359-3368.
[3] SHAN C F, GONG S G, MCOWAN P W. Robust facial expression recognition using local binary patterns[C]//IEEE International Conference on Image Processing. Genova, Italy:IEEE, 2005.
[4] DALAL N, TRIGGS B. Histograms of oriented gradients for human detection[C]//2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05). San Diego, CA, USA: IEEE, 2005: 886-893.
[5] WOLD S, GELADI P, ESBENSEN K, et al. Multi-way principal components-and PLS-analysis[J]. Journal of Chemometrics, 1987, 1(1): 41-56.
[6] LI S, DENG W, DU J. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI: IEEE, 2017: 2584-2593.
[7] BARSOUM E, ZHANG C, FERRER C C, et al. Training deep networks for facial expression recognition with crowd-sourced label distribution[C]//Proceedings of the 18th ACM International Conference on Multimodal Interaction. Tokyo, Japan: ACM, 2016: 279-283.
[8] MOLLAHOSSEINI A, HASANI B, MAHOOR M H. AffectNet: a database for facial expression, valence, and arousal computing in the wild[J]. IEEE Transactions on Affective Computing, 2019, 10(1): 18-31.
[9] WANG K, PENG X, YANG J, et al. Suppressing uncertainties for large-scale facial expression recognition[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA: IEEE, 2020: 6896-6905.
[10] FARZANEH A H, QI X. Facial expression recognition in the wild via deep attentive center loss[C]//2021 IEEE Winter Conference on Applications of Computer Vision (WACV). Waikoloa, HI, USA: IEEE, 2021: 2401-2410.
[11] 张为, 李璞. 基于注意力机制的人脸表情识别网络[J]. 天津大学学报 (自然科学与工程技术版), 2022, 55(7): 706-713.
ZHANG W, LI P. Facial expression recognition network based on attention mechanism[J]. Journal of Tianjin University (Science and Technology), 2022, 55(7): 706-713.
[12] WANG K, PENG X, YANG J, et al. Region attention networks for pose and occlusion robust facial expression recognition[J]. IEEE Transactions on Image Processing, 2020, 29: 4057-4069.
[13] 郑剑, 郑炽, 刘豪, 等. 融合局部特征与两阶段注意力权重学习的面部表情识别[J]. 计算机应用研究, 2022, 39(3): 889-894.
ZHENG J, ZHENG C, LIU H, et al. Deep convolutional neural network fusing local feature and two-stage attention weight learning for facial expression recognition[J]. Applications Research of Computers, 2022, 39(3): 889-894.
[14] GERA D, BALASUBRAMANIAN S. Landmark guidance independent spatio-channel attention and complementary context information based facial expression recognition[J]. Pattern Recognition Letters, 2021, 145: 58-66.
[15] HAN D, YUN S, HEO B, et al. Rethinking channel dimensions for efficient model design[J]. arXiv:2007.00992, 2020.
[16] MISRA D, NALAMADA T, ARASANIPALAI A U, et al. Rotate to attend: convolutional triplet attention module[C]//2021 IEEE Winter Conference on Applications of Computer Vision (WACV). Waikoloa, HI, USA: IEEE, 2021: 3138-3147.
[17] WANG X, GIRSHICK R, GUPTA A, et al. Non-local neural networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 7794-7803.
[18] CAO Y, XU J, LIN S, et al. GCNet: non-local networks meet squeeze-excitation networks and beyond[C]//2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). Seoul, Korea (South): IEEE, 2019: 1971-1980.
[19] WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]//15th European Conference on Computer Vision (ECCV 2018). Cham: Springer International Publishing, 2018: 3-19.
[20] HOU Q, ZHOU D, FENG J. Coordinate attention for efficient mobile network design[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[21] XU J, ZOU Y, TAN Y, et al. Chip pad inspection method based on an improved YOLOv5 algorithm[J]. Sensors, 2022, 22(17): 6685.
[22] ZHANG X, ZHOU X, LIN M, et al. ShuffleNet: an extremely efficient convolutional neural network for mobile devices[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT: IEEE, 2018: 6848-6856.
[23] MA H, YANG H, HUANG D. Boundary guided context aggregation for semantic segmentation[J]. arXiv:2110.14587, 2021.
[24] ZHANG Y, WANG C, DENG W. Relative uncertainty learning for facial expression recognition[C]//Advances in Neural Information Processing Systems, 2021: 17616-17627.
[25] SHE J, HU Y, SHI H, et al. Dive into ambiguity: latent distribution mining and pairwise uncertainty estimation for facial expression recognition[J].arXiv:2104.00232, 2021.
[26] FAN X, DENG Z, WANG K, et al. Learning discriminative representation for facial expression recognition from uncertainties[C]//2020 IEEE International Conference on Image Processing (ICIP). Abu Dhabi, United Arab Emirates: IEEE, 2020: 903-907.
[27] WEN Z, LIN W, WANG T, et al. Distract your attention: multi-head cross attention network for facial expression recognition[J]. arXiv:2109.07270, 2021.
[28] GUO Y, ZHANG L, HU Y, et al. MS-Celeb-1M: a dataset and benchmark for large-scale face recognition[C]//European Conference on Computer Vision (ECCV 2016). Cham: Springer International Publishing, 2016: 87-102.
[29] MA F, SUN B, LI S. Facial expression recognition with visual transformers and attentional selective fusion[J]. IEEE Transactions on Affective Computing, 2023, 14(2): 1236-1248.
[30] LI H, SUI M, ZHAO F, et al. MVT: mask vision transformer for facial expression recognition in the wild[J]. arXiv:2106. 04520, 2021.
[31] 封红旗, 黄伟铠, 张登辉. 结合显著特征筛选和ViT的面部表情识别方法[J]. 计算机工程与应用, 2023, 59(22):136-143.
FENG H Q, HUANG W K, ZHANG D H. Facial expression recognition with distinguishing feature filtering and ViT[J]. Computer Engineering and Applications, 2023, 59(22): 136-143.
[32] SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-CAM: visual explanations from deep networks via gradient-based localization[C]//2017 IEEE International Conference on Computer Vision (ICCV), 2017: 618-626.
[33] MAATEN L, HINTON G. Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9(11): 2579-2605.