关键点引导与显著帧增强的情感识别网络

doi:10.3778/j.issn.1002-8331.2403-0291

摘要/Abstract

摘要： 针对表情与姿态两类情感线索空间占比差异及时间峰值异步的问题，提出一种关键点引导与显著帧增强的情感识别网络。在空间关键点引导子网中，为捕获帧内表情-姿态的情感相关性和互补性信息，基于跨模态注意力和残差结构构建空间关键点引导机制分别获取表情引导语义和姿态引导语义。在时间显著帧增强子网中，为了减少帧间表情-姿态情感峰值异步带来的冗余信息，根据表情引导语义和姿态引导语义度量情感区分度和离散度，提出时间显著帧增强策略实现引导语义序列的时空特征增强。在FABO和CAER视频数据集上的实验结果表明：提出网络的情感识别准确率分别达到95.31%和89.78%，比基线网络分别提高了11.50和13.66个百分点；与相关方法相比，提出方法在自然场景动态视频数据集和静态图片数据集上均具有较好的情感识别性能。

关键词: 面部表情, 肢体姿态, 情感识别, 关键点引导, 显著帧增强

Abstract: Aiming at the problems of spatial proportion difference and temporal peak asynchrony of emotion cues between facial expression and bodily posture, a key-points guidance and significant-frames enhancement (KGSE-ER) network is proposed for emotion recognition. In the spatial key-points guidance subnet, to capture the intra-frame emotional correlation and complementary information between facial expression and bodily posture, a spatial key-points guidance mechanism is constructed to obtain facial expression guidance semantics and bodily posture guidance semantics based on a cross-modal attention and a residual structure. In the temporal significant-frames enhancement subnet, to reduce the inter-frame redundant information caused by temporal peak asynchrony between facial expression and bodily posture, the emotional discrimination and dispersion are measured according to the facial expression guidance semantics and the bodily posture guidance semantics, and a temporal significant-frames enhancement strategy is proposed to enhance spatiotemporal features of cue-guided semantic sequences. The experimental results on FABO and CAER video datasets show that the emotion recognition accuracy of the proposed network reaches 95.31% and 89.78% respectively, which is 11.50 and 13.66 percentage points higher than the baseline network. Compared with related methods, the proposed network has better emotion recognition performance on both natural scene dynamic video datasets and static image datasets.

Key words: facial expression, bodily posture, emotion recognition, key-points guidance, significant-frames enhancemen

黄忠, 张丹妮, 任福继, 胡敏, 刘娟. 关键点引导与显著帧增强的情感识别网络[J]. 计算机工程与应用, 2025, 61(18): 142-156.

HUANG Zhong, ZHANG Danni, REN Fuji, HU Min, LIU Juan. Key-Points Guidance and Significant-Frames Enhancement for Emotion Recognition Network[J]. Computer Engineering and Applications, 2025, 61(18): 142-156.

参考文献

[1] LEONG S C, TANG Y M, LAI C H, et al. Facial expression and body gesture emotion recognition: a systematic review on the use of visual data in affective computing[J]. Computer Science Review, 2023, 48: 100545.
[2] 缪裕青, 董晗, 张万桢, 等. 一种多任务学习的跨模态视频情感分析方法[J]. 计算机工程与应用, 2023, 59(12): 141-147.
MIAO Y Q, DONG H, ZHANG W Z, et al. Cross-modal video emotion analysis method based on multi-task learning[J]. Computer Engineering and Applications, 2023, 59(12): 141-147.
[3] ZHAO R, LIU T S, HUANG Z X, et al. Spatial-temporal graphs plus transformers for geometry-guided facial expression recognition[J]. IEEE Transactions on Affective Computing, 2023, 14(4): 2751-2767.
[4] BARROS P, SCIUTTI A. Across the universe: biasing facial representations toward non-universal emotions with the face-STN[J]. IEEE Access, 2022, 10: 103932-103947.
[5] GUO Y F, HUANG J, XIONG M F, et al. Facial expressions recognition with multi-region divided attention networks for smart education cloud applications[J]. Neurocomputing, 2022, 493: 119-128.
[6] WANG S M, SHUAI H, LIU C G, et al. Bias-based soft label learning for facial expression recognition[J]. IEEE Transactions on Affective Computing, 2023, 14(4): 3257-3268.
[7] 张华忠, 潘曰凯, 涂晓光, 等. 融合三维人脸动态信息和光流信息的人脸表情识别[J]. 计算机科学, 2024, 51(S1): 594-600.
ZHANG H Z, PAN Y K, TU X G, et al. Facial expression recognition integrating 3D facial dynamic information and optical flow information[J]. Computer Science, 2024, 51(S1):594-600.
[8] 姚鸿勋, 邓伟洪, 刘洪海, 等. 情感计算与理解研究发展概述[J]. 中国图象图形学报, 2022, 27(6): 2008-2035.
YAO H X, DENG W H, LIU H H, et al. An overview of research development of affective computing and understanding[J]. Journal of Image and Graphics, 2022, 27(6): 2008-2035.
[9] SUN N, SONG Y, LIU J X, et al. Appearance and geometry transformer for facial expression recognition in the wild[J]. Computers and Electrical Engineering, 2023, 107: 108583.
[10] LENZONI S, BOZZONI V, BURGIO F, et al. Recognition of emotions conveyed by facial expression and body postures in myotonic dystrophy (DM)[J]. Cortex, 2020, 127: 58-66.
[11] BLYTHE E, GARRIDO L, LONGO M R. Emotion is perceived accurately from isolated body parts, especially hands[J]. Cognition, 2023, 230: 105260.
[12] MAHFOUDI M A, MEYER A, GAUDIN T, et al. Emotion expression in human body posture and movement: a survey on intelligible motion factors, quantification and validation[J]. IEEE Transactions on Affective Computing, 2023, 14(4): 2697-2721.
[13] NOROOZI F, CORNEANU C A, KAMI?SKA D, et al. Survey on emotional body gesture recognition[J]. IEEE Transactions on Affective Computing, 2021, 12(2): 505-523.
[14] GUNES H, PICCARDI M. A bimodal face and body gesture database for automatic analysis of human nonverbal affective behavior[C]//Proceedings of the 18th International Conference on Pattern Recognition. Piscataway: IEEE, 2006: 1148-1153.
[15] WEI J, HU G Y, YANG X Y, et al. Learning facial expression and body gesture visual information for video emotion recognition[J]. Expert Systems with Applications, 2024, 237: 121419.
[16] KOSTI R, ALVAREZ J M, RECASENS A, et al. Context based emotion recognition using EMOTIC dataset[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(11): 2755-2766.
[17] WANG S M, SHUAI H, ZHU L, et al. Expression complementary disentanglement network for facial expression recognition[J]. Chinese Journal of Electronics, 2024, 33(3): 742-752.
[18] LI W X, DONG X, WANG Y H. Human emotion recognition with relational region-level analysis[J]. IEEE Transactions on Affective Computing, 2023, 14(1): 650-663.
[19] WU S C, ZHOU L, HU Z X, et al. Hierarchical context-based emotion recognition with scene graphs[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(3): 3725-3739.
[20] QI H X, HAN Z F. Emotion recognition and management in the tourism industry during emergency events using improved convolutional neural network[J]. IEEE Access, 2024, 12: 32660-32667.
[21] PALASH M, BHARGAVA B. EMERSK-explainable multimodal emotion recognition with situational knowledge[J]. IEEE Transactions on Multimedia, 2023, 26: 2785-2794.
[22] ZAGHBANI S, BOUHLEL M S. Multi-task CNN for multi-cue affects recognition using upper-body gestures and facial expressions[J]. International Journal of Information Technology, 2022, 14(1): 531-538.
[23] THUSEETHAN S, RAJASEGARAR S, YEARWOOD J. EmoSeC: emotion recognition from scene context[J]. Neurocomputing, 2022, 492: 174-187.
[24] WANG Z L, LAO L J, ZHANG X Y, et al. Context-dependent emotion recognition[J]. Journal of Visual Communication and Image Representation, 2022, 89: 103679.
[25] GAO Q Q, ZENG H X, LI G, et al. Graph reasoning-based emotion recognition network[J]. IEEE Access, 2021, 9: 6488-6497.
[26] ZHOU S W, WU X M, JIANG F, et al. Emotion recognition from large-scale video clips with cross-attention and hybrid feature weighting neural networks[J]. International Journal of Environmental Research and Public Health, 2023, 20(2): 1400.
[27] LE N, NGUYEN K, NGUYEN A, et al. Global-local attention for emotion recognition[J]. Neural Computing and Applications, 2022, 34(24): 21625-21639.
[28] 郭靖圆, 董乙杉, 刘晓文, 等. 注意力机制与Involution算子改进的人脸表情识别[J]. 计算机工程与应用, 2023, 59(23): 95-103.
GUO J Y, DONG Y S, LIU X W, et al. Facial expression recognition based on attention mechanism and involution[J]. Computer Engineering and Applications, 2023, 59(23): 95-103.
[29] 胡敏, 胡鹏远, 葛鹏, 等. 基于面部运动单元和时序注意力的视频表情识别方法[J]. 计算机辅助设计与图形学学报, 2023, 35(1): 108-117.
HU M, HU P Y, GE P, et al. Video expression recognition method based on facial motion unit and temporal attention[J]. Journal of Computer-Aided Design & Computer Graphics, 2023, 35(1): 108-117.
[30] BARROS P, WERMTER S. Developing crossmodal expression recognition based on a deep neural model[J]. Adaptive Behavior, 2016, 24(5): 373-396.
[31] CHEN L F, LI M, WU M, et al. Coupled multimodal emotional feature analysis based on broad-deep fusion networks in human-robot interaction[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(7): 9663-9673.
[32] LI M, CHEN L F, WU M, et al. Broad-deep network-based fuzzy emotional inference model with personal information for intention understanding in human-robot interaction[J]. Annual Reviews in Control, 2024, 57: 100951.
[33] NGUYEN D, NGUYEN K, SRIDHARAN S, et al. Deep spatio?temporal feature fusion with compact bilinear pooling for multimodal emotion recognition[J]. Computer Vision and Image Understanding, 2018, 174: 33-42.
[34] ILYAS C, NUNES R, NASROLLAHI K, et al. Deep emotion recognition through upper body movements and facial expression[C]//Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications. SCITEPRESS-Science and Technology Publications, 2021: 669-679.
[35] LIN B J, LIN Y T, LIU C C, et al. Mental status detection for schizophrenia patients via deep visual perception[J]. IEEE Journal of Biomedical and Health Informatics, 2022, 26(11): 5704-5715.
[36] 陈彩华. 基于语音、表情与姿态的三模态普通话情感识别[J]. 控制工程, 2020, 27(11): 2023-2029.
CHEN C H. Tri-modal mandarin emotion recognition based on speech, facial expression and body gesture[J]. Control Engineering of China, 2020, 27(11): 2023-2029.
[37] VERMA B, CHOUDHARY A. Affective state recognition from hand gestures and facial expressions using Grassmann manifolds[J]. Multimedia Tools and Applications, 2021, 80(9): 14019-14040.
[38] ZHANG Z C, WANG L J, YANG J F. Weakly supervised video emotion detection and prediction via cross-modal temporal erasing network[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2023: 18888-18897.
[39] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale[J]. arXiv:2010.11929v2, 2020.
[40] SHI L, ZHANG Y, CHENG J, et al. Decoupled spatial-temporal attention network for skeleton-based action recognition[J]. arXiv:2007.03263, 2020.
[41] YUN W L, QI M S, WANG C M, et al. Weakly-supervised temporal action localization by inferring salient snippet-feature[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2024: 6908-6916.
[42] LEE J, KIM S Y, KIM S, et al. Context-aware emotion recognition networks[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2019: 10142-10151.
[43] SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-CAM: visual explanations from deep networks via gradient-based localization[J]. International Journal of Computer Vision, 2020, 128(2): 336-359.