Multimodal Emotional Analysis of Semantic Information Enhancement Under Bimodal Bidirectional Perception

doi:10.3778/j.issn.1002-8331.2311-0069

Abstract

Abstract: In response to the problem of uneven distribution of emotional information among different modalities, which makes it difficult to obtain deeper multimodal emotional semantic information, this paper proposes a multimodal emotional analysis method for semantic information enhancement under bimodal bidirectional perception. Firstly, the text visual modality and text audio modality are fused separately to capture the interdependence in bimodal discourse and obtain bidirectional contextual information between modalities. Secondly, considering that bimodal fusion generates more redundant information, a gating mechanism is adopted to select effective emotional features to enhance the ability to identify key emotional information. Finally, the information between multiple modalities is modeled through a cross modal information exchange mechanism to obtain semantic information enhanced modal feature vectors. The proposed model is evaluated on the publicly available multimodal sentiment analysis dataset CMU-MOSI, and experimental results show that the sentiment analysis results of the model are superior to most existing advanced multimodal sentiment analysis methods, which can effectively improve the performance of sentiment analysis.

Key words: multimodal sentiment analysis, bimodal bidirectional perception, gating mechanism, information enhancement

摘要： 针对不同模态间存在情感信息分布不均匀，难以获得更深层次的多模态情感语义信息问题，提出了一种双模态双向感知下语义信息增强的多模态情感分析方法。对文本-视觉模态、文本-音频模态分别融合，捕获双模态话语中的相互依赖关系，获得模态间双向的上下文感知信息；考虑到双模态在融合时产生较多冗余信息，采用门控机制选择有效的情感特征，以提升识别关键情感信息的能力；通过跨模态信息交互机制对多种模态间的信息进行建模，得到语义信息增强的模态特征向量。在公开的多模态情感分析数据集CMU-MOSI上对所提出的模型进行评估，实验结果表明，该模型的情感分析结果优于大多数现有先进的多模态情感分析方法，能够有效提升情感分析的性能。

关键词: 多模态情感分析, 双模态双向感知, 门控机制, 信息增强

QU Haicheng, XU Bo. Multimodal Emotional Analysis of Semantic Information Enhancement Under Bimodal Bidirectional Perception[J]. Computer Engineering and Applications, 2024, 60(19): 139-147.

曲海成, 徐波. 双模态双向感知下语义信息增强的多模态情感分析[J]. 计算机工程与应用, 2024, 60(19): 139-147.

References

[1] PEÑA D, AGUILERA A, DONGO I, et al. A framework to evaluate fusion methods for multimodal emotion recognition[J]. IEEE Access, 2023, 11: 10218-10237.
[2] ZHAO S, JIA G, YANG J, et al. Emotion recognition from multiple modalities: fundamentals and methodologies[J]. IEEE Signal Processing Magazine, 2021, 38(6): 59-73.
[3] GANDHI A, ADHVARYU K, PORIA S, et al. Multimodal sentiment analysis: a systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions[J]. Information Fusion, 2023, 91: 424-444.
[4] HUANG C, ZHANG J, WU X, et al. TeFNA: text-centered fusion network with crossmodal attention for multimodal sentiment analysis[J]. Knowledge-Based Systems, 2023, 269: 110502.
[5] HAN W, CHEN H, GELBUKH A, et al. Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis[C]//Proceedings of the 2021 International Conference on Multimodal Interaction, 2021: 6-15.
[6] ZADEH A, CHEN M, PORIA S, et al. Tensor fusion network for multimodal sentiment analysis[J]. arXiv:1707.07250, 2017.
[7] LIU Z, SHEN Y, LAKSHMINARASIMHAN V B, et al. Efficient low-rank multimodal fusion with modality-specific factors[J]. arXiv:1806.00064, 2018.
[8] BALTRU?AITIS T, AHUJA C, MORENCY L P. Multimodal machine learning: a survey and taxonomy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(2): 423-443.
[9] JANGRA A, MUKHERJEE S, JATOWT A, et al. A survey on multi-modal summarization[J]. ACM Computing Surveys, 2023, 55(13S): 1-36.
[10] NGIAM J, KHOSLA A, KIM M, et al. Multimodal deep learning[C]//Proceedings of the 28th International Conference on Machine Learning (ICML-11), 2011: 689-696.
[11] NGUYEN D, NGUYEN K, SRIDHARAN S, et al. Deep spatio-temporal feature fusion with compact bilinear pooling for multimodal emotion recognition[J]. Computer Vision and Image Understanding, 2018, 174: 33-42.
[12] HAZARIKA D, ZIMMERMANN R, PORIA S. MISA: modality-invariant and-specific representations for multimodal sentiment analysis[C]//Proceedings of the 28th ACM International Conference on Multimedia, 2020: 1122-1131.
[13] YU W, XU H, YUAN Z, et al. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2021: 10790-10797.
[14] YU J, JIANG J, XIA R. Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2019, 28: 429-439.
[15] XIAO G, TU G, ZHENG L, et al. Multimodality sentiment analysis in social Internet of things based on hierarchical attentions and CSAT-TCN with MBM network[J]. IEEE Internet of Things Journal, 2020, 8(16): 12748-12757.
[16] ZHAO J, YANG F. Fusion with GCN and SE-ResNeXt network for aspect based multimodal sentiment analysis[C]//Proceedings of the 2023 IEEE 6th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), 2023: 336-340.
[17] OU Y, CHEN Z, WU F. Multimodal local-global attention network for affective video content analysis[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 31(5): 1901-1914.
[18] ZADEH A, LIANG P P, PORIA S, et al. Multi-attention recurrent network for human communication comprehension[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
[19] TSAI Y H H, BAI S, LIANG P P, et al. Multimodal transformer for unaligned multimodal language sequences[C]//Proceedings of the 57th Annual Meeting of? The Association for Computational Linguistics, 2019.
[20] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of the International Conference on Machine Learning, 2021: 8748-8763.
[21] ZHU R, HAN C, QIAN Y, et al. Exchanging-based multimodal fusion with Transformer[J]. arXiv:2309.02190, 2023.
[22] JI Y, WANG J, GONG Y, et al. MAP: multimodal uncertainty-aware vision-language pre-training model[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 23262-23271.
[23] BANNUR S, HYLAND S, LIU Q, et al. Learning to exploit temporal structure for biomedical vision-language processing[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 15016-15027.
[24] WU W, WANG X, LUO H, et al. Bidirectional cross-modal knowledge exploration for video recognition with pre-trained vision-language models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 6620-6630.
[25] ZADEH A, ZELLERS R, PINCUS E, et al. Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages[J]. IEEE Intelligent Systems, 2016, 31(6): 82-88.
[26] SAHAY S, OKUR E, KUMAR S H, et al. Low rank fusion based transformers for multimodal sequences[J]. arXiv:2007.02038, 2020.
[27] XU M, LIANG F, SU X, et al. CMJRT: cross-modal joint representation Transformer for multimodal sentiment analysis[J]. IEEE Access, 2022, 10: 131671-131679.
[28] FU Z, LIU F, WANG H, et al. LMR-CBT: learning modality-fused representations with CB-transformer for multimodal emotion recognition from unaligned multimodal sequences[J]. arXiv:2112.01697, 2021.
[29] LV F, CHEN X, HUANG Y, et al. Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 2554-2562.
[30] WANG Y, LI Y, BELL P, et al. Cross-attention is not enough: incongruity-aware multimodal sentiment analysis and emotion recognition[J]. arXiv:2305.13583, 2023.