计算机工程与应用 ›› 2024, Vol. 60 ›› Issue (19): 139-147.DOI: 10.3778/j.issn.1002-8331.2311-0069

• 模式识别与人工智能 • 上一篇    下一篇

双模态双向感知下语义信息增强的多模态情感分析

曲海成,徐波   

  1. 辽宁工程技术大学  软件学院,辽宁  葫芦岛  125105
  • 出版日期:2024-10-01 发布日期:2024-09-30

Multimodal Emotional Analysis of Semantic Information Enhancement Under Bimodal Bidirectional Perception

QU Haicheng, XU Bo   

  1. School of Software, Liaoning Technical University, Huludao, Liaoning 125105, China
  • Online:2024-10-01 Published:2024-09-30

摘要: 针对不同模态间存在情感信息分布不均匀,难以获得更深层次的多模态情感语义信息问题,提出了一种双模态双向感知下语义信息增强的多模态情感分析方法。对文本-视觉模态、文本-音频模态分别融合,捕获双模态话语中的相互依赖关系,获得模态间双向的上下文感知信息;考虑到双模态在融合时产生较多冗余信息,采用门控机制选择有效的情感特征,以提升识别关键情感信息的能力;通过跨模态信息交互机制对多种模态间的信息进行建模,得到语义信息增强的模态特征向量。在公开的多模态情感分析数据集CMU-MOSI上对所提出的模型进行评估,实验结果表明,该模型的情感分析结果优于大多数现有先进的多模态情感分析方法,能够有效提升情感分析的性能。

关键词: 多模态情感分析, 双模态双向感知, 门控机制, 信息增强

Abstract: In response to the problem of uneven distribution of emotional information among different modalities, which makes it difficult to obtain deeper multimodal emotional semantic information, this paper proposes a multimodal emotional analysis method for semantic information enhancement under bimodal bidirectional perception. Firstly, the text visual modality and text audio modality are fused separately to capture the interdependence in bimodal discourse and obtain bidirectional contextual information between modalities. Secondly, considering that bimodal fusion generates more redundant information, a gating mechanism is adopted to select effective emotional features to enhance the ability to identify key emotional information. Finally, the information between multiple modalities is modeled through a cross modal information exchange mechanism to obtain semantic information enhanced modal feature vectors. The proposed model is evaluated on the publicly available multimodal sentiment analysis dataset CMU-MOSI, and experimental results show that the sentiment analysis results of the model are superior to most existing advanced multimodal sentiment analysis methods, which can effectively improve the performance of sentiment analysis.

Key words: multimodal sentiment analysis, bimodal bidirectional perception, gating mechanism, information enhancement