融合反事实推理的多模态情感分析算法研究

doi:10.3778/j.issn.1002-8331.2404-0424

摘要/Abstract

摘要： 在多模态情感分析中，现有的情感识别模型大多在测试集上准确率高，但在实际应用中，模型情感识别准确率比测试集中低，即模型的泛化能力不强。许多研究表明，这种现象归因于：多模态情感分析模型在训练阶段严重依赖于文本模态，进而学习到了文本模态和情感标签之间关于社会、种族、性别等刻板印象，也称为文本偏见。提出一种基于反事实推理的多模态情感分析模型（counterfactual reasoning for multimodal sentiment analysis，CRFM），构建因果图并分析情感识别结果的因果效应，利用反事实推理从情感识别结果总效应中去除文本偏见直接效应，从而去除文本偏见不良影响，提高模型情感识别准确率，进而增强模型泛化能力。利用MOSI和MOSEI两种数据集，分别与六种基线模型进行对比实验，CRFM准确率达87.05%，优于其他基线模型。此外，CRFM在两种数据集中准确率更稳定，模型泛化能力更强。

关键词: 多模态情感分析, 反事实推理, 因果效应, 因果图

Abstract: In multimodal sentiment analysis, existing sentiment recognition models often achieve high accuracy on the test set, but their performance in real-world applications is lower, indicating weak generalization ability. Many studies have attributed this phenomenon to the heavy reliance of multimodal sentiment analysis models on the textual during the training phase, resulting in the learning of stereotypical biases related to social, racial, and gender factors, known as textual biases. A counterfactual reasoning for multimodal sentiment analysis (CRFM) model is proposed, constructing a causal diagram and analyziing the causal effects of sentiment recognition results. Counterfactual reasoning is utilized to remove the direct effects of textual biases from the total effect of sentiment recognition results, thereby removing the adverse effects of textual biases, improving the accuracy of sentiment recognition, and enhancing the model’s generalization ability. Comparative experiments are conducted with six baseline models in both MOSI and MOSEI datasets. The CRFM accuracy reaches 87.05%, which is better than other baseline models. In addition, CRFM has a more stable accuracy and stronger model generalization ability in both datasets.

Key words: multimodal sentiment analysis, counterfactual reasoning, causal effects, causal diagram

王淑娴, 杨海, 冯程, 李雪. 融合反事实推理的多模态情感分析算法研究[J]. 计算机工程与应用, 2025, 61(15): 199-208.

WANG Shuxian, YANG Hai, FENG Cheng, LI Xue. Multimodal Sentiment Analysis Based on Counterfactual Reasoning[J]. Computer Engineering and Applications, 2025, 61(15): 199-208.

参考文献

[1] CHAKRABORTY D, RUDRAPAL D, BHATTACHARYA B. A multimodal sentiment analysis approach for tweets by comprehending co-relations between information modalities[J]. Multimedia Tools and Applications, 2024, 83(17): 50061-50085.
[2] WANG Z H, HE S M, XU G, et al. Will sentiment analysis need subculture? A new data augmentation approach[J]. Journal of the Association for Information Science and Technology, 2024, 75(6): 655-670.
[3] CALISKAN A, BRYSON J J, NARAYANAN A. Semantics derived automatically from language corpora contain human-like biases[J]. Science, 2017, 356(6334): 183-186.
[4] RUDINGER R, MAY C, VAN DURME B. Social bias in elicited natural language inferences[C]//Proceedings of the First ACL Workshop on Ethics in Natural Language Processing. Stroudsburg: ACL, 2017: 74-79.
[5] LIANG P, WU C, MOTENCY L. Towards understanding and mitigating social biases in language models[C]//Proceedings of the International Conference on Machine Learning, 2021: 6565-6576.
[6] ZADEH A, CHEN M H, PORIA S, et al. Tensor fusion network for multimodal sentiment analysis[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2017: 1103-1114.
[7] HE X T, PENG Y X. Fine-grained image classification via combining vision and language[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 7332-7340.
[8] WU Y Q, KUANG K, ZHANG Y T, et al. De-biased court’s view generation with causality[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2020: 763-780.
[9] ZENG X J, LI Y L, ZHAI Y C, et al. Counterfactual generator: a weakly-supervised method for named entity recognition[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2020: 7270-7280.
[10] YU J F, JIANG J, XIA R. Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2019, 28: 429-439.
[11] PéREZ-ROSAS V, MIHALCEA R, MORENCY L P. Utterance-level muItimodal sentiment analysis[C]//Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics(Volume 1: Long Papers), 2013: 973-982.
[12] PORIA S, CAMBRIA E, GELBUKH A. Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2015: 2539-2544.
[13] MAO Y Y, LIU Q, ZHANG Y. Sentiment analysis methods, applications, and challenges: a systematic literature review[J]. Journal of King Saud University - Computer and Information Sciences, 2024, 36(4): 102048.
[14] NOJAVANASGHARI B, GOPINATH D, KOUSHIK J, et al. Deep multimodal fusion for persuasiveness prediction[C]//Proceedings of the 18th ACM International Conference on Multimodal Interaction. New York: ACM, 2016: 284-288.
[15] YU Y H, LIN H F, MENG J N, et al. Visual and textual sentiment analysis of a microblog using deep convolutional neural networks[J]. Algorithms, 2016, 9(2): 41.
[16] HUSSAIN M S, CALVO R A, AGHAEI POUR P. Hybrid fusion approach for detecting affects from multichannel physiology[C]//Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction. Berlin: Springer, 2011: 568-577.
[17] CHEN M H, WANG S, LIANG P P, et al. Multimodal sentiment analysis with word-level fusion and reinforcement learning[C]//Proceedings of the 19th ACM International Conference on Multimodal Interaction. New York: ACM, 2017: 163-171.
[18] GUNES H, PICCARDI M. Bi-modal emotion recognition from expressive face and body gestures[J]. Journal of Network and Computer Applications, 2007, 30(4): 1334-1345.
[19] FENG F L, ZHANG J Z, HE X N, et al. Empowering language understanding with counterfactual reasoning[C]//Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Stroudsburg: ACL, 2021: 2226-2236.
[20] LIU X, YIN D, FENG Y S, et al. Everything has a cause: leveraging causal inference in legal text analysis[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg: ACL, 2021: 1928-1941.
[21] WEI J, ZOU K. Easy data augmentation techniques for boosting performance on text classification tasks[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019: 6382-6388.
[22] QIU R H, WANG S, CHEN Z, et al. CausalRec: causal inference for visual debiasing in visually-aware recommendation[C]//Proceedings of the 29th ACM International Conference on Multimedia. New York: ACM, 2021: 3844-3852.
[23] NIU Y L, TANG K H, ZHANG H W, et al. Counterfactual VQA: a cause-effect look at language bias[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 12695-12705.
[24] 扆雅欣, 孙欣伊, 谭红叶. 基于反事实推理的阅读理解去偏方法[J]. 软件导刊, 2023, 22(1): 152-157.
YI Y X, SUN X Y, TAN H Y. Counterfactual inference for reading comprehension debiasing[J]. Software Guide, 2023, 22(1): 152-157.
[25] 肖之仪. 融合因果推理的细粒度情感分析算法的设计与实现[D]. 北京: 北京邮电大学, 2023.
XIAO Z Y. Design and implementation of fine-grained sentiment analysis algorithm based on causal reasoning[D]. Beijing: Beijing University of Posts and Telecommunications, 2023.
[26] RAHMAN W, HASAN M K, LEE S W, et al. Integrating multimodal information in large pretrained transformers[C]//Proceedings of the Conference Association for Computational Linguistics Meeting, 2020: 2359-2369.
[27] DEVLIN J, CHANG M, LEE M. BERT: Pre-training of deep bidirectional transformers for language understanding [C]//Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2019: 4171-4186.
[28] SUN T, WANG W J, JING L, et al. Counterfactual reasoning for out-of-distribution multimodal sentiment analysis[C]//Proceedings of the 30th ACM International Conference on Multimedia. New York: ACM, 2022: 15-23.
[29] ZUHEROS C, MARTíNEZ-CáMARA E, HERRERA-VIEDMA E, et al. Sentiment analysis based multi-person multi-criteria decision making methodology using natural language proce-ssing and deep learning for smarter decision aid. Case study of restaurant choice using TripAdvisor reviews[J]. Information Fusion, 2021, 68: 22-36.
[30] GAO K, ANANDHAN P, KUMAR R. Analysis and evaluation of the regional air quality index forecasting based on web-text sentiment analysis method[J]. Environmental Impact Assessment Review, 2021, 87: 106514.
[31] CADENE R, BEN-YOUNES H, CORD M, et al. MUREL: multimodal relational reasoning for visual question answering[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 1989-1998.
[32] CHHAJRO M A, LUHANA K K, ALI WAGAN A, et al. Text sentiment analysis from students’ comments using various machine learning and deep learning techniques: a comparative analysis[J]. International Journal of Data Analysis Techniques and Strategies, 2023, 15(4): 323-338.
[33] LE X Q, CHU J D, DENG S Y, et al. CiteOpinion: evidence-based evaluation tool for academic contributions of research papers based on citing sentences[J]. Journal of Data and Information Science, 2019, 4(4): 26-41.