基于多模态信息融合的中文隐式情感分析

doi:10.3778/j.issn.1002-8331.2406-0108

摘要/Abstract

摘要： 隐式情感表达中缺乏显式情感词，给隐式情感分析带来一定的挑战。为有效解决此问题，借助外部信息是有效解决隐式情感分析的方法之一。与现有的主要借助单一文本信息的研究不同，提出一种融合多模态信息（包括语音和视频）的隐式情感分析方法。通过从语音中提取音调、强度等声学特征，以及从视频中捕捉面部表情等视觉特征，辅助理解隐式情感。利用BiLSTM网络挖掘各单模态内部的上下文信息；结合多头互注意力机制分别捕捉与文本相关的语音和视觉特征，并通过迭代优化，减少非文本模态的低阶冗余信息。此外，通过设计以文本为中心的交叉注意融合模块，强化隐式文本特征表示，并处理模态间的异质性，增强隐式情感分析的综合性能。在CMU-MOSI、CMU-MOSEI、MUMETA数据集上的实验结果表明，所提出的模型优于其他基线模型。这种针对隐式情感分析的多模态处理策略，充分利用语音和视觉外部知识，更全面、准确地捕捉隐式情感表达，有效提升了隐式情感分析的准确率。

关键词: 隐式情感分析, 深度神经网络, 多模态, 注意力机制, 特征融合

Abstract: The lack of explicit sentiment words in implicit sentiment expressions poses certain challenges to implicit sentiment analysis. In order to solve this problem, the paper can resort to external information, which is one of the methods for addressing implicit sentiment analysis. Different from the existing research that draws on single textual information, an implicit sentiment analysis method that incorporates multimodal information (including speech and video) is proposed. The method aids in understanding implicit sentiment by extracting acoustic features such as tone and intensity from speech and capturing visual features such as facial expressions from video. BiLSTM network is utilized to mine the contextual information within each unimodal state. The text-related speech and visual features are captured separately by combining the multi-head mutual attention mechanism, and are iteratively optimized to reduce the low-order redundant information of non-textual modalities. In addition, the comprehensive performance of implicit sentiment analysis is enhanced by designing a text-centered cross-attention fusion module to strengthen the implicit text feature representation and deal with inter-modal heterogeneity. Experimental results on CMU-MOSI, CMU-MOSEI, and MUMETA datasets show that the proposed model outperforms other baseline models. This multimodal processing strategy for implicit sentiment analysis makes full use of speech and visual external knowledge to capture implicit sentiment expressions more comprehensively and accurately, effectively improving the accuracy of implicit sentiment analysis.

Key words: implicit sentiment analysis, deep neural networks, multimodal, attention mechanism, feature fusion

张换香, 李梦云, 张景. 基于多模态信息融合的中文隐式情感分析[J]. 计算机工程与应用, 2025, 61(2): 179-190.

ZHANG Huanxiang, LI Mengyun, ZHANG Jing. Implicit Sentiment Analysis for Chinese Texts Based on Multimodal Information Fusion[J]. Computer Engineering and Applications, 2025, 61(2): 179-190.

参考文献

[1] LIU B. Sentiment analysis and opinion mining[M]. Cham: Springer, 2022.
[2] LIAO J, WANG M, CHEN X, et al. Dynamic commonsense knowledge fused method for Chinese implicit sentiment analysis[J]. Information Processing & Management, 2022, 59(3): 102934.
[3] ZHANG D, ZHANG M, GUO T, et al. In your face: sentiment analysis of metaphor with facial expressive features[C]//Proceedings of the 2021 International Joint Conference on Neural Networks, 2021: 1-8.
[4] 冯亚琴, 沈凌洁, 胡婷婷, 等. 利用语音与文本特征融合改善语音情感识别[J].数据采集与处理, 2019, 34(4): 625-631.
FENG Y Q, SHEN L J, HU T T, et al. Using speech and text features fusion to improve speech emotion recognition[J].Journal of Data Acquisition& Processing, 2019, 34(4): 625-631.
[5] 范习健, 杨绪兵, 张礼, 等. 一种融合视觉和听觉信息的双模态情感识别算法[J]. 南京大学学报 (自然科学版), 2021, 57(2): 309-317.
FAN X J, YANG X B, ZHANG L, et al. Emotion recognition based on visual and auditory information[J]. Journal of Nanjing University (Natural Sciences), 2021, 57(2): 309-317.
[6] LU Y, SAKAMOTO K, SHIBUKI H, et al. Construction of a multilingual annotated corpus for deeper sentiment understanding in social media[J]. Information and Media Technologies, 2017, 12: 111-171.
[7] TURNEY P, NEUMAN Y, ASSAF D, et al. Literal and metaphorical sense identification through concrete and abstract context[C]//Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, 2011: 680-690.
[8] LIAO J, WANG S, LI D. Identification of fact-implied implicit sentiment based on multi-level semantic fused representation[J]. Knowledge-Based Systems, 2019, 165: 197-207.
[9] GANDY L, ALLAN N, ATALLAH M, et al. Automatic identification of conceptual metaphors with limited knowledge[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2013: 328-334.
[10] REN J, XIA F, CHEN X, et al. Matching algorithms: fundamentals, applications and challenges[J].IEEE Transactions on Emerging Topics in Computational Intelligence, 2021, 5(3): 332-350.
[11] NISSIM M, MARKERT K. Syntactic features and word similarity for supervised metonymy resolution[C]//Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, 2003: 56-63.
[12] LI Z Y, ZOU Y C, ZHANG C, et al. Learning implicit sentiment in aspect-based sentiment analysis with supervised contrastive pre-training[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2021: 246-256.
[13] XIANG C, REN Y, JI D. Identifying implicit polarity of events by using an attention-based neural network model[J]. IEEE Access, 2019, 7: 133170-133177.
[14] KIM Y. Convolutional neural networks for sentence classification[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2014: 1746-1751.
[15] GRAVES A, JAITLY N, MOHAMED A. Hybrid speech recognition with deep bidirectional LSTM[C]//Proceedings of the 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, 2013: 273-278.
[16] FU L, LIU S. A syntax-based BSGCN model for Chinese implicit sentiment analysis with multi-classification[C]//Proceedings of the 2022 IEEE 16th International Conference on Application of Information and Communication Technologies, 2022: 1-7.
[17]    LI X, LI Q. Using BERT and word definitions for implicit sentiment analysis[C]//Proceedings of the 2022 IEEE 24th International Conference on High Performance Computing & Communications, 2022: 1063-1068.
[18]    ZHOU X, WAN X, XIAO J. Attention-based LSTM network for cross-lingual sentiment classification[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016: 247-256.
[19]    张军, 张丽, 沈凡凡, 等. RoBERTa融合BiLSTM及注意力机制的隐式情感分析[J]. 计算机工程与应用, 2022, 58(23): 142-150.
ZHANG J, ZHANG L, SHEN F F, et al. Implicit sentiment analysis based on RoBERTa fused with BiLSTM and attention mechanism[J].Computer Engineering and Applications, 2022, 58(23): 142-150.
[20] 李嘉伟, 张顺香, 李书羽, 等.基于文本图表征的中文隐式情感分析模型[J/OL].数据分析与知识发现: 1-16[2024-05-09].http://kns.cnki.net/kcms/detail/10.1478.G2.20231225.
0956.004.html.
LI J W, ZHANG S X, LI S Y, et al. Chinese implicit sentiment analysis model based on text graph representation[J/OL]. Data Analysis and Knowledge Discovery, 1-16[2024-05-09].http://kns.cnki.net/kcms/detail/10.1478.G2.20231225.
0956.004.html.
[21] 马圆圆, 禹龙, 田生伟, 等.融合RoBERTa和注意力机制的隐喻方面级情感分析[J].小型微型计算机系统, 2023, 44(10): 2236-2241.
MA Y Y, YU L, TIAN S W, et al. Metaphorical aspect sentiment analysis based on RoBERTa and attention mechanism[J]. Journal of Chinese Computer Systems, 2023, 44(10): 2236-2241.
[22] FORCEVILLE C, URIOS-APARISI E.Multimodal metaphor[M]. Berlin:De Gruyter Mouton, 2009.
[23]    SHUTOVA E, KIELA D, MAILLARD J. Black holes and white rabbits: metaphor identification with visual features[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016: 160-170.
[24]    CHEN M, UBUL K, XU X, et al. Connecting text classification with image classification: a new preprocessing method for implicit sentiment text classification[J]. Sensors, 2022, 22(5): 1899.
[25]    STEEN G, DORST A G, HERRMANN J B, et al. A method for linguistic metaphor identification: from MIP to MIPVU[M]. Amsterdam: Benjamins, 2010.
[26]    NOZZA D, FERSINI E, MESSINA E. A multi-view sentiment corpus[C]//Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, 2017: 273-280.
[27]    ZHANG D, LIN H, YANG L, et al. Construction of a Chinese corpus for the analysis of the emotionality of metaphorical expressions[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018: 144-150.
[28]    STEEN G J. Visual metaphor: structure and process[M]. Amsterdam: Benjamins, 2018.
[29]    ZHANG D, ZHANG M, ZHANG H, et al. Multimet: a multimodal dataset for metaphor understanding[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 3214-3225.
[30]    DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the Conference of the North-American-Chapter of the Association-for-Computational-Linguistics-Human Language Technologies, 2019: 4171-4186.
[31] MCFEE B, RAFFEL C, LIANG D, et al. Librosa: audio and music signal analysis in python[C]//Proceedings of the 14th Python in Science Conference, 2015: 18-25.
[32]    ZHANG K, ZHANG Z, LI Z, et al. Joint face detection and alignment using multitask cascaded convolutional networks[J].IEEE Signal Processing Letters, 2016, 23(10):1499-1503.
[33]    BALTRUSAITIS T, ZADEH A, LIM Y C, et al. Openface 2.0: facial behavior analysis toolkit[C]//Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition , 2018: 59-66.
[34]    ZADEH A, ZELLERS R, PINCUS E, et al. Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages[J]. IEEE Intelligent Systems, 2016, 31(6): 82-88.
[35]    ZADEH A A B, LIANG P P, PORIAO S, et al. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018: 2236-2246.
[36]    PORIA S, CAMBRIA E, HAZARIKA D, et al. Context-dependent sentiment analysis in user-generated videos[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017: 873-883.
[37]    CHEN M, LI X. SWAFN: sentimental words aware fusion network for multimodal sentiment analysis[C]//Proceedings of the 28th International Conference on Computational Linguistics, 2020: 1067-1077.
[38]    WANG L, PENG J, ZHENG C, et al. A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning[J].Information Processing & Management, 2024, 61(3): 103675.
[39]    孙杰, 车文刚, 高盛祥. 面向多模态情感分析的多通道时序卷积融合[J].计算机科学与探索, 2024, 18(11): 3041-3050.
SUN J, CHE W G, GAO S X. Multi-channel temporal convolution fusion for multimodal sentiment analysis[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(11): 3041-3050.
[40]    HU G, LIN T E, ZHAO Y, et al. UniMSE: towards unified multimodal sentiment analysis and emotion recognition[J]. arXiv:2211.11256, 2022.
[41]    CAI S, YUAN J, LI L. A mutual implicit sentiment analysis model with bundle-aware contrastive learning[C]//Proceedings of the 2023 IEEE International Conference on Acoustics, Speech and Signal Processing, 2023: 1-5.