Multimodal Aspect-Level Sentiment Analysis Based on Multi-Granularity View Dynamic Fusion

doi:10.3778/j.issn.1002-8331.2309-0082

Abstract

Abstract: To solve the problems of inadequate feature extraction, low data information utilization, and ignoring the complex interaction in multimodal data for aspect sentiment analysis, a multi-granularity view dynamic fusion model (MVDFM) is proposed. Firstly, text and image data are encoded from two perspectives of coarse-grained and fine-grained, so as to fully capture data features and enhance the information representation ability of the model. Secondly, multi-granularity view features of text and image are extracted, and dynamic gated self-attention mechanism is designed to reduce the noise of fine-grained text and image views to further ensure the quality of feature extraction. Finally, in order to excavate the complementarity and consistency between multiple views at different granularity, a triple-view factorized bilinear pooling mechanism is proposed to carry out two-stage dynamic fusion of multi-granularity view features to obtain the final target aspect sentiment polarity. The experimental results show that the accuracy and F1 values of the model on the public data sets Twitter-2015 and Twitter-2017 reach 78.69% and 74.48%, and 72.77% and 71.61%, respectively. Compared with the best baseline model, the improvement is 0.55, 0.88 percentage points and 1.67, 2.45 percentage points, respectively. This method can make full use of the information contained in the multimodal data, and effectively mine the key parts related to the target aspect words to improve the effect of aspect-level emotion prediction.

Key words: multimodal aspect-level sentiment analysis, dynamic gated attention, multi-granularity views, dynamic fusion

摘要： 为了解决以往多模态方面级情感分析研究中存在的特征提取不充分、数据噪声未被有效处理以及多模态数据中的复杂交互被忽视等问题，提出了一种多粒度视图动态融合模型（multi-granularity view dynamic fusion model，MVDFM）。从粗粒度和细粒度两个视角，对文本和图像数据进行向量化编码，以便充分捕捉数据特征，增强模型信息表达能力；提取文本、图像的多粒度视图特征，并设计动态门控自注意力机制，对细粒度级的文本、图像视图进行降噪，进一步保证特征提取质量；为了挖掘不同粒度上多视图之间的互补性和一致性，提出一种三视图分解高阶池化机制，对多粒度视图特征进行两阶段动态融合，得到最终的目标方面词情感极性。实验结果表明，该模型在公共数据集Twitter-2015和Twitter-2017上的准确率和F1值分别达到了78.69%、74.48%以及72.77%、71.61%，相较于最优基线模型分别提升了0.55、0.88个百分点，以及1.67、2.45个百分点。说明该方法能够充分利用多模态数据中包含的深层语义信息，并有效挖掘与目标方面词相关的重要信息，从而提高方面级情感预测效果。

关键词: 多模态方面级情感分析, 动态门控注意力, 多粒度视图, 动态融合

YANG Ying, QIAN Xinyu, WANG Hening. Multimodal Aspect-Level Sentiment Analysis Based on Multi-Granularity View Dynamic Fusion[J]. Computer Engineering and Applications, 2024, 60(22): 172-183.

杨颖, 钱馨雨, 王合宁. 结合多粒度视图动态融合的多模态方面级情感分析[J]. 计算机工程与应用, 2024, 60(22): 172-183.

References

[1] JIANG L, YU M, MING Z, et al. Target-dependent Twitter sentiment classification[C]//Proceedings of the Meeting of the Association for Computational Linguistics, 2011: 151-160.
[2] ZHOU J, ZHAO J, HUANG J X, et al. MASAD: a large-scale dataset for multimodal aspect-based sentiment analysis [J]. Neurocomputing, 2021, 455: 47-58.
[3] XU N, MAO W J, CHEN G D. Multi-interactive memory network for aspect based multimodal sentiment analysis[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence, 2019: 371-378.
[4] YU J F, JIANG J. Adapting BERT for target-oriented multimodal sentiment classification[C]//Proceedings of the 28th International Joint Conference on Artificial Intelligence, 2019: 5408-5414.
[5] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st Conference on Neural Information Processing Systems, 2017: 6000-6010.
[6] YU J F, CHEN K, XIA R. Hierarchical interactive multimodal transformer for aspect-based multimodal sentiment analysis[J]. IEEE Transactions on Affective Computing, 2023, 14(3): 1966-1978.
[7] YU Z, YU J, XIANG C C, et al. Beyond bilinear: generalized multimodal factorized high-order pooling for visual question answering[J]. IEEE Transactions on Neural Networks and Learning Systems, 2018, 29(12): 5947-5959.
[8] PANG B, LEE L, VAITHYANATHAN S. Thumbs up? Sentiment classification using machine learning techniques[C]//Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing, 2002: 79-86.
[9] READ J. Using emoticons to reduce dependency in machine learning techniques for sentiment classification[C]//Proceedings of the Student Research Workshop at the Meeting of the Association for Computational Linguistics, Ann Arbor, 2005: 43-48.
[10] WAGNER J, ARORA P, CORTES S, et al. DCU: aspect-based polarity classification for SemEval task 4[C]//Proceedings of the 2014 International Conference on Computational Linguistics, 2014.
[11] XU H, ZHANG F, WANG W. Implicit feature identification in Chinese reviews using explicit topic mining model[J]. Knowledge-Based Systems, 2015, 76: 166-175.
[12] HOCHREITER S, SCHMIDHUBER J. Long short-term memory[J]. Neural Computation, 1997, 9(8): 1735-1780.
[13] TANG D Y, QIN B, FENG X C, et al. Effective LSTMs for target-dependent sentiment classification[C]//Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers, Osaka, 2016: 3298-3307.
[14] WANG Y, HUANG M, ZHAO L. Attention-based LSTM for aspect-level sentiment classification[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016: 606-615.
[15] MA D H, LI S J, ZHANG X D, et al. Interactive attention networks for aspect-level sentiment classification[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence, 2017: 4068-4074.
[16] ZHANG C, LI Q C, SONG D W. Aspect-based sentiment classification with aspect-specific graph convolutional networks[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong Kong, China, 2019: 4568-4578.
[17] KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[C]//Proceedings of the 5th International Conference on Learning Representations, 2017.
[18] 李雪莲, 王碧, 李立鑫, 等. 融合抽象语义表示和依存语法的方面级情感分析[J]. 数据分析与知识发现, 2024, 8(1): 55-68.
LI X L, WANG B, LI L X, et al. Integrating abstract meaning representation and dependency grammar for aspect-based sentiment analysis[J]. Data Analysis and Knowledge Discovery, 2024, 8(1): 55-68.
[19] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistic, Minneapolis, 2019: 4171-4186.
[20] 刘路路, 杨燕, 王杰. ABAFN: 面向多模态的方面级情感分析模型[J]. 计算机工程与应用, 2022, 58(10): 193-199.
LIU L L, YANG Y, WANG J. ABAFN: aspect-based sentiment analysis model for multimodal[J]. Computer Engineering and Applications, 2022, 58(10): 193-199.
[21] YU J F, JIANG J, XIA R. Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification[J]. IEEE/ACM Transactions on Audio, Speech and Language Processing, 2020, 28: 429-439.
[22] YANG L, NA J C, YU J F. Cross-modal multitask transformer for end-to-end multimodal aspect-based sentiment analysis[J]. Information Processing and Management, 2022, 59(5): 103038.
[23] JU X C, ZHANG D, XIAO R, et al. Joint multi-modal aspect-sentiment analysis with auxiliary cross-modal relation detection[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021: 4395-4405.
[24] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
[25] SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the inception architecture for computer vision[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 2818-2826.
[26] ADAM P, SAM G, SOUMITH C, et al. Automatic differentiation in PyTorch[C]//Proceedings of the 31st Conference on Neural Information Processing Systems, 2017.
[27] KINGMA D P, BA J. Adam: a method for stochastic optimization[C]//Proceedings of the 3rd International Conference on Learning Representations, 2015.
[28] CHEN P, SUN Z Q, BING L D, et al. Recurrent attention network on memory for aspect sentiment analysis[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, 2017: 452-461.
[29] YU J F, WANG J M, XIA R, et al. Targeted multimodal sentiment classification based on coarse-to-fine grained image-target matching[C]//Proceedings of the 31st International Joint Conference on Artificial Intelligence, Vienna, 2022: 4482-4488.