Target-Oriented Interaction Graph Neural Networks for Multimodal Aspect-Level Sentiment Analysis

doi:10.3778/j.issn.1002-8331.2308-0083

Abstract

Abstract: For multimodal aspect level sentiment analysis task, in addition to extracting text and image representations, it is also necessary to combine them with aspect semantic information. However, the interaction between aspects and text and image information is not sufficiently processed by the previous relevant methods. Even if the global correlation between modes is established by using the attention mechanism, it is difficult to express their interaction in a fine granularity. In order to fully carry out fine-grained information interaction between multiple modes, a target-oriented interactive graph neural network is proposed, which is modeled around the relationship between text, image and aspect. Firstly, cross-attention is used to obtain the global representation of aspect-oriented text and image. Then a multimodal interaction graph is established to connect the local and global representation nodes of different modes. Finally, the graph attention network is used to fully integrate the features in the two granularities. Experiments on two benchmark datasets show that this model has better effect on emotion classification than the model using only attention mechanism.

Key words: multimodal aspect-level sentiment analysis, attention mechanism, cross-modal attention, target-oriented interaction, graph attention networks

摘要： 对于多模态方面级情感分析任务，除了需要提取出文本和图像的表示，还需要将它们与方面语义信息相结合处理。然而，以往的相关方法对方面与文本和图像信息之间的交互处理不够充分，即使使用注意力机制建立起模态全局之间的关联，也难以在细粒度表达出它们的交互。为了充分进行多模态之间细粒度上的信息交互，提出一种面向目标交互图神经网络，围绕文本、图像和方面三者的关系建模，采用交叉注意力获取面向方面目标的文本和图像全局表示；建立多模态交互图，以连接不同模态的局部及全局表示节点；使用图注意力网络在粗细两个粒度上充分融合特征。在两个基准数据集上进行实验，结果表明该模型相比于仅使用注意力机制的模型，具有更佳的情感分类效果。

关键词: 多模态方面级情感分析, 注意力机制, 交叉注意力, 面向目标交互, 图注意力网络

ZHANG Lixia, WANG Kaixuan, PANG Zichao, LIANG Yun. Target-Oriented Interaction Graph Neural Networks for Multimodal Aspect-Level Sentiment Analysis[J]. Computer Engineering and Applications, 2024, 60(23): 136-145.

张丽霞, 汪凯旋, 庞梓超, 梁云. 面向目标交互图神经网络的多模态方面级情感分析[J]. 计算机工程与应用, 2024, 60(23): 136-145.

References

[1] ZHOU J, HUANG J X, CHEN Q, et al. Deep learning for aspect-level sentiment classification: survey, vision, and challenges[J]. IEEE Access, 2019, 7: 78454-78483.
[2] 李阳, 王石, 朱俊武, 等. 方面级情感分析综述[J]. 计算机科学, 2023, 50(1): 24-30.
LI Y, WANG S, ZHU J W, et al. ?Summarization of aspect-level sentiment analysis[J]. Computer Science, 2023, 50(1): 24-30.
[3] MADHOUSHI Z, HAMDAN A R, ZAINUDIN S. Aspect-based sentiment analysis methods in recent years[J]. Asia-Pacific Journal of Information Technology and Multimedia, 2019, 7(2): 79-96.
[4] HONG W, LI M. A review of research on text emotional analysis methods[J]. Computer Engineering and Science, 2019, 41(4): 750-757.
[5] LIU B. Sentiment analysis and opinion mining[M]//Synthesis Lectures on Human Language Technologies. Cham: Springer, 2012.
[6] WILSON T, WIEBE J, HOFFMANN P. Recognizing contextual polarity in phrase-level sentiment analysis[C]//Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, 2005: 347-354.
[7] ESULI A, SEBASTIANI F. SentiWordNet: a publicly available lexical resource for opinion mining[C]//Proceedings of the 15th International Conference on Language Resources and Evaluation, 2006.
[8] BAHDANAU D, CHO K, BENGIO Y. Neural machine translation by jointly learning to align and translate[J]. arXiv:1409.0473, 2014.
[9] DONG L, WEI F, TAN C, et al. Adaptive recursive neural network for target-dependent twitter sentiment classification[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 2014: 49-54.
[10] MA Y, CHENG C L. Joint left and right attention mechanism for aspect-level text sentiment analysis[J]. Application Research of Computers, 2021, 38(6): 1753-1758.
[11] KIRITCHENKO S, ZHU X, CHERRY C, et al. NRC-Canada-2014: detecting aspects and sentiment in customer reviews[C]//Proceedings of the 8th International Workshop on Semantic Evaluation, 2014: 437-442.
[12] XUE W, LI T. Aspect based sentiment analysis with gated convolutional networks[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018: 2514-2523.
[13] TANG D, QIN B, FENG X, et al. Effective LSTMs for target-dependent sentiment classification[C]//Proceedings of the COLING 2016, 26th International Conference on Computational Linguistics: Technical Papers, 2016: 3298-3307.
[14] WANG Y, HUANG M, ZHU X, et al. Attention-based LSTM for aspect-level sentiment classification[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016: 606-615.
[15] WU Z, PAN S, CHEN F, et al. A comprehensive survey on graph neural networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(1): 4-24
[16] ZHANG C, LI Q, SONG D. Aspect-based sentiment classification with aspect-specific graph convolutional networks[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019: 4568-4578.
[17] ZHOU J, HUANG J X, HU Q V, et al. SK-GCN: modeling syntax and knowledge via graph convolutional network for aspect-level sentiment classification[J]. Knowledge-Based Systems, 2020, 205: 106292.
[18] LIANG B, SU H, GUI L, et al. Aspect-based sentiment analysis via affective knowledge enhanced graph convolutional networks[J]. Knowledge-Based Systems, 2022, 235: 107643.
[19] 刘路路, 杨燕, 王杰. ABAFN: 面向多模态的方面级情感分析模型[J]. 计算机工程与应用, 2022, 58(10): 193-199.
LIU L L, YANG Y, WANG J. ABAFN：aspect-based sentiment analysis model for multimodal[J]. Computer Engineering and Applications, 2022, 58(10): 193-199.
[20] XU N, MAO W, CHEN G. Multi-interactive memory network for aspect based multimodal sentiment analysis[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2019: 371-378.
[21] YU J, JIANG J, XIA R. Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2019, 28: 429-439.
[22] YU J, JIANG J. Adapting BERT for target-oriented multimodal sentiment classification[C]//Proceedings of the 28th International Joint Conference on Artificial Intelligence, 2019: 5408-5414.
[23] WANG J, GU D, YANG C, et al. Targeted aspect based multimodal sentiment analysis: an attention capsule extraction and multi-head fusion network[J]. arXiv:2103.07659, 2021.
[24] JU X, ZHANG D, XIAO R, et al. Joint multi-modal aspect sentiment analysis with auxiliary cross-modal relation detection[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021: 4395- 4405.
[25] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[J]. arXiv:1810.04805, 2018.
[26] 袁勋, 刘蓉, 刘明. 融合多层注意力的方面级情感分析模型[J]. 计算机工程与应用, 2021, 57(22): 147-152.
YUAN X, LIU R, LIU M. Aspect-level sentiment analysis model incorporating multi-layer attention[J]. Computer Engineering and Applications, 2021, 57(22): 147-152.
[27] KINGMA D P, BA J. ADAM: a method for stochastic optimization[J]. arXiv:1412.6980, 2014.
[28] CHEN P, SUN Z, BING L, et al. Recurrent attention network on memory for aspect sentiment analysis[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017: 452-461.
[29] MA D, LI S, ZHANG X, et al. Interactive attention networks for aspect-level sentiment classification[J]. arXiv:1709.00893, 2017.
[30] LI X, BING L, LAM W, et al. Transformation networks for target-oriented sentiment classification[J]. arXiv:1805. 01086, 2018.
[31] FAN F F, FENG Y S, ZHAO D Y. Multi-grained attention network for aspect-level sentiment classification[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018: 3433-3442.