ABAFN：Aspect-Based Sentiment Analysis Model for Multimodal

doi:10.3778/j.issn.1002-8331.2108-0056

Abstract

Abstract: With the continuous development of the Internet, user reviews for e-commerce products are increasing. It is of great significance to study the sentiment orientation of these user reviews for guiding the update iteration of products. In the past, the task of aspect-based sentiment analysis usually only involves the text modality. However, user-generated comment data generally include not only plain text, but also a large number of image and text data. For the kind of multimodal data including text and image, this paper proposes a novel aspect-based multimodal sentiment analysis model ABAFN（aspect-based attention and fusion network）. Firstly, the model combines the pre-trained language model BERT and bi-directional long short-term memory network to obtain the textual representation of text and aspect. At the same time, the pre-trained ResNet is used to extract image features to generate visual representation. Then, textual representation and visual representation are weighted based on aspect using attention mechanism. Finally, the weighted representations of the two modalities are fused to perform the sentiment label classification task. Experiments on the Multi-ZOL dataset show that the performance of the ABAFN model exceeds the results in the currently known papers.

Key words: aspect-based multimodal sentiment analysis, BERT, bi-directional long short-term memory network, ResNet, attention mechanism

摘要： 随着互联网的不断发展，面向电商产品的用户评论日益增加。研究这些用户评论的情感导向，对于指导产品的更新迭代具有重要意义。以往的方面级情感分析任务通常只涉及文本模态，然而用户的评论数据一般不仅包括纯文本，还包括大量的图文数据。针对这种包括文本和图片的多模态数据，提出了一种新的方面级多模态情感分析模型ABAFN（aspect-based attention and fusion network）。模型结合预训练语言模型BERT和双向长短时记忆网络来获得文本和方面词的上下文表示，同时利用预训练残差网络ResNet提取图片特征生成视觉表示；利用注意力机制基于方面词对上下文表示和视觉表示进行加权；将两个模态加权后的表示级联融合执行情感标签分类任务。在Multi-ZOL数据集上的实验表明，ABAFN模型的性能超过了目前已知文献的结果。

关键词: 方面级多模态情感分析, BERT, 双向长短时记忆网络, ResNet, 注意力机制

LIU Lulu, YANG Yan, WANG Jie. ABAFN：Aspect-Based Sentiment Analysis Model for Multimodal[J]. Computer Engineering and Applications, 2022, 58(10): 193-199.

刘路路, 杨燕, 王杰. ABAFN：面向多模态的方面级情感分析模型[J]. 计算机工程与应用, 2022, 58(10): 193-199.

References

[1] XU N，MAO W，CHEN G.Multi-interactive memory network for aspect based multimodal sentiment analysis[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence，Hawaii，Jan 27-Feb 1，2019.Menlo Park：AAAI，2019：371-378.
[2] TANG D，QIN B，FENG X，et al.Effective LSTMs for target-dependent sentiment classification[C]//Proceedings of the 26th International Conference on Computational Linguistics：Technical Papers，Osaka，Dec 11-16，2016.New York：ACM，2016：3298-3307.
[3] WANG Y，HUANG M，ZHAO L.Attention-based LSTM for aspect-level sentiment classification[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing，Austin，Nov 1-4，2016.Stroudsburg：ACL，2016：606-615.
[4] CHEN P，SUN Z，BING L，et al.Recurrent attention network on memory for aspect sentiment analysis[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing，Copenhagen，Sep 9-11，2017.Stroudsburg：ACL，2017：452-461.
[5] MA D，LI S，ZHANG X，et al.Interactive attention networks for aspect-level sentiment classification[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence，Melbourne，Aug 19-25，2017.San Francisco：Morgan Kaufmann，2017：4068-4074.
[6] DEVLIN J，CHANG M W，LEE K，et al.BERT：pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，Minneapolis，Jun 2-7，2019：4171-4186.
[7] SONG Y，WANG J，JIANG T，et al.Attentional encoder network for targeted sentiment classification[J].arXiv：1902.
09314，2019.
[8] SHENOY A，SARDANA A.Multilogue-Net：a context-aware RNN for multi-modal emotion detection and sentiment analysis in conversation[C]//2nd Grand-Challenge and Workshop on Multimodal Language（Challenge-HML），Seattle，Jul 5-10，2020：19-28.
[9] KUMAR A，VEPA J.Gated mechanism for attention based multimodal sentiment analysis[C]//2020 IEEE International Conference on Acoustics，Speech and Signal Processing，Barcelona，May 4-9，2020.Piscataway：IEEE，2020：4477-4481.
[10] TRUONG Q T，LAUW H W.VistaNet：visual aspect attention network for multimodal sentiment analysis[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence，Hawaii，Jan 27-Feb 1，2019.Menlo Park：AAAI，2019：305-312.
[11] YADAV A，VISHWAKARMA D K.A deep multi-level attentive network for multimodal sentiment analysis[J].arXiv：2012.08256，2020.
[12] YU J，JIANG J，XIA R.Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification[J].IEEE/ACM Transactions on Audio，Speech，and Language Processing，2019，28：429-439.
[13] YU J，JIANG J.Adapting BERT for target-oriented multimodal sentiment classification[C]//Proceedings of the 28th International Joint Conference on Artificial Intelligence，Macao，Aug 10-16，2019.San Francisco：Morgan Kaufmann，2019：5408-5414.
[14] VASWANI A，SHAZEER N，PARMAE N，et al.Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems，Long Beach，Dec 3-9，2017.Cambridge：MIT Press，2017：6000-6010.
[15] TANG D，QIN B，LIU T.Aspect level sentiment classification with deep memory network[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing，Austin，Nov 1-4，2016.Stroudsburg：ACL，2016：214-224.
[16] XU N，MAO W，CHEN G.A co-memory network for multimodal sentiment analysis[C]//Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval，Ann Arbor，Jul 8-12，2018.New York：ACM，2018：929-932.