Aspect-Level Sentiment Analysis Model Incorporating Multi-layer Attention

doi:10.3778/j.issn.1002-8331.2104-0019

Abstract

Abstract:

Aspect sentiment analysis aims to analyze the sentiment polarity of a specific aspect in a given text. In order to solve the problem of insufficient introduction of aspect emotional attention in current research methods, this paper proposes an aspect level emotion classification model based on the fusion of BERT and Multi-Layer Attention（BMLA）. Firstly, the model extracts the multi-layer aspect emotional attention information from the inner part of BERT, and designs the multi-layer aspect attention by fusing the encoded aspect information with the representation vector of the hidden layer of BERT, then cascades the multi-layer aspect attention with the encoded output text, finally enhances the long dependency relationship between sentences and aspect words. Experiments on the SemEval2014 Task4 and the AI Challenger 2018 datasets show that the proposed model is effective to enhance the weight of the target aspect and interact in context for aspect sentiment classification.

Key words: natural language processing, aspect sentiment analysis, BERT, multi-layer attention, dependency

摘要：

方面情感分析旨在分析给定文本中特定方面的情感极性。针对目前的研究方法存在对方面情感注意力引入不足问题，提出了一种融合BERT和多层注意力的方面级情感分类模型（BERT and Multi-Layer Attention，BMLA）。模型首先提取BERT内部多层方面情感注意力信息，将编码后的方面信息与BERT隐藏层表征向量融合设计了多层方面注意力，然后将多层方面注意力与编码输出文本进行级联，进而增强了句子与方面词之间的长依赖关系。在SemEval2014 Task4和AI Challenger 2018数据集上的实验表明，强化目标方面权重并在上下文进行交互对方面情感分类是有效的。

关键词: 自然语言处理, 方面情感分析, BERT, 多层注意力, 依赖关系

YUAN Xun, LIU Rong, LIU Ming. Aspect-Level Sentiment Analysis Model Incorporating Multi-layer Attention[J]. Computer Engineering and Applications, 2021, 57(22): 147-152.

袁勋，刘蓉，刘明. 融合多层注意力的方面级情感分析模型[J]. 计算机工程与应用, 2021, 57(22): 147-152.

References

[1] 徐戈，王厚峰.自然语言处理中主题模型的发展[J].计算机学报，2011，34（8）：1423-1436.
XU G，WANG H F.Development of topic model in natural language processing[J].Chinese Journal of Computers，2011，34（8）：1423-1436.
[2] LIU B，ZHANG L.A survey of opinion mining and sentiment analysis[M]//Mining text data.Boston：Springer，2012：415-463.
[3] 杨立公，朱俭，汤世平.文本情感分析综述[J].计算机应用，2013，33（6）：1574-1607.
YANG L G，ZHU J，TANG S P.A survey of text sentiment analysis[J].Journal of Computer Applications，2013，33（6）：1574-1607.
[4] 赵妍妍，秦兵，刘挺.文本情感分析[J].软件学报，2010，21（8）：1834-1848.
ZHAO Y Y，QIN B，LIU T.Text sentiment analysis[J].Journal of Software，2010，21（8）：1834-1848.
[5] 杨丽，吴雨茜，王俊丽，等.循环神经网络研究综述[J].计算机应用，2018，38（S2）：1-6.
YANG L，WU Y X，WANG J L，et al.Summary of research on recurrent neural network[J].Journal of Computer Applications，2018，38（S2）：1-6.
[6] SAK H，SENIOR A，BEAUFAYS F.Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition[J].arXiv：1402.1128，2014.
[7] KIM Y.Convolutional neural networks for sentence classification[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing，2014：1746-1751.
[8] JOHNSON R，ZHANG T.Deep pyramid convolutional neural networks for text categorization[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics，2017：562-570.
[9] WANG Y，HUANG M，ZHU X，et al.Attention-based LSTM for aspect-level sentiment classification[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing，2016：606-615.
[10] MA D，LI S，ZHANG X，et al.Interactive attention networks for aspect-level sentiment classification[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence，2017：4068-4074.
[11] QIU X，SUN T，XU Y，et al.Pre-trained models for natural language processing：a survey[J].arXiv：2003.08271，2020.
[12] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems，2017：6000-6010.
[13] DEVLIN J，CHANG M W，LEE K，et al.BERT：pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2019：4171-4186.
[14] SUN C，HUANG L，QIU X.Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2019：380-385.
[15] XU H，LIU B，SHU L，et al.BERT post-training for review reading comprehension and aspect-based sentiment analysis[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language，2019：2324-2335.
[16] LIU X，HE P，CHEN W，et al.Multi-task deep neural networks for natural language understanding[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics，2019：4487-4496.
[17] GAO Z，FENG A，SONG X，et al.Target-dependent sentiment classification with BERT[J].IEEE Access，2019，7：154290-154299.
[18] SONG Y，WANG J，LIANG Z，et al.Utilizing BERT intermediate layers for aspect based sentiment analysis and natural language inference[J].arXiv：2002.04815，2020.
[19] HE K，ZHANG X，REN S，et al.Spatial pyramid pooling in deep convolutional networks for visual recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2015，37（9）：1904-1916.
[20] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[21] WAGNER J，ARORA P，VAíLLO S C，et al.DCU：aspect-based polarity classification for SemEval Task4[C]//Proceedings of the 8th International Workshop on Semantic Evaluation，2014：223-229.
[22] DAI Z，DAI W，LIU Z，et al.Multi-task multi-head attention memory network for fine-grained sentiment analysis[C]//CCF International Conference on Natural Language Processing and Chinese Computing.Cham：Springer，2019：609-620.