Implicit Sentiment Analysis Based on RoBERTa Fused with BiLSTM and Attention Mechanism

doi:10.3778/j.issn.1002-8331.2203-0505

Abstract

Abstract: Implicit sentiment analysis is one of the research hotspots in the field of natural language processing, which makes the traditional text sentiment analysis methods not applicable due to their implicit expressions and lack of obvious sentiment words. To address the difficulties of capturing the hidden sentiment of sentence semantics in the task of implicit sentiment analysis, a fusion model named RBLA is proposed, which is based on the pre-trained model RoBERTa fused with bidirectional long short-term memory network and attention mechanism. The proposed model RBLA can capture the semantics and syntactic features of words in the implicit sentiment sentences using the pre-trained model RoBERTa. Moreover, the proposed model also can learn forward and backward semantic information of sentences via using the bidirectional long short-term memory network, which can capture the dependencies between sentences and extract the deep-level features of the text. And RBLA still takes advantage of the attention mechanism to calculate the sentiment weights. In addition, to obtain implicit sentiment classification results, RBLA performs normalization via using the normalized function softmax. The experimental results show that RBLA can achieve better results in terms of accuracy, recall and F1 value compared with several existing typical implicit sentiment classification models.

Key words: natural language processing, implicit sentiment analysis, RoBERTa, attention mechanism, bi-directional long short term memory（BiLSTM）

摘要： 隐式情感分析是自然语言处理的研究热点之一，由于其表达隐晦且缺少显示情感词，使得传统的文本情感分析方法不再适用。针对隐式情感分析中句子语义的隐藏情感捕捉困难问题，提出了基于RoBERTa融合双向长短期记忆网络及注意力机制的RBLA模型。该模型使用RoBERTa预训练模型捕获隐式情感句中字词的语义特征，再使用双向长短期记忆网络学习句子的正反向语义信息，以捕获句子间的依赖关系，实现对文本深层次特征的提取。使用注意力机制进行情感权重计算，通过softmax函数进行归一化处理，得出隐式情感分类结果。实验结果表明，与现有的几种典型隐式情感分类模型相比较，RBLA模型在精确率、召回率和F1值上均取得了较好效果。

关键词: 自然语言处理, 隐式情感分析, RoBERTa, 注意力机制, 双向长短期记忆网络（BiLSTM）

ZHANG Jun, ZHANG Li, SHEN Fanfan, TAN Hai, HE Yanxiang. Implicit Sentiment Analysis Based on RoBERTa Fused with BiLSTM and Attention Mechanism[J]. Computer Engineering and Applications, 2022, 58(23): 142-150.

张军, 张丽, 沈凡凡, 谭海, 何炎祥. RoBERTa融合BiLSTM及注意力机制的隐式情感分析[J]. 计算机工程与应用, 2022, 58(23): 142-150.

References

[1] 陈静.基于语言知识的文本情感分析技术研究[D].合肥：中国科学技术大学，2019.
CHEN J.Research on text sentiment analysis techniques based on language knowledge[D].Hefei：University of Science and Technology of China，2019.
[2] TANG L P，YAO M H，CHEN X G，et al.Research and implementation of multi-mechanism emotion computing model[C]//Proceedings of 2011 AASRI Conference on Artificial Intelligence and Industry Application，Male Maldives，2011：279-283.
[3] 刘全，梁斌，徐进，等.一种用于基于方面情感分析的深度分层网络模型[J].计算机学报.2018，41（12）：2637-2652.
LIU Q，LIANG B，XU J，et al.A deep hierarchical network model for aspect sentiment analysis[J].Chinese Journal of Computers，2018，41（12）：2637-2652.
[4] 王丽亚，刘昌辉，蔡敦波，等.CNN-BiGRU网络中引入注意力机制的中文文本情感分析[J].计算机应用，2019，39（10）：2841-2846.
WANG L Y，LIU C H，CAI D B，et al.Chinese text sentiment analysis with attention mechanism in CNN-BiGRU network[J].Journal of Computer Applications，2019，39（10）：2841-2846.
[5] 翟社平，杨媛媛，邱程，等.基于注意力机制Bi-LSTM算法的双语文本情感分析[J].计算机应用与软件，2019，36（12）：251-255.
ZHAI S P，YANG Y Y，QIU C，et al.Bilingual text emotion analysis based on attention mechanism Bi-LSTM algorithm[J].Computer Applications and Software，2019，36（12）：251-255.
[6] 陶永才，张鑫倩，石磊，等.面向短文本情感分析的多特征融合方法研究[J].小型微型计算机系统，2020，41（6）：1126-1132.
TAO Y C，ZHANG X Q，SHI L，et al.Research on multi-feature fusion method for sentiment analysis of short text[J].Journal of Chinese Mini-Micro Computer Systems，2020，41（6）：1126-1132.
[7] 杨善良，常征.基于图注意力神经网络的中文隐式情感分析[J].计算机工程与应用，2021，57（24）：161-167.
YANG S L，CHANG Z.Chinese implicit sentiment analysis based on graph attention neural network[J].Computer Engineering and Applications，2021，57（24）：161-167.
[8] ZHANG L，LIU B.Identifying noun product features that imply opinions[C]//Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics：Human Language Technologies，Portland Oregon，USA，2011：575-580.
[9] 赵容梅，熊熙，琚生根，等.基于混合神经网络的中文隐式情感分析[J].四川大学学报（自然科学版），2020，57（2）：264-270.
ZHAO R M，XIONG X，JU S G，et al.Chinese implicit emotion analysis based on hybrid neural network[J].Journal of Sichuan University（Natural Science Edition），2020，57（2）：264-270.
[10] SUNDERMEYER M，SCHLUTER R，NEY H.LSTM neural networks for language modeling[C]//Proceedings of Thirteenth Annual Conference of the International Speech Communication Association，Portland Oregon，USA，2012：1-4.
[11] 黄山成，韩东红，乔百友，等.基于ERNIE2.0-BiLSTM-Attention的隐式情感分析方法[J].小型微型计算机系统，2021，42（12）：2485-2489.
HUANG S C，HAN D H，QIAO B Y，et al.Implicit emotion analysis based on ERNIE2.0-BiLSTM-Attention[J].Journal of Chinese Mini-Micro Computers Systems，2021，42（12）：2485-2489.
[12] 陈秋嫦，赵晖，左恩光，等.上下文感知的树递归神经网络下隐式情感分析[J].计算机工程与应用，2022，58（7）：167-175.
CHEN Q C，ZHAO H，ZUO E G，et al.Implicit sentiment analysis based on context aware tree recurrent neural network[J].Computer Engineering and Applications，2022，58（7）：167-175.
[13] LIU Y H，OTT M，GOYAL N，et al.RoBERTa：a robustly optimized BERT pretraining approach[EB/OL].[2020-07-09].https：//arxiv.org/pdf/1907.11692.pdf.
[14] DEVLIN J，CHANG M W，LEE K，et al.BERT：pre-training of deep bidirectional transformers for language understanding[EB/OL].[2020-07-09].https：//arxiv.org/pdf/1810.04805.pdf.
[15] WOLF T，DEBUT L，SANH V，et al.Transformers：state-of-the-art natural language processing[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing：System Demonstrations，2020：38-45.
[16] MIKOLOV T，KARAFIAT M，BURGET L，et al.Recurrent neural network based language model[C]//Proceedings INTERSPEECH 2010：11th Annual Conference of the International Speech Communication Association，Makuhari，Chiba，Japan，2010：1045-1048.
[17] LU X Y，ROY B V.Ensemble sampling[C]//Annual Conference on Neural Information Processing Systems，Long Beach，USA，2017：3260-3268.
[18] MA J，LI Z L.Data augmentation for chinese text classification using back-translation[J].Journal of Physics：Conference Series，2020，1651（1）：012039.
[19] WANG Y Q，HUANG M L，ZHAO L，et al.Attention-based LSTM for aspect-level sentiment classification[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing，Austin，Texas，USA，2016：606-615.
[20] WANG F K，MAMO T，CHENG X B.Bi-directional long short-term memory recurrent neural network with attention for stack voltage degradation from proton exchange membrane fuel cells[J].Journal of Power Sources，2020，461：228170.