Microblog Sentiment Analysis Based on BERT and Hierarchical Attention

doi:10.3778/j.issn.1002-8331.2107-0448

Abstract

Abstract: Microblog sentiment analysis aims to dig out netizens’ views and opinions on specific events, which is an important content of online public opinion monitoring. The current microblog sentiment analysis models generally static word embedding methods such as Word2Vector or GloVe, which cannot solve the polysemy problems well. In addition, the Attention mechanism at word level fails to fully consider the importance of text hierarchy and fails to capture the relationship between sentences. Aiming at these problems, it proposes a model named BERT-HAN（bidirectional encoder representations from transformers-hierarchical Attention networks）, which is based on BERT and hierarchical Attention mechanism. Firstly, the dynamic character vector containing the context semantics is generated by BERT. Then, two levels of BiGRU are used to obtain sentence representation and document representation respectively. Local Attention mechanism is introduced in the sentence representation layer to capture the important characters in each sentence, and the global Attention mechanism is introduced in the document representation layer to distinguish the importance of different sentences. Finally, the emotions are classified by Softmax. The experimental results show that the BERT-HAN model proposed in this paper can effectively improve the Macro F1 and Micro F1 values of microblog sentiment analysis, which has great practical value.

Key words: deep learning, sentiment analysis, feature extraction, word embedding, attention mechanism

摘要： 微博情感分析旨在挖掘网民对特定事件的观点和看法，是网络舆情监测的重要内容。目前的微博情感分析模型一般使用Word2Vector或GloVe等静态词向量方法，不能很好地解决一词多义问题；另外，使用的单一词语层Attention机制未能充分考虑文本层次结构的重要性，对句间关系捕获不足。针对这些问题，提出一种基于BERT和层次化Attention的模型BERT-HAN（bidirectional encoder representations from transformers-hierarchical Attention networks）。通过BERT生成蕴含上下文语意的动态字向量；通过两层BiGRU分别得到句子表示和篇章表示，在句子表示层引入局部Attention机制捕获每句话中重要的字，在篇章表示层引入全局Attention机制以区分不同句子的重要性；通过Softmax对情感进行分类。实验结果表明，提出的BERT-HAN模型能有效提升微博情感分析的Macro F1和Micro F1值，具有较大的实用价值。

关键词: 深度学习, 情感分析, 特征提取, 词向量, 注意力机制

ZHAO Hong, FU Zhaoyang, ZHAO Fan. Microblog Sentiment Analysis Based on BERT and Hierarchical Attention[J]. Computer Engineering and Applications, 2022, 58(5): 156-162.

赵宏, 傅兆阳, 赵凡. 基于BERT和层次化Attention的微博情感分析研究[J]. 计算机工程与应用, 2022, 58(5): 156-162.

References

[1] YUAN J，SHI J，CHE J，et al.Modeling and simulation analysis of public opinion polarization in a dynamic network environment[J].Concurrency and Computation：Practice and Experience，2020，32（19）：e5771.
[2] 陈兴蜀，常天祐，王海舟，等.基于微博数据的“新冠肺炎疫情”舆情演化时空分析[J].四川大学学报（自然科学版），2020，57（2）：409-416.
CHEN Xingshu，CHANG Tianyou，WANG Haizhou，et al.Spatial and temporal analysis on public opinion evolution of epidemic situation about novel coronavirus pneumonia based on micro-blog data[J].Journal of Sichuan University（Natural Science Edition），2020，57（2）：409-416.
[3] PANG B，LEE L.A sentimental education：sentiment analysis using subjectivity summarization based on minimum cuts[C]//Proc of the ACL 2004.Morristown：ACL，2004：271-278.
[4] ALHARBI N M，ALGHAMDI N S，ALKHAMMASH E H，et al.Evaluation of sentiment analysis via word embedding and RNN variants for amazon online reviews[J].Mathematical Problems in Engineering，2021：532-543.
[5] LEE G T，KIM C O，SONG M.Semisupervised sentiment analysis method for online text reviews[J].Journal of Information Science，2021，47（3）：387-403.
[6] 余同瑞，金冉，韩晓臻，等.自然语言处理预训练模型的研究综述[J].计算机工程与应用，2020，56（23）：12-22.
YU Tongrui，JIN Ran，HAN Xiaozhen，et al.Review of pre-training models for natural language processing[J].Computer Engineering and Applications，2020，56（23）：12-22.
[7] KIM Y.Convolutional neural networks for sentence classification[C]//Proceedings of 2014 Conference on Empirical Methods in Natural Language Processing，2014：1746-1751.
[8] 刘龙飞，杨亮，张绍武，等.基于卷积神经网络的微博情感倾向性分析[J].中文信息学报，2015，29（6）：159-165.
LIU Longfei，YANG Liang，ZHANG Shaowu，et al.Convolutional neural networks for Chinese micro-blog sentiment analysis[J].Journal of Chinese Information Processing，2015，29（6）：159-165.
[9] MIKOLOV T，SUTSKEVER I，CHEN K，et al.Distributed representations of words and phrases and their compositionality[C]//Advances in Neural Information Processing Systems，2013：3111-3119.
[10] SAK H，SENIOR A W，BEAUFAYS F.Long short-term memory recurrent neural network architectures for large scale acoustic modeling[C]//Proceedings of the 15th Annual Conference of the International Speech Communication Association.Minneapolis：ISCA，2014：338-342.
[11] 方炯焜，陈平华，廖文雄.结合GloVe和GRU的文本分类模型[J].计算机工程与应用，2020，56（20）：98-103.
FANG Jiongkun，CHEN Pinghua，LIAO Wenxiong.Text classification model based on GloVe and GRU[J].Computer Engineering and Applications，2020，56（20）：98-103.
[12] 田竹.基于深度特征提取的文本情感极性分类研究[D].济南：山东大学，2017.
TIAN Zhu.Research on sentiment analysis based on deep feature representation[D].Jinan：Shandong University，2017.
[13] TANG D，QIN B，LIU T.Document modeling with gated recurrent neural network for sentiment classification[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing，2015：1422-1432.
[14] BAHDANAU D，CHO K，BENGIO Y.Neural machine translation by jointly learning to align and translate[C]//Proc of the 3rd International Conference on Learning Rrepresentations，2015：1-15.
[15] LUONG M T，PHAM H，MANNING C D.Effective approaches to attention-based neural machine translation[C]//Proc of Conference on Empirical Methods in Natural Language Processing，2015：1412-1421.
[16] YANG Z，YANG D，DYER C，et al.Hierarchical attention networks for document classification[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2016：1480-1489.
[17] DEVLIN J，CHANG M W，LEE K，et al.BERt：pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies.Stroudsburg，PA：Association for Computational Linguistics，2019：4171-4186.
[18] SUTSKEVER I，VINYALS O，LE Q V.Sequence to sequence learning with neural networks[C]//Advances in Neural Information Processing Systems，2014：3104-3112.
[19] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems，2017：5998-6008.
[20] DEY R，SALEM F M.Gate-variants of gated recurrent unit（GRU） neural networks[C]//2017 IEEE 60th International Midwest Symposium on Circuits and Systems （MWSCAS），2017：1597-1600.