Sentiment Representation Learning for Non-IID Document

doi:10.3778/j.issn.1002-8331.2106-0227

Abstract

Abstract: Documents where words/sentences are coupled with each other and the heterogeneous meanings under different contexts are called non-independent and non-identical distributed（non-IID） document. Sentiment of a non-IID document is hard to be captured and represented by the existing methods. The non-IID document representation method for sentiment analysis models the coupling relations and the heterogeneous meaning which are hierarchically exist in a document；and embeds these characteristics in the vector representation. This method can be implemented by a multi-scale and hierarchical deep neural network with an attention mechanism. The network captures word/sentence couplings by a multi-scale convolutional-recurrent structure and reveals the heterogeneous meanings of words/sentences in a document by the attention mechanism. To avoid over-fitting and enhance sentiment-related information in the representation, the network further hierarchically integrates the network-learned implicit features with artificial explicit features, which are designed by sentiment priors. Extensive experiments demonstrate that the non-IID document representation method can enable significantly better sentiment analysis performance.

Key words: non-identical distributed（non-IID） document, textual data representation, sentiment analysis, deep learning

摘要： 非独立同分布文本的情感分析往往极具挑战，因其是一类包含词句间耦合关系和同词（句）多义性特点的复杂文本。现有方法中，几乎没有可以全面捕获非独立同分布文本特性的方法用于情感分析。面向情感分析的非独立同分布文本表示学习方法对文本中层次化存在的耦合关系和多义性问题进行建模，将这些决定着情感极性的非独立同分布特点嵌入到文本的向量表示中。非独立同分布文本表示学习方法通过一种带注意力机制的多尺度层次化深度神经网络实现。该神经网络利用多尺度卷积循环结构捕获文本中的耦合关系，利用注意力机制消除文本中的多义性。同时，该神经网络层次化地融合了由深度学习生成的隐式特征表示和由文本情感先验知识构造的显示特征表示，以防止数据过拟合问题并强化情感表示能力。充分的实验表明，非独立同分布文本表示学习方法可以显著增强文本情感分析的性能。

关键词: 非独立同分布文本, 文本数据表示, 情感分析, 深度学习

LI Qian, GUO Hongyu, ZHENG Yangfei, LIU Yulong, LI Shanhai, WU Yanxiong. Sentiment Representation Learning for Non-IID Document[J]. Computer Engineering and Applications, 2022, 58(24): 180-188.

李倩, 郭红钰, 郑扬飞, 刘玉龙, 李山海, 吴艳雄. 非独立同分布文本情感表示学习方法[J]. 计算机工程与应用, 2022, 58(24): 180-188.

References

[1] MENG Y，HUANG J X，ZHANG Y，et al.On the power of pre-trained text representations：models and applications in text mining[C]//ACM SIGKDD International Conference on Knowledge Discovery and Data Mining，2021：4052-4053.
[2] 杨立月，王移芝.微博情感分析的情感词典构造及分析方法研究[J].计算机技术与发展，2019，29（2）：13-18.
YANG L Y，WANG Y Z.Research on construction and analysis of emotion dictionary in emotion analysis of micro-blog[J].Computer Technology and Development，2019，29（2）：13-18.
[3] DEVLIN J，CHANG M W，LEE K，et al.Bert：pre-training of deep bidirectional transformers for language understanding[J].arXiv：1810.04805，2018.
[4] PHAN M H，PHILIP O O.Modelling context and syntactical features for aspect-based sentiment analysis[C]//Annual Meeting of the Association for Computational Linguistics，2020：3211-3220.
[5] TANG D，QIN B，LIU T.Document modeling with gated recurrent neural network for sentiment classification[C]//Conference on Empirical Methods in Natural Language Processing，2015：1422-1432.
[6] LIU F G，ZHENG L L，ZHENG J Z.HieNN-DWE：a hierarchical neural network with dynamic word embeddings for document level sentiment classification[J].Neurocomputing，2020，403：21-32.
[7] KIM Y.Convolutional neural networks for sentence classi-fication[C]//Conference on Empirical Methods in Natural Language Processing，2014：1746-1751.
[8] BHATIA P，JI Y，EISENSTEIN J.Better document-level sentimentanalysis from rst discourse parsing[C]//Conference on Empirical Methods in Natural Language Processing，2015：2212-2218.
[9] CHEN M.Efficient vector representation for documents through corruption[C]//The International Conference on Learning Representations，2017：1-13.
[10] ARORA S，LIANG Y，MA T.A simple but tough-to-beat baseline for sentence embeddings[C]//The International Conference on Learning Representations，2017：1-16.
[11] MARGARIT H，SUBRAMANIAM R.A batch-normalized recurrent network for sentiment classification[C]//Conference on Neural Information Processing Systems，2016：2-8.
[12] 朱晓霞，宋嘉欣，张晓缇.基于主题挖掘技术的文本情感分析综述[J].情感理论与实践，2019，42（11）：156-163.
ZHU X X，SONG J X，ZHANG X T.Review of text emotion analysis based on topic mining technology[J].Information Studies：Theory and Application，2019，42（11）：156-163.
[13] LIU F，ZHENG J，ZHENG L，et al.Combining attention-based bidirectional gated recurrent neural network and two-dimensional convolutional neural network for document-level sentiment classification[J].Neurocomputing，2020，371：39-50.
[14] PETERS M E，NEUMANN M，IYYER M，et al.Deep contextualized word representations[C]//The Annual Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2018：2227-2237.
[15] MOGHADDAM S，ESTER M.Opinion digger：an unsu-pervised opinion miner from unstructured product reviews[C]//The Conference on Information and Knowledge Management，2010：1825-1828.
[16] HUANG M，QIAN Q，ZHU X.Encoding syntactic knowledge in neural networks for sentiment classification[J].ACM Transactions on Information Systems，2017，35（3）：26.
[17] TABOADA M，BROOKE J，TOFILOSKI M，et al.Lexicon based methods for sentimentanalysis[J].Computational Linguistics，2011，37（2）：267-307.
[18] LIU B，ZHANG L.A survey of opinion mining and sen-timent analysis[M]//Mining text data.[S.l.]：Springer，2012：415-463.
[19] QIAN Q，HUANG M，LEI J，et al.Linguistically regularized LSTMs for sentiment classification[C]//Annual Meeting of the Association for Computational Linguistics，2017：1679-1689.
[20] KE P，HAOZHE J，SIYANG L，et al.Sentilare：linguistic knowledge enhanced language representation for sentiment analysis[C]//Conference on Empirical Methods in Natural Language Processing，2020：6975-6988.
[21] TIAN H，GAO C，XIAO X Y，et al.SKEP：sentiment knowledge enhanced pre-training for sentiment analysis[C]//Annual Meeting of the Association for Computational Linguistics，2020：4067-4076.
[22] DUC H P，ANH C L.Learning multiple layers of knowledge representation for aspect based sentiment analysis[J].Data & Knowledge Engineering，2018，114：26-39.
[23] LI K，LI C，GE J，et al.Leveraging multiple features for document sentiment classification[J].Information Sciences，2020，518：39-55.
[24] SZEGEDY C，LIU W，JIA Y，et al.Going deeper with convolutions[C]//The IEEE/CVF Computer Vision and Pattern Recognition Conference，2015：1-9.
[25] BACCIANELLA S，ESULI A，SEBASTIANI F.Senti-wordnet 3.0：an enhanced lexical resource for sentiment analysis and opinion mining[C]//International Conference on Language Resources and Evaluation，2010：2200-2204.
[26] DIAO Q，QIU M，WU C Y，et al.Jointly modeling aspects，ratings and sentiments for movie recommendation（jmars）[C]//ACM SIGKDD International Conference on Knowledge Discovery and Data Mining，2014：193-202.
[27] JINDAL N，LIU B.Opinion spam and analysis[C]//The ACM Conference on Web Search and Data Mining，2008：219-230.
[28] MANNING C，SURDEANU M，BAUER J，et al.The stanford corenlp natural language processing toolkit[C]//Annual Meeting of the Association for Computational Linguistics，2014：55-60.
[29] MIKOLOV T，SUTSKEVER I，CHEN K，et al.Distributed representations of words and phrases and their compositionality[C]//Conference on Neural Information Processing Systems，2013：3111-3119.
[30] KINGMA D，BA J.Adam：a method for stochastic opti-mization[C]//The International Conference on Learning Representations，2015：1-15.
[31] YANG Z C，YANG D Y，DYER C，et al.Hierarchical attention networks for document classification[C]//The Annual Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2016：1480-1489.