Survey of Short Text Classification Methods Based on Deep Learning

doi:10.3778/j.issn.1002-8331.2209-0048

Abstract

Abstract: From five aspects of CNN, RNN, CNN-RNN, GCN and other deep learning methods, the research status of their application in short text classification is comprehensively analyzed, their advantages and disadvantages are compared, and the commonly used labeled datasets are summarized. The results show that：At present, the application research of deep learning in short text classification mainly focuses on the improvement of efficient algorithms and the expansion of text information. At the same time, the research on constructing labeled datasets for model testing is in the initial stage, mostly for specific fields such as movie reviews, commodity reviews, news, etc., which needs continuous improvement. In the future, the research will focus on algorithm improvement, information expansion and their mutual integration, to explore some specific applications with good classification effect in practice.

Key words: short text, text classification, deep learning, text representation

摘要： 从CNN、RNN、CNN-RNN、GCN及其他深度学习方法五方面，全面分析了深度学习在短文本分类应用中的研究现状，比较了各自的优缺点，总结了常用的标签数据集。结果表明：目前深度学习在短文本分类中的应用研究主要集中在高效算法改进以及文本信息拓展两方面；对模型检验中构建标签数据集的研究也处于起步阶段，大多是针对影评、商品评论、新闻等特定领域的，还需不断完善；基于深度学习的短文本分类方法研究，今后在理论研究方面将重点关注算法改进、信息拓展以及二者的相互融合，在实践中探索某些分类效果较好的特定领域应用。

关键词: 短文本, 文本分类, 深度学习, 文本表示

GAN Yating, AN Jianye, XU Xue. Survey of Short Text Classification Methods Based on Deep Learning[J]. Computer Engineering and Applications, 2023, 59(4): 43-53.

淦亚婷, 安建业, 徐雪. 基于深度学习的短文本分类方法研究综述[J]. 计算机工程与应用, 2023, 59(4): 43-53.

References

[1] 吴思慧，陈世平.结合TFIDF的Self-Attention-Based Bi-LSTM的垃圾短信识别[J].计算机系统应用，2020，29（9）：171-177.
WU S H，CHEN S P.Spam message recognition based on TFIDF and Self-Attention-Based Bi-LSTM[J].Computer Application Systems，2020，29（9）：171-177.
[2] 谢秦，张清华，王国胤.基于相似度量的自适应三支垃圾邮件过滤器[J].计算机研究与发展，2019，56（11）：2410-2423.
XIE Q，ZHANG Q H，WNAG G Y.An adaptive three-way spam filter with similarity measure[J].Journal of Computer Research and Development，2019，56（11）：2410-2423.
[3] XU G，YU Z，YAO H，et al.Chinese text sentiment analysis based on extended sentiment dictionary[J].IEEE Access，2019，7：43749-43762.
[4] BIAN T，XIAO X，XU T，et al.Rumor detection on social media with bi-directional graph convolutional networks[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence，2020：549-556.
[5] 马哲坤，涂艳.基于知识图谱的网络舆情突发话题内容监测研究[J].情报科学，2019，37（2）：33-39.
MA Z K，TU Y.Online emerging topic content monitoring based on knowledge graph[J].Information Science，2019，37（2）：33-39.
[6] 郑捷.NLP汉语自然语言处理原理与实践[M].北京：电子工业出版社，2017：3-5.
ZHEN J.Principles and practices of NLP Chinese natural language processing[M].Beijing：Publishing House of Electronics Industry，2017：3-5.
[7] CHEN M，JIN X，SHEN D.Short text classification improved by learning multi-granularity topics[C]//22nd International Joint Conference on Artificial Intelligence，2011.
[8] LEE K，PALSETIA D，NARAYANAN R，et al.Twitter trending topic classification[C]//2011 IEEE 11th International Conference on Data Mining Workshops，2011：251-258.
[9] 胡勇军，江嘉欣，常会友.基于LDA高频词扩展的中文短文本分类[J].现代图书情报技术，2013（6）：42-48.
HU Y J，JIANG J X，CHANG H Y.A new method of key words extraction for Chinese short-text classification[J].New Technology of Library and Information Service，2013（6）：42-48.
[10] 张志飞，苗夺谦，高灿.基于LDA主题模型的短文本分类方法[J].计算机应用，2013，33（6）：1587-1590.
ZHANG Z F，MIAO D Q，GAO C.Short text classification using latent Dirichlet allocation[J].Journal of Computer Applications，2013，33（6）：1587-1590.
[11] ZHOU F G，ZHANG F，YANG B R，et al.Research on short text classification algorithm based on statistics and rules[C]//2010 3rd International Symposium on Electronic Commerce and Security，2010：3-7.
[12] 刘琴，袁家政，翁长虹.基于深度学习的短文本分类研究综述[C]//中国计算机用户协会网络应用分会2017年第二十一届网络新技术与应用年会论文集，2017：17-21.
LIU Q，YUAN J Z，WENG C H.Survey of short text classification based on deep learning[C]//Proceedings of the 21st Annual Conference on New Network Technologies and Applications 2017，Network Application Branch of China Computer Users Association，2017：17-21.
[13] LECUN Y，BOTTOU L，BENGIO Y，et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE，1998，86（11）：2278-2324.
[14] LIPTON Z C，BERKOWITZ J，ELKAN C.A critical review of recurrent neural networks for sequence learning[J].arXiv：1506.00019，2015.
[15] BRUNA J，ZAREMBA W，SZLAM A，et al.Spectral networks and locally connected networks on graphs[J].arXiv：1312.6203，2013.
[16] KIM Y.Convolutional neural networks for sentence classifification[J].arXiv：1408.5882，2014.
[17] CONNEAU A，SCHWENK H，BARRAULT L，et al.Very deep convolutional networks for text classification[J].arXiv：1606.01781，2016.
[18] LE H T，CERISARA C，DENIS A.Do convolutional networks need to be deep for text classification?[J].arXiv：1707.04108，2017.
[19] JOHNSON R，ZHANG T.Deep pyramid convolutional neural networks for text categorization[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers），2017：562-570.
[20] GUO J，YUE B，XU G，et al.An enhanced convolutional neural network model for answer selection[C]//Proceedings of the 26th International Conference on World Wide Web Companion，2017：789-790.
[21] WANG H，HE J，ZHANG X，et al.A short text classification method based on n-gram and CNN[J].Chinese Journal of Electronics，2020，29（2）：248-254.
[22] WANG P，XU B，XU J，et al.Semantic expansion using word embedding clustering and convolutional neural network for improving short text classification[J].Neurocomputing，2016，174：806-814.
[23] SOTTHISOPHA N，VATEEKUL P.Improving short text classification using fast semantic expansion on multichannel convolutional neural network[C]//2018 19th IEEE/ACIS International Conference on Software Engineering，Artificial Intelligence，Networking and Parallel/Distributed Computing，2018：182-187.
[24] WANG J，WANG Z，ZHANG D，et al.Combining knowledge with deep convolutional neural networks for short text classification[C]//26th International Joint Conference on Artificial Intelligence，2017：2915-2921.
[25] WANG H，TIAN K，WU Z，et al.A short text classification method based on convolutional neural network and semantic extension[J].International Journal of Computational Intelligence Systems，2021，14（1）：367-375.
[26] HOCHREITER S，SCHMIDHUBER J.Long short-term memory[J].Neural computation，1997，9（8）：1735-1780.
[27] GERS F A，SCHMIDHUBER J，CUMMINS F.Learning to forget：continual prediction with LSTM[J].Neural computation，2000，12（10）：2451-2471.
[28] CHO K，VAN MERRI?NBOER B，GULCEHRE C，et al.Learning phrase representations using RNN encoder-decoder for statistical machine translation[J].arXiv：1406.1078，2014.
[29] 刘建伟，宋志妍.循环神经网络研究综述[J].控制与决策，2022，37（11）：2753-2768.
LIU J W，SONG Z Y.Overview of recurrent neural networks[J].Control and Decision，2022，37（11）：2753-2768.
[30] LIU P，QIU X，HUANG X.Recurrent neural network for text classification with multi-task learning[J].arXiv：1605.
05101，2016.
[31] TAI K S，SOCHER R，MANNING C D.Improved semantic representations from tree-structured long short-term memory networks[J].arXiv：1503.00075，2015.
[32] ZHANG Y，XU H，XU K.Chinese short text classification based on dependency syntax information[C]//2021 5th International Conference on Compute and Data Analysis，2021：133-138.
[33] ZHOU Y，XU B，XU J，et al.Compositional recurrent neural networks for Chinese short text classification[C]//2016 IEEE/WIC/ACM International Conference on Web Intelligence，2016：137-144.
[34] GAO M X，LI J Y.Chinese short text classification method based on word embedding and long short-term memory neural network[C]//2021 International Conference on Artificial Intelligence，Big Data and Algorithms，2021：91-95.
[35] KALJAHI R，FOSTER J.Any-gram kernels for sentence classification：a sentiment analysis case study[J].arXiv：1712.07004，2017.
[36] 宋明，刘彦隆.Bert在微博短文本情感分类中的应用与优化[J].小型微型计算机系统，2021，42（4）：714-718.
SONG M，LIU Y L.Application and optimization of Bert in sentiment classification of Weibo short text[J].Journal of Chinese Computer Systems，2021，42（4）：714-718.
[37] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems，2017，30：5998-6008.
[38] YANG Z，YANG D，DYER C，et al.Hierarchical attention networks for document classification[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2016：1480-1489.
[39] ZHOU Y，XU J，CAO J，et al.Hybrid attention networks for Chinese short text classification[J].Computación y Sistemas，2017，21（4）：759-769.
[40] 陶志勇，李小兵，刘影，等.基于双向长短时记忆网络的改进注意力短文本分类方法[J].数据分析与知识发现，2019，3（12）：21-29.
TAO Z Y，LI X B，LIU Y，et al.Classifying short texts with improved-attention based bidirectional long memory network[J].Data Analysis and Knowledge Discovery，2019，3（12）：21-29.
[41] 吴小华，陈莉，魏甜甜，等.基于Self-Attention和Bi-LSTM的中文短文本情感分析[J].中文信息学报，2019，33（6）：100-107.
WU X H，CHEN L，WEI T T，et al.Sentiment analysis of Chinese short text based on self-attention and Bi-LSTM[J].Journal of Chinese Information Processing，2019，33（6）：100-107.
[42] 陈立潮，秦杰，陆望东，等.自注意力机制的短文本分类方法[J].计算机工程与设计，2022，43（3）：728-734.
CHEN L C，QIN J，LU W D，et al.Short text classification method based on self-attentinmechanism[J].Computer Engineering and Design，2022，43（3）：728-734.
[43] 石磊，王明宇，宋哲理，等.自注意力机制和BiGRU相结合的文本分类研究[J].小型微型计算机系统，2022，43（12）：2541-2548.
SHI L，WNAG P Y，SONG Z L，et al.Text classification research with the combination of self-attention mechanism and BiGRU[J].Journal of Chinese Computer Systems，2022，43（12）：2541-2548.
[44] LAI S，XU L，LIU K，et al.Recurrent convolutional neural networks for text classification[C]//29th AAAI Conference on Artificial Intelligence，2015.
[45] XU J，CAI Y，WU X，et al.Incorporating context-relevant concepts into convolutional neural networks for short text classification[J].Neurocomputing，2020，386：42-53.
[46] HAO M，XU B，LIANG J Y，et al.Chinese short text classification with mutual-attention convolutional neural networks[J].ACM Transactions on Asian and Low-Resource Language Information Processing，2020，19（5）：1-13.
[47] CHEN J，HU Y，LIU J，et al.Deep short text classification with knowledge powered attention[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence，2019：6252-6259.
[48] SHE X，ZHANG D.Text classification based on hybrid CNN-LSTM hybrid model[C]//2018 11th International Symposium on Computational Intelligence and Design，2018，2：185-189.
[49] 郑承宇，王新，王婷，等.基于Stacking-Bert集成学习的中文短文本分类算法[J].科学技术与工程，2022，22（10）：4033-4038.
ZHENG C Y，WANG X，WNAG T，et al.Chinese short text classification algorithm based on Stacking-Bert ensemble learning[J].Science Technology and Engineering，2022，22（10）：4033-4038.
[50] 李颖.基于BERT-DPCNN的垃圾弹幕识别改进及应用[D].上海：上海师范大学，2020.
LI Y.Improvement and application of barrage recognition based on BERT-DPCNN[D].Shanghai：Shanghai Normal University，2020.
[51] DEFFERRARD M，BRESSON X，VANDERGHEYNST P.Convolutional neural networks on graphs with fast localized spectral filtering[C]//Advances in Neural Information Processing Systems，2016，29：3844-3852.
[52] KIPF T N，WELLING M.Semi-supervised classification with graph convolutional networks[J].arXiv：1609.02907，2016.
[53] VELI?KOVI? P，CUCURULL G，CASANOVA A，et al.Graph attention networks[J].arXiv：1710.10903，2017.
[54] 徐冰冰，岑科廷，黄俊杰，等.图卷积神经网络综述[J].计算机学报，2020，43（5）：755-780.
XU B B，CEN K Y，HUANG J J，et al.A survey on graph convolutional neural network[J].Chinese Journal of Computers，2020，43（5）：755-780.
[55] YAO L，MAO C，LUO Y.Graph convolutional networks for text classification[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence，2019：7370-7377.
[56] YANG T，HU L，SHI C，et al.HGAT：heterogeneous graph attention networks for semi-supervised short text classification[J].ACM Transactions on Information Systems，2021，39（3）：32.
[57] LIU X，YOU X，ZHANG X，et al.Tensor graph convolutional networks for text classification[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence，2020：8409-8416.
[58] LI R，CHEN H，FENG F，et al.Dual graph convolutional networks for aspect-based sentiment analysis[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing（Volume 1：Long Papers），2021：6319-6329.
[59] LIN Y，MENG Y，SUN X，et al.BertGCN：transductive text classification by combining GCN and BERT[J].arXiv：2105.
05727，2021.
[60] GAO W，HUANG H.A gating context-aware text classification model with BERT and graph convolutional networks[J].Journal of Intelligent & Fuzzy Systems，2021，40（3）：4331-4343.
[61] HUANG L，MA D，LI S，et al.Text level graph neural network for text classification[J].arXiv：1910.02356，2019.
[62] YANG M，ZHAO W，CHEN L，et al.Investigating the transferring capability of capsule networks for text classification[J].Neural Networks，2019，118：247-261.
[63] 王超凡，琚生根，孙界平，等.融入多尺度特征注意力的胶囊神经网络及其在文本分类中的应用[J].中文信息学报，2022，36（1）：65-74.
WANG C F，JU S G，SUN J P，et al.Capsule network with multi-scale feature attention for text classification[J].Journal of Chinese Information Processing，2022，36（1）：65-74.
[64] IYYER M，MANJUNATHA V，BOYD-GRABER J，et al.Deep unordered composition rivals syntactic methods for text classification[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing（Volume 1：Long Papers），2015：1681-1691.
[65] JOULIN A，GRAVE E，BOJANOWSKI P，et al.Bag of tricks for efficient text classification[J].arXiv：1607.01759，2016.
[66] ZHANG X，ZHAO J，LECUN Y.Character-level convolutional networks for text classification[C]//Advances in Neural Information Processing Systems，2015，28：649-657.
[67] LIU Z，ZHANG L，TU C，et al.Statistical and semantic analysis of rumors in Chinese social media[J].Scientia Sinica Informationis，2015，45（12）：1536.
[68] QIU X，GONG J，HUANG X.Overview of the NLPCC 2017 shared task：Chinese news headline categorization[C]//National CCF Conference on Natural Language Processing and Chinese Computing.Cham：Springer，2017：948-953.
[69] PANG B，LEE L.Seeing stars：exploiting class relationships for sentiment categorization with respect to rating scales[J].arXiv：cs/0506075，2005.
[70] LI X，ROTH D.Learning question classifiers[C]//19th International Conference on Computational Linguistics，2002.
[71] SOCHER R，PERELYGIN A，WU J，et al.Recursive deep models for semantic compositionality over a sentiment treebank[C]//Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing，2013：1631-1642.
[72] WANG F，WANG Z，LI Z，et al.Concept-based short text classification and ranking[C]//Proceedings of the 23rd ACM International Conference on Information and Knowledge Management，2014：1069-1078.
[73] PHAN X H，NGUYEN L M，HORIGUCHI S.Learning to classify short and sparse text & web with hidden topics from large-scale data collections[C]//Proceedings of the 17th International Conference on World Wide Web，2008：91-100.
[74] PANG B，LEE L.A sentimental education：sentiment analysis using subjectivity summarization based on minimum cuts[J].arXiv：cs/0409058，2004.
[75] MAAS A，DALY R E，PHAM P T，et al.Learning word vectors for sentiment analysis[C]//Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics：Human Language Technologies，2011：142-150.
[76] WU C Y，DIAO Q，QIU M，et al.Jointly modeling aspects，ratings and sentiments for movie recommendation[C]//Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining，2014：193-202.