Enhanced Contextual Neural Topic Model for Short Texts

doi:10.3778/j.issn.1002-8331.2212-0259

Abstract

Abstract: Most of the current topic models are modeled based on word co-occurrence information of their own texts, and do not introduce topic sparsity constraints to improve the model’s topic extraction ability. In addition, short texts have the problem of word co-occurrence sparsity, which seriously affects accuracy of short text topic modeling. In response to the above problems, an enhanced context neural topic model (ECNTM) is proposed. ECNTM implements sparsity constraints on the topic based on the topic controller to filter out irrelevant topics. At the same time, the input of the model becomes the splicing of BOW vector and SBERT sentence embedding. In the Gaussian decoder, the topic on the word is embedded in the embedding space. The distribution is treated as a multivariate Gaussian distribution or a Gaussian mixture distribution, which explicitly enriches the limited context information of short texts and solves the problem of sparse word co-occurrence features in short texts. Experimental results on four public datasets of WS, Reuters, KOS and 20 NewsGroups show that this model has significantly improved compared with the benchmark model in terms of perplexity, topic consistency, and text classification accuracy, which proves the introduction of topic sparsity constraints and rich effectiveness of contextual information to short text topic modeling.

Key words: neural subject model, short text, sparsity constraint, variational auto-encoder, topic modeling

摘要： 目前的主题模型大多数基于自身文本的词共现信息进行建模，并没有引入主题的稀疏约束来提升模型的主题抽取能力，此外短文本本身存在词共现稀疏的问题，该问题严重影响了短文本主题建模的准确性。针对以上问题，提出了一种增强上下文神经主题模型（enhanced context neural topic model，ECNTM）。ECNTM基于主题控制器对主题进行稀疏性约束，过滤掉不相关的主题，同时模型的输入变成BOW向量和SBERT句子嵌入的拼接，在高斯解码器中，通过在嵌入空间中将单词上的主题分布处理为多元高斯分布或高斯混合分布，显式地丰富了短文本有限的上下文信息，解决了短文本词共现特征稀疏问题。在WS、Reuters、KOS、20 NewsGroups四个公开数据集上的实验结果表明，该模型在困惑度、主题一致性以及文本分类准确率上相较基准模型均有明显提升，证明了引入主题稀疏约束特性以及丰富的上下文信息到短文本主题建模的有效性。

关键词: 神经主题模型, 短文本, 稀疏约束, 变分自编码器, 主题建模

LIU Gang, WANG Tongli, TANG Hongwei, ZHAN Kai, YANG Wenli. Enhanced Contextual Neural Topic Model for Short Texts[J]. Computer Engineering and Applications, 2024, 60(1): 154-164.

刘刚, 王同礼, 唐宏伟, 战凯, 杨雯莉. 面向短文本的增强上下文神经主题模型[J]. 计算机工程与应用, 2024, 60(1): 154-164.

References

[1] KINGMA D P, WELLING M. Auto-encoding variational Bayes[C]//Proceedings of the International Conference on Learning Representations, 2014: 1-14.
[2] REZAEE M, FERRARO F. A discrete variational recurrent topic model without the reparametrization trick[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020: 13831-13843.
[3] PANWAR M, SHAILABH S, AGGARWAL M, et al. TAN-NTM: topic attention networks for neural topic modeling[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 3865-3880.
[4] TIAN R, MAO Y, ZHANG R. Learning VAE-LDA models with rounded reparameterization trick[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020: 1315-1325.
[5] LEE Y Y, KE H, YEN T Y, et al. Combining and learning word embedding with WordNet for semantic relatedness and similarity measurement[J]. Journal of the Association for Information Science and Technology, 2020, 71(6): 657-670.
[6] ZHAO H, DU L, LIU G, et al. Leveraging meta information in short text aggregation[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019: 4042-4049.
[7] ZHU Q, FENG Z, LI X. GraphBTM: graph enhanced autoencoded variational inference for biterm topic model[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018: 4663-4672.
[8] PENG M, XIE Q, ZHANG Y, et al. Neural sparse topical coding[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018: 2332-2340.
[9] LI X, ZHANG J, OUYANG J. Dirichlet multinomial mixture with variational manifold regularization: topic modeling over short texts[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2019: 7884-7891.
[10] HUANG J, PENG M, LI P, et al. Improving biterm topic model with word embeddings[J]. World Wide Web, 2020, 23(6): 3099-3124.
[11] DIENG A B, RUIZ F J R, BLEI D M. Topic modeling in embedding spaces[J]. Transactions of the Association for Computational Linguistics, 2020, 8: 439-453.
[12] GUPTA P, CHAUDHARY Y, BUETTNER F, et al. Document informed neural autoregressive topic models with distributional prior[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2019: 6505-6512.
[13] LI C, WANG H, ZHANG Z, et al. Topic modeling for short texts with auxiliary word embeddings[C]//Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, 2016: 165-174.
[14] ZHAO X, WANG D, ZHAO Z, et al. A neural topic model with word vectors and entity vectors for short texts[J]. Information Processing & Management, 2021, 58(2): 102455.
[15] MIAO Y, LEI Y, BLUNSOM P. Neural variational inference for text processing[C]//Proceedings of the 33rd International Conference on Machine Learning, 2016: 1727-1736.
[16] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018: 4171-4186.
[17] REIMERS N, GUREVYCH I. Sentence-BERT: sentence embeddings using siamese BERT-Networks[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019: 3982-3992.
[18] 胡铭菲, 左信, 刘建伟. 深度生成模型综述[J]. 自动化学报, 2022, 48(1): 40-74.
HU M F, ZUO X, LIU J W. A survey of deep generative models[J]. Acta Automatica Sinica, 2022, 48(1): 40-74.
[19] MA J, SAUL L K, SAVAGE S, et al. Identifying suspicious URLs: an application of large-scale online learning[C]//Proceedings of the 26th Annual International Conference on Machine Learning. Montreal, Quebec, Canada: Association for Computing Machinery, 2009: 681-688.
[20] BLEI D M, LAFFERTY J D. Dynamic topic models[C]//Proceedings of the 23rd International Conference on Machine Learning. Pittsburgh, Pennsylvania, USA: Association for Computing Machinery, 2006: 113-120.
[21] MIAO Y, GREFENSTETTE E, BLUNSOM P. Discovering discrete latent topics with neural variational inference[C]//International Conference on Machine Learning, 2017: 2410-2419.
[22] ZHANG Y, WANG Z, YU Y, et al. LF-LDA: a supervised topic model for multi-label documents classification[J]. International Journal of Data Warehousing and Mining, 2018, 14(2): 18-36.