Multi-Label Text Classification Based on Shared Semantic Space

doi:10.3778/j.issn.1002-8331.2203-0284

Abstract

Abstract: In the multi-label text classification task, each given document corresponds to a set of related labels. At present, it mainly faces the following three problems：（1） the joint modeling of label-text and label-label relationships is inadequate; （2） the semantic mining of the label itself is insufficient; （3） the utilization of the internal structure information of the label is ignored. To solve the above problems, this study proposes a multi-label text classification method based on joint attention and shared semantic space. The proposed joint multi-head attention mechanism synchronously models the relationship between labels and relationship between labels and documents simultaneously, that avoids error transmission and uses the interaction information between them. The proposed decouple shared semantic space embedding method improves the method of using labels semantic information, and uses the encoder of shared parameters to extract the semantic representation of labels and documents, reducing its deviation in the phase of modeling correlation. The proposed hierarchical hinting method based on prior knowledge relies on the prior knowledge in the pre-trained model to exploit the labels hierarchy information. Experimental results show that the proposed method is superior to the existing state-of-the-art multi-label text classification methods in public datasets.

Key words: multi-label text classification, attention mechanism, label representation, pre-trained model, semantic embedding

摘要： 在多标签文本分类任务中，每个给定的文档都对应一组相关标签。目前主要面临以下三方面问题：（1）对标签-文本和标签-标签关系的联合建模不充分；（2）对标签本身语义的挖掘不足；（3）忽略了对标签内部结构信息的利用。对于以上问题，提出了一种基于联合注意力和共享语义空间的多标签文本分类方法。提出了融合多头注意力机制，该方法旨在同步地对标签与文档的关系和标签之间的关系进行建模，利用两者交互信息的同时避免误差传递。提出了解耦的共享语义空间嵌入方法，改进了利用标签语义信息的方法，使用共享参数的编码器提取标签和文档的语义表示，减少其在建模相关性阶段的偏差。提出了一种基于先验知识的层次提示方法，利用预训练模型中的先验知识增强标签层次结构信息。实验结果表明，该方法在公开数据集上优于目前最先进的多标签文本分类模型。

关键词: 多标签文本分类, 注意力机制, 标签表示, 预训练模型, 语义嵌入

SUN Kun, QIN Bowen, SANG Jitao, YU Jian. Multi-Label Text Classification Based on Shared Semantic Space[J]. Computer Engineering and Applications, 2023, 59(12): 100-105.

孙坤, 秦博文, 桑基韬, 于剑. 基于共享语义空间的多标签文本分类[J]. 计算机工程与应用, 2023, 59(12): 100-105.

References

[1] GOPAL S，YANG Y.Multilabel classification with meta-level features[C]//Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval，2010：315-322.
[2] KUMAR A，IRSOY O，ONDRUSKA P，et al.Ask me anything：dynamic memory networks for natural language processing[C]//International Conference on Machine Learning，2016：1378-1387.
[3] CAMBRIA E，OLSHER D，RAJAGOPAL D.SenticNet 3：a common and common-sense knowledge base for cognition-driven sentiment analysis[C]//Twenty-Eighth AAAI Conference on Artificial Intelligence，2014.
[4] KIM Y.Convolutional neural networks for sentence classification[J].arXiv：1408.5882，2014.
[5] LIU P，QIU X，HUANG X.Recurrent neural network for text classification with multi-task learning[J].arXiv：1605.
05101，2016.
[6] CHEN G，YE D，XING Z，et al.Ensemble application of convolutional and recurrent neural networks for multi-label text categorization[C]//2017 International Joint Conference on Neural Networks，2017：2377-2383.
[7] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[J].arXiv：1706.03762v2，2017.
[8] ZHANG X，ZHANG Q W，YAN Z，et al.Enhancing label correlation feedback in multi-label text classification via multi-task learning[J].arXiv：2106.03103，2021.
[9] ZHANG Q W，ZHANG X M，ZHAO Y，et al.Correlation-guided representation for multi-label text classification[C]//Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence，Montreal，Canada，2021.
[10] GUO H，LI X，ZHANG L，et al.Label-aware text representation for multi-label text classification[C]//2021 IEEE International Conference on Acoustics，Speech and Signal Processing（ICASSP），2021：7728-7732.
[11] MA Q，YUAN C，ZHOU W，et al.Label-specific dual graph neural network for multi-label text classification[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing（Volume 1：Long Papers），2021：3855-3864.
[12] YANG P C，XU S.SGM：sequence generation model for multi-label classification[J].arXiv：1806.04822，2018.
[13] LIU H，YUAN C，WANG X.Label-wise document pre-training for multi-label text classification[C]//CCF International Conference on Natural Language Processing and Chinese Computing.Cham：Springer，2020：641-653.
[14] LIU H，CHEN G，LI P，et al.Multi-label text classification via joint learning from label embedding and label correlation[J].Neurocomputing，2021，460：385-398.
[15] LEWIS D D，YANG Y，RUSSELL-ROSE T，et al.Rcv1：a new benchmark collection for text categorization research[J].Journal of Machine Learning Research，2004，5（4）：361-397.
[16] DEVLIN J，CHANG M W，LEE K，et al.BERT：pre-training of deep bidirectional transformers for language understanding[J].arXiv：1810.04805，2018.
[17] SUN C，QIU X，XU Y，et al.How to fine-tune BERT for text classification?[C]//China National Conference on Chinese Computational Linguistics.Cham：Springer，2019.
[18] YOU R，ZHANG Z，WANG Z，et al.Attentionxml：label tree-based attention-aware deep model for high-performance extreme multi-label text classification[J].arXiv：1811.01727，2018.
[19] WANG G，LI C，WANG W，et al.Joint embedding of words and labels for text classification[J].arXiv：1805.04174，2018.
[20] PAPPAS N，HENDERSON J.Gile：a generalized input-label embedding for text classification[J].Transactions of the Association for Computational Linguistics，2019，7：139-155.
[21] KURATA G，XIANG B，ZHOU B.Improved neural network-based multi-label classification with better initialization leveraging label co-occurrence[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2016：521-526.
[22] LIU J，CHANG W C，WU Y，et al.Deep learning for extreme multi-label text classification[C]//Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval，2017：115-124.
[23] ZHANG W，YAN J，WANG X，et al.Deep extreme multi-label learning[C]//Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval，2018：100-107.
[24] DU C，CHEN Z，FENG F，et al.Explicit interaction model towards text classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2019：6359-6366.
[25] XIAO L，HUANG X，CHEN B，et al.Label-specific document representation for multi-label text classification[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing（EMNLP-IJCNLP），2019：466-475.
[26] LOSHCHILOV I，HUTTER F.Decoupled weight decay regularization[J].arXiv：1711.05101，2017.