Multi-Label Text Classification Combining Bidirectional Attention and Contrast Enhancement Mechanism

doi:10.3778/j.issn.1002-8331.2311-0079

Abstract

Abstract: A multi-label text classification model that integrates bidirectional attention and contrast enhancement mechanism is proposed to address the issues of missing semantic information during text sequence growth and ignoring rich knowledge from existing instances when predicting specific labels. Firstly, combining BERT word embedding, the CTransformer model is used to obtain the global dependency relationship and local structural information of the sequence. Simultaneously, bidirectional attention and label embedding are used to further generate final text and label representations, text information is interacted with label information to obtain more comprehensive semantic information. Then, a contrast enhancement mechanism is used for KNN instance retrieval, a multi-label contrastive learning objective is designed to make the model aware of the KNN classification process and improve the quality of retrieving neighbor instances during the inference process. Finally, the classifier performs text classification based on label representation and text representation. To evaluate the performance of the model, it is tested on three publicly available English datasets, and the experimental results show that the proposed model outperforms P@K and nDCG@K compared to other mainstream baseline models in terms of indicators.

Key words: multi-label text classification, attention mechanism, contrast enhancement mechanism, CTransformer model, multi-label comparative learning

摘要： 现有多标签文本分类模型面对文本序列增长时语义信息缺失，预测特定标签时忽略已有实例中的丰富知识等问题不能很好地解决，由此提出一种融合双向注意力和对比增强机制的多标签文本分类模型。结合BERT词嵌入，利用CTransformer模型分别获取序列的全局依赖关系和局部结构信息；使用双向注意力和标签嵌入进一步生成最终文本表示和标签表示，将文本信息与标签信息进行交互，得到更为全面的综合语义信息；利用对比增强机制进行KNN实例检索，设计一个多标签对比学习目标，使模型意识到KNN分类过程，并提高推理过程中检索邻居实例的质量。分类器根据标签表示和文本表示进行文本分类。为了评估模型的性能，在三个公开英文数据集上进行测试，实验结果表明提出的模型在P@K和nDCG@K指标上均优于其他主流基线模型。

关键词: 多标签文本分类, 注意力机制, 对比增强机制, CTransformer模型, 多标签对比学习

LI Jiandong, FU Jia, LI Jiaqi. Multi-Label Text Classification Combining Bidirectional Attention and Contrast Enhancement Mechanism[J]. Computer Engineering and Applications, 2024, 60(16): 105-115.

李建东, 傅佳, 李佳琦. 融合双向注意力和对比增强机制的多标签文本分类[J]. 计算机工程与应用, 2024, 60(16): 105-115.

References

[1] GOPAL S, YANG Y. Multilabel classification with meta-level features[C]//Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2010: 315-322.
[2] CHEN G, YE D, XING Z, et al. Ensemble application of convolutional and recurrent neural networks for multi-label text categorization[C]//Proceedings of the 2017 International Joint Conference on Neural Networks, 2017: 2377-2383.
[3] NAM J, MENCIA E L, KIM H J, et al. Maximizing subset accuracy with recurrent neural networks in multi-label classification[C]//Advances in Neural Information Processing Systems 30, 2017.
[4] CHEN W, XU B. Semi-supervised Chinese word segmentation based on bilingual information[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015: 1207-1216.
[5] QIAN Q, TIAN B, HUANG M, et al. Learning tag embeddings and tag-specific composition functions in recursive neural network[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2015: 1365-1374.
[6] BAI S, KOLTER J Z, KOLTUN V. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling[J]. arXiv:1803.01271, 2018.
[7] WANG G, LI C, WANG W, et al. Joint embedding of words and labels for text classification[J]. arXiv:1805.04174, 2018.
[8] LI Y, YANG Y. Label embedding for multi-label classification via dependence maximization[J]. Neural Processing Letters, 2020, 52: 1651-1674.
[9] BOUTELL M R, LUO J, SHEN X, et al. Learning multi-label scene classification[J]. Pattern Recognition, 2004, 37(9): 1757-1771.
[10] READ J, PFAHRINGER B, HOLMES G, et al. Classifier chains for multi-label classification[J]. Machine Learning, 2011, 85: 333-359.
[11] ZHANG M L, ZHOU Z H. ML-KNN: a lazy learning approach to multi-label learning[J]. Pattern Recognition, 2007, 40(7): 2038-2048.
[12] ELISSEEFF A, WESTON J. A kernel method for multi-labelled classification[C]//Advances in Neural Information Processing Systems 14, 2001.
[13] KALCHBRENNER N, GREFENSTETTE E, BLUNSOM P. A convolutional neural network for modelling sentences[J]. arXiv:1404.2188, 2014.
[14] KURATA G, XIANG B, ZHOU B. Improved neural network-based multi-label classification with better initialization leveraging label co-occurrence[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016: 521-526.
[15] SABOUR S, FROSST N, HINTON G E. Dynamic routing between capsules[C]//Advances in Neural Information Processing Systems 30, 2017.
[16] 倪斌, 陆晓蕾, 童逸琦, 等. 胶囊神经网络在期刊文本分类中的应用[J]. 南京大学学报(自然科学版), 2021, 57(5): 750-756.
NI B, LU X L, TONG Y Q, et al. Automated journal text classification based on capsule neural network[J]. Journal of Nanjing University (Natural Science Edition), 2021, 57(5): 750-756.
[17] GUO L, ZHANG D, WANG L, et al. CRAN: a hybrid CNN-RNN attention-based model for text classification[C]//Proceedings of the 37th International Conference on Conceptual Modeling, Xi’an, Oct 22-25, 2018. Cham: Springer, 2018: 571-585.
[18] ZHOU P, SHI W, TIAN J, et al. Attention-based bidirectional long short-term memory networks for relation classification[C]//Proceedings of the 54th Annual Meeting of the Association For Computational Linguistics (Volume 2: Short papers), 2016: 207-212.
[19] YANG P, SUN X, LI W, et al. SGM: sequence generation model for multi-label classification[J]. arXiv:1806.04822, 2018.
[20] YOU R, ZHANG Z, WANG Z, et al. AttentionXML: label tree-based attention-aware deep model for high-performance extreme multi-label text classification[C]//Advances in Neural Information Processing Systems 32, 2019.
[21] XIAO L, HUANG X, CHEN B, et al. Label-specific document representation for multi-label text classification[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019: 466-475.
[22] 冯海, 马甲林, 许林杰, 等. 融合标签嵌入和知识感知的多标签文本分类方法[J]. 南京大学学报 (自然科学), 2023, 59(2): 273-281.
FENG H, MA J L, XU L J, et al. Multi-label text classification method combining label embedding and knowledge-aware[J]. Journal of Nanjing University (Natural Science Edition), 2023, 59(2): 273-281.
[23] 杨春霞, 马文文, 陈启岗, 等. 融合CNN-SAM与GAT的多标签文本分类模型[J]. 计算机工程与应用, 2023, 59(5): 106-114.
YANG C X, MA W W, CHEN Q G, et al. Multi-label text classification model combining CNN-SAM and GAT[J]. Journal of Computer Engineering and Applications, 2023, 59(5): 106-114.
[24] KHANDELWAL U, LEVY O, JURAFSKY D, et al. Generalization through memorization: nearest neighbor language models[J]. arXiv:1911.00172, 2019.
[25] KHANDELWAL U, FAN A, JURAFSKY D, et al. Nearest neighbor machine translation[J]. arXiv:2010.00710, 2020.
[26] ZHENG X, ZHANG Z, GUO J, et al. Adaptive nearest neighbor machine translation[J]. arXiv:2105.13022, 2021.
[27] LEWIS D D, YANG Y, RUSSELL-ROSE T, et al. RCV1: a new benchmark collection for text categorization research[J]. Journal of Machine Learning Research, 2004, 5: 361-397.
[28] BLOM-HANSEN J. Studying power and influence in the European Union: exploiting the complexity of post-Lisbon legislation with EUR-Lex[J]. European Union Politics, 2019, 20(4): 692-706.
[29] DU C, CHEN Z, FENG F, et al. Explicit interaction model towards text classification[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence, 2019: 6359-6366.
[30] WANG R, RIDLEY R, QU W, et al. A novel reasoning mechanism for multi-label text classification[J]. Information Processing & Management, 2021, 58(2): 102441.
[31] MA Q, YUAN C, ZHOU W, et al. Label-specific dual graph neural network for multi-label text classification[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021: 3855-3864.
[32] MARAGHEH H K, GHAREHCHOPOGH F S, MAJIDZADEH K, et al. A new hybrid based on long short-term memory network with spotted hyena optimization algorithm for multi-label text classification[J]. Mathematics, 2022, 10(3): 488.