Multi-Label Text Classification Model Integrating GAT and Head-Tail Label

doi:10.3778/j.issn.1002-8331.2305-0368

Abstract

Abstract: The existing multi-label text classification based on neural networks has two main shortcomings：firstly, there is a long tailed distribution of label text frequency in the existing data, and secondly, the correlation between global labels is rarely obtained from the graph structure. To address the above issues, this paper proposes a multi-label text classification model (GATTN) that integrates GAT and head and tail label classifiers. This model utilizes Bi-LSTM with attention mechanism to obtain the feature vector representation of the text. Simultaneously, the correlation between different labels is transformed into an edge weighted graph containing global information, and a multi-layer graph attention network is used to learn the correlation between head labels. Then it interacts with the text context semantic information by dot product, and gets the feature representation with text semantics and header tag information to improve the generalization ability of the model. The experimental results on two publicly available English datasets, AAPD, RCV1-V2 and EUR-Lex, demonstrate that this model outperforms other baseline models in multi-label text classification for long tailed distribution of data.

Key words: multi-label text classification, graph attention networks (GAT), head and tail label, many-shot learning

摘要： 现有基于神经网络的多标签文本分类主要存在两方面的不足，一是现有的数据中标签-文本频率存在长尾分布，二是很少从图结构中获取全局标签之间的关联性，针对以上问题，提出了一种融合GAT与头尾标签分类器的多标签文本分类模型（GATTN），该模型利用带有注意力机制的Bi-LSTM得到文本的特征向量表示；同时把不同标签之间的关联性转换为包含了全局信息的边加权图，利用多层的图注意力网络来学习头标签之间的关联性。然后将其与文本上下文语义信息进行点积交互，得到具有文本语义和头标签信息的特征表示提高模型的泛化能力。在AAPD、RCV1-V2和EUR-Lex三个公开英文数据集上的实验结果证明，该模型针对数据长尾分布的多标签文本分类效果优于其他基线模型。

关键词: 多标签文本分类, 图注意力网络, 头尾标签, 多样本学习

YANG Chunxia, HUANG Yukun, YAN Han, WU Yalei. Multi-Label Text Classification Model Integrating GAT and Head-Tail Label[J]. Computer Engineering and Applications, 2024, 60(15): 150-160.

杨春霞, 黄昱锟, 闫晗, 吴亚雷. 融合GAT与头尾标签的多标签文本分类模型[J]. 计算机工程与应用, 2024, 60(15): 150-160.

References

[1] CAMBRIA E, HOWARD N. Common and common-sense knowledge integration for concept-level sentiment analysis[C]//Proceedings of the Twenty-Seventh International Florida Artificial Intelligence Research Society Conference (FLAIRS 2014), 2014.
[2] YANG Z, YANG D, DYER C, et al. Hierarchical attention networks for document classification[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016: 1480-1489.
[3] SUN R, JIANG J, TAN Y F, et al. Using syntactic and semantic relation analysis in question answering[C]//Fourteenth Text Retrieval Conference, 2005.
[4] KUMAR A, IRSOY O, ONDRUSKA P, et al. Ask me anything: dynamic memory networks for natural language processing[J]. arXiv:1506.07285, 2015.
[5] 肖琳, 陈博理, 黄鑫. 基于标签语义注意力的多标签文本分类[J]. 软件学报, 2020, 31(4): 1079-1089.
XIAO L, CHEN B L, HUANG X. Multi-label text classification based on label semantic information[J]. Journal of Software, 2020, 31(4): 1079-1089.
[6] WEI T, LI Y F. Does tail label help for large-scale multi-label learning[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 31(7): 2315-2324.
[7] LIU Z, MIAO Z, ZHAN X, et al. Large-scale long-tailed recognition in an open world[J]. arXiv:1904.05160, 2019.
[8] CAO K, WEI C, GAIDON A, et al. Learning imbalanced datasets with label-distribution-aware margin loss[C]//Advances in Neural Information Processing Systems, 2019: 1567-1578.
[9] YUAN M, XU J, LI Z. Long tail multi-label learning[C]//2019 IEEE Second International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), 2019.
[10] XIAO L, ZHANG X, JING L, et al. Does head label help for long-tailed multi-label text classification[J]. arXiv:2101.
09704, 2021.
[11] 王浩镔, 胡平. 采用多级特征的多标签长文本分类算法[J]. 计算机工程与应用, 2021, 57(15): 193-199.
WANG H B, HU P. Multi-label long text classification algorithm using multi-level features[J]. Computer Engineering and Applications, 2021, 57(15): 193-199.
[12] KURATA G, XIANG B, ZHOU B. Improved neural network-based multi-label classification with better initialization leveraging label co-occurrence[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016: 521-526.
[13] ZHANG W, YAN J, WANG X, et al. Deep extreme multi-label learning[J]. arXiv:1704.03718, 2017.
[14] DU C, CHEN Z, FENG F, et al. Explicit interaction model towards text classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2019: 6359-6366.
[15] PAPPAS N, HENDERSON J. GILE: a generalized input-label embedding for text classification[J]. Transactions of the Association for Computational Linguistics, 2019, 7(1): 139-155.
[16] MACAVANEY S, DERNONCOURT F, CHANG W, et al. Interaction matching for long-tail multi-label classification[J]. arXiv:2005.08805, 2020.
[17] BYRD J, LIPTON Z C. What is the effect of importance weighting in deep learning[C]//Proceedings of the 36th International Conference on Machine Learning, 2019.
[18] BOWYER K W, HALL L O, CHAWLA N V, et al. SMOTE: synthetic minority over-sampling technique[J]. Journal of Artificial Intelligence Research, 2002, 16(1): 321-357.
[19] CUI Y, JIA M , LIN T Y, et al. Class-balanced loss based on effective number of samples[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019: 9268-9277.
[20] ZHOU B, CUI Q, WEI X S, et al. BBN: bilateral-branch network with cumulative learning for long-tailed visual recognition[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020: 9719-9728.
[21] KANG B, XIE S, ROHRBACH M, et al. Decoupling represe ntation and classifier for long-tailed recognition[C]//International Conference on Learning Representations, 2019.
[22] HARIHARAN B, GIRSHICK R. Low-shot visual recognition by shrinking and hallucinating features[J]. arXiv:1606.
02819, 2016.
[23] GIDARIS S, KOMODAKIS N. Dynamic few-shot visual learning without forgetting[J]. arXiv:1804.09458, 2018.
[24] PENG Z, QI Z, ZHENG S, et al. Text classification improved by integrating bidirectional LSTM with two-dimen sional max pooling[J]. arXiv:1611.06639, 2016.
[25] TAN Z, WANG M, XIE J, et al. Deep semantic role labeling with self-attention[C]//Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[26] SNELL J, SWERSKY K, ZEMEL R S. Prototypical networks for few-shot learning[J]. arXiv:1703.05175, 2017.
[27] QI H, BROWN M, LOWE D G. Low-shot learning with imprinted weights[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 5822-5830.