Multi-Label Text Classification Model Combining CNN-SAM and GAT

doi:10.3778/j.issn.1002-8331.2109-0195

Abstract

Abstract: The existing research methods of multi-label text classification based on neural network have two shortcomings：one is that they can not fully extract text information features, and the other is that they rarely mine the association between global labels from graph structure data. To solve the above two problems, this paper proposes a multi-label text classification model（CS-GAT） integrating convolutional neural network self attention mechanism and graph attention network. The model uses multi-layer convolutional neural network and self attention mechanism to fully extract and fuse the local and global information of the text, so as to obtain a more comprehensive feature vector representation. At the same time, the relevance between different text labels is transformed into an edge weighted graph with global information. The multi-layer graph attention mechanism is used to automatically learn the degree of association between different labels, and then interact with the text context semantic information to obtain the global label information representation with text semantic connection. Finally, the adaptive fusion strategy is used to further extract the feature information of the two models to improve the generalization ability of the model. The experimental results on three open English data sets, AAPD, RCV1-V2 and EUR-Lex, show that the multi-label classification effect achieved by this model is significantly better than other mainstream baseline models.

Key words: multi-label text classification, multi-layer convolutional neural network, self attention mechanism, multi-headed graph attention mechanism

摘要： 现有基于神经网络的多标签文本分类研究方法存在两方面不足，一是不能全面提取文本信息特征，二是很少从图结构数据中挖掘全局标签之间的关联性。针对以上两个问题，提出融合卷积神经网络-自注意力机制（CNN-SAM）与图注意力网络（GAT）的多标签文本分类模型（CS-GAT）。该模型利用多层卷积神经网络与自注意力机制充分提取文本局部与全局信息并进行融合，得到更为全面的特征向量表示；同时将不同文本标签之间的关联性转变为具有全局信息的边加权图，利用多层图注意力机制自动学习不同标签之间的关联程度，将其与文本上下文语义信息进行交互，获取具有文本语义联系的全局标签信息表示；使用自适应融合策略进一步提取两者特征信息，提高模型的泛化能力。在AAPD、RCV1-V2与EUR-Lex三个公开英文数据集上的实验结果表明，该模型所达到的多标签分类效果明显优于其他主流基线模型。

关键词: 多标签文本分类, 多层卷积神经网络, 自注意力机制, 多头图注意力机制

YANG Chunxia, MA Wenwen, CHEN Qigang, GUI Qiang. Multi-Label Text Classification Model Combining CNN-SAM and GAT[J]. Computer Engineering and Applications, 2023, 59(5): 106-114.

杨春霞, 马文文, 陈启岗, 桂强. 融合CNN-SAM与GAT的多标签文本分类模型[J]. 计算机工程与应用, 2023, 59(5): 106-114.

References

[1] PARWEZ M A，ABULAISH M.Multi-label classification of microblogging texts using convolution neural network[J].IEEE Access，2019，7：68678-68691.
[2] 刘敬学，孟凡荣，周勇，等.字符级卷积神经网络短文本分类算法[J].计算机工程与应用，2019，55（5）：135-142.
LIU Jingxue，MENG Fanrong，ZHOU Yong，et al.Character-level convolution neural networks for short text classification[J].Computer Engineering and Applications，2019，55（5）：135-142.
[3] LIU Z，LU C，HUANG H，et al.Text classification based on multi-granularity attention hybrid neural network[J].arXiv：2008.05282，2020.
[4] 王浩镔，胡平.采用多级特征的多标签长文本分类算法[J].计算机工程与应用，2021，57（15）：193-199.
WANG Haobin，HU Ping.Multi-label long text classification algorithm based on multi-level features[J].Computer Engineering and Applications，2021，57（15）：193-199.
[5] 肖琳，陈博理，黄鑫，等.基于标签语义注意力的多标签文本分类[J].软件学报，2020，31（4）：1079-1089.
XIAO Lin，CHEN Boli，HUANG Xin，et al.Multi-lable text classification method based on label semantic information[J].Journal of Software，2020，31（4）：1079-1089.
[6] ZHANG M L，ZHOU Z H.A review on multi-label learning algorithms[J].IEEE Transactions on Knowledge and Data Engineering，2014，26（8）：1819-1837.
[7] KIM Y.Convolutional neural networks for sentence classification[C]//Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing，2014：1746-1751.
[8] LIU J，CHANG W C，WU Y，et al.Deep learning for extreme multi-label text classification[C]//The 40th International ACM SIGIR Conference，2017：115-124.
[9] YOU R，DAI S，ZHANG Z，et al.AttentionXML：extreme multi-label text classification with multi-label attention based recurrent neural networks[J].arXiv：1811.01727，2018.
[10] YANG Z，LIU G.Hierarchical sequence-to-sequence model for multi-label text classification[J].IEEE Access，2019，7：153012-153020.
[11] XIAO L，HUANG X，CHEN B，et al.Label-specific document representation for multi-label text classification[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing（EMNLP-IJCNLP），2019：466-475.
[12] YOU R，ZHANG Z，WANG Z，et al.AttentionXML：label tree-based attention-aware deep model for high-performance extreme multi-label text classification[C]//Advances in Neural Information Processing Systems，2019：5820-5830.
[13] HUANG X，CHEN B，XIAO L，et al.Label-aware document representation via hybrid attention for extreme multi-label text classification[J].Neural Processing Letters，2022，54（5）：3601-3617.
[14] XIAO L，ZHANG X，JING L，et al.Does head label help for long-tailed multi-label text classification[J].arXiv：2101.09704，2021.
[15] YAO L，MAO C，LUO Y.Graph convolutional networks for text classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2019：7370-7377.
[16] PENNINGTON J，SOCHER R，MANNING C.GloVe：global vectors for word representation[C]//Conference on Empirical Methods in Natural Language Processing，2014：1532-1543.
[17] 金旺，易国洪，洪汉玉，等.基于卷积神经网络的实时车辆检测[J].计算机工程与应用，2021，57（5）：222-228.
JIN Wang，YI Guohong，HONG Hanyu，et al.Real-time vehicle detection based on convolutional neural network[J].Computer Engineering and Applications，2021，57（5）：222-228.
[18] DU C，CHEN Z，FENG F，et al.Explicit interaction model towards text classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2019：6359-6366.
[19] YANG P，SUN X，LI W，et al.SGM：sequence generation model for multi-label classification[C]//Proceedings of the 27th International Conference on Computational Linguistics，2018：3915-3926.