Text Similarity Analysis Algorithm Combining Attention Mechanism and MatchPyramid

doi:10.3778/j.issn.1002-8331.2102-0221

Abstract

Abstract: Text similarity analysis is the core task in the field of natural language processing, and text similarity analysis based on deep text matching model is the main idea of this task. Aiming at the shortcomings of traditional MatchPyramid model in text feature extraction, a text similarity analysis method based on enhanced MatchPyramid model is proposed. In order to reduce the computational complexity of the model, multi-head self-attention mechanism and mutual attention mechanism are added to the input encoding layer, and autoencoder is used to reduce the dimension of the input word vector of dual attention mechanism. Then, the output of the dual attention mechanism is connected with the original word vector to improve the representation ability of the word vector to the key information of the text. Finally, the single channel graph formed by the dot product of the word vector matrix of two texts is mapped to multiple feature subspaces to form a multi-channel graph, and the dense connected convolutional neural network is used to extract the features of the multi-channel graph. The experimental results show that compared with the traditional MatchPyramid model, the accuracy of the proposed model is improved by 1.59?percentage points, and the F1 value is improved by 2.49 percentage points.

Key words: text similarity, attention mechanism, MatchPyramid, convolutional neural network

摘要： 文本相似度分析是自然语言处理领域的核心任务，基于深度文本匹配模型进行文本相似度分析是当前研究该任务的主流思路。针对传统的MatchPyramid模型对文本特征提取的不足之处进行改进，提出了基于增强MatchPyramid模型进行文本相似度分析的方法。该方法在输入编码层加入多头自注意力机制和互注意力机制，同时对双注意力机制的输入词向量使用自编码器做降维处理，以降低模型的计算量。接着将双注意力机制的输出与原始词向量相连接，提升了词向量对文本关键信息的表征能力。最后将两个文本的词向量矩阵点积形成的单通道图映射到多个特征子空间形成了多通道图，使用密集连接的卷积神经网络对多通道图进行特征提取。实验结果表明，相比于传统的MatchPyramid模型，所提出的模型准确率提升了1.59个百分点，F1值提升了2.49个百分点。

关键词: 文本相似度, 注意力机制, MatchPyramid, 卷积神经网络

DAI Xiang, SUN Haichun, ZHU Rongchen, SUN Tianyang. Text Similarity Analysis Algorithm Combining Attention Mechanism and MatchPyramid[J]. Computer Engineering and Applications, 2022, 58(19): 158-165.

代翔, 孙海春, 朱容辰, 孙天杨. 联合注意力机制与MatchPyramid的文本相似度分析算法[J]. 计算机工程与应用, 2022, 58(19): 158-165.

References

[1] 王春柳，杨永辉，邓霏，等.文本相似度计算方法研究综述[J].情报科学，2019，37（3）：158-168.
WANG C L，YANG Y H，DENG F，et al.Review of text similarity calculation methods[J].Information Science，2019，37（3）：158-168.
[2] MIKOLOV T，CHEN K，CORRADO G，et al.Efficient estimation of word representations in vector space[EB/OL].（2013-09-07）[2021-04-10].https：//arxiv.org/abs/1301.3781.
[3] PENNINGTON J，SOCHER R，MANNING C D.Glove：Global vectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing（EMNLP），2014：1532-1543.
[4] 刘继明，于敏敏，袁野.基于句向量的文本相似度计算方法[J].科学技术与工程，2020，20（17）：6950-6955.
LIU J M，YU M M，YUAN Y.Text similarity calculation method based on sentence vector[J].Science，Technology and Engineering，2020，20（17）：6950-6955.
[5] HUANG P S，HE X D，GAO J F，et al.Learning deep structured semantic models for web search using clickthrough data[C]//Proceedings of the 22nd ACM International Conference on Information & Knowledge Management，2013：2333-2338.
[6] SHEN Y L，HE X D，GAO J F，et al.A latent semantic model with convolutional-pooling structure for information retrieval[C]//Proceedings of Conference on Information and Knowledge Management，2014：101-110.
[7] PALANGI H，DENG L，SHEN Y，et al.Semantic modelling with long-short-term memory for information retrieval[EB/OL].（2015-02-27）[2021-04-10].https：//arxiv.org/abs/1412.6629.
[8] HU B，LU Z，LI H，et al.Convolutional neural network architectures for matching natural language sentences[C]//Advances in Neural Information Processing Systems，2014：2042-2050.
[9] YIN W P，HINRICH SCHüTZE，XIANG B，et al.ABCNN：Attention-based convolutional neural network for modeling sentence pairs[J].Transactions of the Association for Computational Linguistics，2016，4：259-272
[10] WAN S，LAN Y，GUO J，et al.A deep architecture for semantic matching with multiple positional sentence representations[C]//Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence，2016：2835-2841.
[11] LIANG P，LAN Y，GUO J，et al.Text matching as image recognition[C]//Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence，2016：2793-2799.
[12] MUELLER J，THYAGARAJAN A.Siamese recurrent architectures for learning sentence similarity[C]//Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence，2016：2786-2792.
[13] PARIKH A P，T?CKSTR?M O，DAS D，et al.A decomposable attention model for natural language inference[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing，2016：2249-2255.
[14] WANG S，JIANG J.A compare-aggregate model for matching text sequences.（2017）[C]//Proceedings of the International Conference on Learning Representations，Toulon，France，April 24-26，2017：1-15.
[15] NIE Y，BANSAL M.Shortcut-stacked sentence encoders for multi-domain inference[C]//Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP，2017：41-45.
[16] CHEN Q，ZHU X，LING Z，et al.Enhanced LSTM for Natural Language Inference[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics，2017：1657-1668.
[17] WANG Z，HAMZA W，FLORIAN R.Bilateral multi-perspective matching for natural language sentences[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence，2017：4144-4150.
[18] TAY Y，TUAN L A，HUI S C.Multi-cast attention networks[C]//Proceedings of International Conference on Knowledge Discovery & Data Mining，2018：2299-2308.
[19] CHEN Q，Hu Q，HUANG J X，et al.CAN：Enhancing sentence similarity modeling with collaborative and adversarial network[C]//Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval，2018：815-824.
[20] KIM S，KANG I，KWAK N.Semantic sentence matching with densely-connected recurrent and coattentive information[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2019：6586-6593.
[21] 赵琪，杜彦辉，芦天亮，等.基于Capsule-BiGRU的文本相似度分析算法[J].计算机工程与应用，2021，57（15）：171-177.
ZHAO Q，DU Y H，LU T L，et al.Algorithm of text similarity analysis based on Capsule-BiGRU[J].Computer Engineering and Applications，2021，57（15）：171-177.
[22] ZHOU L，TANG，JIAN，et al.Jointly considering siamese network and matchpyramid network for text semantic matching[J].IOP Conference Series（Materials Science and Engineering），2019，490（4）：042043.