Factual Consistency Assessment Evaluation Model for Text Summarization Based on Multi-Attention Mechanism

doi:10.3778/j.issn.1002-8331.2208-0167

Abstract

Abstract: Factual consistency of text summarization is consistent with information of the source document and the summarization. Recent researches have shown that there are large number of factual inconsistencies existed in the outputs of abstractive summarization models. It is important to design a method that can detect and evaluate the error of fact inconsistency. Most of existing methods based on natural language inference have insufficient ability to extract key content of source document and infer the information of content. This paper improves the accuracy of consistency assessment model for summarization by multi-attention mechanism. Firstly, the key sentences are selected according to sentence-BERT, which is fine-turning on pre-trained language model. Each evidence and claim are formed into separate sentence pairs and sent to the BERT encoder to obtain the vectors of the sentence pairs representation, then it achieves evidence reasoning by the vectors combined with ESIM. Finally, the graph attention network is used to complete the aggregation of inferential information to obtain the fact consistency assessment result. Experimental results show that this algorithm is compared with several typical algorithms in common datasets in the field, and its feasibility and effectiveness are verified.

Key words: text summarization, factual consistency, BERT, graph attention network, interactive attention

摘要： 文本摘要事实一致性是摘要内容与源文档内容的信息一致。最近的研究表明，文本摘要模型生成的摘要存在较多与原文事实不一致的问题，设计能够检测并评估出事实不一致错误的方法至关重要。目前基于自然语言推理的方法存在对源文档内容提取简单，推理信息交互不充分等问题。提出多注意力机制的文本摘要事实一致性评估模型，利用预训练模型微调的sentence-BERT模型挑选源文档中的关键句，然后将摘要句与关键句组合成句子对，输入BERT模型编码获得向量表示结合ESIM进行句子对的推理，利用图注意力网络完成推理信息的聚合，提高文本摘要事实一致性评估模型的准确率。实验结果表明，该算法与多个典型算法在在领域内常用的数据集进行实验比较，其可行性和有效性得到验证。

关键词: 文本摘要, 事实一致性, BERT, 图注意力网络, 交互注意力

WEI Chuyuan, ZHANG Xinxian, WANG Zhiyuan, LI Jinzhe, LIU Jie. Factual Consistency Assessment Evaluation Model for Text Summarization Based on Multi-Attention Mechanism[J]. Computer Engineering and Applications, 2023, 59(7): 163-170.

魏楚元, 张鑫贤, 王致远, 李金哲, 刘杰. 多注意力机制的文本摘要事实一致性评估模型[J]. 计算机工程与应用, 2023, 59(7): 163-170.

References

[1] SEE A，LIU P J，MANNING C D.Get to the point：summarization with pointer-generator networks[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics，2017：1073-1083.
[2] DEVLIN J，CHANG M W，LEE K，et al.BERT：pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2019：4171-4186.
[3] KRYSCINSKI W，MCCANN B，XIONG C，et al.Evaluating the factual consistency of abstractive text summarization[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing，2020：9332-9346.
[4] CAO Z，WEI F，LI W，et al.Faithful to the original：fact aware neural abstractive summarization[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence，2018.
[5] 谢逸宽.生成式摘要中的事实不一致性问题研究[D].成都：电子科技大学，2022.
XIE Y K.Research on the faithfulness problem of abstractive summarization[D].Chengdu：University of Electronic Science and Technology of China，2022.
[6] 林泓宇.文本摘要生成中的事实性问题研究[D].成都：电子科技大学，2022.
LIN H Y.Research on factual problems in text summarization[D].Chengdu：University of Electronic Science and Technology of China，2022.
[7] PAPINENI K，ROUKOS S，WARD T，et al.Bleu：a method for automatic evaluation of machine translation[C]//Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics，2002：311-318.
[8] LIN C Y.Rouge：a package for automatic evaluation of summaries[C]//Proceedings of the Text Summarization Branches Out，2004：74-81.
[9] ZHANG T，KISHORE V，WU F，et al.BERTScore：evaluating text generation with BERT[C]//Proceedings of the International Conference on Learning Representations，2019.
[10] ZHAO W，PEYRARD M，LIU F，et al.MoverScore：text generation evaluating with contextualized embeddings and earth mover distance[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing（EMNLP-IJCNLP），2019：563-578.
[11] GOODRICH B，RAO V，LIU P J，et al.Assessing the factual accuracy of generated text[C]//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining，2019：166-175.
[12] ETZIONI O，BANKO M，SODERLAND S，et al.Open information extraction from the Web[J].Communications of the ACM，2008，51（12）：68-74.
[13] NAN F，NALLAPATI R，WANG Z，et al.Entity-level factual consistency of abstractive text summarization[C]//Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics，2021：2727-2733.
[14] WANG A，CHO K，LEWIS M.Asking and answering questions to evaluate the factual consistency of summaries[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics，2020：5008-5020.
[15] DURMUS E，HE H，DIAB M.FEQA：a question answering evaluation framework for faithfulness assessment in abstractive summarization[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics，2020：5055-5070.
[16] FALKE T，RIBEIRO L F R，UTAMA P A，et al.Ranking generated summaries by correctness：an interesting but challenging application for natural language inference[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics，2019：2214-2220.
[17] MISHRA A，PATEL D，VIJAYAKUMAR A，et al.Looking beyond sentence-level natural language inference for downstream tasks[J].arXiv：2009.09099，2020.
[18] ZHANG S，NIU J，WEI C.Fine-grained factual consistency assessment for abstractive summarization models[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing，2021：107-116.
[19] REIMERS N，GUREVYCH I.Sentence-BERT：sentence embeddings using siamese BERT-networks[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing（EMNLP-IJCNLP），2019：3982-3992.
[20] LIU Y，LAPATA M.Text summarization with pretrained encoders[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing（EMNLP-IJCNLP），2019：3730-3740.
[21] CHEN Q，ZHU X，LING Z H，et al.Enhanced LSTM for natural language inference[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics，2017：1657-1668.
[22] ZHOU J，HAN X，YANG C，et al.GEAR：graph-based evidence aggregating and reasoning for fact verification[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics，2019：892-901.
[23] HERMANN K M，KOCISKY T，GREFENSTETTE E，et al.Teaching machines to read and comprehend[C]//Advances in Neural Information Processing Systems，2015.
[24] TRISCHLER A，WANG T，YUAN X，et al.NewsQA：a machine comprehension dataset[C]//Proceedings of the 2nd Workshop on Representation Learning for NLP，2017：191-200.
[25] LEWIS M，LIU Y，GOYAL N，et al.BART：denoising sequence-to-sequence pre-training for natural language generation，translation，and comprehension[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics，2020：7871-7880.
[26] RAJPURKAR P，JIA R，LIANG P.Know what you don’t know：unanswerable questions for SQuAD[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics，2018：784-789.
[27] DEMSZKY D，GUU K，LIANG P.Transforming question answering datasets into natural language inference datasets[J].arXiv：1809.02922，2018.
[28] RAJPURKAR P，ZHANG J，LOPYREV K，et al.SQuAD：100，000+ questions for machine comprehension of text[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing，2016：2383-2392.