Multi-Hop Knowledge Base Question Answering with Pre-Trained Language Model Feature Enhancement

doi:10.3778/j.issn.1002-8331.2311-0459

Abstract

Abstract: Knowledge base question answering (KBQA) is a challenging and popular research direction. The main challenge of multi-hop knowledge base question answering is the inconsistency between unstructured natural language questions and structured knowledge base reasoning paths. The multi-hop knowledge base question answering model based on graph retrieval is good at grasping the topological structure of the graph, but ignores the text information carried by the nodes and edges in the graph. In order to fully learn the text information of knowledge base triples, this paper constructs the text form of knowledge base triples and proposes three feature enhancement models RBERT, CBERT, and GBERT based on non-graph retrieval. The three feature models respectively use feedforward neural networks, deep pyramid convolutional networks, and graph attention networks to enhance features. The three models significantly improve feature representation capabilities and question and answer accuracy. RBERT has the simplest structure, CBERT is the fastest in training, and GBERT has the best performance. Experimental comparisons are conducted on the datasets MetaQA, WebQSP and CWQ, the three models are significantly better than the current mainstream models on the two indicators of Hits@1 and F1, and are also significantly better than other BERT improved models.

Key words: knowledge base question answering, pre-trained language model, feature enhancement

摘要： 知识库问答（knowledge base question answering，KBQA）是一个具有挑战性的热门研究方向，多跳知识库问答主要的挑战是非结构化的自然语言问题与结构化的知识库推理路径存在不一致性，基于图检索的多跳知识库问答模型善于把握图的拓扑结构，但忽略了图中结点和边携带的文本信息。为了充分学习知识库三元组的文本信息，构造了知识库三元组的文本形式，并提出了三个基于非图检索的特征增强模型RBERT、CBERT、GBERT，它们分别使用前馈神经网络、深层金字塔卷积网络、图注意力网络增强特征。三个模型显著提高了特征表示能力和问答准确率，其中RBERT结构最简单，CBERT训练最快，GBERT性能最优。在数据集MetaQA、WebQSP和CWQ上进行实验对比，在Hits@1和F1两个指标上三个模型明显优于目前的主流模型，也明显优于其他BERT的改进模型。

关键词: 多跳知识库问答, 预训练语言模型, 特征增强

WEI Qianqiang, ZHAO Shuliang, LU Danqi, JIA Xiaowen, YANG Shilong. Multi-Hop Knowledge Base Question Answering with Pre-Trained Language Model Feature Enhancement[J]. Computer Engineering and Applications, 2024, 60(22): 184-196.

魏谦强, 赵书良, 卢丹琦, 贾晓文, 杨世龙. 预训练语言模型特征增强的多跳知识库问答[J]. 计算机工程与应用, 2024, 60(22): 184-196.

References

[1] 曹书林, 史佳欣, 侯磊, 等. 知识库问答研究进展与展望[J]. 计算机学报, 2023, 46(3): 512-539.
CAO S L, SHI J X, HOU L, et al. Research progress and prospect of knowledge base question answering[J]. Chinese Journal of Computers, 2023, 46(3): 512-539.
[2] YAN Y, LI R, WANG S, et al. Large-scale relation learning for question answering over knowledge bases with pre-trained language models[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021: 3653-3660.
[3] ZHANG W, DU T, WANG J. Deep learning over multi-field categorical data: a case study on user response prediction[C]//Advances in Information Retrieval: Proceedings of the 38th European Conference on IR Research, 2016: 45-57.
[4] JOHNSON R, ZHANG T. Deep pyramid convolutional neural networks for text categorization[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017: 562-570.
[5] VELICKOVIC P, CUCURULL G, CASANOVA A, et al. Graph attention networks[C]//Proceedings of the 6th International Conference on Learning Representations, 2018: 340-354.
[6] GU Y, KASE S E, VANNI M T, et al. Beyond i.i.d.: three levels of generalization for question answering on knowledge bases[C]//Proceedings of the Web Conference 2021, 2021: 3477-3488.
[7] PEREZ J, ARENAS M, GUTIERREZ C. Semantics and complexity of SPARQL[C]//Proceedings of the 5th International Semantic Web Conference, 2006: 30-43.
[8] YIH S W, CHANG M W, HE X, et al. Semantic parsing via staged query graph generation: question answering with knowledge base[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, 2015: 1321-1331.
[9] LAN Y, JIANG J. Query graph generation for answering multi-hop complex questions from knowledge bases[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 969-974.
[10] DAS R, ZAHEER M, THAI D, et al. Case-based reasoning for natural language queries over knowledge bases[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021: 9594-9611.
[11] DAS R, GODBOLE A, DHULIAWALA S, et al. A simple approach to case-based reasoning in knowledge bases[C]// Proceedings of the 2020 Conference on Automated Knowledge Base Construction, 2020.
[12] SUN H T, DHINGRA B, ZAHEER M, et al. Open domain question answering using early fusion of knowledge bases and text[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018: 4231-4242.
[13] SUN H T, BEDRAX-WEISS T, COHEEN W W. PullNet: open domain question answering with iterative retrieval on knowledge bases and text[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019: 2380-2390.
[14] SAXENA A, TRIPATHI A, TALUKDAR P. Improving multi-hop question answering over knowledge graphs using knowledge base embeddings[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 4498-4507.
[15] HE G L, LAN Y S, JIANG J, et al. Improving multi-hop knowledge base question answering by learning intermediate supervision signals[C]//Proceedings of the 14th ACM International Conference on Web Search and Data Mining, 2021: 553-561.
[16] SAXENA A, KOCHSIEK A, GEMULLA R. Sequence-to-sequence knowledge graph completion and question answering[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022: 2814-2828.
[17] ZHANG J, ZHANG X, YU J, et al. Subgraph retrieval enhanced model for multi-hop knowledge base question answering[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022: 5773-5784.
[18] OGUZ B, CHEN X, KARPUKHIN V, et al. UniK-QA: unified representations of structured and unstructured knowledge for open-domain question answering[C]//Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, 2022: 1535-1546.
[19] YU D, ZHANG S, NG P, et al. DecAF: Joint decoding of answers and logical forms for question answering over knowledge bases[C]//Proceedings of the 11th International Conference on Learning Representations, 2023.
[20] HAVELIWALA T H. Topic-sensitive PageRank[C]//Proceedings of the 11th International Conference on World Wide Web, 2002: 517-526.
[21] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019: 4171-4186.
[22] WU S, HE Y. Enriching pre-trained language model with entity information for relation classification[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019: 2361-2364.
[23] KIM Y. Convolutional neural networks for sentence classification[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, 2014: 1746-1751.
[24] JOHNSON R, ZHANG T. Effective use of word order for text categorization with convolutional neural networks[C]// Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics, 2015: 103-112.
[25] ZHANG Y, DAI H, KOZAREVA Z, et al. Variational reasoning for question answering with knowledge graph[C]//Proceedings of the 32nd AAAI Conference on Artificial InTelligence, 2018.
[26] TALMOR A, BERANT J. The Web as a knowledge-base for answering complex questions[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018: 641-651.
[27] MILLER A, FISCH A, DODGE J, et al. Key-value memory networks for directly reading documents[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016: 1400-1409.
[28] ANDERSEN R, CHUNG F, LANG K. Local graph partitioning using PageRank vectors[C]//Proceedings of the 2006 47th Annual IEEE Symposium on Foundations of Computer Science, 2006: 475-486.
[29] COHEN W W, SUN H, HOFER R A, et al. Scalable neural methods for reasoning with a symbolic knowledge base[C]//Proceedings of the 8th International Conference on Learning Representations, 2020: 26-30.
[30] PAT V, HAITIAN S, LIVIO B S, et al. Adaptable and interpretable neural memory over symbolic knowledge[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics, 2021: 3678-3691.
[31] CHEN Y, WU L, ZAKI M J. Bidirectional attentive memory networks for question answering over knowledge bases[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019: 2913-2923.
[32] SEN P, SAFFARI, OLIYA A. Expanding end-to-end question answering on differentiable knowledge graphs with intersection[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021: 8805-8812.
[33] SHI J, CAO S, HOU L, et al. TransferNet-an effective and transparent framework for multi-hop question answering over relation graph[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021: 4149-4158.
[34] SHUANG C, QIAN L, ZHIWEI Y, et al. ReTraCk: a flexible and efficient framework for knowledge base question answering[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 325-336.
[35] DAS R, GODBOLE A, NAIK A, et al. Knowledge base question answering by case-based reasoning over subgraphs[C]//Proceedings of the 39th International Conference on Machine Learning, 2022: 4777-4793.
[36] BHUTANI N, ZHENG X, JAGADISH H V. Learning to answer complex questions over knowledge bases with query composition[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019: 739-748.
[37] TAN Y, MIN D, LI Y, et al. Evaluation of ChatGPT as a question answering system for answering complex questions[J]. arXiv:2303.07992, 2023.
[38] JIANG J, ZHOU K, DONG Z, et al. StructGPT: a general framework for large language model to reason over structured data[J]. arXiv:2305.09645, 2023.
[39] ATIF F, EL KHATIB O, DIFALLAH D. BeamQA: multi-hop knowledge graph question answering with sequence-to-sequence prediction and beam search[C]//Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023: 781-790.
[40] LUO H, TANG Z, PENG S, et al. ChatKBQA: a generate-then-retrieve framework for knowledge base question answering with fine-tuned large language models[J]. arXiv:2310.08975, 2023.
[41] RAFFEL C, SHAZEER N, ROBERTS A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer[J]. The Journal of Machine Learning Research, 2020, 21(140): 1-67.
[42] HUDSON D, MANNING C D. Learning by abstraction: the neural state machine[C]//Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019: 5901-5914.
[43] 张鹤译, 王鑫, 韩立帆, 等. 大语言模型融合知识图谱的问答系统研究[J]. 计算机科学与探索, 2023, 17(10): 2377-2388.
ZHANG H Y, WANG X, HAN L F, et al. Research on question answering system on joint of knowledge graph and large language models[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(10): 2377-2388.