MSIM： Multi-Stage Inference Knowledge Graph Question Answering Model with Integrated Attention Mechanism

doi:10.3778/j.issn.1002-8331.2409-0192

Abstract

Abstract: Multi-hop knowledge graph question answering (KGQA) aims to retrieve relevant entities from a knowledge graph based on user queries. To address the heterogeneity between node embeddings and question embeddings, this paper introduces a message encoder that integrates these embeddings by adding spatial position encoding to the graph structure. A novel node embedding initialization method, relation frequency-inverse general entity frequency(RF-IGEF) is also proposed to enhance the initialization process, mitigating issues like low weight extraction and synonym relation overshadowing. The resulting MSIM model outperforms benchmarks on popular KGQA datasets, achieving significant improvements in H@1 and F@1 metrics. Specifically, MSIM shows up to 1.2 and 2 percentage points gains in H@1 and F@1, respectively, on the WebQuestionsSP dataset, and up to 1.1 and 0.3 percentage points on the ComplexWebQuestions dataset. It also scores 97.5% and 100% on the Meta-QA 1-hop and 2-hop datasets.

Key words: knowledge graph, graph neural network, question answering inference model, attention mechnism

摘要： 多跳知识图谱问答任务是根据用户输入的自然语言提问从知识图谱中检索对应的实体。鉴于现有方法在使用知识图谱的节点嵌入与问题文本指令嵌入时存在异构问题，提出了消息编码器来融合两种异构的嵌入，此编码器通过对图结构增加空间位置编码使序列嵌入与图消息融合。并引入了一种新颖的节点嵌入初始化策略——关系频率-逆实体频率（RF-IGEF）。从而改进节点嵌入初始化策略，防止KGQA嵌入初始化方法存在的嵌入提取权重过小以及同义关系被覆盖等缺陷。结合以上两种方法提出的MSIM模型在流行知识图谱问答数据集对比H@1和F@1这两个关键性能指标，MSIM模型均展现出优于基准模型的表现。具体来说，与近两年的模型相比，MSIM模型在WebQuestionSP数据集中H@1指标上最高提升了1.2个百分点，在F@1指标上最高提升了2个百分点。在ComplexWebQuestions数据集中H@1指标上最高提升了1.1个百分点，在F@1指标上最高提升了0.3个百分点。在Meta-QA1-hop、2-hop数据集中分别取得97.5%与100%的优良成绩。

关键词: 知识图谱, 图神经网络, 问答推理模型, 注意力机制

QIU Tianbo, ZHANG Dong, LI Guanyu. MSIM： Multi-Stage Inference Knowledge Graph Question Answering Model with Integrated Attention Mechanism[J]. Computer Engineering and Applications, 2025, 61(24): 144-153.

邱天搏, 张东, 李冠宇. MSIM:融合注意力机制的多阶段推理知识图谱问答模型[J]. 计算机工程与应用, 2025, 61(24): 144-153.

References

[1] MAVROMATIS C, KARYPIS G. ReaRev: adaptive reasoning for question answering over knowledge graphs[C]//Proceedings of the Findings of the Association for Computational Linguistics. Stroudsburg: ACL, 2022: 2447-2458.
[2] QIAO Z L, YE W, ZHANG T, et al. Exploiting hybrid semantics of relation paths for multi-hop question answering over knowledge graphs[C]//Proceedings of the 29th International Conference on Computational Linguistics, 2022: 1813-1822.
[3] ATZENI M, BOGOJESKA J, LOUKAS A. SQALER: scaling question answering by decoupling multi-hop and logical reasoning[C]//Advances in Neural Information Processing Systems, 2021: 12587-12599.
[4] PARK J, CHOI H K, KO J, et al. Relation-aware language-graph transformer for question answering[C]//Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, 2023: 13457-13464.
[5] ZHANG X K, BOSSELUT A, YASUNAGA M, et al. GreaseLM: graph REASoning enhanced language models for question answering[J]. arXiv:2201.08860, 2022.
[6] BOLLACKER K, EVANS C, PARITOSH P, et al. Freebase: a collaboratively created graph database for structuring human knowledge[C]//Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. New York: ACM, 2008: 1247-1250.
[7] VRANDE?I? D, KR?TZSCH M. Wikidata: a free collaborative knowledgebase[J]. Communications of the ACM, 2014, 57(10): 78-85.
[8] CHOI H K, LEE S, CHU J, et al. NuTrea: neural tree search for context-guided multi-hop KGQA[C]//Proceedings of the 37th International Conference on Neural Information Processing Systems, 2024: 1559-1570.
[9] PéREZ J, ARENAS M, GUTIERREZ C. Semantics and complexity of SPARQL[J]. ACM Transactions on Database Systems, 2009, 34(3): 16-60.
[10] BERANT J, CHOU A, FROSTIG R, et al. Semantic parsing on freebase from question-answer pairs[C]//Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2013: 1533-1544.
[11] REDDY S, LAPATA M, STEEDMAN M. Large-scale semantic parsing without question-answer pairs[J]. Transactions of the Association for Computational Linguistics, 2014, 2: 377-392.
[12] DAS R, ZAHEER M, THAI D, et al. Case-based reasoning for natural language queries over knowledge bases[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2021: 9594-9611.
[13] YU D H, ZHANG S, NG P, et al. DecAF: joint decoding of answers and logical forms for question answering over knowledge bases[J]. arXiv:2210.00063, 2022.
[14] XIONG G M, BAO J W, ZHAO W. Interactive-KBQA: multi-turn interactions for knowledge base question answering with large language models[C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2024: 10561-10582.
[15] SUN H T, BEDRAX-WEISS T, COHEN W. PullNet: open domain question answering with iterative retrieval on knowledge bases and text[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Stroudsburg: ACL, 2019: 2380-2390.
[16] SUN H T, DHINGRA B, ZAHEER M, et al. Open domain question answering using early fusion of knowledge bases and text[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2018: 4231-4242.
[17] HE G L, LAN Y S, JIANG J, et al. Improving multi-hop knowledge base question answering by learning intermediate supervision signals[C]//Proceedings of the 14th ACM International Conference on Web Search and Data Mining. New York: ACM, 2021: 553-561.
[18] MILLER A, FISCH A, DODGE J, et al. Key-value memory networks for directly reading documents[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2016: 1400-1409.
[19] LAN Y S, WANG S H, JIANG J. Knowledge base question answering with topic units[C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, 2019: 5046-5052.
[20] BHUTANI N, ZHENG X Y, JAGADISH H V. Learning to answer complex questions over knowledge bases with query composition[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management. New York: ACM, 2019: 739-748.
[21] WANG K, DUAN F, WANG S, et al. Knowledge-driven CoT: exploring faithful reasoning in LLMs for knowledge-intensive question answering[J]. arXiv:2308. 13259, 2023.
[22] LI T L, MA X G, ZHUANG A, et al. Few-shot in-context learning on knowledge base question answering[C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2023: 6966-6980.
[23] YIH W, RICHARDSON M, MEEK C, et al. The value of semantic parse labeling for knowledge base question answ-ering[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016: 201-206.
[24] TALMOR A, BERANT J. The web as a knowledge-base for answering complex questions[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg: ACL, 2018: 641-651.
[25] ZHANG Y, DAI H, KOZAREVA Z, et al. Variational rea-soning for question answering with knowledge graph[C]//Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, 2018: 5946-5953.
[26] SHI J X, CAO S L, HOU L, et al. TransferNet: an effective and transparent framework for multi-hop question answering over relation graph[J]. arXiv:2104. 07302, 2021.