结合语言模型双编码和坐标注意力卷积的知识图谱补全

doi:10.3778/j.issn.1002-8331.2404-0273

摘要/Abstract

摘要： 知识图谱补全（KGC）旨在学习知识图谱中的现有知识实现对缺失三元组的补全。近期的相关研究表明，将语言模型（LM）应用于KGC任务能够改善模型在结构稀疏的知识图谱上的推理性能。针对现有结合LM的KGC模型性能仅依赖于LM捕获的语义特征，没有同时考虑知识图谱的结构信息和语义信息的问题，提出一种结合语言模型双编码和坐标注意的知识图谱补全方法LDCA。在编码时，通过引入掩码预训练的语言模型双编码结构，充分学习实体和关系的语义特征；在解码时，使用坐标注意力机制的卷积神经网络捕获实体和关系组合嵌入的跨通道信息、方向感知信息和位置感知信息。在WN18RR和FB15K-237数据集上的实验结果表明，LDCA模型在MR、MRR、Hits@1、Hits@3和Hits@10上的整体性能优于基准模型，验证了所提出模型的有效性和先进性。

关键词: 语言模型（LM）, 掩码预训练, 坐标注意力机制, 卷积神经网络

Abstract: Knowledge graph completion (KGC) aims to complete missing triples by learning existing knowledge in the knowledge graph. Recent studies have shown that applying language models (LM) to KGC tasks can improve the model’s inference performance on sparsely structured knowledge graphs. Aiming at the problem that the performance of existing KGC models combined with LMs only relies on the semantic features captured by LMs, without taking into account both the structural and semantic information of knowledge graphs, a knowledge graph completion method LDCA combining language model dual encoding and coordinate attention is proposed. In coding, the dual coding structure of language model with mask pre-training is introduced to fully learn the semantic features of entities and relationships. In decoding, a convolutional neural network using a coordinate attention mechanism captures cross-channel information, directional perception information, and location-aware information embedded by a combination of entities and relationships. Experimental results on WN18RR and FB15K-237 datasets show that the overall performance of LDCA model is superior to that of the benchmark model on MR, MRR, Hits@1, Hits@3 and Hits@10, which verifies the validity and advance of the proposed model.

Key words: language model (LM), mask pre-training, coordinate attention mechanism, convolutional neural network

王瑄, 王晓霞, 陈晓. 结合语言模型双编码和坐标注意力卷积的知识图谱补全[J]. 计算机工程与应用, 2025, 61(14): 206-213.

WANG Xuan, WANG Xiaoxia, CHEN Xiao. Combining Language Model Dual Encoder and Coordinate Attention Convolution for Knowledge Graph Completion[J]. Computer Engineering and Applications, 2025, 61(14): 206-213.

参考文献

[1] TAMA?AUSKAIT? G, GROTH P. Defining a knowledge graph development process through a systematic review[J]. ACM Transactions on Software Engineering and Methodology, 2023, 32(1): 1-40.
[2] PELLISSIER T T, WEIKUM G, SUCHANEK F. YAGO 4: a reasonable knowledge base[C]//Proceedings of the 17th International Conference on European Semantic Web Conference, 2020: 583-596.
[3] ALMOUSA M, BENLAMRI R, KHOURY R. A novel word sense disambiguation approach using WordNet knowledge graph[J]. Computer Speech & Language, 2022, 74: 101337.
[4] SINGH K, LYTRA I, RADHAKRISHNA A S, et al. No one is perfect: analysing the performance of question answering components over the DBpedia knowledge graph[J]. Journal of Web Semantics, 2020, 65: 100594.
[5] ARNAOUT H, RAZNIEWSKI S, WEIKUM G, et al. Negative knowledge for open-world wikidata[C]//Proceedings of the Web Conference. New York: ACM, 2021: 544-551.
[6] ISSA S, ADEKUNLE O, HAMDI F, et al. Knowledge graph completeness: a systematic literature review[J]. IEEE Access, 2021, 9: 31322-31339.
[7] 张明星, 张骁雄, 刘姗姗, 等. 利用知识图谱的推荐系统研究综述[J]. 计算机工程与应用, 2023, 59(4): 30-42.
ZHANG M X, ZHANG X X, LIU S S, et al. Review of recommendation systems using knowledge graph[J]. Computer Engineering and Applications, 2023, 59(4): 30-42.
[8] 于梦波, 杜建强, 罗计根, 等. 基于知识表示学习的知识图谱补全研究进展[J]. 计算机工程与应用, 2023, 59(18): 59-73.
YU M B, DU J Q, LUO J G, et al. Research progress of knowledge graph completion based on knowledge representation learning[J]. Computer Engineering and Applications, 2023, 59(18): 59-73.
[9] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[J]. arXiv:1810.04805, 2018.
[10] YAO L, MAO C S, LUO Y. KG-BERT: BERT for knowledge graph completion[J]. arXiv:1909.03193, 2019.
[11] ZHANG Z Y, LIU X Q, ZHANG Y, et al. Pretrain-KGE: learning knowledge representation from pretrained language models[C]//Findings of the Association for Computational Linguistics. Stroudsburg: ACL, 2020: 259-266.
[12] BORDES A, USUNIER N, GARCIA-DURAN A, et al. Translating embeddings for modeling multi-relational data[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems, 2013: 2787-2795.
[13] CHURCH K W. Word2Vec[J]. Natural Language Engineering, 2017, 23(1): 155-162.
[14] WANG Z, ZHANG J W, FENG J L, et al. Knowledge graph embedding by translating on hyperplanes[C]//Proceedings of the 28th AAAI Conference on Artificial Intelligence, 2014: 1112 - 1119.
[15] LIN Y, LIU Z, SUN M, et al. Learning entity and relation embeddings for knowledge graph completion[C]//Proceedings of the 29th AAAI Conference on Artificial Intelligence, 2015: 2181-2187.
[16] SUN Z Q, DENG Z H, NIE J Y, et al. RotatE: knowledge graph embedding by relational rotation in complex space[J]. arXiv:1902.10197, 2019.
[17] YANG B S, YIH W T, HE X D, et al. Embedding entities and relations for learning and inference in knowledge bases[J]. arXiv:1412.6575, 2014.
[18] TROUILLON T, WELBL J, RIEDEL S, et al. ComplEx embeddings for simple link prediction[C]//Proceedings of the International Conference on Machine Learning, 2016: 2071-2080.
[19] TRAN H N, TAKASU A. Multi-partition embedding interaction with block term format for knowledge graph completion[J]. arXiv:2006.16365, 2020.
[20] TRAN H N, TAKASU A. MEIM: multi-partition embedding interaction beyond block term format for efficient and expressive link prediction[J]. arXiv:2209.15597, 2022.
[21] SCHLICHTKRULL M, KIPF T N, BLOEM P, et al. Modeling relational data with graph convolutional networks[C]//Proceedings of the 15th International Conference on the Semantic Web. Cham: Springer International Publishing, 2018: 593-607.
[22] DETTMERS T, MINERVINI P, STENETORP P, et al. Convolutional 2D knowledge graph embeddings[J]. arXiv:1707. 01476, 2017.
[23] NGUYEN D Q, NGUYEN D Q, NGUYEN T D, et al. A convolutional neural network-based model for knowledge base completion and its application to search personalization[J]. Semantic Web, 2019, 10(5): 947-960.
[24] MOHAMED H A, PILUTTI D, JAMES S, et al. Locality-aware subgraphs for inductive link prediction in knowledge graphs[J]. Pattern Recognition Letters, 2023, 167: 90-97.
[25] KIM B, HONG T, KO Y, et al. Multi-task learning for knowledge graph completion with pre-trained language models[C]//Proceedings of the 28th International Conference on Computational Linguistics, 2020: 1737-1743.
[26] CHEN S X, LIU X D, GAO J F, et al. HittER: hierarchical transformers for knowledge graph embeddings[J]. arXiv:2008.12813, 2020.
[27] BOLLACKER K, EVANS C, PARITOSH P K, et al. Freebase: a collaboratively created graph database for structuring human knowledge[C]//Proceedings of the ACM SIGMOD International Conference on Management of Data, 2008: 1247-1280.
[28] ABBOUD R, CEYLAN I, LUKASIEWICZ T, et al. BoxE: a box embedding model for knowledge base completion[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020: 9649-9661.
[29] WANG Y C, GE X O, WANG B, et al. GreenKGC: a lightweight knowledge graph completion method[C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2023: 10596-10613.
[30] LUO Y, YANG C M, LI B, et al. CP tensor factorization for knowledge graph completion[C]//Proceedings of the Knowledge Science, Engineering and Management. Cham: Springer International Publishing, 2022: 240-254.
[31] VASHISHTH S, SANYAL S, NITIN V, et al. InteractE: improving convolution-based knowledge graph embeddings by increasing feature interactions[J]. arXiv:1911.00219, 2019.
[32] XIE X, ZHANG N, LI Z, et al. From discrimination to generation: knowledge graph completion with generative transformer[J]. arXiv:2022.02113, 2020.