Knowledge Graph Embedding Model Based on k-Order Sampling and Graph Attention Networks

doi:10.3778/j.issn.1002-8331.2208-0339

Abstract

Abstract: Knowledge graph embedding (KGE) aims to map entities and relations of knowledge graph into a low-dimensional space to obtain its vector representation. Existing KGE models only consider the first-order neighbors, which influence the accuracy of reasoning and prediction tasks in knowledge graph. In order to solve this problem, a novel KGE model based on [k]-order sampling algorithm and graph attention networks is proposed. Firstly, a [k]-order sampling algorithm is proposed to obtain the neighbors’ features of a central entity by aggregating [k]-order neighborhood in the pruned subgraph. Then, the graph attention networks are introduced to learn the attention values of the central entity’s neighbors, and the new entity embedding is obtained by the weighted sum of neighbors’ features. Finally, the ConvKB is used as a decoder to analyze the global embedding property of a triple. Evaluation experiments on several datasets, WN18RR, FB15k-237, NELL-995, Kinship, reveal that the model performs better than the state-of-the-art models on the task of link prediction. Besides, the influence on the model hit rate while changing order [k] or sampling coefficient [b] has been discussed.

Key words: knowledge graph embedding, [k]-order sampling algorithm, graph attention networks, pruned subgraph, link prediction

摘要： 知识图谱表示（KGE）旨在将知识图谱中的实体和关系映射到低维度向量空间而获得其向量表示。现有的KGE模型只考虑一阶近邻，这影响了知识图谱中推理和预测任务的准确性。为了解决这一问题，提出了一种基于[k]阶采样算法和图注意力网络的KGE模型。[k]阶采样算法通过聚集剪枝子图中的[k]阶邻域来获取中心实体的邻居特征。引入图注意力网络来学习中心实体邻居的注意力值，通过邻居特征加权和得到新的实体向量表示。利用ConvKB作为解码器来分析三元组的全局表示特征。在WN18RR、FB15k-237、NELL-995、Kinship数据集上的评价实验表明，该模型在链接预测任务上的性能明显优于最新的模型。此外，还讨论了阶数[k]和采样系数[b]的改变对模型命中率的影响。

关键词: 知识图谱表示, [k]阶采样算法, 图注意力网络, 剪枝子图, 链接预测

LIU Wenjie, YAO Junfei, CHEN Liang. Knowledge Graph Embedding Model Based on k-Order Sampling and Graph Attention Networks[J]. Computer Engineering and Applications, 2024, 60(2): 113-120.

刘文杰, 姚俊飞, 陈亮. k阶采样和图注意力网络的知识图谱表示模型[J]. 计算机工程与应用, 2024, 60(2): 113-120.

References

[1] GUO D, TANG D, DUAN N, et al. Dialog-to-action: conversational question answering over a large-scale knowledge base[C]//Advances in Neural Information Processing Systems, 2018:2946-2955.
[2] ZHANG N Y, CHEN X, SUN Z L, et al. Long-tail relation extraction via knowledge graph embeddings and graph convolution networks[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Jun 2-7, 2019: 3016-3025.
[3] ZHU Z H, YU J, HU Y, et al. Mucko: multi-layer cross-modal knowledge reasoning for fact-based visual question answering[C]//Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. San Francisco: Morgan Kaufmann, 2020: 1097-1103.
[4] WANG Q, MAO Z, WANG B, et al. Knowledge graph embedding: a survey of approaches and applications[J]. IEEE Transactions on Knowledge and Data Engineering, 2017, 29（12）: 2724-2743.
[5] JI S, PAN S, CAMBRIA S, et al. A survey on knowledge graphs: representation, acquisition, and applications[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 33（2）: 494-514.
[6] BORDES A, USUNIER A, et al. Translating embeddings for modeling multi-relational data[C]//Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013, Nevada, Dec 5-8, 2013: 2787-2795.
[7] LIN Y K, LIU Z Y. Modeling relation paths for representation learning of knowledge bases[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Sep 17-21, 2015: 705-714.
[8] NICKEL M, TRESP V, KRIEGEL H. A three-way model for collective learning on multi-relational data[C]//Proceedings of the 28th International Conference on Machine Learning, Washington, Jun 28-Jul 2, 2011. Madison: Omnipress, 2011: 809-816.
[9] DETTMERS T, MINERVINI P, et al. Convolutional 2D knowledge graph embeddings[C]//Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, Louisiana, Feb 2-7, 2018: 1811-1818.
[10] SCHLICHTKRULL M S, KIPF T N, BLOEM P, et al. Modeling relational data with graph convolutional networks[C]//The Semantic Web-15th International Conference, Heraklion, Jun 3-7, 2018. Berlin: Springer, 2018: 593-607.
[11] VASHISHTH S, SANYAL S, NITIN V, et al. Composition-based multi-relational graph convolutional networks[C]//8th International Conference on Learning Representations, Ethiopia, Apr 26-30, 2020.
[12] THOMAS N, WELLING M. Semi-supervised classification with graph convolutional networks[C]//5th International Conference on Learning Representations, Toulon, Apr 24-26, 2017.
[13] VELICKOVIC P, CUCURULL G. Graph attention networks[J]. arXiv:1710.10903, 2017.
[14] NATHANI D, CHAUHAN J, SHARMA C, et al. Learning attention-based embeddings for relation prediction in knowledge graphs[C]//Proceedings of the 57th Conference of the Association for Computational Linguistics, Florence, Jul 28-Aug 2, 2019: 4710-4723.
[15] NGUYEN D Q, NGUYEN T D. A novel embedding model for knowledge base completion based on convolutional neural network[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Louisiana, Jun 1-6, 2018: 327-333.
[16] LIN Y K, LIU Z Y, SUN M S, et al. Learning entity and relation embeddings for knowledge graph completion[C]//Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Texas, Jan 25-30, 2015: 2181-2187.
[17] JI G L, HE S Z, XU L, et al. Knowledge graph embedding via dynamic mapping matrix[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, Beijing, Jul 26-31, 2015: 687-696.
[18] WANG Z, ZHANG J W, FENG J L, et al. Knowledge graph embedding by translating on hyperplanes[C]//Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, Canada, Jul 27-31, 2014: 1112-1119.
[19] GUO L B, SUN Z Q, HU W. Learning to exploit long-term relational dependencies in knowledge graphs[C]//Proceedings of the 36th International Conference on Machine Learning, California, Jun 9-15, 2019: 2505-2514.
[20] YANG B S, HE X D. Embedding entities and relations for learning and inference in knowledge bases[C]//3rd International Conference on Learning Representations, San Diego, May 7-9, 2015.
[21] TROUILLON T, RIEDEL J. Complex embeddings for simple link prediction[C]//Proceedings of the 33nd International Conference on Machine Learning, New York, Jun 19-24, 2016: 2071-2080.
[22] NICKEL M, ROSASCO L. Holographic embeddings of knowledge graphs[C]//Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, Feb 12-17, 2016: 1955-1961.
[23] SHANG C, TANG Y. End-to-end structure-aware convolutional networks for knowledge base completion[C]//The Thirty-Third AAAI Conference on Artificial Intelligence, Honolulu, Jan 27-Feb 1, 2019: 3060-3067.
[24] SHEIKH N, QIN X. Knowledge graph embedding using graph convolutional networks with relation-aware attention[J]. arXiv:2102.07200, 2021.
[25] WANG M H, LI H. Learning knowledge graph embeddings by multi-attention mechanism for link prediction[C]//Algorithms and Architectures for Parallel Processing-21st International Conference, Dec 3-5, 2021. Berlin: Springer, 2021: 33-49.
[26] ZHANG Z, ZHUANG F Z. Relational graph neural network with hierarchical attention for knowledge graph completion[C]//The Thirty-Fourth AAAI Conference on Artificial Intelligence, New York, 2021: 9612-9619.
[27] TOUTANOVA K, CHEN D Q. Representing text for joint embedding of text and knowledge bases[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Sep 17-21, 2015: 1499-1509.
[28] BOLLACKER K D, EVANS C, PARITOSH P K, et al. Freebase: a collaboratively created graph database for structuring human knowledge[C]//Proceedings of the ACM SIGMOD International Conference on Management of Data, Vancouver, Jun 10-12, 2008. New York: ACM, 2008: 1247-1250.
[29] XIONG W H, THIEN H. DeepPath: a reinforcement learning method for knowledge graph reasoning[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Sep 9-11, 2017: 564-573.
[30] CARLSON A, BETTERIDGE J, KISIEL B, et al. Toward an architecture for never-ending language learning[C]//Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, Atlanta, Jul 11-15, 2010.
[31] LIN X V, SOCHER R, XIONG C, et al. Multi-hop knowledge graph reasoning with reward shaping[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Oct 31-Nov 4, 2018: 3243-3253.