联合预训练模型和层级注意力的知识超图链接预测

doi:10.3778/j.issn.1002-8331.2401-0234

摘要/Abstract

摘要： 知识超图（knowledge hypergraph，KHG）是超图结构的知识图谱。现有知识超图链接预测模型主要存在以下不足：模型输入时将实体和关系简单地表示为嵌入层的ID（索引），而没有考虑实体和关系之间复杂的联系和语义；编码时只考虑位置和角色信息，而忽略了实体邻域结构和多元关系间的联系，导致实体和关系表示能力不足；模型训练时采样的负样本质量不够高，不能帮助模型高效学习样本特征。针对以上问题，提出一种联合预训练模型和层级注意力的知识超图链接预测模型（link prediction in knowledge hypergraph combining pretrained model and hierarchical attention，LPPH）。该模型引入预训练模型和简化的团式展开方法初始化超图嵌入，将实体和关系之间复杂联系和语义融入至实体和关系嵌入中；编码时使用层级注意力机制聚合实体邻域结构信息以增强实体表示，并使用实体-关系融合操作增强关系表示；提出一种基于过滤机制和主动学习的负样本选择策略，实现模型的高效训练。真实数据集上的大量实验结果验证了LPPH能有效提高知识超图链接预测的效果。

关键词: 知识超图, 链接预测, 预训练模型, 层级注意力

Abstract: Knowledge hypergraph (KHG) is the knowledge graph of hypergraph structure. Existing methods of KHG link prediction mainly have the following shortcomings: Entities and relations are simply represented as the ID (index) of the embedding layer during input, without considering the complex relations and semantics between them. Only location and role information are taken into account when encoding, while disregarding the connection between entity neighborhood structure and multiple relations, resulting in insufficient representation ability of entities and relations. The quality of negative samples sampled during model training is not high enough to help the model learn sample features efficiently. To address these issues, a link prediction method in knowledge hypergraph combining pretrained model and hierarchical attention (LPPH) is proposed. The model utilizes a pretrained model and a simplified clique expansion method to initialize hypergraph embeddings, integrating complex relations and semantics between entities and relation into their respective embeddings. When encoding, a hierarchical attention mechanism is employed to aggregate information from entity neighborhood structures for enhanced entity representation, and an entity-relation fusion operation is used to enhance relation representation. Finally, a negative sample selection strategy based on filtering mechanism and active learning is proposed to achieve efficient training of the model. Extensive experimental results on real datasets demonstrate that LPPH can effectively improve the efficiency of knowledge hypergraph link prediction.

Key words: knowledge hypergraph, link prediction, pretrained model, hierarchical attention

庞俊, 梅杰, 林晓丽, 王蒙湘. 联合预训练模型和层级注意力的知识超图链接预测[J]. 计算机工程与应用, 2025, 61(10): 133-144.

PANG Jun, MEI Jie, LIN Xiaoli, WANG Mengxiang. Link Prediction in Knowledge Hypergraph Combining Pretrained Model and Hierarchical Attention[J]. Computer Engineering and Applications, 2025, 61(10): 133-144.

参考文献

[1] GU Y, YU K Q, SONG Z, et al. Distributed hypergraph processing using intersection graphs[J]. IEEE Transactions on Knowledge and Data Engineering, 2022, 34(7): 3182-3195.
[2] YU D H, GU Y, XIONG C Y, et al. CompleQA: benchmarking the impacts of knowledge graph completion methods on question answering[C]//Findings of the Association for Computational Linguistics: EMNLP 2023. Stroudsburg: ACL, 2023: 12748-12755.
[3] BORDES A, USUNIER N, GARCIA-DURáN A, et al. Translating embeddings for modeling multi-relational data[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems, 2013: 2787-2795.
[4] GROVER A, LESKOVEC J. node2vec: scalable feature learning for networks[C]//Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York: ACM, 2016: 855-864.
[5] 梁梅霖, 段友祥, 昌伦杰, 等. 邻域信息分层感知的知识图谱补全方法[J]. 计算机工程与应用, 2024, 60(2): 147-153.
LIANG M L, DUAN Y X, CHANG L J, et al. Knowledge graph completion method based on neighborhood hierarchical perception[J]. Computer Engineering and Applications, 2024, 60(2): 147-153.
[6] WANG Z, ZHANG J W, FENG J L, et al. Knowledge graph embedding by translating on hyperplanes[C]//Proceedings of the 28th AAAI Conference on Artificial Intelligence, 2014: 1112-1119.
[7] LIN Y K, LIU Z Y, SUN M S, et al. Learning entity and relation embeddings for knowledge graph completion[C]//Proceedings of the 29th AAAI Conference on Artificial Intelligence, 2015: 2181-2187.
[8] XIAO H, HUANG M L, ZHU X Y, et al. From one point to a manifold[C]//Proceedings of the 25th International Joint Conference on Artificial Intelligence, 2016: 1315-1321.
[9] SUN Z, DENG Z H, NIE J Y, et al. RotatE: knowledge graph embedding by relational rotation in complex space[C]//Proceedings of the 6th International Conference on Learning Representations, 2018.
[10] CHAMI I, WOLF A, JUAN D C, et al. Low-dimensional hyperbolic knowledge graph embeddings[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2020: 6901-6914.
[11] NICKEL M, TRESP V, KRIEGEL H P, et al. A three-way model for collective learning on multi-relational data[C]//Proceedings of the 28th International Conference on International Conference on Machine Learning, 2011: 809-816.
[12] YANG B, YIH S W, HE X, et al. Embedding entities and relations for learning and inference in knowledge bases[C]//Proceedings of the 3rd International Conference on Learning Representations, 2015.
[13] TROUILLON T, WELBL J, RIEDEL S, et al. Complex embeddings for simple link prediction[C]//Proceedings of the 33rd International Conference on Machine Learning, 2016: 2071-2080.
[14] KAZEMI S M, POOLE D, KAZEMI S M, et al. SimplE embedding for link prediction in knowledge graphs[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018: 4289-4300.
[15] BALAZEVIC I, ALLEN C, HOSPEDALES T. TuckER: tensor factorization for knowledge graph completion[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Stroudsburg: ACL, 2019: 5184-5193.
[16] DETTMERS T, MINERVINI P, STENETORP P, et al. Convolutional 2D knowledge graph embeddings[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence and 30th Innovative Applications of Artificial Intelligence Conference and 8th AAAI Symposium on Educational Advances in Artificial Intelligence, 2018: 1811-1818.
[17] NGUYEN D Q, NGUYEN T D, NGUYEN D Q, et al. A novel embedding model for knowledge base completion based on convolutional neural network[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg: ACL, 2018: 327-333.
[18] JIANG X T, WANG Q, WANG B. Adaptive convolution for multi-relational learning[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg: ACL, 2019: 978-987.
[19] BALA?EVI? I, ALLEN C, HOSPEDALES T M. Hypernetwork knowledge graph embeddings[C]//Proceedings of the 28th International Conference on Artificial Neural Networks. Cham: Springer, 2019: 553-565.
[20] VASHISHTH S, SANYAL S, NITIN V, et al. InteractE: improving convolution-based knowledge graph embeddings by increasing feature interactions[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(3): 3009-3016.
[21] SCHLICHTKRULL M, KIPF T N, BLOEM P, et al. Modeling relational data with graph convolutional networks[C]//Proceedings of the 15th International Conference on Semantic Web. Cham: Springer, 2018: 593-607.
[22] WEN J F, LI J X, MAO Y Y, et al. On the representation and embedding of knowledge bases beyond binary relations[C]//Proceedings of the 25th International Joint Conference on Artificial Intelligence, 2016: 1300-1307.
[23] ZHANG R C, LI J P, MEI J J, et al. Scalable instance reconstruction in knowledge bases via relatedness affiliated embedding[C]//Proceedings of the 2018 World Wide Web Conference. New York: ACM, 2018: 1185-1194.
[24] LIU Y, YAO Q M, LI Y, et al. Generalizing tensor decomposition for N-ary relational knowledge bases[C]//Proceedings of the Web Conference 2020. New York: ACM, 2020: 1104-1114.
[25] PANG J, QIN H C, LIU Y, et al. Two birds with one stone: a link prediction model for knowledge hypergraph based on fully-connected tensor decomposition[C]//Proceedings of the 19th International Conference on Advanced Data Mining and Applications. Cham: Springer, 2023: 78-90.
[26] FATEMI B, TASLAKIAN P, VAZQUEZ D, et al. Knowledge hypergraphs: prediction beyond binary relations[C]//Proceedings of the 29th International Joint Conference on Artificial Intelligence, 2020: 2191-2197.
[27] LIU Y, YAO Q M, LI Y, et al. Role-aware modeling for N-ary relational knowledge bases[C]//Proceedings of the Web Conference 2021. New York: ACM, 2021: 2660-2671.
[28] CHEN Z R, WANG X, WANG C X, et al. PosKHG: a position-aware knowledge hypergraph model for link prediction[J]. Data Science and Engineering, 2023, 8(2): 135-145.
[29] GUAN S P, JIN X L, WANG Y Z, et al. Link prediction on N-ary relational data[C]//Proceedings of the World Wide Web Conference 2019. New York: ACM, 2019: 583-593.
[30] GUAN S P, JIN X L, GUO J F, et al. Link prediction on N-ary relational data based on relatedness evaluation[J]. IEEE Transactions on Knowledge and Data Engineering, 2023, 35(1): 672-685.
[31] ROSSO P, YANG D Q, CUDRé-MAUROUX P, et al. Beyond triplets: hyper-relational knowledge graph embedding for link prediction[C]//Proceedings of the Web Conference 2020. New York: ACM, 2020: 1885-1896.
[32] GUAN S P, JIN X L, GUO J F, et al. NeuInfer: knowledge inference on N-ary facts[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2020: 6141-6151.
[33] HU Z W, GUTIéRREZ-BASULTO V, XIANG Z L, et al. HyperFormer: enhancing entity and relation interaction for hyper-relational knowledge graph completion[C]//Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. New York: ACM, 2023: 803-812.
[34] GALKIN M, TRIVEDI P, MAHESHWARI G, et al. Message passing for hyper-relational knowledge graphs[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2020: 7346-7359.
[35] WANG C X, LI Z, WANG X, et al. EnhancE: enhanced entity and relation embedding for knowledge hypergraph link prediction[C]//Proceedings of the ACM Web Conference 2023. New York: ACM, 2023: 115-118.
[36] WANG C X, WANG X, LI Z, et al. HyConvE: a novel embedding model for knowledge hypergraph link prediction with convolutional neural networks[C]//Proceedings of the ACM Web Conference 2023. New York: ACM, 2023: 188-198.
[37] LI Z, WANG C X, WANG X, et al. HJE: joint convolutional representation learning for knowledge hypergraph completion[J]. IEEE Transactions on Knowledge and Data Engineering, 2024, 36(8): 3879-3892.
[38] YADATI N, NIMISHAKAVI M, YADAV P, et al. HyperGCN: a new method of training graph convolutional networks on hypergraphs[C]//Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019: 1511-1522.
[39] 郭正山, 左劼, 段磊, 等. 面向知识超图链接预测的生成对抗负采样方法[J]. 计算机研究与发展, 2022, 59(8): 1742-1756.
GUO Z S, ZUO J, DUAN L, et al. A generative adversarial negative sampling method for knowledge hypergraph link prediction[J]. Journal of Computer Research and Development, 2022, 59(8): 1742-1756.