Multi-Embedding Representation Entity Alignment Method Based on Image Fusion Information

doi:10.3778/j.issn.1002-8331.2304-0260

Abstract

Abstract: Entity alignment is the key step of knowledge graph fusion technology. However, existing methods fail to make full use of graph data when processing cross-language graph. Hence, the paper proposes a multi-embedding representation entity alignment method based on image fusion. This method obtains text embeddings from different angles, enriches text embeddings with image data, and realizes multi-modal information fusion to complete entity alignment across linguistic knowledge graph. The image generation model is used to solve the problem of incomplete entity image coverage, and the high-quality entity image information is obtained by the iterative strategy to expand the seed sequence pairs in the cross-language knowledge graph. In order to better apply the knowledge graph fusion process in the real world, the method transforms the alignment phase into a binary graph matching problem. The proposed method is experimentally analyzed on a public data set, and the experimental results show the good performance of the method. The ablation experiment also verifies the effectiveness of each module, and provides the parameter selectivity for different situations.

Key words: entity alignment, knowledge graph, knowledge fusion, cross-language

摘要： 实体对齐是知识图谱融合技术的关键环节，然而现有方法在处理跨语言图谱时未能充分利用图谱数据，在此提出一种方法融合图像信息的多嵌入表示实体对齐方法。该方法从不同角度获取文本嵌入，并利用图像数据丰富文本嵌入，实现多模态信息融合以完成跨语言图谱的实体对齐任务。通过图像生成模型解决实体图像覆盖不完全问题，结合迭代策略获得高质量实体图像信息以扩充跨语言知识图谱中种子序列对。为了更好适用现实世界真实知识图谱融合过程，该方法将对齐阶段转换为二分图匹配问题。提出的方法在公开数据集上进行了实验分析，实验结果表明了方法的良好性能，还通过消融实验验证各模块的有效性，并针对不同情况提供了参数的可选择性。

关键词: 实体对齐, 知识图谱, 知识融合, 跨语言

LIU Chunmei, GAO Yongbin, YU Wenjun. Multi-Embedding Representation Entity Alignment Method Based on Image Fusion Information[J]. Computer Engineering and Applications, 2024, 60(15): 111-121.

刘春梅, 高永彬, 余文俊. 融合图像信息的多嵌入表示实体对齐方法[J]. 计算机工程与应用, 2024, 60(15): 111-121.

References

[1] BIZER C, LEHMANN J, KOBILAROV G, et al. Dbpedia-a crystallization point for the web of data[J]. Journal of Web Semantics, 2009, 7(3): 154-165.
[2] SUCHANEK F M, KASNECI G, WEIKUM G. YAGO: a large ontology from Wikipedia and wordnet[J]. Journal of Web Semantics, 2008, 6(3): 203-217.
[3] BOLLACKER K, EVANS C, PARITOSH P, et al. Freebase: a collaboratively created graph database for structuring human knowledge[C]//Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, 2008: 1247-1250.
[4] MOUSSALLEM D, WAUER M, NGOMO A C N. Machine translation using semantic web technologies: a survey[J]. Journal of Web Semantics, Social Science Electronic Publishing, 2018(51): 1-19.
[5] HUANG X, ZHANG J, LI D, et al. Knowledge graph embedding based question answering[C]//Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, 2019: 105-113.
[6] BORDES A, USUNIER N, GARCIA-DURAN A, et al. Translating embeddings for modeling multi-relational data[C]//Advances in Neural Information Processing Systems, 2013.
[7] CHEN M, TIAN Y, YANG M, et al. Multilingual knowledge graph embeddings for cross-lingual knowledge alignment[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence, 2017: 1511-1517.
[8] SUN Z, HU W, ZHANG Q, et al. Bootstrapping entity alignment with knowledge graph Embedding[C]//Proceedings of the 27th International Joint Conferences on Artificial Intelligence. [S.l.]: AAAI Press, 2018: 4396-4402.
[9] PEI S, YU L, ZHANG X. Improving cross-lingual entity alignment via optimal transport[C]//Proceedings of the 28th International Joint Conferences on Artificial Intelligence, 2019: 3231-3237.
[10] ZHU H, XIE R, LIU Z, et al. Iterative entity alignment via knowledge embeddings[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence, 2017: 4258-4264.
[11] SUN Y, ZHANG X, LYU Y, Relation-specific network for entity alignment[C]//Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021: 1647-1651.
[12] WU Z, PAN S, CHEN F, et al. A comprehensive survey on graph neural networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(1): 4-24.
[13] MARCHEGGIANI D, TITOV I. Encoding sentences with graph convolutional networks for semantic role labeling[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017: 1506-1515.
[14] TRISEDYA B D, QI J, ZHANG R, et al. Entity alignment between knowledge graphs using attribute embeddings[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2019: 297-304.
[15] WANG Z, LV Q, LAN X, et al. Cross-lingual knowledge graph alignment via graph convolutional networks[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018: 349-357.
[16] WU Y, LIU X, FENG Y, et al. Jointly learning entity and relation representations for entity alignment[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019: 240-249.
[17] WU Y, LIU X, FENG Y, et al. Relation-aware entity alignment for heterogeneous knowledge graphs[C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, 2019: 5278-5284.
[18] SUN Z, WANG C, HU W, et al. Knowledge graph alignment network with gated multi-hop neighborhood aggregation[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020: 222-229.
[19] 苏佳林, 王元卓, 靳小龙, 等. 融合语义和结构信息的知识图谱实体对齐[J]. 山西大学学报(自然科学版), 2019, 42(1): 23-30.
SU J L, WANG Y Z, JIN X L, et al. Knowledge graph entity alignment with semantic and structural information[J]. Journal of Shanxi University (Natural Science Edition), 2019, 42(1): 23-30.
[20] CHEN L, LI Z, WANG Y, et al. MMEA: entity alignment for multi-modal knowledge graph[C]//Proceedings of the 13th International Conference on Knowledge Science, Engineering and Management (KSEM’2020), 2020: 134-147.
[21] 车超, 刘迪. 基于双向对齐与属性信息的跨语言实体对齐[J]. 计算机工程, 2022, 48(3): 74-80.
CHE C, LIU D. Cross-language entity alignment based on bidirectional alignment and attribute information[J]. Computer Engineering, 2022, 48(3): 74-80.
[22] ZENG W, ZHAO X, TANG J, . Collective entity alignment via adaptive features[C]//International Conference on Data Engineering, 2020: 1870-1873.
[23] YANG K, LIU S, ZHAO J, et al. Cotsae: co-training of structure and attribute embeddings for entity alignment[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020: 3025-3032.
[24] ZHU Y, LIU H, WU Z, et al. Relation-aware neighborhood matching model for entity alignment[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2021: 4749-4756.
[25] 张清恒. 基于多视图知识图谱嵌入的实体对齐技术研究[D]. 南京: 南京大学, 2020.
ZHANG Q H. Research on entity alignment technology based on multiview knowledge graph embedding[D]. Nanjing: Nanjing University, 2020.
[26] NGUYEN T T, HUYNH T T, YIN H, et al. Entity alignment for knowledge graphs with multi-order convolutional networks[J]. IEEE Transactions on Knowledge and Data Engineering, 2022, 34(9): 4201-4214.
[27] ZENG W, ZHAO X, TANG J, et al. Reinforcement learning-based collective entity alignment with adaptive features[J]. ACM Transactions on Information Systems, 2021, 39(3): 26-31.
[28] POWELL M. The theory of radial basis function approximation in 1990[C]//Advances in Numerical Analysis, 1992.
[29] GE C, LIU X, CHEN L, et al. Make it easy: an effective end-to-end entity alignment framework[C]//Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, New York, NY, USA, 2021: 777-786.
[30] XIANG Y, ZHANG Z, CHEN J, et al. OntoEA: ontology guided entity alignment via joint knowledge graph embedding[C]//Findings of the Association for Computational Linguistics (ACL-IJCNLP 2021), 2021.
[31] CAI W, MA W, ZHAN J, et al. Entity alignment with reliable path reasoning and relation-aware heterogeneous graph transformer[C]//Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, 2022: 1930-1937.
[32] LIN Z X, ZHANG Z H, WANG M. Multi-modal contrastive representation learning for entity alignment[C]//Proceedings of the 29th International Conference on Computational Linguistics, 2022: 2572-2584.
[33] CHEN Z, CHEN J Y, ZHANG W. MEAformer: multi-modal entity alignment transformer for meta modality hybrid[C]//Proceedings of the 31st ACM International Conference on Multimedia, 2022: 3317-3327.
[34] ZHAO X, ZENG W, TANG J, et al. An experimental study of state-of-the-art entity alignment approaches[J]. IEEE Transactions on Knowledge and Data Engineering, 2020, 34(6): 2610-2625.
[35] SRIVASTAVA R K, GREFF K, SCHMIDHUBER J, et al. Highway networks[J]. arXiv:1505.00387, 2015.
[36] MUDGAL S, LI H, REKATSINAS T, et al. Deep learning for entity matching: a design space exploration[C]//Proceedings of the 2018 International Conference on Management of Data, 2018: 19-34.
[37] LIU Z, LIN Y, CAO Y, et al. Swin Transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 10224-10233.
[38] LIU F, CHEN M, ROTH D, et al. Visual pivoting for (unsupervised) entity alignment[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2021: 4257-4266.
[39] LEHMANN J, ISELE R, JAKOB M, et al. Dbpedia: a large-scale, multilingual knowledge base extracted from wikipedia[J]. Semantic Web, 2015, 6(2): 167-195.
[40] YANG H W, ZOU Y, SHI P, et al. Aligning cross-lingual entities with multi-aspect information[C]//Proceedings of the 2019 Conferences on Empirical Methods in Natural Language Processing and the 9th International Joint Conferences on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. [S.l.]: Association of Computational Linguistics, 2019: 4431-4441.
[41] LOSHCHILOV I, HUTTER F. Decoupled weight decay regularization[C]//Proceedings of the International Conference on Learning Representations, 2019.
[42] SUN Z, ZHANG Q, HU W, et al. A benchmarking study of embedding-based entity alignment for knowledge graphs[J]. Proceedings of the VLDB Endowment, 2020(13): 2326-2340.
[43] LIU B, ZHU Y, SONG K, et al. Towards faster and stabilized GAN training for high-fidelity few-shot image synthesis[C]//International Conference on Learning Representations (ICLR), 2021.
[44] LI J, LI D, XIONG C, et al. Blip: bootstrapping language-image pre-training for unified vision-language understanding and generation[C]//International Conference on Machine Learning, 2022: 12888-12900.
[45] SUN Z, HU W, LI C. Cross-lingual entity alignment via joint attribute-preserving embedding[C]//The 16th International Semantic Web Conference, 2017: 628-644.
[46] LI C, CAO Y, HOU L, et al. Semi-supervised entity alignment via joint knowledge embedding model and cross-graph model[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. [S.l.]: Association for Computational Linguistics, 2019: 2723-2732.
[47] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.