Research and Comprehensive Review on Multi-Modal Knowledge Graph Fusion Techniques

doi:10.3778/j.issn.1002-8331.2309-0481

Abstract

Abstract: Multi-modal knowledge graphs (MMKG) integrate various modal information such as vision and text, presenting knowledge structures graphically. With the advancement of artificial intelligence, MMKG have played a significant role in recommendation systems, intelligent Q&A, and knowledge search among other fields. Compared to traditional knowledge graphs, MMKG can understand and present knowledge in multiple dimensions, possessing superior representation and application capabilities. To delve deep into the study of MMKG, this review first conducts a detailed analysis and elucidation of the value and categories of MMKG. Based on different construction methods, it compares and summarizes multi-modal knowledge extraction, representation learning, entity alignment, and other aspects, categorizes multi-modal knowledge integration methods. It analyzes the progress in the applications of MMKG, discusses the limitations of MMKG, and proposes future research directions in the field of MMKG.

Key words: multi-modal knowledge graph, language model, fusion techniques, pretraining techniques

摘要： 多模态知识图谱融合了视觉、文本等多种模态信息，并以图的形式展现知识结构。随着人工智能的发展，多模态知识图谱在推荐系统、智能问答和知识搜索等领域发挥了重要作用。与传统知识图谱相比，多模态知识图谱可以多维度理解和展现知识，有更好的表示和应用能力。为了深入研究多模态知识图谱，对多模态知识图谱价值及类别进行了详细的分析与阐述，根据多模态知识图谱构建中融合方法的不同，从多源异构数据文本转换、表示学习、实体对齐、特征抽取方面进行对比和总结，重点对跨模态知识图谱融合技术分类叙述。对多模态知识图谱的应用进展进行了分析，并探讨了多模态知识图谱的局限性，提出了多模态知识图谱领域今后的研究方向。

关键词: 多模态知识图谱, 语言模型, 融合技术, 预训练技术

CHEN Youren, LI Yong, WEN Ming, SUN Chi. Research and Comprehensive Review on Multi-Modal Knowledge Graph Fusion Techniques[J]. Computer Engineering and Applications, 2024, 60(13): 36-50.

陈囿任, 李勇, 温明, 孙驰. 多模态知识图谱融合技术研究综述[J]. 计算机工程与应用, 2024, 60(13): 36-50.

References

[1] JI S, PAN S, CAMBRIA E, et al. A survey on knowledge graphs: representation, acquisition, and applications[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 33(2): 494-514.
[2] CHEN Y H, LI H, LIU W H, et al. An overview of knowledge graph reasoning: key technologies and applications[J]. Journal of Sensor and Actuator Networks, 2022, 11(4): 78.
[3] SHARMA K, GIANNAKOS M. Multimodal data capabilities for learning: what can multimodal data tell us about learning?[J]. British Journal of Educational Technology, 2020, 51(5): 1450-1484.
[4] ZHU X, LI Z, WANG X, et al. Multi-modal knowledge graph construction and application: a survey[J]. IEEE Transactions on Knowledge & Data Engineering, 2022(1): 1-20.
[5] 陈烨, 周刚, 卢记仓. 多模态知识图谱构建与应用研究综述[J]. 计算机应用研究, 2021, 38(12): 3535-3543.
CHEN Y, ZHOU G, LU J C. Survey on construction and application research for multi-modal knowledge graphs[J]. Application Research of Computers, 2021, 38(12): 3535-3543.
[6] CHEN X, JIA S, XIANG Y. A review: knowledge reasoning over knowledge graph[J]. Expert Systems with Applications, 2020, 141: 112948.
[7] ZHANG N, LI L, CHEN X, et al. Multimodal analogical reasoning over knowledge graphs[C]//Proceedings of the Eleventh International Conference on Learning Representations, 2022.
[8] BOEHM K M, KHOSRAVI P, VANGURI R, et al. Harnessing multimodal data integration to advance precision oncology[J]. Nature Reviews Cancer, 2022, 22(2): 114-126.
[9] WU X, DUAN J, PAN Y, et al. Medical knowledge graph: data sources, construction, reasoning, and applications[J]. Big Data Mining and Analytics, 2023, 6(2): 201-217.
[10] O?ORO-RUBIO D, NIEPERT M, GARCíA-DURáN A, et al. Answering visual-relational queries in web-extracted knowledge graphs[C]//Proceedings of the 1st Conference on Automated Knowledge Base Construction (AKBC), 2019.
[11] LIU Y, LI H, GARCIA-DURAN A, et al. MMKG: multi-modal knowledge graphs[C]//Proceedings of the 16th International Conference on Semantic Web (ESWC), 2019: 459-474.
[12] FERRADA S, BUSTOS B, HOGAN A. IMGpedia: a linked dataset with content-based analysis of Wikimedia images[C]//Proceedings of the 16th International Semantic Web Conference, Vienna, Austria, Oct 21-25, 2017. Cham: Springer, 2017: 84-93.
[13] ALBERTS H T, DESHPANDE Y R, et al. VisualSem: a high-quality knowledge graph for vision and language[C]//Proceedings of the 1st Workshop on Multilingual Representation Learning, 2021: 138-152.
[14] WANG M, WANG H, QI G, et al. Richpedia: a large-scale, comprehensive multi-modal knowledge graph[J]. Big Data Research, 2020, 22: 100159.
[15] LI M, ZAREIAN A, LIN Y, et al. GAIA: a fine-grained multimedia knowledge extraction system[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020: 77-86.
[16] MA Y, WANG Z, LI M, et al. MMEKG: multi-modal event knowledge graph towards universal representation across modalities[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2022: 231-239.
[17] ZHAO W, WU X. Boosting entity-aware image captioning with multi-modal knowledge graph[J]. IEEE Transactions on Multimedia, 2024, 26: 2659-2670.
[18] ZHANG J, WANG J, WANG X, et al. AspectMMKG: a multi-modal knowledge graph with aspect-aware entities[C]//Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, 2023: 3361-3370.
[19] HUANG C, WANG J, WANG S, et al. A review of deep learning in dentistry[J]. Neurocomputing, 2023: 126629.
[20] SUN R, CAO X, ZHAO Y, et al. Multi-modal knowledge graphs for recommender systems[C]//Proceedings of the 29th ACM International Conference on Information & Knowledge Management, 2020: 1405-1414.
[21] HUANG J, CHEN Y, LI Y, et al. Medical knowledge-based network for patient-oriented visual question answering[J]. Information Processing & Management, 2023, 60(2): 103241.
[22] ZHU J, HUANG C, DE MEO P. DFMKE: a dual fusion multi-modal knowledge graph embedding framework for entity alignment[J]. Information Fusion, 2023, 90: 111-119.
[23] SUMMAIRA J, LI X, SHOIB A M, et al. Recent advances and trends in multimodal deep learning: a review[J]. arXiv:2105.11087, 2021.
[24] LEE S J, RHO M. Multimodal deep learning applied to classify healthy and disease states of human microbiome[J]. Scientific Reports, 2022, 12(1): 1-11.
[25] GAO J, LI P, CHEN Z, et al. A survey on deep learning for multimodal data fusion[J]. Neural Computation, 2020, 32(5): 829-864.
[26] DA C, WANG P, YAO C. Levenshtein OCR[C]//Proceedings of the 17th European Conference on Computer Vision. Cham: Springer, 2022: 322-338.
[27] WEN Y, LUO B, ZHAO Y. IMKGA-SM: interpretable multimodal knowledge graph answer prediction via sequence modeling[J]. arXiv:2301.02445, 2023.
[28] LIU Z, XIAO L, CHEN J, et al. A multimodal knowledge graph for medical decision making centred around personal values[C]//Proceedings of the 2023 26th International Conference on Computer Supported Cooperative Work in Design (CSCWD), 2023: 1638-1643.
[29] XU D, XU T, WU S, et al. Relation-enhanced negative sampling for multimodal knowledge graph completion[C]//Proceedings of the 30th ACM International Conference on Multimedia, 2022: 3857-3866.
[30] 孙水发, 李小龙, 李伟生, 等. 图神经网络应用于知识图谱推理的研究综述[J]. 计算机科学与探索, 2023, 17(1): 27-52.
SUN S F, LI X L, LI W S, et al. Review of graph neural networks applied to knowledge graph reasoning[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(1): 27-52.
[31] WANG Z, ZHANG J, FENG J, et al. Knowledge graph embedding by translating on hyperplanes[C]//Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, 2014: 1112-1119.
[32] LIN Y, LIU Z, SUN M, et al. Learning entity and relation embeddings for knowledge graph completion[C]//Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015: 2181-2187.
[33] JI G, HE S, XU L, et al. Knowledge graph embedding via dynamic mapping matrix[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, 2015: 687-696.
[34] FUSAR-POLI P. TRANSD recommendations: improving transdiagnostic research in psychiatry[J]. World Psychiatry, 2019, 18(3): 361-362.
[35] NGUYEN D Q, SIRTS K, QU L, et al. STransE: a novel embedding model of entities and relationships in knowledge bases[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016: 460-466.
[36] EBISU T, ICHISE R. TorusE: knowledge graph embedding on a lie group[C]//Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, 2018: 1819-1826.
[37] HE S, LIU K, JI G, et al. Learning to represent knowledge graphs with Gaussian embedding[C]//Proceedings of the 24th ACM International Conference on Information and Knowledge Management, 2015: 623-632.
[38] XIAO H, HUANG M, HAO Y, et al. TransG: a generative mixture model for knowledge graph embedding[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016: 2316-2325.
[39] WANG Y, WUMAIER A, SUN W, et al. TransH-RA: a learning model of knowledge representation by hyperplane projection and relational attributes[J]. IEEE Access, 2023, 11: 29510-29520.
[40] WAN B, NIU Y, CHEN C, et al. TransRFT: a knowledge representation learning model based on a relational neighborhood and flexible translation[J]. Applied Sciences, 2023, 13(19): 10864.
[41] PEROZZI B, AL-RFOU R, SKIENA S. Deepwalk: online learning of social representations[C]//Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2014: 701-710.
[42] GROVER A, LESKOVEC J. node2vec: scalable feature learning for networks[C]//Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016: 855-864.
[43] LEE J, CHUNG C, WHANG J J. InGram: inductive knowledge graph embedding via relation graphs[J]. arXiv:2305.
19987, 2023.
[44] SOCHER R, CHEN D, MANNING C D, et al. Reasoning with neural tensor networks for knowledge base completion[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems, 2013: 926-934.
[45] YANG B, YIH W, HE X, et al. Embedding entities and relations for learning and inference in knowledge bases[C]//Proceedings of the International Conference on Learning Representations (ICLR), 2015.
[46] LI W, XIAO X, LIU J, et al. Leveraging graph to improve abstractive multi-document summarization[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 6232-6243.
[47] ZHANG Z, ZHANG F, ZHUANG F, et al. Knowledge graph error detection with hierarchical path structure[C]//Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, 2023: 4430-4434.
[48] PARK N, KAN A, DONG X L, et al. Estimating node importance in knowledge graphs using graph neural networks[C]//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019: 596-606.
[49] SCHLICHTKRULL M, KIPF T N, BLOEM P, et al. Modeling relational data with graph convolutional networks[C]//Proceedings of the 15th International Conference on Semantic Web, Heraklion, Crete, Greece, Jun 3-7, 2018. Cham: Springer, 2018: 593-607.
[50] MA T, HUANG L, LU Q, et al. KR-GCN: knowledge-aware reasoning with graph convolution network for explainable recommendation[J]. ACM Transactions on Information Systems, 2023, 41(1): 1-27.
[51] HUANG Z, LI X, YE Y, et al. Multi-view knowledge graph fusion via knowledge-aware attentional graph neural network[J]. Applied Intelligence, 2023, 53(4): 3652-3671.
[52] LIANG S, ZHU A, ZHANG J, et al. Hyper-node relational graph attention network for multi-modal knowledge graph completion[J]. ACM Transactions on Multimedia Computing, Communications and Applications, 2023, 19(2): 1-21.
[53] CHEN M, TIAN Y, YANG M, et al. Multilingual knowledge graph embeddings for cross-lingual knowledge alignment[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence, 2017: 1511-1517.
[54] 王欢, 宋丽娟, 杜方. 基于多模态知识图谱的中文跨模态实体对齐方法[J]. 计算机工程, 2023, 49(12): 88-95.
WANG H, SONG L J, DU F. Chinese cross-modal entity alignment method based on multi-modal knowledge graph[J]. Computer Engineering, 2023, 49(12): 88-95.
[55] CHEN F L, ZHANG D Z, HAN M L, et al. VLP: a survey on vision-language pre-training[J]. Machine Intelligence Research, 2023, 20(1): 38-56.
[56] CHEN X, ZHANG N, LI L, et al. Hybrid transformer with multi-level fusion for multimodal knowledge graph completion[C]//Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022: 904-915.
[57] KRICHEN M. Convolutional neural networks: a survey[J]. Computers, 2023, 12(8): 151.
[58] YU H, SUN H, TAO J, et al. A multi-stage data augmentation and AD-ResNet-based method for EPB utilization factor prediction[J]. Automation in Construction, 2023, 147: 104734.
[59] CHEN T, KORNBLITH S, NOROUZI M, et al. A simple framework for contrastive learning of visual representations[C]//Proceedings of the International Conference on Machine Learning (PMLR), 2020: 1597-1607.
[60] SINHA D, EL-SHARKAWY M. Thin MobileNet: an enhanced MobileNet architecture[C]//Proceedings of the 2019 IEEE 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), 2019: 280-285.
[61] LI R, YU J, LI F, et al. Automatic bridge crack detection using unmanned aerial vehicle and Faster R-CNN[J]. Construction and Building Materials, 2023, 362: 129659.
[62] WANG W, BAO H, DONG L, et al. Image as a foreign language: BEiT pretraining for vision and vision-language tasks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 19175-19186.
[63] YEH C H, HONG C Y, HSU Y C, et al. Decoupled contrastive learning[C]//Proceedings of the 17th European Conference on Computer Vision. Cham: Springer, 2022: 668-684.
[64] RICHEMOND P H, TAM A, TANG Y, et al. The edge of orthogonality: a simple view of what makes BYOL tick[C]//Proceedings of the International Conference on Machine Learning (PMLR), 2023: 29063-29081.
[65] PEHLIVAN H, DALVA Y, DUNDAR A. StyleRes: transforming the residuals for real image editing with StyleGAN[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 1828-1837.
[66] GANGWAR A, GONZáLEZ-CASTRO V, ALEGRE E, et al. Triple-BigGAN: semi-supervised generative adversarial networks for image synthesis and classification on sexual facial expression recognition[J]. Neurocomputing, 2023, 528: 200-216.
[67] RAHALI A, AKHLOUFI M A. End-to-end transformer-based models in textual-based NLP[J]. AI, 2023, 4(1): 54-110.
[68] ZHOU C, LI Q, LI C, et al. A comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT[J]. arXiv:2302.09419, 2023.
[69] FENG Z, ZHANG Z, YU X, et al. ERNIE-ViLG 2. 0: improving text-to-image diffusion model with knowledge-enhanced mixture-of-denoising-experts[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 10135-10145.
[70] KALE A S, PANDYA V, DI TROIA F, et al. Malware classification with Word2Vec, HMM2Vec, BERT, and ELMo[J]. Journal of Computer Virology and Hacking Techniques, 2023, 19(1): 1-16.
[71] YANG Z L, DAI Z H. XLNet: generalized autoregressive pretraining for language understanding[C]//Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019: 5753-5763.
[72] LIU X, ZHENG Y, DU Z, et al. GPT understands, too[J]. arXiv:2103.10385, 2021.
[73] KIM D, HONG S, CHOI Y H. SC VALL-E: style-controllable zero-shot text to speech synthesizer[J]. arXiv:2307.10550, 2023.
[74] LATIF S, CUAYáHUITL H, PERVEZ F, et al. A survey on deep reinforcement learning for audio-based applications[J]. Artificial Intelligence Review, 2023, 56(3): 2193-2240.
[75] YOSHIDA M, TOGO R, OGAWA T, et al. Off-screen sound separation based on audio-visual pre-training using binaural audio[J]. Sensors, 2023, 23(9): 4540.
[76] OORD A, DIELEMAN S, ZEN H, et al. WaveNet: a generative model for raw audio[J]. arXiv:1609.03499, 2016.
[77] KURADA S, KURADA A. Poster: VGGish embeddings based audio classifiers to improve Parkinson’s disease diagnosis[C]//Proceedings of the 2020 IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE), 2020: 9-11.
[78] SUN C, MYERS A, VONDRICK C, et al. VideoBERT: a joint model for video and language representation learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 7464-7473.
[79] LI L, CHEN Y C, CHENG Y, et al. Hero: hierarchical encoder for video+ language omni-representation pre-training[J]. arXiv:2005.00200, 2020.
[80] SU W, ZHU X, CAO Y, et al. VL-BERT: pre-training of generic visual-linguistic representations[C]//Proceedings of the 2020 International Conference on Learning Representations (ICLR), 2020.
[81] QI D, SU L, SONG J, et al. ImageBERT: cross-modal pre-training with large-scale weak-supervised image-text data[J]. arXiv:2001.07966, 2020.
[82] LUO H, JI L, SHI B, et al. UniVL: a unified video and language pre-training model for multimodal understanding and generation[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019: 5100-5111.
[83] TAN H, BANSAL M. LXMERT: learning cross-modality encoder representations from transformers[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019: 5100-5111.
[84] YU F, TANG J, YIN W, et al. ERNIE-ViL: knowledge enhanced vision-language representations through scene graphs[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2021: 3208-3216.
[85] LU J, BATRA D, PARIKH D, et al. ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[C]//Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019: 13-23.
[86] KIM W, SON B, KIM I. VILT: vision-and-language transformer without convolution or region supervision[C]//Proceedings of the International Conference on Machine Learning (PMLR), 2021: 5583-5594.
[87] BAO H, WANG W, DONG L, et al. VL-BEiT: generative vision-language pretraining[J]. arXiv:2206.01127, 2022.
[88] BAO H, WANG W, DONG L, et al. VLMO: unified vision-language pre-training with mixture-of-modality-experts[C]//Advances in Neural Information Processing Systems, 2022, 35: 32897-32912.
[89] ZHU L, YANG Y. ActBERT: learning global-local video-text representations[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 8746-8755.
[90] LI X. IMF: interactive multimodal fusion model for link prediction[C]//Proceedings of the ACM Web Conference, 2023: 2572-2580.
[91] HAO Y, SONG H, DONG L, et al. Language models are general-purpose interfaces[J]. arXiv:2206.06336, 2022.
[92] MITTAL T. M3er: multiplicative multimodal emotion recognition using facial, textual, and speech cues[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020: 1359-1367.
[93] ZADEH A, LIANG P P, MAZUMDER N, et al. Memory fusion network for multi-view sequential learning[C]//Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, 2018: 5634-5641.
[94] ZHANG Y, CHENG H, SHEN Z, et al. Pre-training multi-task contrastive learning models for scientific literature understanding[J]. arXiv:2305.14232, 2023.
[95] CHO J M, LEI J, TAN H, et al. Unifying vision-and-language tasks via text generation[C]//Proceedings of the International Conference on Machine Learning (PMLR), 2021: 1931-1942.
[96] UZKENT B, GARG A, ZHU W, et al. Dynamic inference with grounding based vision and language models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 2624-2633.
[97] AISHWARYA K, MANNAT S, YANN L, et al. MDETR-modulated detection for end-to-end multi-modal understanding[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 1780-1790.
[98] ZHU P, WANG X, ZHU L, et al. Prompt-based learning for unpaired image captioning[J]. IEEE Transactions on Multimedia, 2024, 26: 379-393.
[99] JIAN Y, LIU T, TAO Y, et al. SimVLG: simple and efficient pretraining of visual language generative models[J]. arXiv:2310.03291, 2023.
[100] DESAI K, KAUL G, AYSOLA Z, et al. RedCaps: web-curated image-text data created by the people, for the people[C]//Proceedings of the Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[101] GU J, MENG X, LU G, et al. Wukong: a 100 million large-scale Chinese cross-modal pre-training benchmark[C]//Advances in Neural Information Processing Systems, 2022, 35: 26418-26431.
[102] PAREKH Z, BALDRIDGE J, CER D, et al. Crisscrossed captions: extended intramodal and intermodal semantic similarity judgments for MS-COCO[C]//Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, 2021: 2855-2870.
[103] ZHAN X, WU Y, DONG X, et al. Product1M: towards weakly supervised instance-level product retrieval via cross-modal pretraining[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 11782-11791.
[104] SCHUHMANN C, VENCU R, BEAUMONT R, et al. Laion-400m: open dataset of clip-filtered 400 million image-text pair[C]//Proceedings of the NeurIPS Workshop Datacentric AI, 2021.
[105] MONDAL P, CHAKDER D, RAJ S, et al. Graph convolutional neural network for multimodal movie recommendation[C]//Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing, 2023: 1633-1640.
[106] WU Y, WU X, LI J, et al. MMpedia: a large-scale multi-modal knowledge graph[C]///Proceedings of the International Semantic Web Conference. Cham: Springer, 2023: 18-37.
[107] 梅宏, 杜小勇, 金海, 等. 大数据技术前瞻[J]. 大数据, 2023, 9(1): 1-20.
MEI H, DU X Y, JIN H, et al. Big data technologies forward-looking[J]. Big Data Research, 2023, 9(1): 1-20.
[108] LIU J, ZHU X, LIU F, et al. OPT: omni-perception pre-trainer for cross-modal understanding and generation[J]. arXiv:2107.00249, 2021.
[109] BI K, XIE L, ZHANG H, et al. Accurate medium-range global weather forecasting with 3D neural networks[J]. Nature, 2023, 619(7970): 533-538.
[110] TEAM G, ANIL R, BORGEAUD S, et al. Gemini: a family of highly capable multimodal models[J]. arXiv:2312.11805, 2023.