基于大语言模型的施工安全多模态知识图谱的构建与应用

doi:10.3778/j.issn.1002-8331.2408-0036

摘要/Abstract

摘要： 现有施工安全管理方法难以有效整合文本与图片多模态信息，针对施工现场安全事故的领域内知识表达和推理能力有限，并且处理和应用数据需要广泛的领域知识和专业背景。针对这一问题，提出一种基于多模态大语言模型的多模态知识图谱构建方法。基于施工安全管理的基本理论和实践经验，构建施工安全知识本体，在此基础上运用多模态大模型构建出多模态知识图谱，解决文本与图片多模态整合以及领域内知识表达和推理能力有限的问题。构建出的知识图谱不仅整合了文本中的事故安全知识，还包含了现场图片信息，提升了知识的全面性和实用性。通过计算准确率、召回率、F1值三个指标对抽取结果进行评价，均得到了很高的分数，验证了大模型对于图片抽取的合理性和准确性。在实际应用中，该方法有助于安全管理人员及时的发现施工现场的安全事故，为管理决策和智能推理提供了重要支持。

关键词: 多模态知识图谱, 大语言模型, 安全管理, 知识抽取, 本体构建

Abstract: It is difficult for existing construction safety management methods to effectively integrate multi-modal information of text and picture, and the knowledge expression and reasoning ability in the field of construction site safety accidents are limited, and the processing and application of data require a wide range of domain knowledge and professional background. To solve this problem, this paper proposes a multimodal knowledge graph construction method based on multimodal large language model. Through three steps of data collection and preprocessing, ontology construction at concept level and instance level, the multimodal knowledge graph is constructed to solve the problems of multimodal integration of text and picture and the limited knowledge expression and reasoning ability in the field. The knowledge map constructed not only integrates the accident safety knowledge in the text, but also includes the scene picture information, which improves the comprehensiveness and practicality of the knowledge. Three indexes of accuracy, recall rate and F1 value are used to evaluate the extraction results, and high scores are obtained, which verify the rationality and accuracy of the large model for image extraction. In practical application, this method is helpful for safety managers to discover safety accidents on construction site in time, and provides important support for management decision-making and intelligent reasoning.

Key words: multimodal knowledge graph, large language model, safety management, knowledge extraction, ontology construction

董磊, 吴福居, 史健勇, 潘龙飞. 基于大语言模型的施工安全多模态知识图谱的构建与应用[J]. 计算机工程与应用, 2025, 61(9): 325-333.

DONG Lei, WU Fuju, SHI Jianyong, PAN Longfei. Construction and Application of Multimodal Knowledge Graph in Construction Safety Field Based on Large Language Model[J]. Computer Engineering and Applications, 2025, 61(9): 325-333.

参考文献

[1] ZHU Z, REN X, CHEN Z.Integrated detection and tracking of workforce and equipment from construction jobsite videos[J].Automation in Construction, 2017, 81:161-171.
[2] LIU W Y, ?MENG Q F, ?LI Z, et al.Applications of computer vision in monitoring the unsafe behavior of construction workers: current status and challenges[J].Buildings, 2021, 11(9): 409.
[3] 吴宛凝, 王莉.基于改进YOLOv7的电力施工现场异常行为检测分析[J].电子技术, 2024, 53(1): 422-424.
WU W N, WANG L.Analysis of abnormal behavior detection in electric power construction sites based on improved YOLOv7[J].Electronic Technology, 2024, 53(1): 422-424.
[4] 王震宇, 朱学芳, 杨睿.基于多模态大语言模型的关系抽取研究[J]. 数据分析与知识发现, 2025(1): 90-99.
WANG Z Y, ZHU X F, YANG R. Extracting relations based on multimodal large language model [J]. Data Analysis and Knowledge Discovery, 2025(1): 90-99.
[5] 朱文情.面向多源数据的起重吊装作业知识图谱构建与应用研究[D]. 徐州: 中国矿业大学, 2024.
ZHU W Q.Research on the construction and application of knowledge graph for lifting and hositing operations based on multi-source data[D]. Xuzhou: China University of Mining and Technology, 2024.
[6] 陈烨, 周刚, 卢记仓.多模态知识图谱构建与应用研究综述[J].计算机应用研究, 2021, 38(12): 3535-3543.
CHEN Y, ZHOU G, LU J C. Survey on construction and application research for multi-modal knowledge graphs[J].Application Research of Computers, 2021, 38(12): 3535-3543.
[7] HUANG J, CHEN Y, LI Y, et al.Medical knowledge-based network for patient-oriented visual question answering[J].Information Processing & Management, 2023, 60(2): 103241.
[8] DANG F, TANG J, LI S.MOOC-KG: a MOOC knowledge graph for cross-platform online learning resources[C]//Proceedings of the International Conference on Electronics Information and Emergency Communication (ICEIEC), Beijing, China, 2019: 1-8.
[9] 沈扬, 田丹, 刘昊, 等.面向混凝土坝施工管理的知识图谱智能构建[J].水力发电学报, 2022, 41(9): 118-128.
SHEN Y, TIAN D, LIU H, et al.Knowledge graph intelligent establishment for concrete dam construction management[J].Journal of Hydroelectric Engineering, 2022, 41(9): 118-128.
[10] 周逸凡, 段浩, 赵红莉, 等.水文模型知识图谱构建与应用[J].水利学报, 2024, 55(1): 80-91.
ZHOU Y F, DUAN H, ZHAO H L, et al. Hydrological modeling knowledge graph construction and application[J]. Journal of Hydraulic Engineering, 2024, 55(1): 80-91.
[11] YAN J, LV T, YU Y.Construction and recommendation of a water affair knowledge graph[J].Sustaionability, 2018, 10(10): 3429
[12] 钟逸, 冷彦, 陈思慧, 等.基于大语言模型RAG架构的电池加速研究:现状与展望[J]. 储能科学与技术, 2024, 13 (9): 3214-3225.
ZHONG Y, LENG Y, CHEN S H, et al.Accelerating battery research with retrieval-augmented large language models: present and future[J].Energy Storage Science and Technology, 2024, 13 (9): 3214-3225.
[13] 陈囿任, 李勇, 温明, 等.多模态知识图谱融合技术研究综述[J]. 计算机工程与应用, 2024, 60(13): 36-50.
CHEN Y R, LI Y, WEN M, et al.Research and comprehensive review on multi-modal knowledge graph fusion techniques[J].Computer Engineering and Applications, 2024, 60(13): 36-50.
[14] FANG W, MA L, LOVE E P, et al.Knowledge graph for identifying hazards on construction sites: integrating computer vision with ontology[J]. Automation in Construction, 2020, 119:103310.
[15] CHEN Q H, LONG D B, YANG C, et al. Knowledge graph improved dynamic risk analysis method for behavior-basedsafety management on a construction site[J]. Journal of Management in Engineering, 2023, 39(4).
[16] WEBSON A, PAVLICK E.Do promptbased models really understand the meaning of their prompts[J]. arXiv:2109.01247, 2021.
[17] 冯钧, 畅阳红, 陆佳民, 等. 基于大语言模型的水工程调度知识图谱的构建与应用[J]. 计算机科学与探索, 2024, 18(6):1637-1647.
FENG J, CHANG Y H, LU J M, et al. Construction and application of knowledge graph for water engineering scheduling based on large language model[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(6):1637-1647.
[18] 刘静, 郭龙腾. GPT-4对多模态大模型在多模态理解、生成、交互上的启发[J]. 中国科学基金, 2023, 37(5): 793-802.
LIU J, GUO L T.Inspiration of GPT-4 on multimodal foundation models in multimodal understanding, generation, and interaction[J]. Bulletin of National Natural Science Foundation of China, 2023, 37(5): 793-802.
[19] ZHAO W X, ZHOU K, LI J Y, et al. A survey of large language models[J]. arXiv:2303.18223, 2023.
[20] JACOBS G, HOSTE V.SENTiVENT: enabling supervised information extractionof company-specific events in economicand financial news[J]. Language Resources and Evaluation, 2022, 56(1): 225-257.
[21] LI Y, ZAKHOZHYI V, ZHU D, et al.Domain specific knowledge graphs as a service to the public: powering social-impact funding in the US[C]//Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020: 2793-2801.
[22] BROWN T, MANN B, RYDER N, et al. Language models are few-shot learners[C]//Advances in Neural Information Processing Systems, 2020: 1877-1901.
[23] 张才科, 李小龙, 郑胜, 等. 基于大语言模型的知识图谱构建及应用研究[J]. 计算机科学与探索, 2024, 18(10): 2656-2667.
ZHANG C K, LI X L, ZHENG S, et al.Research on construction and application of knowledge graph based on large language model[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(10): 2656-2667.
[24] 王键霖. 面向小麦生产领域的知识图谱融合方法研究[D].郑州: 河南农业大学, 2024.
WANG J L.Research on knowledge graph fusion method for fields of wheat production[D]. Zhengzhou: Henan Agricultural University, 2024.
[25] 陈俞源.基于BERT模型的建筑事故隐患分类研究[D]. 西安:西安建筑科技大学, 2021.
CHEN Y Y.Research on the classification of hidden danger of building accidents based on BERT model[D]. Xi’an: Xi’an University of Architecture and Technology, 2021.
[26] LI W C, PAN W M. Enhancing chain-of-thought reasoning in large language models through text style diversity and prompt fusion[C]//Proceedings of the 2023 2nd International Conference on Urban Planning and Regional Economy (UPRE 2023), 2023.
[27] 支凤稳, 陈佳琪, 孙若阳, 等. 基于知识图谱的数字记忆领域知识发现研究[J]. 情报科学, 2024, 42(9): 123-134.
ZHI F W, CHEN J Q, SUN R Y. et al.Research on knowledge discovery in the field of digital memory based on knowledge graph[J]. Information Science, 2024, 42(9): 123-134.
[28] 沈阳, 李洪磊, 陈杰. 图数据模型及其在疫情追溯领域的应用研究[J]. 软件导刊, 2021, 20(2): 13-17.
SHEN Y, LI H L, CHEN J. Study and analysis of epidemic situation based on graph database [J]. Software Guide, 2021, 20(2): 13-17.