视觉语言大模型驱动的多模态工业缺陷检测与智能决策

doi:10.3778/j.issn.1002-8331.2507-0389

摘要/Abstract

摘要： 缺陷检测是工业领域的重要应用场景。针对传统检测效率低、工业数据需智能分析处理、单模态系统泛化能力不足及大模型存在幻觉等问题，提出并实现一种增强型工业缺陷智检系统，支持多源异构数据协同处理与高效决策。系统微调工业异常检测多模态大模型，通过改进的跨模态特征对齐算法与提示学习，实现图像、文本等多源数据语义融合，同步输出缺陷识别的语义描述；构建工业数据知识库，借助RAG检索增强生成，抑制模型幻觉，提升检测可信度与决策效果；结合Depth-Anything-V2生成高一致性深度图，支持缺陷三维量化分析，突破二维检测局限；基于自然语言驱动的Excel智能分析模块，自动提取质检表格数据并可视化；OCR智检模块融合PaddleOCR与ErnieBot，实现工业文档文本提取与语义理解。通过智能体统筹核心功能模块给出综合决策建议。在金属、螺丝等5类典型工业零件测试中，系统缺陷类型平均识别精确率为95.26%，位置定位平均误差2.9像素，其拓展了工业质检自动化水平与分析维度，为制造业智能化转型提供了实用技术方案。

关键词: 工业缺陷检测, 多模态融合, 大视觉语言模型（LVLM）, 深度图分析, 检索提高生成（RAG）, 数据决策, 智能体

Abstract: Defect detection is an important application scenario in the industrial field. Aiming at the problems of low traditional detection efficiency, intelligent analysis and processing of industrial data, insufficient generalization ability of single-modal system and hallucination of large models, an enhanced industrial defect intelligent inspection system is proposed and implemented to support multi-source heterogeneous data collaborative processing and efficient decision-making. The system fine-tunes a multimodal large model for industrial anomaly detection. Through improved cross-modal feature alignment algorithms and prompt learning, it achieves semantic fusion of multi-source data such as images and texts, and synchronously outputs semantic descriptions of defect recognition results. It constructs an industrial data knowledge base, uses RAG retrieval to enhance generation, suppresses model hallucinations and improves detection credibility and decision-making effects. Combined with Depth-Anything-V2, it generates highly consistent depth maps to support 3D quantitative analysis of defects, breaking through the limitations of traditional 2D detection. A natural language-driven Excel intelligent analysis module automatically extracts and visualizes quality inspection table data. The OCR intelligent inspection module integrates PaddleOCR and ErnieBot to realize text extraction and semantic understanding of industrial documents. Finally, the agent integrates the core functional modules to provide comprehensive decision-making advice. Tests on 5 typical industrial parts such as metals and screws show that the system’s average recognition accuracy for defect types reaches 95.26%, with an average error of 2.9 pixels in defect location positioning. It expands the automation level and analysis dimensions of industrial quality inspection, providing a practical technical solution for the intelligent transformation of manufacturing.

Key words: industrial defect detection, multimodal fusion, large vision-language model (LVLM), depth map analysis, retrieval-augmented generation (RAG), decision-making, agent

董禹彤, 侯惠芳, 龚明明, 陈自康. 视觉语言大模型驱动的多模态工业缺陷检测与智能决策[J]. 计算机工程与应用, 2026, 62(8): 176-188.

DONG Yutong, HOU Huifang, GONG Mingming, CHEN Zikang. Multimodal Industrial Defect Detection and Intelligent Decision-Making Driven by Large Vision-Language Models[J]. Computer Engineering and Applications, 2026, 62(8): 176-188.

参考文献

[1] BERGMANN P，FAUSER M，SATTLEGGER D，et al.MVTec AD：a comprehensive real-world dataset for unsupervised anomaly detection[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway：IEEE，2020：9584-9592.
[2] SHIN J，HAMZA M，KIM H，et al.Lightweight Swin Transformer for high-precision industrial defect detection in smart manufacturing[J].Alexandria Engineering Journal，2025，130：227-240.
[3] YANG F Q，ZHENG Y L，LIAO K Y，et al.Defect detection in YOLOv8 based on a hybrid domain attention mechanism[J].Signal，Image and Video Processing，2025，19（13）：1158.
[4] SUN Y Y，YAN H Q，SHANG Z K，et al.MCH-YOLOv12：research on surface defect detection algorithm for aluminum profiles based on improved YOLOv12[J].Sensors，2025，25（17）：5389.
[5] JIA Z T，WANG M，ZHAO S M.A review of deep learning-based approaches for defect detection in smart manufacturing[J].Journal of Optics，2024，53（2）：1345-1351.
[6] RADFORD A，KIM J W，HALLACY C，et al.Learning transferable visual models from natural language supervision[C]//Proceedings of the 38th International Conference on Machine Learning，2021：8748-8763.
[7] GIRDHAR R，EL-NOUBY A，LIU Z，et al.ImageBind one embedding space to bind them all[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway：IEEE，2023：15180-15190.
[8] YANG T L，CHANG L Y，YAN J D，et al.A survey on foundation-model-based industrial defect detection[EB/OL].[2025-06-08].https：//arxiv.org/abs/2502.19106.
[9] HE H，LI G，GENG Z J，et al.Analyzing and boosting the power of fine-grained visual recognition for multi-modal large language models[EB/OL].[2025-06-10].https：//arxiv.org/abs/2501.15140.
[10] GU Z P，ZHU B K，ZHU G B，et al.AnomalyGPT：detecting industrial anomalies using large vision-language models[C]//Proceedings of the 38th AAAI Conference on Artificial Intelligence.Palo Alto：AAAI，2024：1932-1940.
[11] 蒋章雷，李念雪，李云鹏，等.基于OPC UA的多源异构工业数据协同管理信息模型构建技术研究[J].机床与液压，2024，52（22）：8-15.
JIANG Z L，LI N X，LI Y P，et al.Research on information model construction technology of multi-source heterogeneous industrial data collaborative management based on OPC UA[J].Machine Tool & Hydraulics，2024，52（22）：8-15.
[12] XIANG A，HUANG Z X，GAO X T，et al.BridgeNet：a unified multimodal framework for bridging 2D and 3D industrial anomaly detection[EB/OL].[2025-06-14].https：//arxiv.org/abs/2507.19253.
[13] LIU H T，LI C Y，WU Q Y，et al.Visual Instruction Tuning[C]//Advances in Neural Information Processing Systems 36，2023：34892-34916.
[14] HU E J，SHEN Y L，WALLIS P，et al.LoRA：low-rank adaptation of large language models[C]//Proceedings of the 10th International Conference on Learning Representations，2022.
[15] 苏佳，贾泽，秦一畅，等.面向工业表面缺陷检测的改进YOLOv8算法[J].计算机工程与应用，2024，60（14）：187-196.
SU J，JIA Z，QIN Y C，et al.Improved YOLOv8 algorithm for industrial surface defect detection[J].Computer Engineering and Applications，2024，60（14）：187-196.
[16] CHEN T，KORNBLITH S，NOROUZI M，et al.A simple framework for contrastive learning of visual representations[C]//Proceedings of the 37th International Conference on Machine Learning，2020：1597-1607.
[17] 王亮.检索增强生成（RAG）驱动的知识服务：原理、范式及评估[J].科技与出版，2025（4）：37-46.
WANG L.Retrieval-augmented generation（RAG）-driven knowledge service：principles，paradigms，and evaluation[J].Science-Technology & Publication，2025（4）：37-46.
[18] ZAVRTANIK V，KRISTAN M，SKO?AJ D.DRAEM-a discriminatively trained reconstruction embedding for surface anomaly detection[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision.Piscataway：IEEE，2022：8310-8319.
[19] CUI L，SHEN Y J，YANG K，et al.A unified model for multi-class anomaly dection[C]//Advances in Neural Information Processing Systems 35，2022：4571-4584.
[20] YAO X C，LI R Q，ZHANG J，et al.Explicit boundary guided semi-push-pull contrastive learning for supervised anomaly detection[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway：IEEE，2023：24490-24499.
[21] ZAVRTANIK V，KRISTAN M，SKO?AJ D.DSR—a dual subspace re-projection network for surface anomaly detection[C]//Proceedings of the 17th European Conference on Computer Vision.Cham：Springer ，2022：539-554.
[22] GUO J，LU S，JIA L Z，et al.ReContrast：domain-specific anomaly detection via contrastive reconstruction[C]//Advances in Neural Information Processing Systems 36，2023.
[23] 余鸿铭，陈兴无，万敏，等.复杂边缘目标的亚像素定位[J].计算机仿真，2009，26（12）：190-192.
YU H M，CHEN X W，WAN M，et al.Sub-pixel edge of complex edge target[J].Computer Simulation，2009，26（12）：190-192.
[24] 周大可，田径，杨欣.结合局部平面参数预测的无监督单目图像深度估计[J].中国图象图形学报，2021，26（1）：165-175.
ZHOU D K，TIAN J，YANG X.Unsurpervised monocular image depth estimation based on the prediction of local plane parameters[J].Journal of Image and Graphics，2021，26（1）：165-175.
[25] YANG L H，KANG B Y，HUANG Z L，et al.Depth anything V2[EB/OL].[2025-07-02].https：//arxiv.org/abs/2406.
09414.
[26] DOSOVITSKIY A，BEYER L，KOLESNIKOV A，et al.An image is worth 16x16 words：transformers for image recognition at scale[C]//Proceedings of the 9th International Conference on Learning Representations，2021.
[27] HORNUNG A，WURM K M，BENNEWITZ M，et al.OctoMap：an efficient probabilistic 3D mapping framework based on octrees[J].Autonomous Robots，2013，34（3）：189-206.
[28] RAASVELDT M，MüHLEISEN H.DuckDB：an embeddable analytical database[C]//Proceedings of the 2019 International Conference on Management of Data.New York：ACM，2019：1981-1984.
[29] LI G R，KANG G L，LIU W，et al.Content-consistent matching for domain adaptive semantic segmentation[C]//Proceedings of the 16th European Conference on Computer Vision.Cham：Springer，2020：440-456.
[30] LIAO M H，WAN Z Y，YAO C，et al.Real-time scene text detection with differentiable binarization[C]//Proceedings of the 37th AAAI Conference on Artificial Intelligence.Palo Alto：AAAI，2020：11474-11481.
[31] DU Y K，CHEN Z N，JIA C Y，et al.SVTR：scene text recognition with a single visual model[C]//Proceedings of the 31st International Joint Conference on Artificial Intelligence，2022：884-890.