融合多模态信息与大语言模型的生成式命名实体识别方法

doi:10.3778/j.issn.1002-8331.2503-0243

摘要/Abstract

摘要： 针对现有基于序列标注的多模态命名实体识别方法存在模型架构复杂、跨模态信息融合不充分等问题，提出一种基于大语言模型的生成式多模态命名实体识别方法。该方法设计包含三阶段处理流程：采用模态特异性编码器分别提取图文特征，继而设计跨模态注意力机制实现噪声鲁棒的特征融合，通过指令微调的大语言模型实现结构化实体生成。该方法的创新性体现为：有效挖掘大语言模型内隐的语义推理能力；构建具有跨模态一致性的高层语义表征。在Twitter-2015和Twitter-2017基准数据集上的实验表明，该方法F1值分别达到62.7%和66.8%，显著优于现有生成式方法。

关键词: 命名实体识别（NER）, 大语言模型（LLMs）, 多模态融合, 信息抽取

Abstract: A generative multimodal named entity recognition method based on large language models proposed to address the problems of complex model architecture and insufficient cross-modal information fusion in existing sequence label-based multimodal named entity recognition methods. This method design includes a three-stage processing flow：initially employing modality-specific encoders to independently capture visual and textual features, subsequently devising a cross-modal attention mechanism for noise-resistant feature fusion, and ultimately utilizing instruction-tuned large language models to produce structured entity outputs. The methodological novelty manifests in two aspects： effectively mining the implicit semantic reasoning ability of large language models; building high-level semantic representations with cross-modal consistency. Experiments on the benchmark datasets Twitter-2015 and Twitter-2017 show that this method achieves F1 values of 62.7% and 66.8%, respectively, which are significantly better than existing generative methods.

Key words: named entity recognition (NER), large language models (LLMs), multimodal fusion, information extraction

胡慧云, 葛杨, 崔凌潇, 唐琳, 齐思洋, 孔竣达, 肖波. 融合多模态信息与大语言模型的生成式命名实体识别方法[J]. 计算机工程与应用, 2026, 62(8): 231-240.

HU Huiyun, GE Yang, CUI Lingxiao, TANG Lin, QI Siyang, KONG Junda, XIAO Bo. Generative Named Entity Recognition Method Integrating Multimodal Information and Large Language Models[J]. Computer Engineering and Applications, 2026, 62(8): 231-240.

参考文献

[1] LI J，SUN A X，HAN J L，et al.A survey on deep learning for named entity recognition[J].IEEE Transactions on Knowledge and Data Engineering，2022，34（1）：50-70.
[2] FEI H，WU S Q，LI J Y，et al.LasUIE：unifying information extraction with latent adaptive structure-aware generative language model[C]//Advances in Neural Information Proceesing Systems 35，2022：15460-15475.
[3] SCH?FER H，IDRISSI?YAGHIR A，HORN P，et al.Cross-language transfer of high?quality annotations：combining neural machine translation with cross-linguistic span alignment to apply NER to clinical texts in a low-resource language[C]//Proceedings of the 4th Clinical Natural Language Processing Workshop，2022：53-62.
[4] BIE? M，GILSKI M，MACIEJEWSKA M，et al.RecipeNLG：a cooking recipes dataset for semi-structured text generation[C]//Proceedings of the 13th International Conference on Natural Language Generation，2020：22-28.
[5] 杨书新，刘天扬，黄伟东.基于MVBCN-FLW的中文法律文书命名实体识别[J].计算机工程与应用，2025，61（2）：219-226.
YANG S X，LIU T Y，HUANG W D.Chinese legal document named entity recognition based on MVBCN-FLW[J].Computer Engineering and Applications，2025，61（2）：219-226.
[6] 王旭阳，赵丽婕，张继远.位置标签增强的中文医学命名实体级联识别[J].计算机工程与应用，2024，60（2）：121-128.
WANG X Y，ZHAO L J，ZHANG J Y.Enhanced cascading recognition with positional labels for Chinese medicine named entity[J].Computer Engineering and Applications，2024，60（2）：121-128.
[7] MOON S，NEVES L，CARVALHO V.Multimodal named entity recognition for short social media posts[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2018：852-860.
[8] WU Z W，ZHENG C M，CAI Y，et al.Multimodal representation with embedded visual guiding objects for named entity recognition in social media posts[C]//Proceedings of the 28th ACM International Conference on Multimedia.New York：ACM，2020：1038-1046.
[9] 束文豪，奚雪峰，崔志明，等.图神经网络在命名实体识别中的应用研究[J].计算机工程与应用，2023，59（19）：52-65.
SHU W H，XI X F，CUI Z M，et al.Study of named entity recognition based on graph neural network[J].Computer Engineering and Applications，2023，59（19）：52-65.
[10] YU J F，LI Z Y，WANG J M，et al.Grounded multimodal named entity recognition on social media[C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics，2023：9141-9154.
[11] SUI Y，BU F Y，HU Y T，et al.Trigger-GNN：a trigger-based graph neural network for nested named entity recognition[C]//Proceedings of the 2022 International Joint Conference on Neural Networks.Piscataway：IEEE，2022：1-8.
[12] LI J Y，LI H，SUN D，et al.LLMs as bridges：reformulating grounded multimodal named entity recognition[C]//Findings of the Association for Computational Linguistics：ACL 2024，2024：1302-1318.
[13] WU Z H，PAN S R，CHEN F W，et al.A comprehensive survey on graph neural networks[J].IEEE Transactions on Neural Networks and Learning Systems，2021，32（1）：4-24.
[14] KADDOUR J，HARRIS J，MOZES M，et al.Challenges and applications of large language models[EB/OL].[2025?02-20].https：//arxiv.org/abs/2307.10169.
[15] WANG X Y，GUI M，JIANG Y，et al.ITA：image-text alignments for multi-modal named entity recognition[C]//Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2022：3176-3189.
[16] CHEN X，ZHANG N Y，LI L，et al.Good visual guidance make a better extractor：hierarchical visual prefix for multimodal entity and relation extraction[C]//Findings of the Association for Computational Linguistics：NAACL 2022，2022：1607-1618.
[17] WANG X Y，CAI J，JIANG Y，et al.Named entity and relation extraction with multi-modal retrieval[C]//Findings of the Association for Computational Linguistics：EMNLP 2022，2022：5925-5936.
[18] LI J Y，LI H，PAN Z，et al.Prompting ChatGPT in MNER：enhanced multimodal named entity recognition with auxiliary refined knowledge[C]//Findings of the Association for Computational Linguistics：EMNLP 2023，2023：2787-2802.
[19] MA Y B，CAO Y X，HONG Y，et al.Large language model is not a good few-shot information extractor，but a good reranker for hard samples![C]//Findings of the Association for Computational Linguistics：EMNLP 2023，2023：10572-10601.
[20] WANG S H，SUN X F，LI X Y，et al.GPT-NER：named entity recognition via large language models[EB/OL].[2025-02-27].https：//arxiv.org/abs/2304.10428.
[21] ASHOK D，LIPTON Z C.PromptNER：prompting for named entity recognition[EB/OL].[2025?03?02].https：//arxiv.org/abs/2305.15444.
[22] CAI C R，WANG Q L，LIANG B，et al.In-context learning for few?shot multimodal named entity recognition[C]//Findings of the Association for Computational Linguistics：EMNLP 2023，2023：2969-2979.
[23] RADFORD A，KIM J W，HALLACY C，et al.Learning transferable visual models from natural language supervision[C]//Proceedings of the 38th International Conference on Machine Learning，2021：8748-8763.
[24] BA J L，KIROS J R，HINTON G E.Layer normalization[EB/OL].[2025-03-05].https：//arxiv.org/abs/1607.06450.
[25] ALAYRAC J B，DONAHUE J，LUC P，et al.Flamingo：a visual language model for few-shot learning[C]//Advances in Neural Information Processing Systems 35，2022：23716-23736.
[26] LU D，NEVES L，CARVALHO V，et al.Visual attention model for name tagging in multimodal social media[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics，2018：1990-1999.
[27] ZHANG Q，FU J L，LIU X Y，et al.Adaptive co-attention network for named entity recognition in tweets[C]//Proceedings of the 32th AAAI Conference on Artificial Intelligence.Palo Alto：AAAI，2018.
[28] LOSHCHILOV I，HUTTER F.Decoupled weight decay regularization[EB/OL].[2025?03?09].https：//arxiv.org/abs/1711.
05101.
[29] WANG P，YANG A，MEN R，et al.OFA：unifying architectures，tasks，and modalities through a simple sequence-to-sequence learning framework[C]//Proceedings of the 39th International Conference on Machine Learning，2022：23318-23340.