Attention Adaptive Model with Word Information Embeding for Named Entity Recognition

doi:10.3778/j.issn.1002-8331.2112-0193

Abstract

Abstract: Lack of word segmentation information, interference of out of vocabulary words and irrelevant words are the main problems faced by character-level Chinese named entity recognition. In this paper, an attention adaptive model with word information embeding for Chinese named entity recognition is proposed. Based on the discovery of new words, the integration of character vector embedding and word-level information embedding is used as the input of the model, which reduces the influence of out of vocabulary words on the model, enhances the significance of entity features and makes them easier for learners to acquire. At the same time, the dynamic scaling factor is introduced into the attention mechanism to adaptively adjust the attention distribution of related entities and irrelevant words, which reduces the interference of irrelevant words to the model. The experimental comparison of this method on public datasets proves the effectiveness of the method.

Key words: Chinese named entity recognition, attention mechanism, dynamic scaling factor, out-of-vocabulary words

摘要： 缺少分词信息及未登录词、无关词干扰是字符级中文命名实体识别面临的主要问题，提出了融合词信息嵌入的注意力自适应中文命名实体识别模型，在新词发现的基础上，将字向量嵌入与词级信息嵌入融合作为模型输入，减少了未登录词对模型的影响，并增强了实体特征的显著性，使实体特征更容易被学习器获取；同时，在注意力机制中引入动态缩放因子，自适应地调整相关实体和无关词的注意力分布，一定程度上减小了无关词对模型的干扰。将该方法在公共数据集上进行实验，实验结果证明了方法的有效性。

关键词: 中文命名实体识别, 注意力机制, 动态缩放因子, 未登录词

ZHAO Ping, DOU Quansheng, TANG Huanling, JIANG Ping, CHEN Shuzhen. Attention Adaptive Model with Word Information Embeding for Named Entity Recognition[J]. Computer Engineering and Applications, 2023, 59(8): 167-174.

赵萍, 窦全胜, 唐焕玲, 姜平, 陈淑振. 融合词信息嵌入的注意力自适应命名实体识别[J]. 计算机工程与应用, 2023, 59(8): 167-174.

References

[1] YU B，ZHENYU Z，LIU T，et al.Beyond word attention：using segment attention in neural relation extraction[C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence，2019：5401-5407.
[2] PIZZATO L A，MOLLá D，PARIS C.Pseudo relevance feedback using named entities for question answering[C]//Proceedings of the Australasian Language Technology Workshop，2006：83-90.
[3] BABYCH B，HARTLEY A.Improving machine translation quality with automatic named entity recognition[C]//Proceedings of the 7th International EAMT Workshop on MT and Other Language Technology Tools，Improving MT Through Other Language Technology Tools：Resources and Tools for Building MT，Association for Computational Linguistics，2003：1-8.
[4] BERGER A，LAFFERTY J.Information retrieval as statistical translation[C]//ACM SIGIR Forum.New York，NY，USA：ACM，2017：219-226.
[5] LIU L，SHANG J，REN X，et al.Empower sequence labeling with task-aware neural language model[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2018.
[6] YANG J，TENG Z，ZHANG M，et al.Combining discrete and neural features for sequence labeling[C]//International Conference on Intelligent Text Processing and Computational Linguistics.Cham：Springer，2016：140-154.
[7] HE H，SUN X.F-score driven max margin neural network for named entity recognition in Chinese social media[J].arXiv：1611.04234，2016.
[8] HE H，SUN X.A unified model for cross-domain and semi-supervised named entity recognition in Chinese social media[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2017.
[9] HIGASHIYAMA S，UTIYAMA M，SUMITA E，et al.Incorporating word attention into character-based word segmentation[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，Volume 1（Long and Short Papers），2019：2699-2709.
[10] LIU W，FU X，ZHANG Y，et al.Lexicon enhanced Chinese sequence labeling using bert adapter[J].arXiv：2105.
07148，2021.
[11] LIU Z，ZHU C，ZHAO T.Chinese named entity recognition with a sequence labeling approach：based on characters，or based on words?[C]//International Conference on Intelligent Computing.Berlin，Heidelberg：Springer，2010：634-640.
[12] LI H，HAGIWARA M，LI Q，et al.Comparison of the impact of word segmentation on name tagging for Chinese and Japanese[C]//Proceedings of the Ninth International Conference on Language Resources and Evaluation（LREC’14），2014：2532-2536.
[13] ZHANG Y，YANG J.Chinese NER using lattice LSTM[J].arXiv：1805.02023，2018.
[14] LIU W，XU T，XU Q，et al.An encoding strategy based word-character LSTM for Chinese NER[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，Volume 1（Long and Short Papers），2019：2379-2389.
[15] GUI T，ZOU Y，ZHANG Q，et al.A lexicon-based graph neural network for Chinese NER[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing（EMNLP-IJCNLP），2019：1040-1050.
[16] SUI D，CHEN Y，LIU K，et al.Leverage lexical knowledge for Chinese named entity recognition via collaborative graph network[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing（EMNLP-IJCNLP），2019：3830-3840.
[17] LI X，YAN H，QIU X，et al.FLAT：Chinese NER using flat-lattice transformer[J].arXiv：2004.11795，2020.
[18] DEVLIN J，CHANG M W，LEE K，et al.Bert：pretraining of deep bidirectional transformers for language understanding[J].arXiv：1810.04805，2018.
[19] YAN H，DENG B，LI X，et al.TENER：adapting transformer encoder for named entity recognition[J].arXiv：1911.04474，2019.
[20] JIA C，SHI Y，YANG Q，et al.Entity enhanced BERT pre-training for Chinese NER[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing（EMNLP），2020：6384-6396.
[21] LAMPLE G，BALLESTEROS M，SUBRAMANIAN S，et al.Neural architectures for named entity recognition[J].arXiv：1603.01360，2016.
[22] HUANG Z，XU W，YU K.Bidirectional LSTM-CRF models for sequence tagging[J].arXiv：1508.01991，2015.
[23] GUI T，MA R，ZHANG Q，et al.CNN-based Chinese NER with lexicon rethinking[C]//Proceedings of IJCAI，2019：4982-4988.
[24] MA R，PENG M，ZHANG Q，et al.Simplify the usage of lexicon in Chinese NER[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics，2020.