Chinese Named Entity Recognition by Integrating Multi-Heads Attention Mechanism and Character and Words Fusion

doi:10.3778/j.issn.1002-8331.2104-0265

Abstract

Abstract: Named entity recognition（NER） is an important basic task in natural language processing and Chinese named entity recognition（CNER） is particularly difficult because of word segmentation ambiguity and polysemy. To solve these problems, a multi-heads attention mechanism（Multi-Attention） and character and words fusion CNER model is proposed. The model is abbreviated as CWA-CNER. Firstly, the character vector and its words vector are connected together. The words are the possible words containing the character in the sentence. Then the connected vector are input into bidirectional long short-term memory（BiLSTM） neural network to further extract contextual semantic information. Secondly, Multi-Attention is used to capture the tightness of the connection between elements in the sentence, and finally the entity labeling is carried out through conditional random field（CRF）. The model is tested on Boson dataset, 1998 and 2014 People’s Daily corpus, and their F1 values are all more than 90%. The results show that the model is effective.

Key words: named entity recognition（NER）, multi-heads attention mechanism, character and words fusion

摘要： 命名实体识别（named entity recognition，NER）是自然语言处理中重要的基础任务，而中文命名实体识别（Chinese named entity recognition，CNER）因分词歧义和一词多义等问题使其尤显困难。针对这些问题，提出多头注意力机制（multi-heads attention mechanism，Multi-Attention）与字词融合的中文命名实体识别模型（CWA-CNER）。将汉语文本字向量与其在句中可能成词的词向量进行拼接，并将其送入长短时记忆网络（bidirectional long short-term memory neural network，BiLSTM）提取上下文语义信息，进而利用多头注意力机制捕获句中元素间联系的紧密程度，最后通过条件随机场（conditional random field，CRF）进行实体标注。该模型在Boson数据集，1998和2014年《人民日报》三种语料上进行实验，其F1值均达到90%以上，结果表明了模型的有效性。

关键词: 命名实体识别（NER）, 多头注意力机制, 字词融合

ZHAO Dandan, HUANG Degen, MENG Jiana, GU Feng, ZHANG Pan. Chinese Named Entity Recognition by Integrating Multi-Heads Attention Mechanism and Character and Words Fusion[J]. Computer Engineering and Applications, 2022, 58(7): 142-149.

赵丹丹, 黄德根, 孟佳娜, 谷丰, 张攀. 多头注意力与字词融合的中文命名实体识别[J]. 计算机工程与应用, 2022, 58(7): 142-149.

References

[1] LI J，SUN A，HAN J，et al.A survey on deep learning for named entity recognition[J].IEEE Transactions on Knowledge and Data Engineering，2020，34（1）：50-70.
[2] XIA C，ZHANG C，YANG T，et al.Multi-grained named entity recognition[C]//Proceedings of the 57th Conference of the Association for Computational Linguistics，2019：1430-1440.
[3] 王宁，葛瑞芳，苑春法，等.中文金融新闻中公司名的识别[J].中文信息学报，2002（2）：1-6.
WANG N，GE R F，YUAN C F，et al.Company name identification in Chinese financial domain[J].Journal of Chinese Information Processing，2002（2）：1-6.
[4] COLLINS M，SINGER Y.Unsupervised models for named entity classification[C]//Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora，1999：100-110.
[5] BIKEL D M，SCHWARTZ R，WEISCHEDEL R M.An algorithm that learns what’s in a name[J].Machine Learning，1999，34（1）：211-231.
[6] RATNAPARKHI A.A maximum entropy model for part-of-speech tagging[C]//Proceedings of Empirical Methods in Natural Language Processing，1996：133-142.
[7] CRISTIANINI N，SHAWE-TAYLOR J.An introduction to support vector machines and other kernel-based learning methods[M].[S.l.]：Cambridge University Press，2000.
[8] 马建红，张炳斐，张少光，等.基于主动MCNN-SCRF的新能源汽车命名实体识别[J].计算机工程与应用，2019，55（7）：23-29.
MA J H，ZHANG B F，ZHANG S G，et al.Named entity recognition for new energy vehicles based on active MCNN-SCRF[J].Computer Engineering and Applications，2019，55（7）：23-29.
[9] 何玉洁，杜方，史英杰，等.基于深度学习的命名实体识别研究综述[J].计算机工程与应用，2021，57（11）：21-36.
HE Y J，DU F，SHI Y J，et al.Survey of named entity recognition based on deep learning[J].Computer Engineering and Applications，2021，57（11）：21-36.
[10] ZHANG Y，YANG J.Chinese NER using lattice LSTM[C]//Proceedings of the 56th Annual Meeting of the Computational Linguistics，2018：1554-1564.
[11] ZHOU J T，ZHANG H，JIN D，et al.Roseq：Robust sequence labeling[J].IEEE Transactions on Neural Networks and Learning Systems，2019，31（7）：2304-2314.
[12] 刘小安，彭涛.基于卷积神经网络的中文景点识别研究[J].计算机工程与应用，2020，56（4）：140-145.
LIU X A，PENG T.Research on Chinese scenic spot named entity recognition based on convolutional neural network[J].Computer Engineering and Applications，2020，56（4）：140-145.
[13] 李博，康晓东，张华丽，等.采用Transformer-CRF的中文电子病历命名实体识别[J].计算机工程与应用，2020，56（5）：153-159.
LI B，KANG X D，ZHANG H L，et al.Named entity recognition in Chinese electronic medical records using Transformer-CRF[J].Computer Engineering and Applications，2020，56（5）：153-159.
[14] 殷章志，李欣子，黄德根，等.融合字词模型的中文命名实体识别研究[J].中文信息学报，2019，33（11）：95-100.
YIN Z Z，LI X Z，HUANG D G，et al.Chinese named entity recognition ensembled with character[J].Journal of Chinese Information Processing，2019，33（11）：95-100.
[15] YAN H，DENG B，LI X，et al.TENER：Adapting transformer encoder for name entity recognition[J].arXiv：1911.04474，2019.
[16] 陈茹，卢先领.融合空洞卷积神经网络与层次注意力机制的中文命名实体识别[J].中文信息学报，2020，34（8）：70-77.
CHEN R，LU X L.Combing iterated dilated convolutions neural network and hierarchical attention network for Chinese named entity recognition[J].Journal of Chinese Information Processing，2020，34（8）：70-77.
[17] 刘心惠，陈文实，周爱，等.基于联合模型的多标签文本分类研究[J].计算机工程与应用，2020，56（14）：111-117.
LIU X H，CHEN W S，ZHOU A，et al.Multi-label text classification based on joint model[J].Computer Engineering and Applications，2020，56（14）：111-117.
[18] 刘宇鹏，栗冬冬.基于BLSTM-CNN-CRF的中文命名实体识别方法[J].哈尔滨理工大学学报，2020，25（1）：115-120.
LIU Y P，LI D D.Chinese named entity recognition method based on bi-directional LSTM-CNN-CRF[J].Journal of Harbin University of Science and Technology，2020，25（1）：115-120.
[19] 杨贺羽，杜洪波，朱立军.基于顺序遗忘编码和Bi-LSTM的命名实体识别算法[J].计算机应用与软件，2020，37（2）：213-217.
YANG H Y，DU H B，ZHU L J.Named entity recognition algorithm based on ordinally forgetting encoding and BiLSTM[J].Computer Applications and Software，2020，37（2）：213-217.