BERT-LCRF Named Entity Recognition Method Oriented Clock Domain

doi:10.3778/j.issn.1002-8331.2102-0110

Abstract

Abstract: Named entity recognition is a key step in constructing a knowledge graph in the clock domain. However, the current clock domain has problems such as the small number of labeled samples, which leads to the low accuracy of named entity recognition in the clock domain. To this end, this paper uses the pre-trained language model BERT to extract the features of the text in the clock domain, and then uses the linear chain conditional random field（Linear-CRF） method for sequence labeling, and proposes a BERT-LCRF named entity recognition model. The results of comparative experiments show that the model can fully learn the feature information of the clock domain, improve the accuracy of sequence labeling, and then improve the effect of named entity recognition in the clock domain.

Key words: named entity recognition, pre-training language model, conditional random field, self-attention mechanism, deep learning

摘要： 命名实体识别是构建时钟领域知识图谱的关键步骤，然而目前时钟领域存在标注样本数量少等问题，导致面向时钟领域的命名实体识别精度不高。为此，利用预训练语言模型BERT进行时钟领域文本的特征提取，利用线性链条件随机场（Linear-CRF）方法进行序列标注，提出了一种BERT-LCRF的命名实体识别模型。对比实验结果表明，该模型能够充分学习时钟领域的特征信息，提升序列标注精度，进而提升时钟领域的命名实体识别效果

关键词: 命名实体识别, 预训练语言模型, 条件随机场, 自注意力机制, 深度学习

TANG Huanling, WANG Hui, WEI Hao, ZHAO Honglei, DOU Quansheng, LU Mingyu. BERT-LCRF Named Entity Recognition Method Oriented Clock Domain[J]. Computer Engineering and Applications, 2022, 58(18): 218-226.

唐焕玲, 王慧, 隗昊, 赵红磊, 窦全胜, 鲁明羽. 面向时钟领域的BERT-LCRF命名实体识别方法[J]. 计算机工程与应用, 2022, 58(18): 218-226.

References

[1] ALFONSECA E，MANANDHAR S.An unsupervised method for general named entity recognition and automated concept discovery[C]//Proceedings of the International Conference on GeneralWordNet，2002.
[2] SHINYAMA Y，SEKINE S.Named entity discovery using comparable news articles[C]//20th International Conference on Computational Linguistics，Geneva，Switzerland，2004：848-853.
[3] ETZIONI O，CAFARELLA M，DOWNEY D，et al.Unsupervised named-entity extraction from the Web：an experimental study[J].Artifical Intelligence，2005，165（1）：91-134.
[4] COLLINS M，SINGER Y.Unsupervised models for named entity classification[C]//Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora，1999：100-110.
[5] COLLINS M.Ranking algorithms for named-entity extraction：boosting and the voted perceptron[C]//Proceedings of the 40th Annual Meeting on Association for Computational Lingustics，2002：489-496.
[6] CUCERZAN S，YAROWSKY D.Language independent named entity recognition combining morphological and contextual evidence[C]//Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora，2001：90-99.
[7] CUCCHIARELLI A，VELARDI P.Unsupervised named entity recognition using syntactic and semantic contextual evidence[J].Computational Linguistics，2001，27（1）：123-131.
[8] ZHAO S.Named entity recognition in biomedical texts using an HMM model[C]//Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications，2004：84-87.
[9] HABIB M S，KALITA J.Scalable biomedical named entity recognition：investigation of a database supported SVM approach[J].International Journal of Bioinformatics Research & Applications，2010，6（2）：191-208.
[10] MCCALLUM A，LI W.Early results for named entity recognition with conditional random fields，feature induction and web-enhanced lexicons[C]//Proceedings of the 7th Conference on Neural Language Learning at HLT-NAACL，2003：188-191.
[11] HOCHREITER S，SCHMIDHUBER J.Long short-term memory[J].Neural Computation，1997，9（8）：1735-1780.
[12] HUANG Z，XU W，YU K.Bidirectional LSTM-CRF models for sequence tagging[J].arXiv：1508.01991，2015.
[13] DEVLIN J，CHANG M W，LEE K，et al.BERT：pre-training of deep bidirectional transformers for language understanding[J].arXiv：1810.04805，2018.
[14] 李明扬，孔芳.融入自注意力机制的社交媒体命名实体识别[J].清华大学学报（自然科学版），2019，59（6）：461-467.
LI M Y，KONG F.Combined self-attention mechanism for named entity recognition in social media[J].Journal of Tsinghua University（Science and Technology），2019，59（6）：461-467.
[15] 李博，康晓东，张华丽，等.采用Transformer-CRF的中文电子病历命名实体识别[J].计算机工程与应用，2020，56（5）：153-159.
LI B，KANG X D，ZHANG H L，et al.Named entity recognition in Chinese electronic medical records using Transformer-CRF[J].Computer Engineering and Applications，2020，56（5）：153-159.
[16] 杨培，杨志豪，罗凌，等.基于注意机制的化学药物命名实体识别[J].计算机研究与发展，2018，55（7）：1548-1556.
YANG P，YANG Z H，LUO L，et al.An attention-based approach for chemical compound and drug named entity recognition[J].Journal of Computer Research and Development，2018，55（7）：1548-1556.
[17] KURU O，CAN O A.CharNER：character-level named entity recognition[C]//Proceedings of COLING 2016，the 26th International Conference on Computational Linguistics：Technical Papers，Osaka，Japan，2016：911-921.
[18] ZHANG Y，YANG J.Chinese NER using lattice LSTM[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers），2018：1554-1564.
[19] VASWANI A，SHAZEER N.Attention is all you need[C]//Advances in Neural Information Processing Systems 30：Annual Conference on Neural Information Processing Systems 2017，December4-9，2017，Long Beach，CA，USA，2017：5998-6008.
[20] FORNEY G D.The viterbi algorithm[J].Proceedings of the IEEE，1973，61（3）：268-278.
[21] 刘伟童，刘培玉，刘文锋，等.基于互信息和邻接熵的新词发现算法[J].计算机应用研究，2019，36（5）：1293-1296.
LIU W T，LIU P Y，LIU W F，et al.New word discovery algorithm based on mutual information and branch entropy[J].Application Research of Computers，2019，36（5）：1293-1296.