Research on Domain-Specific Word Vector Generation Based on BERT-CRF

doi:10.3778/j.issn.1002-8331.2103-0531

Abstract

Abstract: How to obtain a high-quality domain-specific word vector representation based on the Chinese BERT word vector for various text analysis tasks based on domain word segmentation is an urgent problem to be solved. This paper proposes a domain-specific word vector generation method based on BERT. A BERT-CRF domain-specific word segmenter is established, and the domain text is combined with the domain text to perform fine-tuning and domain word segmentation learning based on the pre-trained BERT word vector. The domain-specific word vector representation is further obtained through the domain-specific word segmentation decoding results. Experiments show that this method can learn a tokenizer model that meets the requirements of the domain task using only a small amount of domain text, and can obtain a higher-quality domain-specific word vector than the original BERT.

Key words: bidirectional encoder representations from transformers（BERT）, domain tokenizer, domain-specific word vector, conditional random field, word vector visualization

摘要： 如何在中文BERT字向量基础上进一步得到高质量的领域词向量表示，用于各种以领域分词为基础的文本分析任务是一个亟待解决的问题。提出了一种基于BERT的领域词向量生成方法。建立一个BERT-CRF领域分词器，在预训练BERT字向量基础上结合领域文本进行fine-tuning和领域分词学习；通过领域分词解码结果进一步得到领域词向量表示。实验表明，该方法仅利用少量的领域文本就可以学习出符合领域任务需求的分词器模型，并能获得相比原始BERT更高质量的领域词向量。

关键词: BERT, 领域分词器, 领域词向量, 条件随机场, 词向量可视化

GUO Zhendong, LIN Min, LI Chengcheng, ZHAO Jiapeng. Research on Domain-Specific Word Vector Generation Based on BERT-CRF[J]. Computer Engineering and Applications, 2022, 58(21): 156-162.

郭振东, 林民, 李成城, 赵佳鹏. 基于BERT-CRF的领域词向量生成研究[J]. 计算机工程与应用, 2022, 58(21): 156-162.

References

[1] MIKOLOV T，CHEN K，CORRADO G，et al.Efficient estimation of word representations in vector space[J].arXiv：1301.3781，2013.
[2] 彭晓娅，周栋.跨语言词向量研究综述[J].中文信息学报，2020，34（2）：1-15.
PENG X Y，ZHOU D.Survey of cross-lingual word embedding[J].Journal of Chinese Information Processing，2020，34（2）：1-15.
[3] DEVLIN J，CHANG M W，LEE K，et al.Bert：pre-training of deep bidirectional transformers for language understanding[J].arXiv：1810.04805，2018.
[4] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[J].arXiv：1706.03762，2017.
[5] PENNINGTON J，SOCHER R，MANNING C D.Global vectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing，2014.
[6] PETERS M E，NEUMANN M，IYYER M，et al.Deep contextualized word representations[J].arXiv：1802.05365，2018.
[7] RADFORD A，NARASIMHAN K，SALIMANS T，et al.Improving language understanding by generative pre-training[R].Open AI，2018.
[8] CUI Y M，CHE W X.Revisiting pre-trained models for Chinese natural language processing[J].arXiv：2004.13922，2020.
[9] 苏剑林.提速不掉点：基于词颗粒度的中文WoBERT[EB/OL].[2020-09-18].https：//kexue.fm/archives/7758.
SU J L.Speeding up without losing points：Chinese WoBERT based on word granularity[EB/OL].[2020-09-18].https：//kexue.fm/archives/7758.
[10] LAFFERTY J，MCCALUM A，PEREIRA F C N.Conditional random fields：probabilistic models for segmenting and labeling sequence data[C]//Proceedings of 18th International Conference on Machine Learning，2001.
[11] 唐琳，郭崇慧，陈静锋.中文分词技术研究综述[J].现代图书情报技术，2020，4（2）：1-17.
TANG L，GUO C H，CHEN J F.Review of Chinese word segmentation studies[J].Data Analysis and Knowledge Discovery，2020，4（2）：1-17.
[12] 王子牛，姜猛，高建瓴，等.基于BERT的中文命名实体识别方法[J].计算机科学，2019，46（z2）：138-142.
WANG Z N，JIANG M，GAO J L，et al.Chinese named entity recognition method based on BERT[J].Computer Science，2019，46（z2）：138-142.
[13] 王堃，林民，李艳玲.端到端对话系统意图语义槽联合识别研究综述[J].计算机工程与应用，2020，56（14）：14-25.
WANG K，LIN M，LI Y L.Review of research on joint intent detection and semantic slot filling in end to end dialogue system[J].Computer Engineering and Applications，2020，56（14）：14-25.
[14] 谢腾，杨俊安，刘辉.基于BERT-BiLSTM-CRF模型的中文实体识别[J].计算机系统应用，2020，29（7）：48-55.
XIE T，YANG J A，LIU H.Chinese entity recognition based on BERT-BiLSTM-CRF model[J].Computer Systems & Applications，2020，29（7）：48-55.
[15] 王莉军，周越，桂婕，等.基于BiLSTM-CRF的中医文言文文献分词模型研究[J].计算机应用研究，2020，37（11）：3359-3362.
WANG L J，ZHOU Y，GUI J，et al.Chinese medicine literature word segmentation based on BiLSTM-CRF[J].Application Research of Computers，2020，37（11）：3359-3362.
[16] 段瑞雪，巢文宇，张仰森.预训练语言模型BERT在下游任务中的应用[J].北京信息科技大学学报（自然科学版），2020，35（6）：77-83.
DUAN R X，CHAO W Y，ZHANG Y S.Application of pre-trained language model BERT in downstream tasks[J].Journal of Beijing Information Science & Technology University，2020，35（6）：77-83.