Research on Improved BERT’s Chinese Multi-relation Extraction Method

doi:10.3778/j.issn.1002-8331.2011-0199

Abstract

Abstract:

There are few studies on extracting multiple triples from text sentences when constructing triples, and most of them are based on English context. For this reason, a BERT-based Chinese multi-relation extraction model BCMRE is proposed, which consists of relation classification and element extraction. Two mission models are connected in series. BCMRE predicts the possible relationships through the relationship classification task, fuses the predicted relationship code into the word vector, copies an instance of each relationship, and then enters the element extraction task to predict the triplet through named entity recognition. BCMRE adds different pre-models based on the characteristics of the two tasks. Word vectors are designed to optimize the shortcomings of BERT in Chinese characters when processing Chinese. Different loss functions are designed to make the model better. BERT’s multi-head and self-attention mechanism are used to fully extract the feature completes the extraction of triples. BCMRE compares experiments with other models and changes to different pre-models. It has achieved relatively good results under the F1 evaluation, which proves that the model can effectively improve the effect of extracting multi-relational triples.

Key words: Named Entity Recognition（NER）, relationship extraction, pre-model, classification, serial task, Bidirectional Encoder Representations from Transformers（BERT） model

摘要：

构建三元组时在文本句子中抽取多个三元组的研究较少，且大多基于英文语境，为此提出了一种基于BERT的中文多关系抽取模型BCMRE，它由关系分类与元素抽取两个任务模型串联组成。BCMRE通过关系分类任务预测出可能包含的关系，将预测关系编码融合到词向量中，对每一种关系复制出一个实例，再输入到元素抽取任务通过命名实体识别预测三元组。BCMRE针对两项任务的特点加入不同前置模型；设计词向量优化BERT处理中文时以字为单位的缺点；设计不同的损失函数使模型效果更好；利用BERT的多头与自注意力机制充分提取特征完成三元组的抽取。BCMRE通过实验与其他模型，以及更换不同的前置模型进行对比，在F1的评估下取得了相对较好的结果，证明了模型可以有效性提高抽取多关系三元组的效果。

关键词: 命名实体识别, 关系抽取, 前置模型, 分类, 串联任务, BERT模型

HUANG Meigen, LIU Jiale, LIU Chuan. Research on Improved BERT’s Chinese Multi-relation Extraction Method[J]. Computer Engineering and Applications, 2021, 57(21): 234-240.

黄梅根，刘佳乐，刘川. 基于BERT的中文多关系抽取方法研究[J]. 计算机工程与应用, 2021, 57(21): 234-240.

References

[1] PUJARA J，MIAO H，GETOOR L，et al.Knowledge graph identification[C]//International Semantic Web Conference.Berlin，Heidelberg：Springer，2013：542-557.
[2] DONG L，WEI F，ZHOU M，et al.Question answering over freebase with multi-column convolutional neural networks[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing（Volume 1：Long Papers），2015：260-269.
[3] ZHANG F，YUAN N J，LIAN D，et al.Collaborative knowledge base embedding for recommender systems[C]//Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining，2016：353-362.
[4] ZHANG L，ZHAO H.Named entity recognition for Chinese microblog with convolutional neural network[C]//2017 13th International Conference on Natural Computation，Fuzzy Systems and Knowledge Discovery（ICNC-FSKD），2017：87-92.
[5] BACH N，BADASKAR S.A review of relation extraction[J].Literature Review for Language and Statistics II，2007，2：1-15.
[6] 王传栋，徐娇，张永.实体关系抽取综述[J].计算机工程与应用，2020，56（12）：25-36.
WANG C D，XU J，ZHANG Y.Survey of entity relation extraction[J].Computer Engineering and Applications，2020，56（12）：25-36.
[7] KAMBHATLA N.Combining lexical，syntactic，and semantic features with maximum entropy models for extracting relations[C]//Proceedings of the ACL 2004 on Interactive Poster and Demonstration Sessions，2004.
[8] OUDAH M，SHAALAN K.NERA 2.0：improving coverage and performance of rule-based named entity recognition for Arabic[J].Natural Language Engineering，2017，23（3）：441-472.
[9] ABE N，MAMITSUKA H.Query learning strategies using boosting and bagging[C]//Proceedings of the Fifteenth International Conference on Machine Learning（ICML 1998），Madison，Wisconsin，USA，July 24-27，1998.
[10] GLASS M，BARKER K.Bootstrapping relation extraction using parallel news articles[C]//Proceedings of the IJCAI Workshop on Learning by Reading and its Applications in Intelligent Question-Answering，Barcelona，2011.
[11] HASEGAWA T，SEKINE S，GRISHMAN R.Discovering relations among named entities from large corpora[C]//Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics（ACL-04），2004：415-422.
[12] ATTARDI G.Deepnl：a deep learning NLP pipeline[C]//Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing，2015：109-115.
[13] WANG S，ZHANG Y，CHE W，et al.Joint extraction of entities and relations based on a novel graph scheme[C]//Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence，2018：4461-4467.
[14] ZENG D，LIU K，CHEN Y，et al.Distant supervision for relation extraction via piecewise convolutional neural networks[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing，2015：1753-1762.
[15] HOCHREITER S，SCHMIDHUBER J.Long short-term memory[J].Neural Computation，1997，9（8）：1735-1780.
[16] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems，2017：5998-6008.
[17] JOHNSON S，SHEN S，LIU Y.CWPC_BiAtt：character-word-position combined BiLSTM-attention for Chinese named entity recognition[J].Information，2020，11（1）：45.
[18] DEVLIN J，CHANG M W，LEE K，et al.BERT：pre-training of deep bidirectional transformers for language understanding[J].arXiv：1810.04805，2018.
[19] JIANG S，ZHAO S，HOU K，et al.A BERT-BiLSTM-CRF model for Chinese electronic medical records named entity recognition[C]//2019 12th International Conference on Intelligent Computation Technology and Automation（ICICTA），2019：166-169.
[20] LAN Z，CHEN M，GOODMAN S，et al.Albert：a lite bert for self-supervised learning of language representations[J].arXiv：1909.11942，2019.
[21] LIU Y，JI L，HUANG R，et al.An attention-gated convolutional neural network for sentence classification[J].Intelligent Data Analysis，2019，23（5）：1091-1107.