Research on Cascaded Labeling Framework for Relation Extraction with Self-Attention

doi:10.3778/j.issn.1002-8331.2205-0325

Abstract

Abstract: Named entity recognition and relation extraction are two essential subtasks in natural language processing and knowledge graph construction. In order to solve the problems of error transmission and entity sharing in the process of relation extraction, an entity relation extraction cascading labeling framework, Att-CasRel, which incorporates self-attention mechanism, is proposed. It can not only solve the cascading errors, but also solve the problem of multiple relational triples sharing the same entity in the same sentence. Based on Bert model, CB-Bert suitable for Chinese medical field is obtained by retraining text of CMeIE data set. Self-attention mechanism is incorporated in tail entity recognition stage to enhance feature expression of encoding vector of head entity and improve feature extraction capability of the model. Experimental results on CMeIE data set show that the proposed framework achieves better performance than independent extraction model extraction and other joint extraction models.

Key words: natural language processing, relation extraction, self-attention mechanism, knowledge graph, Bert

摘要： 命名实体识别和关系抽取是自然语言处理和知识图谱构建中两个十分重要的子任务。针对关系抽取过程中容易出现的错误传递和实体共享的缺陷，提出了一种融合自注意力机制的实体关系抽取级联标记框架Att-CasRel，不仅解决了级联错误，还能够解决同一个句子中多个关系三元组共享相同实体的问题。在Bert模型的基础上，使用CMeIE数据集的文本进行再训练得到适用于中文医疗领域的CB-Bert，并在尾实体识别阶段融入自注意力机制来增强头实体编码向量的特征表达，提高了模型的特征提取能力。在CMeIE数据集上的实验结果表明，该标记框架相较于独立抽取的模型以及其他联合抽取模型取得了更好的效果。

关键词: 自然语言处理, 关系抽取, 自注意力机制, 知识图谱, Bert

XIAO Lizhong, ZANG Zhongxing, SONG Saisai. Research on Cascaded Labeling Framework for Relation Extraction with Self-Attention[J]. Computer Engineering and Applications, 2023, 59(3): 77-83.

肖立中, 臧中兴, 宋赛赛. 融合自注意力的关系抽取级联标记框架研究[J]. 计算机工程与应用, 2023, 59(3): 77-83.

References

[1] 鄂海红，张文静，肖思琪，等.深度学习实体关系抽取研究综述[J].软件学报，2019，30（6）：1793-1818.
E H H，ZHANG W J，XIAO S Q，et al.Survey of entity relationship extraction based on deep learning[J].Journal of Software，2019，30（6）：1793-1818.
[2] LI Q，JI H.Incremental joint extraction of entity mentions and relations[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers），Baltimore，Maryland，USA，June 22-27，2014.Stroudsburg：ACL，2014：402-412.
[3] MIWA M，SASAKI Y.Modeling joint entity and relation extraction with table representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing（EMNLP），Doha，Qatar，October 25-29，2014.Stroudsburg：ACL，2014：1858-1869.
[4] REN X，WU Z，HE W，et al.Cotype：joint extraction of typed entities and relations with knowledge bases[C]//Proceedings of the 26th International Conference on World Wide Web，Perth，Australia，April 3-7，2017.Switzerland，Republic and Canton of Geneva：International World Wide Web Conferences Steering Committee，2017：1015-1024.
[5] ZHENG S，WANG F，BAO H，et al.Joint extraction of entities and relations based on a novel tagging scheme[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers），Vancouver，Canada，July 30-August 4，2017.Stroudsburg：ACL，2017：1227-1236.
[6] ZENG X，ZENG D，HE S，et al.Extracting relational facts by an end-to-end neural model with copy mechanism[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers），Melbourne，Australia，July 15-20，2018.Stroudsburg：ACL，2018：506-514.
[7] FU T J，LI P H，MA W Y.Graphrel：modeling text as relational graphs for joint entity and relation extraction[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics，Florence，Italy，July 28-August 2，2019.Stroudsburg：ACL，2019：1409-1418.
[8] 冯钧，张涛，杭婷婷.重叠实体关系抽取综述[J].计算机工程与应用，2022，58（1）：1-11.
FENG J，ZHANG T，HANG T T.Survey of overlapping entities and relations extraction[J].Computer Engineering and Applications，2022，58（1）：1-11.
[9] DONG X，GABRILOVICH E，HEITZ G，et al.Knowledge vault：a web-scale approach to probabilistic knowledge fusion[C]//Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining，New York，USA，August 24-27，2014.New York：ACM，2014：601-610.
[10] MINTZ M，BILLS S，SNOW R，et al.Distant supervision for relation extraction without labeled data[C]//Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP：Volume 2，Suntec，Singapore，August 2-7，2009.Stroudsburg：ACL，2009：1003-1011.
[11] MIWA M，BANSAL M.End-to-end relation extraction using LSTMs on sequences and tree structures[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers），Berlin，Germany，August 7-12，2016.Stroudsburg：ACL，2016：1105-1116.
[12] DAI D，XIAO X，LYU Y，et al.Joint extraction of entities and overlapping relations using position-attentive sequence labeling[C]//Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence，Honolulu，Hawaii，USA，January 27-February 1，2019.Menlo Park，CA：AAAI，2019：6300-6308.
[13] 乔晶晶，温政，段利国，等.基于实体图路径聚合的多实体关系抽取[J].计算机工程与设计，2020，41（8）：2309-2314.
QIAO J J，WEN Z，DUAN L G，et al.Multi-entity’s relation extraction based on walk aggregation in entity graph[J].Computer Engineering and Design，2020，41（8）：2309-2314.
[14] ZHAO K，XU H，CHENG Y，et al.Representation iterative fusion based on heterogeneous graph neural network for joint entity and relation extraction[J].Knowledge-Based Systems，2021，219：106888.
[15] LIU J，CHEN S，WANG B，et al.Attention as relation：learning supervised multi-head self-attention for relation extraction[C]//Proceedings of the Twenty-Ninth International Joint Conferences on Artificial Intelligence，Yokohama，Japan，January 7-15，2021.San Francisco，CA：Morgan Kaufmann，2021：3787-3793.
[16] LAI T，CHENG L，WANG D，et al.RMAN：relational multi-head attention neural network for joint extraction of entities and relations[J].Applied Intelligence，2022，52（3）：3132-3142.
[17] DEVLIN J，CHANG M W，LEE K，et al.Bert：pretraining of deep bidirectional transformers for language understanding[C]//Proceedings of NAACL-HLT，Minneapolis，Minnesota，June 2-7，2019.Stroudsburg：ACL，2019：4171-4186.
[18] GURURANGAN S，MARASOVI? A，SWAYAMDIPTA S，et al.Don’t stop pretraining：adapt language models to domains and tasks[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics，July 5-10，2020.Stroudsburg：ACL，2020：8342-8360.
[19] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Proceeding of the 31st Conference on Neural Information Processing Systems，Long Beach，CA，USA，December 4-9，2017.Cambridge：MIT Press，2017：5998-6008.
[20] KINGMA D P，BA J.Adam：a method for stochastic optimization[C]//Proceedings of the 3rd International Conference on Learning Representations，San Diego，CA，USA，May 7-9，2015.