Distant Supervision Relation Extraction Based on Reinforcement Learning with Noisy Network

doi:10.3778/j.issn.1002-8331.2105-0173

Abstract

Abstract: Aiming at the noisy labeling problem in the current distant supervision relation extraction task, this paper proposes a reinforcement learning strategy to design a noisy indicator. By interacting with the environment composed of relation classifier and noisy data, the false positive instances and false negative instances of each relation category are dynamically identified, and the correct relation labels are redistributed, thus, the noisy data is transformed into useful training samples, which is helpful to improve the performance of the distant supervision relation extraction model. In addition, in the process of training, noise is added to the weight of policy network to balance the exploration and utilization of policy network, so as to enhance the exploration ability of noisy indicator and make the noisy indicator more accurately select sentences that can correctly express entity relationship. The experimental results on freebase aligned NYT public dataset show that the proposed method can significantly improve the performance of the distant supervision relation extraction model, which shows that the model has the ability to recognize and correct noisy data labels, and can better learn the relation features.

Key words: distant supervision relation extraction, reinforcement learning, noisy network, false negative instances

摘要： 针对目前远程监督关系抽取任务中存在的错误标注问题，提出使用强化学习策略设计噪声指示器，通过与由关系分类器和噪声数据组成的环境相交互，动态识别每个关系类别的假正例与假负例，并为其重新分配正确的关系标签，从而将噪声数据转换成有用的训练样本，有利于提高远程监督关系抽取模型的性能；另外，在训练过程中，通过在策略网络权重上添加噪声，平衡策略网络的探索和利用问题，从而增强噪声指示器的探索能力，使噪声指示器更准确地选择出能够正确表达实体-关系的句子。在Freebase对齐NYT公共数据集上的实验结果表明，提出的方法可以显著提高远程监督关系抽取模型的性能，表明模型拥有识别并纠正噪声数据标签的能力，可以更好地学习关系特征。

关键词: 远程监督关系抽取, 强化学习, 噪声网络, 假负例

XIE Binhong, WANG Enhui, ZHANG Yingjun. Distant Supervision Relation Extraction Based on Reinforcement Learning with Noisy Network[J]. Computer Engineering and Applications, 2022, 58(23): 169-177.

谢斌红, 王恩慧, 张英俊. 结合噪声网络的强化学习远程监督关系抽取[J]. 计算机工程与应用, 2022, 58(23): 169-177.

References

[1] MINTZ M，BILLS S，SNOW R，et al.Distant supervision for relation extraction without labeled data[C]//Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP，2009：1003-1011.
[2] ZENG D，LIU K，CHEN Y，et al.Distant supervision for relation extraction via piecewise convolutional neural networks[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing，2015：1753-1762.
[3] LIN Y，SHEN S，LIU Z，et al.Neural relation extraction with selective attention over instances[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics，2016：2124-2133.
[4] HAN X，YU P，LIU Z，et al.Hierarchical relation extraction with coarse-to-fine grained attention[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing，2018：2236-2245.
[5] FENG J，HUANG M，ZHAO L，et al.Reinforcement learning for relation classification from noisy data[C]//Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence，2018：5779-5786.
[6] YANG K，HE L，DAI X，et al.Exploiting noisy data in distant supervision relation classification[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2019：3216-3225.
[7] QIN P，XU W，WANG W Y.DSGAN：generative adversarial training for distant supervision relation extraction[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics，2018：496-505.
[8] QIN P，XU W，WANG W.Robust distant supervision relation extraction via deep reinforcement learning[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics，2018：2137-2147.
[9] ZENG D，LIU K，LAI S，et al.Relation classification via convolutional deep neural network[C]//Proceedings of the 25th International Conference on Computational Linguistics，2014：2335-2344.
[10] XU K，FENG Y，HUANG S，et al.Semantic relation classification via convolutional neural networks with simple negative sampling[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing，2015：536-540.
[11] SOCHER R，HUVAL B，MANNING C D，et al.Semantic compositionality through recursive matrix-vector spaces[C]//Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning，2012：1201-1211.
[12] HASHIMOTO K，MIWA M，TSURUOKA Y，et al.Simple customization of recursive neural networks for semantic relation classification[C]//Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing，2013：1372-1376.
[13] YAN X，MOU L，LI G，et al.Classifying relations via long short term memory networks along shortest dependency paths[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing，2015：1785-1794.
[14] NGUYEN T H，GRISHMAN R.Combining neural networks and log-linear models to improve relation extraction[J].arXiv：1511.05926，2015.
[15] JI G，LIU K，HE S，et al.Distant supervision for relation extraction with sentence-level attention and entity descriptions[C]//Proceedings of the 31st AAAI Conference on Artificial Intelligence，2017：3060-3066.
[16] YE Z，LIN Z.Distant supervision relation extraction with intra-bag and inter-bag attentions[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2019：2810-2819.
[17] ZENG X，HE S，LIU K，et al.Large scaled relation extraction with reinforcement learning[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence，2018：5658-5665.
[18] WU S，FAN K，ZHANG Q.Improving distantly supervised relation extraction with neural noise converter and conditional optimal selector[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence，2019：7273-7280.
[19] SHANG Y，HUANG H，MAO X，et al.Are noisy sentences useless for distant supervised relation extraction?[C]//Proceedingsof the AAAI Conference on Artificial Intelligence，2020.
[20] CHEN T，WANG N，HE M，et al.Reducing wrong labels for distantly supervised relation extraction with reinforcement learning[J].IEEE Access，2020，8：81320-81330.
[21] RIEDEL S，YAO L，MCCALLUM A.Modeling relations and their mentions without labeled text[J].Machine Learning and Knowledge Discovery in Databases，2010：148-163.
[22] FORTUNATO M，AZAR M G，PIOT B，et al.Noisy networks for exploration[C]//Proceedings of the 6th International Conference on Learning Representations，2018：1-21.
[23] KIM Y.Convolutional neural networks for sentence classification[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing，2014：1746-1751.
[24] WILLIAMS R J.Simple statistical gradient-following algorithms for connectionist reinforcement learning[J].Machine Learning，1992，8（3/4）：229-256.
[25] MIKOLOV T，CHEN K，CORRADO G，et al.Efficient estimation of word representations in vector space[C]//Proceedings of the 1st International Conference on Learning Representations，2013：1-12.
[26] FINKEL J R，GRENAGER T，MANNING C.Incorporating non-local information into information extraction systems by gibbs sampling[C]//Proceedings of the 43rd Annual Meeting of the ACL，2005.