融合动作退出和软奖励的强化学习知识推理方法

doi:10.3778/j.issn.1002-8331.2308-0215

摘要/Abstract

摘要： 针对深度强化学习推理方法中存在的过拟合以及稀疏奖励的问题，提出了一种融合动作退出和软奖励的强化学习知识推理方法（knowledge reasoning method of reinforcement learning integrating action withdrawal and soft reward，AS-KRL）。AS-KRL使用门控循环神经网络（GRU）对历史路径信息进行编码，为智能体的动作选择提供当前节点的全局信息；引入动作退出策略随机隐藏部分神经元后再构建策略网络，提高模型路径搜索的成功率，还避免了可能出现的过拟合问题；通过策略网络指导智能体进行动作选择，调用评分函数计算智能体所选三元组的相似度得分，并将所得分数作为智能体的奖励，有效解决稀疏奖励问题。为验证该方法的有效性，在FB15K-237和NELL-995数据集上进行实验，将实验结果与TransE、MINERVA、HRL等9种主流方法进行对比分析，结果表明该方法在链接预测任务上的Hits@k平均提升了0.027，MRR平均提升了0.056。

关键词: 知识推理, 强化学习, 动作退出算法, 门控循环神经网络, 软奖励机制

Abstract: Aiming at the problems of overfitting and sparse reward in deep reinforcement learning reasoning methods, a knowledge reasoning method of reinforcement learning integrating action withdrawal and soft reward is proposed (AS-KRL). AS-KRL uses gated recurrent unit (GRU) to encode the historical path information and provide the global information of the current node for the agent’s action selection. By introducing the action exit strategy to hide some neurons randomly, the strategy network is constructed to improve the success rate of model path search and avoid the possible overfitting problem. The strategy network is used to guide the agent to make action selection, and the score function is called to calculate the similarity score of the triplet selected by the agent, and the score is taken as the reward of the agent, which effectively solves the sparse reward problem. To verify the effectiveness of the proposed method, experiments are carried out on FB15K-237 and NELL-995 datasets. The experimental results are compared with those of 9 mainstream methods such as TransE, MINERVA and HRL. The results show that the proposed method improves Hits@k by an average of 0.027 and MRR by an average of 0.056 on the link prediction task.

Key words: knowledge reasoning, reinforcement learning, action dropout, gated recurrent unit (GRU), soft reward mechanism

孙崇, 王海荣, 荆博祥, 马赫. 融合动作退出和软奖励的强化学习知识推理方法[J]. 计算机工程与应用, 2024, 60(24): 158-165.

SUN Chong, WANG Hairong, JING Boxiang, MA He. Knowledge Reasoning Method of Reinforcement Learning Integrating Action Withdrawal and Soft Reward[J]. Computer Engineering and Applications, 2024, 60(24): 158-165.

参考文献

[1] 杨东华, 何涛, 王宏志, 等. 面向知识图谱的图嵌入学习研究进展[J]. 软件学报, 2022, 33(9): 3370-3390.
YANG D H, HE T, WANG H Z, et al. Survey on knowledge graph embedding learning[J]. Journal of Software, 2022, 33(9): 3370-3390.
[2] 官赛萍, 靳小龙, 贾岩涛, 等. 面向知识图谱的知识推理研究进展[J]. 软件学报, 2018, 29(10): 2966-2994.
GUAN S P, JIN X D, JIA Y T, et al. Knowledge reasoning over knowledge graph: a survey[J]. Journal of Software, 2018, 29(10): 2966-2994.
[3] WU W Q, ZHU Z F, QI J T, et al. A dynamic graph expansion network for multi-hop knowledge base question answering[J]. Neurocomputing, 2023, 515: 37-47.
[4] SARABI S, HAN Q, VRIES B D, et al. Methodology for development of an expert system to derive knowledge from existing nature-based solutions experiences[J]. MethodsX, 2023, 10: 101978.
[5] CUI H, PENG T, XIAO F, et al. Incorporating anticipation embedding into reinforcement learning framework for multi-hop knowledge graph question answering[J]. Information Sciences, 2022, 619: 745-761.
[6] CHEN X J, JIA S B, XIANG Y. A review: knowledge reasoning over knowledge graph[J]. Expert Systems with Applications, 2020, 141(C): 112948.
[7] LIU H, ZHOU S W, CHEN C F, et al. Dynamic knowledge graph reasoning based on deep reinforcement learning[J]. Knowledge-Based Systems, 2022, 241: 108235.
[8] BORDES A, USUNIER N, GARCIADURAN A, et al. Translating embeddings for modeling multi-relational data[C]//Proceedings of the Conference on Neural Information Processing Systems, 2013: 2787-2795.
[9] WANG Z, ZHANG J W, FENG J L, et al. Knowledge graph embedding by translating on hyperplanes[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2014: 1112-1119.
[10] LIN Y K, LIU Z Y, SUN M S, et al. Learning entity and relation embeddings for knowledge graph completion[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2015: 2181-2187.
[11] JI G L, HE S Z, XU L H, et al. Knowledge graph embedding via dynamic mapping matrix[C]//Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2015: 687-696.
[12] WANG Q, MAO Z D, WANG B, et al. Knowledge graph embedding: a survey of approaches and applications[J]. IEEE Transactions on Knowledge and Data Engineering, 2017, 29(12): 2724-2743.
[13] LAO N, COHEN W W. Relational retrieval using a combination of?path-constrained random walks[J]. Machine Learning, 2010, 81(1): 53-67.
[14] TROUILLON T, WELBL J, RIEDEL S, et al. Complex embeddings for simple link prediction[C]//Proceedings of the 33rd International Conference on Machine Learning, 2016: 2071-2080.
[15] DETTMERS T, MINERVINI P, STENETORP P, et al. Convolutional 2D knowledge graph embeddings[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2018: 1811-1818.
[16] WANG Z K, LI L J, ZHEN D J. SRGCN: graph-based multi-hop reasoning on knowledge graphs[J]. Neurocomputing, 2021, 454: 280-290.
[17] ZHANG Y, MENG F D, ZHANG J C, et al. MKGN: a multi-dimensional knowledge enhanced graph network for multi-hop question and answering[J]. IEICE Transactions on Information and Systems, 2022, E105D(4): 807-819.
[18] LI D A, SHUYI M, ZHAO B F, et al. ConvHiA: convolutional network with hierarchical attention for knowledge graph multi-hop reasoning[J]. International Journal of Machine Learning and Cybernetics, 2023, 14: 1-15.
[19] DU Z X, ZHOU C, YAO J C, et al. CogKR: cognitive graph for multi-hop knowledge reasoning[J]. IEEE Transactions on Knowledge and Data Engineering, 2021, 35(2): 1283-1295.
[20] WANG Z K, LI L J, ZENG D J. Hierarchical multihop reasoning on knowledge graphs[J]. IEEE Intelligent Systems, 2022, 37(1): 71-78.
[21] XIA Y, LUO J Y, LAN M J, et al. Reason more like human: incorporating meta information into hierarchical reinforcement learning for knowledge graph reasoning[J]. Applied Intelligence, 2022, 53: 13293-13308.
[22] SHANG B, ZHAO Y L, WANG C X, et al. Multi-hop knowledge reasoning with deep reinforcement learning[C]//Proceedings of the 2022 7th International Conference on Computational Intelligence and Applications (ICCIA), 2022: 27-31.
[23] TIWARI P, ZHU H Y, PANDEY H M. DAPath: distance-aware knowledge graph reasoning based on deep reinforcement learning[J]. Neural Networks, 2021, 135(5/6): 1-12.
[24] XIA Y, LAN M J, LUO J Y, et al. Iterative rule-guided reasoning over sparse knowledge graphs with deep reinforcement learning[J]. Information Processing and Management, 2022, 59(5): 103040.
[25] LI S Y, WANG H, PAN R, et al. MemoryPath: a deep reinforcement learning framework for incorporating memory component into knowledge graph reasoning[J]. Neurocomputing, 2021, 419: 273-286.
[26] MANDANA S, STEVEN K, ZHANG C X, et al. Heterogeneous relational reasoning in knowledge graphs with reinforcement learning[J]. Information Fusion, 2022, 88: 12-21.
[27] GAO C M, WANG S Q, LI S J, et al. CIRS: bursting filter bubbles by counterfactual interactive recommender system[J]. arXiv:2204.01266, 2022.
[28] SRIVASTAVA N, HINTON G E, KRIZHEVSKY A, et al. Dropout: a simple way to prevent neural networks from overfitting[J]. The Journal of Machine Learning Research, 2014, 15(1): 1929-1958.
[29] TOUTANOVA K, CHEN D Q, PANTEL P, et al. Representing text for joint embedding of text and knowledge bases[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2015: 1499-1509.
[30] XIONG W H, HOANG T, WANG Y W. DeepPath: a reinforcement learning method for knowledge graph reasoning[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017.
[31] BALAZEVIC L, ALLEN C, HOSPEDALES T M. TuckER: tensor factorization for knowledge graph completion[J]. arXiv:1901.09590, 2019.
[32] NGUYEN D Q, NGUYEN T D. A novel embedding model for knowledge base completion based on convolutional neural network[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018.
[33] DAS R, DHULIAWALA S, ZAHEER M, et al. Go for a walk and arrive at the answer: reasoning over paths in knowledge bases using reinforcement learning[J]. arXiv:1711.05851, 2017.
[34] NICKEL M, ROSASCO L, POGGIN T A. Holographic embeddings of knowledge graphs[C]//Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 2016: 1955-1961.
[35] SUN Z Q, DENG Z H, NIE J Y, et al. RotatE: knowledge graph embedding by relational rotation in complex space[C]//Proceedings of the International Conference on Learning Representations, 2019.