Knowledge Reasoning Method Combining Knowledge Representation with Deep Reinforcement Learning

doi:10.3778/j.issn.1002-8331.2104-0430

Abstract

Abstract:

Knowledge reasoning is an important method to solve the problem of lack of knowledge in the knowledge graph. The knowledge reasoning method in the large-scale knowledge graph still has the problems of poor interpretability, low reasoning accuracy and efficiency. This paper proposes a method RLPTransE that combines knowledge representation with deep reinforcement learning. Firstly, it uses the knowledge representation learning method to map the knowledge graph to the vector space containing the semantic information of the triples, and establishes a reinforcement learning environment in the space. Then, it trains through the single-step optimization strategy network and the multi-step reasoning strategy network to enable the reinforcement learning agent to efficiently mine the reasoning rules and complete the reasoning in the process interacting with the environment. According to experimental results on public datasets, compared with state-of-the-art methods, the proposed method achieves better performance in reasoning tasks for large-scale datasets.

Key words: knowledge reasoning, deep reinforcement learning, knowledge representation, path control, rule mining

摘要：

知识推理是解决知识图谱中知识缺失问题的重要方法，针对大规模知识图谱中知识推理方法仍存在可解释性差、推理准确率和效率偏低的问题，提出了一种将知识表示和深度强化学习相结合的方法RLPTransE。利用知识表示学习方法，将知识图谱映射到含有三元组语义信息的向量空间中，并在该空间中建立强化学习环境。通过单步择优策略网络和多步推理策略网络的训练，使强化学习智能体在与环境交互过程中，高效挖掘推理规则进而完成推理。在公开数据集上的实验结果表明，相比于其他先进方法，该方法在大规模数据集推理任务中取得更好的表现。

关键词: 知识推理, 深度强化学习, 知识表示, 路径控制, 规则挖掘

SONG Haonan, ZHAO Gang, WANG Xingfen. Knowledge Reasoning Method Combining Knowledge Representation with Deep Reinforcement Learning[J]. Computer Engineering and Applications, 2021, 57(19): 189-197.

宋浩楠，赵刚，王兴芬. 融合知识表示和深度强化学习的知识推理方法[J]. 计算机工程与应用, 2021, 57(19): 189-197.

References

[1] BOLLACKER K，EVANS C，PARITOSH P，et al.Freebase：A collaboratively created graph database for structuring human knowledge[C]//Proceedings of 2008 ACM SIGMOD International Conference on Management of Data，2008：1247-1250.
[2] AUER S，BIZER C，KOBILAROV G，et al.DBpedia：A nucleus for a Web of open data[C]//Proceedings of the 6th Int’l Semantic Web Conference，2007：722-735.
[3] CARLSON A，BETTERIDGE J，KISIEL B，et al.Toward an architecture for never-ending language learning[C]//Proceedings of the 24th AAAI Conference on Artificial Intelligence，2010：1306-1313.
[4] 张仲伟，曹雷，陈希亮，等.基于神经网络的知识推理研究综述[J].计算机工程与应用，2019，55（12）：8-19.
ZHANG Z W，CAO L，CHEN X L，et al.Survey of knowledge reasoning based on neural networks[J].Computer Engineering and Applications，2019，55（12）：8-19.
[5] 官赛萍，靳小龙，贾岩涛，等.面向知识图谱的知识推理研究进展[J].软件学报，2018，29（10）：2966-2994.
GUAN S P，JIN X L，JIA Y T，et al.Knowledge reasoning over knowledge graph：A survey[J].Journal of Software，2018，29（10）：2966-2994.
[6] BORDES A，USUNIER N，GARCIA-DURAN A，et al.Translating embeddings for modeling multi-relational data[C]//Advances in Neural Information Processing Systems，2013：2787-2795.
[7] WANG Z，ZHANG J，FENG J，et al.Knowledge graph embedding by translating on hyperplanes[C]//Proceedings of the 28th Conference on Artificial Intelligence（AAAI），2014：1112-1119.
[8] 阮小芸，廖健斌，李祥，等.基于人才知识图谱推理的强化学习可解释推荐研究[J].数据分析与知识发现，2020，5（6）：36-50.
RUAN X Y，LIAO J B，LI X，et al.Interpretable recommendation of reinforcement learning based on talent knowledge graph reasoning[J].Data Analysis and Knowledge Discovery，2020，5（6）：36-50.
[9] LIN Y，LIU Z，SUN M，et al.Learning entity and relation embeddings for knowledge graph completion[C]//Proceedings of the 29th Conference on Artificial Intelligence，2015：2181-2187.
[10] JI G，HE S，XU L，et al.Knowledge graph embedding via dynamic mapping matrix[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing，2015：687-696.
[11] EBISU T，ICHISE R.Toruse：Knowledge graph embedding on a lie group[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence，2018：1819.
[12] SUN Z Q，DENG Z H，NIE J Y，et al.Rotate：Knowledge graph embedding by relational rotation in complex space[J].arXiv：1902.10197，2019.
[13] ZHANG S，TAY Y，YAO L N，et al.Quaternion knowledge graph embeddings[C]//Proceedings of the 33rd Conference on Neural Information Processing Systems，2019：2735.
[14] 刘峤，李杨，段宏，等.知识图谱构建技术综述[J].计算机研究与发展，2016，53（3）：582-600.
LIU Q，LI Y，DUAN H，et al.Knowledge graph construction techniques[J].Journal of Computer Research and Development，2016，53（3）：582-600.
[15] LAO N，COHEN W W.Relational retrieval using a combination of path-constrained random walks[J].Machine Learning，2010，81（1）：53-67.
[16] LAO N，MITCHELL T，COHEN W W.Random walk inference and learning in a large scale knowledge base[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing，Association for Computational Linguistics，2011：529-539．
[17] GARDNER M，MITCHELL T.Efficient and expressive knowledge base completion using subgraph feature extraction[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing，2015：1488-1498.
[18] GARDNER M，TALUKDAR P，KRISHNAMURTHY J，et al.Incorporating vector space similarity in random walk inference over knowledge bases[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing（EMNLP），2014：397-406.
[19] DAS R，NEELAKANTAN A，BELANGER D，et al.Chains of reasoning over entities，relations，and text using recurrent neural networks[J].arXiv：1607.01426，2016.
[20] CHEN W，XIONG W，YAN X，et al.Variational knowledge graph reasoning[J].arXiv：1803.06581，2018.
[21] LIN Y，LIU Z，LUAN H，et al.Modeling relation paths for representation learning of knowledge bases[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing，2015：705-714.
[22] 陈海旭，周强，刘学军.一种结合路径信息和嵌入模型的知识推理方法[J].小型微型计算机系统，2020，41（6）：1147-1151.
CHEN H X，ZHOU Q，LIU X J.Knowledge graph reasoning combining path information and embedding model[J].Journal of Chinese Computer Systems，2020，41（6）：1147-1151.
[23] JIA Y，WANG Y，JIN X，et al.Path-specific knowledge graph embedding[J].Knowledge-Based Systems，2018，151：37-44．
[24] WU Y B，ZHU D H，LIAO X W，et al.Knowledge graph reasoning based on paths of tensor factorization[J].Pattern Recognition and Artificial Intelligence，2017，30（5）：473-480．
[25] XIONG W，HOANG T，WANG W Y.Deeppath：A reinforcement learning method for knowledge graph reasoning[J].arXiv：1707.06690，2017.
[26] DAS R，DHULIAWALA S，ZAHEER M，et al.Go for a walk and arrive at the answer：Reasoning over paths in knowledge bases using reinforcement learning[J].arXiv：1711.05851，2017.
[27] LIN X V，SOCHER R，XIONG C.Multi-hop knowledge graph reasoning with reward shaping[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing，2018.
[28] LI R，CHENG X.DIVINE：A generative adversarial imitation learning framework for knowledge graph reasoning[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing（EMNLP-IJCNLP），2019.
[29] WANG H，LI S，PAN R，et al.Incorporating graph attention mechanism into knowledge graph reasoning based on deep reinforcement learning[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing（EMNLP-IJCNLP），2019.
[30] WILLIAMS R J.Simple statistical gradient-following algorithms for connectionist reinforcement learning[J].Machine Learning，1992，8（3/4）：229-256.
[31] TOUTANOVA K，CHEN D，PANTEL P，et al.Representing text for joint embedding of text and knowledge bases[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing，2015：1499-1509.