Design of Reinforcement Learning Reward Function for Trajectory Planning of Robot Manipulator

doi:10.3778/j.issn.1002-8331.2102-0307

Abstract

Abstract: Aiming at the problems of low learning efficiency of robotic manipulator trajectory planning methods based on deep reinforcement learning and poor robustness of planning strategies, this paper proposes a robotic manipulator trajectory planning method based on voice reward function. The voice instructions are defined as the different states of planning task, and modeled using the Markov chain. It provides the global guidance for the trajectory planning, reduces the blindness of deep reinforcement learning. Meanwhile, the proposed method combines the global information based on the voice and local information of relative distance to design the reward function, considering the degree of fitness between the relative distance and voice guidance. Experimental results demonstrate that the proposed reward function improves the robustness and convergence rate of manipulator trajectory planningeff ectively.

Key words: deep reinforcement learning, robot manipulator, trajectory planning, voicereward function

摘要： 针对基于深度强化学习的机械臂轨迹规划方法学习效率较低，规划策略鲁棒性差的问题，提出了一种基于语音奖励函数的机械臂轨迹规划方法，利用语音定义规划任务的不同状态，并采用马尔科夫链对状态进行建模，为轨迹规划提供全局指导，降低深度强化学习优化的盲目性。提出的方法结合了基于语音的全局信息和基于相对距离的局部信息来设计奖励函数，在每个状态根据相对距离与语音指导的契合程度对机械臂进行奖励或惩罚。实验证明，设计的奖励函数能够有效地提升基于深度强化学习的机械臂轨迹规划的鲁棒性和收敛速度。

关键词: 深度强化学习, 机械臂, 轨迹规划, 语音奖励函数

JIN Dongyin, LI Yue, SHAO Zhenzhou, SHI Zhiping, GUAN Yong. Design of Reinforcement Learning Reward Function for Trajectory Planning of Robot Manipulator[J]. Computer Engineering and Applications, 2022, 58(19): 302-308.

靳栋银, 李跃, 邵振洲, 施智平, 关永. 面向机械臂轨迹规划的强化学习奖励函数设计[J]. 计算机工程与应用, 2022, 58(19): 302-308.

References

[1] CHETTIBI T.Smooth point-to-point trajectory planning for robot manipulators by using radial basis functions[J].Robotica，2019，37（3）：539-559.
[2] NUBERT J L，K?HLER J，BERENZ V，et al.Safe and fast tracking on a robot manipulator：Robust MPC and neural network control[J].IEEE Robotics and Automation Letters，2020，5（2）：3050-3057.
[3] VAN M，DO X P，MAVROVOUNIOTIS M.Self-tuning fuzzy pid-nonsingular fast terminal sliding mode control for robust fault tolerant control of robot manipulators[J].ISA Transactions，2020，96：60-68.
[4] NUBERT J，KOHLER J，BERENZ V，et al.Safe and fast tracking on a robot manipulator：Robust mpc and neural network control[J].IEEE Robotics and Automation Lettersvol，2020，5（2）：3050-3057.
[5] ALJALAL M，IBRAHIM S，DJEMAL R，et al.Comprehensive review on brain-controlled mobile robots and robotic arms based on electroencephalography signals[J].Intelligent Service Robotics，2020，13：539-563.?
[6] LE Q D，KANG H J.Finite-time fault-tolerant control for a robot manipulator based on synchronous terminal sliding mode control[J].Appl Sci，2020，10：2998.
[7] LI T，ZHENG S，SHU X，et al.Self-recognition grasping operation with a vision-based redundant manipulator system[J].Appl Sci，2019，9：5172.
[8] PIAZZI A，VISIOLI A.Global minimum-jerk trajectory planning of robot manipulators[J].IEEE Transactions on Industrial Electronics，2000，47（1）：140-149.
[9] LIU S P，SHI X P.Optimization of the trajectory planning of robot manipulators taking into account the dynamics of the system[J].Journal of Changsha University，2003，33（7）：883-894.
[10] SARAMAGO S F P，JUNIOR V S.Optimal trajectory planning of robot manipulators in the presence of moving obstacles[J].Mechanism & Machine Theory，2000，35（8）：1079-1094.
[11] MILICA L，NASTASE A，ANDREI G.Optimal path planning for a new type of 6rss parallel robot based on virtual displacements expressed through hermite polynomials[J].Mechanism and Machine Theory，2018，61：14-33.
[12] KAVRAKI L E，SVESTKA P，LATOMBE J C，et al.Probabilistic roadmaps for path planning in high-dimensional configuration spaces[J].IEEE Transaction on Robot Automation，1996，12（4）：566-580.
[13] LAVALLE S M.Rapidly-exploring random trees：a new tool for path planning[R].Iowa State University，1998.
[14] MNIH V，KAVUKCUOGLU K，SILVER D，et al.Human-level control through deep reinforcement learning[J].Nature，2015，518：529.
[15] VAN HASSELT H，GUEZ A，SILVER D.Deep reinforcement learning with double q?learning[C]//Proceedingds of the Thirtieth AAAI Conference on Artificial Intelligence，2016.
[16] WANG Z，SCHAUL T，HESSEL M，et al.Dueling network architectures for deep reinforcement learning[J].arXiv：1511.06581，2015.
[17] HESSEL M，MODAYIL J，VAN HASSELT H，et al.Rainbow：Combining improvements in deep reinforcement learning[C]//Proceedingds of the 32nd AAAI Conference on Artificial Intelligence，2018.
[18] LILLICRAP T P，HUNT J J，PRITZEL A，et al.Continuous control with deep reinforcement learning[J].arXiv：1509.
02971，2015.
[19] MNIH V，BADIA A P，MIRZA M，et al.Asynchronous methods for deep reinforcement learning[C]//Proceedings of International Conference on Machine Learning（ICML），2016：1928-1937.
[20] HEESS N，SRIRAM S，LEMMON J，et al.Emergence of locomotion behaviours in rich environments[J].arXiv：1707.02286，2017.
[21] 黄麒元，王致杰，王东伟，等.马尔科夫理论及其在预测中的应用综述[J].技术与市场，2015（9）：12-13.
HUANG Q Y，WANG Z J，WANG D W，et al.Summary of markov theory and its application in forecasting[J].Technology and Market，2015（9）：12-13.