面向机械臂轨迹规划的强化学习奖励函数设计

doi:10.3778/j.issn.1002-8331.2102-0307

计算机工程与应用 ›› 2022, Vol. 58 ›› Issue (19): 302-308.DOI: 10.3778/j.issn.1002-8331.2102-0307

面向机械臂轨迹规划的强化学习奖励函数设计

靳栋银，李跃，邵振洲，施智平，关永

1.首都师范大学信息工程学院，北京 100048
2.首都师范大学轻型工业机械臂与安全验证北京市重点实验室，北京 100048
3.河北工业职业技术学院计算机技术系，石家庄 050000
4.首都师范大学成像技术北京市高精尖创新中心，北京 100048

出版日期:2022-10-01 发布日期:2022-10-01

Design of Reinforcement Learning Reward Function for Trajectory Planning of Robot Manipulator

JIN Dongyin, LI Yue, SHAO Zhenzhou, SHI Zhiping, GUAN Yong

1.College of Information Engineering, Capital Normal University, Beijing 100048, China
2.Beijing Key Laboratory of Light Industrial Robot and Safety Verification, Capital Normal University, Beijing 100048, China
3.Department of Computer Technology, Hebei College of Industry and Technology, Shijiazhuang 050000, China
4.Beijing Advanced Innovation Center for Imaging Technology, Capital Normal University, Beijing 100048, China

Online:2022-10-01 Published:2022-10-01

摘要/Abstract

摘要： 针对基于深度强化学习的机械臂轨迹规划方法学习效率较低，规划策略鲁棒性差的问题，提出了一种基于语音奖励函数的机械臂轨迹规划方法，利用语音定义规划任务的不同状态，并采用马尔科夫链对状态进行建模，为轨迹规划提供全局指导，降低深度强化学习优化的盲目性。提出的方法结合了基于语音的全局信息和基于相对距离的局部信息来设计奖励函数，在每个状态根据相对距离与语音指导的契合程度对机械臂进行奖励或惩罚。实验证明，设计的奖励函数能够有效地提升基于深度强化学习的机械臂轨迹规划的鲁棒性和收敛速度。

关键词: 深度强化学习, 机械臂, 轨迹规划, 语音奖励函数

Abstract: Aiming at the problems of low learning efficiency of robotic manipulator trajectory planning methods based on deep reinforcement learning and poor robustness of planning strategies, this paper proposes a robotic manipulator trajectory planning method based on voice reward function. The voice instructions are defined as the different states of planning task, and modeled using the Markov chain. It provides the global guidance for the trajectory planning, reduces the blindness of deep reinforcement learning. Meanwhile, the proposed method combines the global information based on the voice and local information of relative distance to design the reward function, considering the degree of fitness between the relative distance and voice guidance. Experimental results demonstrate that the proposed reward function improves the robustness and convergence rate of manipulator trajectory planningeff ectively.

Key words: deep reinforcement learning, robot manipulator, trajectory planning, voicereward function

靳栋银, 李跃, 邵振洲, 施智平, 关永. 面向机械臂轨迹规划的强化学习奖励函数设计[J]. 计算机工程与应用, 2022, 58(19): 302-308.

JIN Dongyin, LI Yue, SHAO Zhenzhou, SHI Zhiping, GUAN Yong. Design of Reinforcement Learning Reward Function for Trajectory Planning of Robot Manipulator[J]. Computer Engineering and Applications, 2022, 58(19): 302-308.

参考文献

[1] CHETTIBI T.Smooth point-to-point trajectory planning for robot manipulators by using radial basis functions[J].Robotica，2019，37（3）：539-559.
[2] NUBERT J L，K?HLER J，BERENZ V，et al.Safe and fast tracking on a robot manipulator：Robust MPC and neural network control[J].IEEE Robotics and Automation Letters，2020，5（2）：3050-3057.
[3] VAN M，DO X P，MAVROVOUNIOTIS M.Self-tuning fuzzy pid-nonsingular fast terminal sliding mode control for robust fault tolerant control of robot manipulators[J].ISA Transactions，2020，96：60-68.
[4] NUBERT J，KOHLER J，BERENZ V，et al.Safe and fast tracking on a robot manipulator：Robust mpc and neural network control[J].IEEE Robotics and Automation Lettersvol，2020，5（2）：3050-3057.
[5] ALJALAL M，IBRAHIM S，DJEMAL R，et al.Comprehensive review on brain-controlled mobile robots and robotic arms based on electroencephalography signals[J].Intelligent Service Robotics，2020，13：539-563.?
[6] LE Q D，KANG H J.Finite-time fault-tolerant control for a robot manipulator based on synchronous terminal sliding mode control[J].Appl Sci，2020，10：2998.
[7] LI T，ZHENG S，SHU X，et al.Self-recognition grasping operation with a vision-based redundant manipulator system[J].Appl Sci，2019，9：5172.
[8] PIAZZI A，VISIOLI A.Global minimum-jerk trajectory planning of robot manipulators[J].IEEE Transactions on Industrial Electronics，2000，47（1）：140-149.
[9] LIU S P，SHI X P.Optimization of the trajectory planning of robot manipulators taking into account the dynamics of the system[J].Journal of Changsha University，2003，33（7）：883-894.
[10] SARAMAGO S F P，JUNIOR V S.Optimal trajectory planning of robot manipulators in the presence of moving obstacles[J].Mechanism & Machine Theory，2000，35（8）：1079-1094.
[11] MILICA L，NASTASE A，ANDREI G.Optimal path planning for a new type of 6rss parallel robot based on virtual displacements expressed through hermite polynomials[J].Mechanism and Machine Theory，2018，61：14-33.
[12] KAVRAKI L E，SVESTKA P，LATOMBE J C，et al.Probabilistic roadmaps for path planning in high-dimensional configuration spaces[J].IEEE Transaction on Robot Automation，1996，12（4）：566-580.
[13] LAVALLE S M.Rapidly-exploring random trees：a new tool for path planning[R].Iowa State University，1998.
[14] MNIH V，KAVUKCUOGLU K，SILVER D，et al.Human-level control through deep reinforcement learning[J].Nature，2015，518：529.
[15] VAN HASSELT H，GUEZ A，SILVER D.Deep reinforcement learning with double q?learning[C]//Proceedingds of the Thirtieth AAAI Conference on Artificial Intelligence，2016.
[16] WANG Z，SCHAUL T，HESSEL M，et al.Dueling network architectures for deep reinforcement learning[J].arXiv：1511.06581，2015.
[17] HESSEL M，MODAYIL J，VAN HASSELT H，et al.Rainbow：Combining improvements in deep reinforcement learning[C]//Proceedingds of the 32nd AAAI Conference on Artificial Intelligence，2018.
[18] LILLICRAP T P，HUNT J J，PRITZEL A，et al.Continuous control with deep reinforcement learning[J].arXiv：1509.
02971，2015.
[19] MNIH V，BADIA A P，MIRZA M，et al.Asynchronous methods for deep reinforcement learning[C]//Proceedings of International Conference on Machine Learning（ICML），2016：1928-1937.
[20] HEESS N，SRIRAM S，LEMMON J，et al.Emergence of locomotion behaviours in rich environments[J].arXiv：1707.02286，2017.
[21] 黄麒元，王致杰，王东伟，等.马尔科夫理论及其在预测中的应用综述[J].技术与市场，2015（9）：12-13.
HUANG Q Y，WANG Z J，WANG D W，et al.Summary of markov theory and its application in forecasting[J].Technology and Market，2015（9）：12-13.

面向机械臂轨迹规划的强化学习奖励函数设计

Design of Reinforcement Learning Reward Function for Trajectory Planning of Robot Manipulator

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	魏婷婷, 袁唯淋, 罗俊仁, 张万鹏. 智能博弈对抗中的对手建模方法及其应用综述[J]. 计算机工程与应用, 2022, 58(9): 19-29.
[2]	高敬鹏, 胡欣瑜, 江志烨. 改进DDPG无人机航迹规划算法[J]. 计算机工程与应用, 2022, 58(8): 264-272.
[3]	赵庶旭, 元琳, 张占平. 多智能体边缘计算任务卸载[J]. 计算机工程与应用, 2022, 58(6): 177-182.
[4]	欧新, 周璐, 张鸷, 吴月家, 赵云斌. 电能表检验台体串联机械臂时间最优轨迹规划[J]. 计算机工程与应用, 2022, 58(6): 250-255.
[5]	邓心, 那俊, 张瀚铎, 王昱林, 张斌. 基于深度强化学习的智能灯个性化调节方法[J]. 计算机工程与应用, 2022, 58(6): 264-270.
[6]	徐博, 周建国, 吴静, 罗威. 可编程数据平面下基于DDPG的路由优化方法[J]. 计算机工程与应用, 2022, 58(3): 143-150.
[7]	温建伟, 姚冰冰, 万剑雄, 李雷孝. 结合深度强化学习的区块链分片系统性能优化[J]. 计算机工程与应用, 2022, 58(19): 116-123.
[8]	吴亚丽, 王君虎, 郑帅龙. 基于改进双重深度Q网络的入侵检测模型[J]. 计算机工程与应用, 2022, 58(16): 102-110.
[9]	高阳, 张晓晖, 高玉儿, 尚婷, 杨启航. 基于神经网络和模糊补偿的水下机械臂控制[J]. 计算机工程与应用, 2022, 58(15): 317-323.
[10]	李庆, 龚远强, 张玮, 张洋, 刘超, 李军, 韩丹, 刘德峰, 梅文豪, 董雪. 用于智能垃圾分拣的注意力YOLOv4算法[J]. 计算机工程与应用, 2022, 58(11): 260-268.
[11]	宋浩楠, 赵刚, 孙若莹. 基于深度强化学习的知识推理研究进展综述[J]. 计算机工程与应用, 2022, 58(1): 12-25.
[12]	牛鹏飞, 王晓峰, 芦磊, 张九龙. 强化学习在车辆路径问题中的研究综述[J]. 计算机工程与应用, 2022, 58(1): 41-55.
[13]	刘建宇，范平清. 基于改进的RRT*-connect算法机械臂路径规划[J]. 计算机工程与应用, 2021, 57(6): 274-278.
[14]	马志豪，朱响斌. 拟双曲动量梯度的对抗深度强化学习研究[J]. 计算机工程与应用, 2021, 57(24): 90-99.
[15]	李宝帅，叶春明. 深度强化学习算法求解作业车间调度问题[J]. 计算机工程与应用, 2021, 57(23): 248-254.