结构化环境中UR3机械臂对于移动物体的抓取研究

doi:10.3778/j.issn.1002-8331.2501-0027

摘要/Abstract

摘要： 针对在结构化环境下机械臂移动抓取自主决策能力不足、环境适应性低、学习效率低以及机械臂协同作业中的协调性和路径规划等问题，将双重回放缓存机制（double replay buffer，DRB）与强化学习柔性动作-评价（soft actor-critic，SAC）算法融合，提出基于DRB-SAC（soft actor-critic with double replay buffer）的深度强化学习移动物体机械臂抓取方法。首先搭建移动物体机械臂抓取系统。然后提出改进的深度强化学习控制策略DRB-SAC。该策略通过马尔可夫决策过程模型构建，定义动作空间和状态空间提供操作和观察环境的框架，确定机械臂控制任务的目标和约束条件，设计训练策略和奖励函数，利用深度神经网络来拟合动作价值函数和策略函数，实现机械臂移动自适应抓取智能决策，并引入双重回放缓存机制进一步增强了算法的稳定性和泛化能力。最后进行仿真和物理实验对比分析，结果表明该方法具有很好的收敛性，在探索环境奖励、抓取动作的完成程度方面具有优越性。

关键词: 强化学习, 移动物体抓取, 双重回放缓存机制, CoppeliaSim仿真

Abstract: In response to the problems of sufficient autonomous decision-making ability for the manipulator movement and grasping in non-structured environments, low adaptability to the environment, low learning efficiency, and poor coordination and path planning in the coordination of manipulators, this paper proposes a novel method to object-grasping by integrating the double replay buffers (DRB) and reinforcement learning soft actor-critic (SAC) algorithm. The proposed method involves setting up a moving object-grasping robotic arm system, introducing an improved deep reinforcement learning control strategy called DRB-SAC, defining action space and state space by using a Markov decision process model, designing training strategy and reward function, utilizing deep neural networks to fit the action value function and policy function for adaptive intelligent decision-making in robotic arm movement, and enhancing stability and generalization ability through the double replay buffers mechanism. Comparative analysis of simulation results and physical experiments demonstrates that this method exhibits good convergence and superiority in exploring environmental rewards as well as completing grasping actions.

Key words: reinforcement learning, moving object grasp, double replay buffer, Coppeliasim simulation

罗国庆, 袁庆霓, 曲鹏举, 吴兴杰. 结构化环境中UR3机械臂对于移动物体的抓取研究[J]. 计算机工程与应用, 2025, 61(16): 106-115.

LUO Guoqing, YUAN Qingni, QU Pengju, WU Xingjie. Grasp of Moving Objects by UR3 Manipulator in Structured Environment[J]. Computer Engineering and Applications, 2025, 61(16): 106-115.

参考文献

[1] MNIH V, KAVUKCUOGLU K, SILVER D, et al. Playing Atari with deep reinforcement learning[J]. arXiv:1312.5602, 2013.
[2] TAMPUU A, MATIISEN T, KODELJA D, et al. Multiagent cooperation and competition with deep reinforcement learning[J]. PLoS One, 2017, 12(4): e0172395.
[3] TSURUMINE Y, CUI Y D, UCHIBE E, et al. Deep reinforcement learning with smooth policy update: application to robotic cloth manipulation[J]. Robotics and Autonomous Systems, 2019, 112: 72-83.
[4] SCHULMAN J, LEVINE S, ABBEEL P, et al. Trust region policy optimization[C]//Proceedings of the 32nd International Conference on Machine Learning, 2015: 1889-1897.
[5] MONTGOMERY W, LEVINE S. Guided policy search as approximate mirror descent[C]//Advances in Neural Information Processing Systems 29, 2016.
[6] MONTGOMERY W, LEVINE S. Guided policy search via approximate mirror descent[C]//proceedings of the 30th Annual Conference on Neural Information Processing Systems, 2016: 4015-4023.
[7] HAARNOJA T, ZHOU A, HARTIKAINEN K, et al. Soft actor-critic algorithms and applications[J]. arXiv:1812.05905, 2018.
[8] COWLEY A, COHEN B, MARSHALL W, et al. Perception and motion planning for pick-and-place of dynamic objects[C]//Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway: IEEE, 2013: 816-823.
[9] MARTURI N, KOPICKI M, RASTEGARPANAH A, et al. Dynamic grasp and trajectory planning for moving objects[J]. Autonomous Robots, 2019, 43(5): 1241-1256.
[10] 李鑫, 沈捷, 曹恺, 等. 深度强化学习的机械臂密集场景多物体抓取方法[J]. 计算机工程与应用, 2024, 60(23): 325-332.
LI X, SHEN J, CAO K, et al. Deep reinforcement learning for manipulator multi-object grasping in dense scenes[J]. Computer Engineering and Applications, 2024, 60(23): 325-332.
[11] CHEN P Z, LU W Q. Deep reinforcement learning based moving object grasping[J]. Information Sciences, 2021, 565: 62-76.
[12] ZHENG X, YUAN S M, CHEN P Z. Robotic autonomous grasping strategy and system for cluttered multi-class objects[J]. International Journal of Control, Automation and Systems, 2024, 22(8): 2602-2612.
[13] XU H Y, WANG Q, MIN H S. CCA-MTFCN: a robotic pushing-grasping collaborative method based on deep reinforcement learning[C]//Proceedings of the 2023 International Conference on Cognitive Systems and Information Processing. Singapore: Springer, 2023: 57-72.
[14] 胡丞熙. 基于深度强化学习的工业机器人抓取结构避障控制方法[J]. 现代计算机, 2024, 30(21): 59-63.
HU C X. The control method of industrial robot grasping structure based on deep reinforcement learning[J]. Modern Computer, 2024, 30(21): 59-63.
[15] ZUO G Y, TONG J Y, WANG Z H, et al. A graph-based deep reinforcement learning approach to grasping fully occluded objects[J]. Cognitive Computation, 2023, 15(1): 36-49.
[16] GAO Y C, ZHOU D K, SHEN Y Q, et al. Dual experience replay-based TD3 for single intersection signal control[J]. The Journal of Supercomputing, 2024, 80(11): 15161-15182.
[17] WEI Z G, XIAO W D, YUAN L, et al. Memory-based soft actor-critic with prioritized experience replay for autonomous navigation[J]. Intelligent Service Robotics, 2024, 17(3): 621-630.
[18] ONORI G, SHAHID A A, BRAGHIN F, et al. Adaptive optimization of hyper-parameters for robotic manipulation through evolutionary reinforcement learning[J]. Journal of Intelligent & Robotic Systems, 2024, 110(3): 108.
[19] PENG J, YUAN Y. Moving object grasping method of mechanical arm based on deep deterministic policy gradient and hindsight experience replay[J]. Journal of Advanced Computational Intelligence and Intelligent Informatics, 2022, 26(1): 51-57.
[20] ZHANG Z Z, CHEN J L, CHEN Z B, et al. Asynchronous episodic deep deterministic policy gradient: toward continuous control in computationally complex environments[J]. IEEE Transactions on Cybernetics, 2021, 51(2): 604-613.
[21] AFZALI S R, SHOARAN M, KARIMIAN G. A modified convergence DDPG algorithm for robotic manipulation[J]. Neural Processing Letters, 2023, 55(8): 11637-11652.
[22] CHEN C J, ZHANG H, PAN Y, et al. Robot autonomous grasping and assembly skill learning based on deep reinforcement learning[J]. The International Journal of Advanced Manufacturing Technology, 2024, 130(11): 5233-5249.
[23] KUO P H, CHEN K L. Two-stage fuzzy object grasping controller for a humanoid robot with proximal policy optimization[J]. Engineering Applications of Artificial Intelligence, 2023, 125: 106694.
[24] CAI Z Y, FENG Z Q, ZHOU L R, et al. Deep-reinforcement-learning-based robot motion strategies for grabbing objects from human hands[J]. Virtual Reality & Intelligent Hardware, 2023, 5(5): 407-421.
[25] XU B Z, HASSAN T, HUSSAIN I. Improving reinforcement learning based moving object grasping with trajectory prediction[J]. Intelligent Service Robotics, 2024, 17(2): 265-276.
[26] CORTES C, MOHRI M, ROSTAMIZADEH A. L2 regularization for learning kernels[C]//Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, 2009: 109-116.